Kernel and RPM related bbclass changes

Kernel and RPM related bbclass changes

[Zoltan Boszormenyi]

I have been carrying some forked bbclass recipes and I think
these can be beneficial to others, not to mention that I could
get rid of the private forks of these.

Please, review.

Best regards,
Zoltán Böszörményi

[PATCH 1/6] package_rpm.bbclass: Handle posttrans scriptlets

[Zoltan Boszormenyi]

From: Zoltán Böszörményi <zboszor@gmail.com>

The "posttrans" scriptlet is the RPM equivalent of the
OPKG "intercept script" concept and probably a cleaner one.

"pretrans" also exists in RPM but there's no equivalent
for it in OPKG.

Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
---
 meta/classes/package_rpm.bbclass | 30 +++++++++++++++++++++---------
 1 file changed, 21 insertions(+), 9 deletions(-)

diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass
index 88d861c0e7..5992d3fd06 100644
--- a/meta/classes/package_rpm.bbclass
+++ b/meta/classes/package_rpm.bbclass
@@ -308,10 +308,11 @@ python write_specfile () {
     srcrconflicts  = ""
     srcrobsoletes  = ""
 
-    srcrpreinst  = []
-    srcrpostinst = []
-    srcrprerm    = []
-    srcrpostrm   = []
+    srcrpreinst   = []
+    srcrpostinst  = []
+    srcrprerm     = []
+    srcrpostrm    = []
+    srcrposttrans = []
 
     spec_preamble_top = []
     spec_preamble_bottom = []
@@ -373,11 +374,11 @@ python write_specfile () {
         splitrconflicts  = localdata.getVar('RCONFLICTS') or ""
         splitrobsoletes  = ""
 
-        splitrpreinst  = localdata.getVar('pkg_preinst')
-        splitrpostinst = localdata.getVar('pkg_postinst')
-        splitrprerm    = localdata.getVar('pkg_prerm')
-        splitrpostrm   = localdata.getVar('pkg_postrm')
-
+        splitrpreinst   = localdata.getVar('pkg_preinst')
+        splitrpostinst  = localdata.getVar('pkg_postinst')
+        splitrprerm     = localdata.getVar('pkg_prerm')
+        splitrpostrm    = localdata.getVar('pkg_postrm')
+        splitrposttrans = localdata.getVar('pkg_posttrans')
 
         if not perfiledeps:
             # Add in summary of per file dependencies
@@ -405,6 +406,7 @@ python write_specfile () {
             srcrpostinst   = splitrpostinst
             srcrprerm      = splitrprerm
             srcrpostrm     = splitrpostrm
+            srcrposttrans  = splitrposttrans
 
             file_list = []
             walk_files(root, file_list, conffiles, dirfiles)
@@ -496,6 +498,11 @@ python write_specfile () {
             scriptvar = wrap_uninstall(splitrpostrm)
             spec_scriptlets_bottom.append(scriptvar)
             spec_scriptlets_bottom.append('')
+        if splitrposttrans:
+            spec_scriptlets_bottom.append('%%posttrans -n %s' % splitname)
+            spec_scriptlets_bottom.append('# %s - posttrans' % splitname)
+            spec_scriptlets_bottom.append(splitrposttrans)
+            spec_scriptlets_bottom.append('')
 
         # Now process files
         file_list = []
@@ -590,6 +597,11 @@ python write_specfile () {
         scriptvar = wrap_uninstall(srcrpostrm)
         spec_scriptlets_top.append(scriptvar)
         spec_scriptlets_top.append('')
+    if srcrposttrans:
+        spec_scriptlets_top.append('%posttrans')
+        spec_scriptlets_top.append('# %s - posttrans' % srcname)
+        spec_scriptlets_top.append(srcrposttrans)
+        spec_scriptlets_top.append('')
 
     # Write the SPEC file
     specfile = open(outspecfile, 'w')
-- 
2.31.1

[PATCH 2/6] kernel-module-split.bbclass: Support zstd-compressed modules

[Zoltan Boszormenyi]

From: Zoltán Böszörményi <zboszor@gmail.com>

Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
---
 meta/classes/kernel-module-split.bbclass | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/meta/classes/kernel-module-split.bbclass b/meta/classes/kernel-module-split.bbclass
index b56dd4a9c7..6c1de4c992 100644
--- a/meta/classes/kernel-module-split.bbclass
+++ b/meta/classes/kernel-module-split.bbclass
@@ -44,17 +44,20 @@ python split_kernel_module_packages () {
     def extract_modinfo(file):
         import tempfile, subprocess
         tempfile.tempdir = d.getVar("WORKDIR")
-        compressed = re.match( r'.*\.([xg])z$', file)
+        compressed = re.match( r'.*\.(gz|xz|zst)$', file)
         tf = tempfile.mkstemp()
         tmpfile = tf[1]
         if compressed:
             tmpkofile = tmpfile + ".ko"
-            if compressed.group(1) == 'g':
+            if compressed.group(1) == 'gz':
                 cmd = "gunzip -dc %s > %s" % (file, tmpkofile)
                 subprocess.check_call(cmd, shell=True)
-            elif compressed.group(1) == 'x':
+            elif compressed.group(1) == 'xz':
                 cmd = "xz -dc %s > %s" % (file, tmpkofile)
                 subprocess.check_call(cmd, shell=True)
+            elif compressed.group(1) == 'zst':
+                cmd = "zstd -dc %s > %s" % (file, tmpkofile)
+                subprocess.check_call(cmd, shell=True)
             else:
                 msg = "Cannot decompress '%s'" % file
                 raise msg
@@ -153,7 +156,7 @@ python split_kernel_module_packages () {
     kernel_package_name = d.getVar("KERNEL_PACKAGE_NAME") or "kernel"
     kernel_version = d.getVar("KERNEL_VERSION")
 
-    module_regex = r'^(.*)\.k?o(?:\.[xg]z)?$'
+    module_regex = r'^(.*)\.k?o(?:\.(gz|xz|zst))?$'
 
     module_pattern_prefix = d.getVar('KERNEL_MODULE_PACKAGE_PREFIX')
     module_pattern_suffix = d.getVar('KERNEL_MODULE_PACKAGE_SUFFIX')
-- 
2.31.1

[PATCH 3/6] kernel-module-split.bbclass: Allow opt-out of split kernel modules

[Zoltan Boszormenyi]

From: Zoltán Böszörményi <zboszor@gmail.com>

For some use cases, a monolithic kernel-modules package containing
all modules built from the kernel sources may be preferred.
For one, download time is shorter and installation time is faster.

Set KERNEL_SPLIT_MODULES="0" for this.

The default is one subpackage per module.

Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
---
 meta/classes/kernel-module-split.bbclass | 17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)

diff --git a/meta/classes/kernel-module-split.bbclass b/meta/classes/kernel-module-split.bbclass
index 6c1de4c992..a29c294810 100644
--- a/meta/classes/kernel-module-split.bbclass
+++ b/meta/classes/kernel-module-split.bbclass
@@ -28,6 +28,7 @@ do_install:append() {
 	install -d ${D}${sysconfdir}/modules-load.d/ ${D}${sysconfdir}/modprobe.d/
 }
 
+KERNEL_SPLIT_MODULES ?= "1"
 PACKAGESPLITFUNCS:prepend = "split_kernel_module_packages "
 
 KERNEL_MODULES_META_PACKAGE ?= "${@ d.getVar("KERNEL_PACKAGE_NAME") or "kernel" }-modules"
@@ -156,18 +157,26 @@ python split_kernel_module_packages () {
     kernel_package_name = d.getVar("KERNEL_PACKAGE_NAME") or "kernel"
     kernel_version = d.getVar("KERNEL_VERSION")
 
+    metapkg = d.getVar('KERNEL_MODULES_META_PACKAGE')
+    splitmods = d.getVar('KERNEL_SPLIT_MODULES')
+    postinst = d.getVar('pkg_postinst:modules')
+    postrm = d.getVar('pkg_postrm:modules')
+
+    if splitmods != '1':
+        etcdir = d.getVar('sysconfdir')
+        d.appendVar('FILES:' + metapkg, '%s/modules-load.d/ %s/modprobe.d/ %s/modules/' % (etcdir, etcdir, d.getVar("nonarch_base_libdir")))
+        d.appendVar('pkg_postinst:%s' % metapkg, postinst)
+        d.prependVar('pkg_postrm:%s' % metapkg, postrm);
+        return
+
     module_regex = r'^(.*)\.k?o(?:\.(gz|xz|zst))?$'
 
     module_pattern_prefix = d.getVar('KERNEL_MODULE_PACKAGE_PREFIX')
     module_pattern_suffix = d.getVar('KERNEL_MODULE_PACKAGE_SUFFIX')
     module_pattern = module_pattern_prefix + kernel_package_name + '-module-%s' + module_pattern_suffix
 
-    postinst = d.getVar('pkg_postinst:modules')
-    postrm = d.getVar('pkg_postrm:modules')
-
     modules = do_split_packages(d, root='${nonarch_base_libdir}/modules', file_regex=module_regex, output_pattern=module_pattern, description='%s kernel module', postinst=postinst, postrm=postrm, recursive=True, hook=frob_metadata, extra_depends='%s-%s' % (kernel_package_name, kernel_version))
     if modules:
-        metapkg = d.getVar('KERNEL_MODULES_META_PACKAGE')
         d.appendVar('RDEPENDS:' + metapkg, ' '+' '.join(modules))
 
     # If modules-load.d and modprobe.d are empty at this point, remove them to
-- 
2.31.1

[PATCH 4/6] kernel.bbclass: Adapt to KERNEL_SPLIT_MODULES != "1" case

[Zoltan Boszormenyi]

From: Zoltán Böszörményi <zboszor@gmail.com>

Extra RDEPENDS and other inter-package references are needed.

Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
---
 meta/classes/kernel.bbclass | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
index 027e66eec7..6dc5387a9b 100644
--- a/meta/classes/kernel.bbclass
+++ b/meta/classes/kernel.bbclass
@@ -98,6 +98,13 @@ python __anonymous () {
         d.appendVar('PACKAGES', ' %s-image-%s' % (kname, typelower))
         d.setVar('FILES:' + kname + '-image-' + typelower, '/' + imagedest + '/' + type + '-${KERNEL_VERSION_NAME}' + ' /' + imagedest + '/' + type)
         d.appendVar('RDEPENDS:%s-image' % kname, ' %s-image-%s' % (kname, typelower))
+        splitmods = d.getVar("KERNEL_SPLIT_MODULES")
+        if splitmods != '1':
+            d.appendVar('RDEPENDS:%s-image' % kname, ' %s-modules' % kname)
+            d.appendVar('RDEPENDS:%s-image-%s' % (kname, typelower), ' %s-modules-${KERNEL_VERSION_PKG_NAME}' % kname)
+            d.setVar('PKG:%s-modules' % kname, '%s-modules-${KERNEL_VERSION_PKG_NAME}' % kname)
+            d.appendVar('RPROVIDES:%s-modules' % kname, '%s-modules-${KERNEL_VERSION_PKG_NAME}' % kname)
+
         d.setVar('PKG:%s-image-%s' % (kname,typelower), '%s-image-%s-${KERNEL_VERSION_PKG_NAME}' % (kname, typelower))
         d.setVar('ALLOW_EMPTY:%s-image-%s' % (kname, typelower), '1')
         d.setVar('pkg_postinst:%s-image-%s' % (kname,typelower), """set +e
-- 
2.31.1

[PATCH 5/6] kernel.bbclass: Use full versions for inter-package dependencies

[Zoltan Boszormenyi]

From: Zoltán Böszörményi <zboszor@gmail.com>

Some Yocto users do use package repositories and sometimes the PR
value is forgotten. Use full versions for inter-package dependencies
for the kernel subpackages.

Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
---
 meta/classes/kernel.bbclass | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)

diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
index 6dc5387a9b..9ec7daa17a 100644
--- a/meta/classes/kernel.bbclass
+++ b/meta/classes/kernel.bbclass
@@ -91,17 +91,18 @@ python __anonymous () {
     kname = d.getVar('KERNEL_PACKAGE_NAME') or "kernel"
     imagedest = d.getVar('KERNEL_IMAGEDEST')
 
+    fullver = d.getVar('EXTENDPKGV')
     for type in types.split():
         if bb.data.inherits_class('nopackages', d):
             continue
         typelower = type.lower()
         d.appendVar('PACKAGES', ' %s-image-%s' % (kname, typelower))
         d.setVar('FILES:' + kname + '-image-' + typelower, '/' + imagedest + '/' + type + '-${KERNEL_VERSION_NAME}' + ' /' + imagedest + '/' + type)
-        d.appendVar('RDEPENDS:%s-image' % kname, ' %s-image-%s' % (kname, typelower))
+        d.appendVar('RDEPENDS:%s-image' % kname, ' %s-image-%s (= %s)' % (kname, typelower, fullver))
         splitmods = d.getVar("KERNEL_SPLIT_MODULES")
         if splitmods != '1':
-            d.appendVar('RDEPENDS:%s-image' % kname, ' %s-modules' % kname)
-            d.appendVar('RDEPENDS:%s-image-%s' % (kname, typelower), ' %s-modules-${KERNEL_VERSION_PKG_NAME}' % kname)
+            d.appendVar('RDEPENDS:%s-image' % kname, ' %s-modules (= %s)' % (kname, fullver))
+            d.appendVar('RDEPENDS:%s-image-%s' % (kname, typelower), ' %s-modules-${KERNEL_VERSION_PKG_NAME} (= %s)' % (kname, fullver))
             d.setVar('PKG:%s-modules' % kname, '%s-modules-${KERNEL_VERSION_PKG_NAME}' % kname)
             d.appendVar('RPROVIDES:%s-modules' % kname, '%s-modules-${KERNEL_VERSION_PKG_NAME}' % kname)
 
@@ -628,12 +629,12 @@ FILES:${KERNEL_PACKAGE_NAME}-image = ""
 FILES:${KERNEL_PACKAGE_NAME}-dev = "/boot/System.map* /boot/Module.symvers* /boot/config* ${KERNEL_SRC_PATH} ${nonarch_base_libdir}/modules/${KERNEL_VERSION}/build"
 FILES:${KERNEL_PACKAGE_NAME}-vmlinux = "/boot/vmlinux-${KERNEL_VERSION_NAME}"
 FILES:${KERNEL_PACKAGE_NAME}-modules = ""
-RDEPENDS:${KERNEL_PACKAGE_NAME} = "${KERNEL_PACKAGE_NAME}-base"
+RDEPENDS:${KERNEL_PACKAGE_NAME} = "${KERNEL_PACKAGE_NAME}-base (= ${EXTENDPKGV})"
 # Allow machines to override this dependency if kernel image files are
 # not wanted in images as standard
-RDEPENDS:${KERNEL_PACKAGE_NAME}-base ?= "${KERNEL_PACKAGE_NAME}-image"
+RDEPENDS:${KERNEL_PACKAGE_NAME}-base ?= "${KERNEL_PACKAGE_NAME}-image (= ${EXTENDPKGV})"
 PKG:${KERNEL_PACKAGE_NAME}-image = "${KERNEL_PACKAGE_NAME}-image-${@legitimize_package_name(d.getVar('KERNEL_VERSION'))}"
-RDEPENDS:${KERNEL_PACKAGE_NAME}-image += "${@oe.utils.conditional('KERNEL_IMAGETYPE', 'vmlinux', '${KERNEL_PACKAGE_NAME}-vmlinux', '', d)}"
+RDEPENDS:${KERNEL_PACKAGE_NAME}-image += "${@oe.utils.conditional('KERNEL_IMAGETYPE', 'vmlinux', '${KERNEL_PACKAGE_NAME}-vmlinux (= ${EXTENDPKGV})', '', d)}"
 PKG:${KERNEL_PACKAGE_NAME}-base = "${KERNEL_PACKAGE_NAME}-${@legitimize_package_name(d.getVar('KERNEL_VERSION'))}"
 RPROVIDES:${KERNEL_PACKAGE_NAME}-base += "${KERNEL_PACKAGE_NAME}-${KERNEL_VERSION}"
 ALLOW_EMPTY:${KERNEL_PACKAGE_NAME} = "1"
-- 
2.31.1

[PATCH 6/6] Support zstd-compressed squashfs and cpio initramfs

[Zoltan Boszormenyi]

From: Zoltán Böszörményi <zboszor@gmail.com>

Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
---
 meta/classes/base.bbclass            |  4 ++++
 meta/classes/image_types.bbclass     |  6 ++++--
 meta/classes/kernel-fitimage.bbclass |  2 +-
 meta/classes/kernel.bbclass          | 10 ++++++++--
 4 files changed, 17 insertions(+), 5 deletions(-)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 11b65171d9..340ebe7d78 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -631,6 +631,10 @@ python () {
         if path.endswith('.lz4'):
             d.appendVarFlag('do_unpack', 'depends', ' lz4-native:do_populate_sysroot')
 
+        # *.zst should DEPEND on zstd-native for unpacking
+        elif path.endswith('.zst'):
+            d.appendVarFlag('do_unpack', 'depends', ' zstd-native:do_populate_sysroot')
+
         # *.lz should DEPEND on lzip-native for unpacking
         elif path.endswith('.lz'):
             d.appendVarFlag('do_unpack', 'depends', ' lzip-native:do_populate_sysroot')
diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index e0eb06c1e3..32d4cd4c76 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -107,6 +107,7 @@ IMAGE_CMD:squashfs = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${
 IMAGE_CMD:squashfs-xz = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-xz ${EXTRA_IMAGECMD} -noappend -comp xz"
 IMAGE_CMD:squashfs-lzo = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-lzo ${EXTRA_IMAGECMD} -noappend -comp lzo"
 IMAGE_CMD:squashfs-lz4 = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-lz4 ${EXTRA_IMAGECMD} -noappend -comp lz4"
+IMAGE_CMD:squashfs-zst = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-zst ${EXTRA_IMAGECMD} -noappend -comp zstd"
 
 IMAGE_CMD:erofs = "mkfs.erofs ${EXTRA_IMAGECMD} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.erofs ${IMAGE_ROOTFS}"
 IMAGE_CMD:erofs-lz4 = "mkfs.erofs -zlz4 ${EXTRA_IMAGECMD} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.erofs-lz4 ${IMAGE_ROOTFS}"
@@ -244,6 +245,7 @@ do_image_squashfs[depends] += "squashfs-tools-native:do_populate_sysroot"
 do_image_squashfs_xz[depends] += "squashfs-tools-native:do_populate_sysroot"
 do_image_squashfs_lzo[depends] += "squashfs-tools-native:do_populate_sysroot"
 do_image_squashfs_lz4[depends] += "squashfs-tools-native:do_populate_sysroot"
+do_image_squashfs_zst[depends] += "squashfs-tools-native:do_populate_sysroot"
 do_image_ubi[depends] += "mtd-utils-native:do_populate_sysroot"
 do_image_ubifs[depends] += "mtd-utils-native:do_populate_sysroot"
 do_image_multiubi[depends] += "mtd-utils-native:do_populate_sysroot"
@@ -262,10 +264,10 @@ IMAGE_TYPES = " \
     btrfs \
     iso \
     hddimg \
-    squashfs squashfs-xz squashfs-lzo squashfs-lz4 \
+    squashfs squashfs-xz squashfs-lzo squashfs-lz4 squashfs-zst \
     ubi ubifs multiubi \
     tar tar.gz tar.bz2 tar.xz tar.lz4 tar.zst \
-    cpio cpio.gz cpio.xz cpio.lzma cpio.lz4 \
+    cpio cpio.gz cpio.xz cpio.lzma cpio.lz4 cpio.zst \
     wic wic.gz wic.bz2 wic.lzma wic.zst \
     container \
     f2fs \
diff --git a/meta/classes/kernel-fitimage.bbclass b/meta/classes/kernel-fitimage.bbclass
index 2ef8f06b14..38e05153e3 100644
--- a/meta/classes/kernel-fitimage.bbclass
+++ b/meta/classes/kernel-fitimage.bbclass
@@ -576,7 +576,7 @@ fitimage_assemble() {
 	#
 	if [ "x${ramdiskcount}" = "x1" ] && [ "${INITRAMFS_IMAGE_BUNDLE}" != "1" ]; then
 		# Find and use the first initramfs image archive type we find
-		for img in cpio.lz4 cpio.lzo cpio.lzma cpio.xz cpio.gz ext2.gz cpio; do
+		for img in cpio.lz4 cpio.lzo cpio.lzma cpio.xz cpio.zst cpio.gz ext2.gz cpio; do
 			initramfs_path="${DEPLOY_DIR_IMAGE}/${INITRAMFS_IMAGE_NAME}.${img}"
 			echo "Using $initramfs_path"
 			if [ -e "${initramfs_path}" ]; then
diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
index 9ec7daa17a..ab60e060c6 100644
--- a/meta/classes/kernel.bbclass
+++ b/meta/classes/kernel.bbclass
@@ -9,6 +9,7 @@ PROVIDES += "${@ "virtual/kernel" if (d.getVar("KERNEL_PACKAGE_NAME") == "kernel
 DEPENDS += "virtual/${TARGET_PREFIX}binutils virtual/${TARGET_PREFIX}gcc kmod-native bc-native bison-native"
 DEPENDS += "${@bb.utils.contains("INITRAMFS_FSTYPES", "cpio.lzo", "lzop-native", "", d)}"
 DEPENDS += "${@bb.utils.contains("INITRAMFS_FSTYPES", "cpio.lz4", "lz4-native", "", d)}"
+DEPENDS += "${@bb.utils.contains("INITRAMFS_FSTYPES", "cpio.zst", "zstd-native", "", d)}"
 PACKAGE_WRITE_DEPS += "depmodwrapper-cross"
 
 do_deploy[depends] += "depmodwrapper-cross:do_populate_sysroot gzip-native:do_populate_sysroot"
@@ -237,7 +238,7 @@ copy_initramfs() {
 	mkdir -p ${B}/usr
 	# Find and use the first initramfs image archive type we find
 	rm -f ${B}/usr/${INITRAMFS_IMAGE_NAME}.cpio
-	for img in cpio cpio.gz cpio.lz4 cpio.lzo cpio.lzma cpio.xz; do
+	for img in cpio cpio.gz cpio.lz4 cpio.lzo cpio.lzma cpio.xz cpio.zst; do
 		if [ -e "${DEPLOY_DIR_IMAGE}/${INITRAMFS_IMAGE_NAME}.$img" ]; then
 			cp ${DEPLOY_DIR_IMAGE}/${INITRAMFS_IMAGE_NAME}.$img ${B}/usr/.
 			case $img in
@@ -266,12 +267,17 @@ copy_initramfs() {
 				xz -df ${B}/usr/${INITRAMFS_IMAGE_NAME}.$img
 				break
 				;;
+			*zst)
+				echo "zst decompressing image"
+				zstd -df ${B}/usr/${INITRAMFS_IMAGE_NAME}.$img
+				break
+				;;
 			esac
 			break
 		fi
 	done
 	# Verify that the above loop found a initramfs, fail otherwise
-	[ -f ${B}/usr/${INITRAMFS_IMAGE_NAME}.cpio ] && echo "Finished copy of initramfs into ./usr" || die "Could not find any ${DEPLOY_DIR_IMAGE}/${INITRAMFS_IMAGE_NAME}.cpio{.gz|.lz4|.lzo|.lzma|.xz) for bundling; INITRAMFS_IMAGE_NAME might be wrong."
+	[ -f ${B}/usr/${INITRAMFS_IMAGE_NAME}.cpio ] && echo "Finished copy of initramfs into ./usr" || die "Could not find any ${DEPLOY_DIR_IMAGE}/${INITRAMFS_IMAGE_NAME}.cpio{.gz|.lz4|.lzo|.lzma|.xz|.zst) for bundling; INITRAMFS_IMAGE_NAME might be wrong."
 }
 
 do_bundle_initramfs () {
-- 
2.31.1

Re: [OE-core] [PATCH 1/6] package_rpm.bbclass: Handle posttrans scriptlets

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 3754 bytes --]

This needs to be better documented and tested.

What does this posttrans thing really do?
Who would want to use it?
Can there be examples?
Can the Postinst test in meta/lib/oeqa/selftest/cases/runtime_test.py be
extended to regression-check that it works?

Alex

On Mon, 23 Aug 2021 at 14:23, Zoltan Boszormenyi via lists.openembedded.org
<zboszor=pr.hu@lists.openembedded.org> wrote:

> From: Zoltán Böszörményi <zboszor@gmail.com>
>
> The "posttrans" scriptlet is the RPM equivalent of the
> OPKG "intercept script" concept and probably a cleaner one.
>
> "pretrans" also exists in RPM but there's no equivalent
> for it in OPKG.
>
> Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
> ---
>  meta/classes/package_rpm.bbclass | 30 +++++++++++++++++++++---------
>  1 file changed, 21 insertions(+), 9 deletions(-)
>
> diff --git a/meta/classes/package_rpm.bbclass
> b/meta/classes/package_rpm.bbclass
> index 88d861c0e7..5992d3fd06 100644
> --- a/meta/classes/package_rpm.bbclass
> +++ b/meta/classes/package_rpm.bbclass
> @@ -308,10 +308,11 @@ python write_specfile () {
>      srcrconflicts  = ""
>      srcrobsoletes  = ""
>
> -    srcrpreinst  = []
> -    srcrpostinst = []
> -    srcrprerm    = []
> -    srcrpostrm   = []
> +    srcrpreinst   = []
> +    srcrpostinst  = []
> +    srcrprerm     = []
> +    srcrpostrm    = []
> +    srcrposttrans = []
>
>      spec_preamble_top = []
>      spec_preamble_bottom = []
> @@ -373,11 +374,11 @@ python write_specfile () {
>          splitrconflicts  = localdata.getVar('RCONFLICTS') or ""
>          splitrobsoletes  = ""
>
> -        splitrpreinst  = localdata.getVar('pkg_preinst')
> -        splitrpostinst = localdata.getVar('pkg_postinst')
> -        splitrprerm    = localdata.getVar('pkg_prerm')
> -        splitrpostrm   = localdata.getVar('pkg_postrm')
> -
> +        splitrpreinst   = localdata.getVar('pkg_preinst')
> +        splitrpostinst  = localdata.getVar('pkg_postinst')
> +        splitrprerm     = localdata.getVar('pkg_prerm')
> +        splitrpostrm    = localdata.getVar('pkg_postrm')
> +        splitrposttrans = localdata.getVar('pkg_posttrans')
>
>          if not perfiledeps:
>              # Add in summary of per file dependencies
> @@ -405,6 +406,7 @@ python write_specfile () {
>              srcrpostinst   = splitrpostinst
>              srcrprerm      = splitrprerm
>              srcrpostrm     = splitrpostrm
> +            srcrposttrans  = splitrposttrans
>
>              file_list = []
>              walk_files(root, file_list, conffiles, dirfiles)
> @@ -496,6 +498,11 @@ python write_specfile () {
>              scriptvar = wrap_uninstall(splitrpostrm)
>              spec_scriptlets_bottom.append(scriptvar)
>              spec_scriptlets_bottom.append('')
> +        if splitrposttrans:
> +            spec_scriptlets_bottom.append('%%posttrans -n %s' % splitname)
> +            spec_scriptlets_bottom.append('# %s - posttrans' % splitname)
> +            spec_scriptlets_bottom.append(splitrposttrans)
> +            spec_scriptlets_bottom.append('')
>
>          # Now process files
>          file_list = []
> @@ -590,6 +597,11 @@ python write_specfile () {
>          scriptvar = wrap_uninstall(srcrpostrm)
>          spec_scriptlets_top.append(scriptvar)
>          spec_scriptlets_top.append('')
> +    if srcrposttrans:
> +        spec_scriptlets_top.append('%posttrans')
> +        spec_scriptlets_top.append('# %s - posttrans' % srcname)
> +        spec_scriptlets_top.append(srcrposttrans)
> +        spec_scriptlets_top.append('')
>
>      # Write the SPEC file
>      specfile = open(outspecfile, 'w')
> --
> 2.31.1
>
>
> 
>
>

[-- Attachment #2: Type: text/html, Size: 4987 bytes --]

Re: [OE-core] [PATCH 3/6] kernel-module-split.bbclass: Allow opt-out of split kernel modules

[Bruce Ashfield]

On Mon, Aug 23, 2021 at 8:23 AM Zoltan Boszormenyi via
lists.openembedded.org <zboszor=pr.hu@lists.openembedded.org> wrote:
>
> From: Zoltán Böszörményi <zboszor@gmail.com>
>
> For some use cases, a monolithic kernel-modules package containing
> all modules built from the kernel sources may be preferred.
> For one, download time is shorter and installation time is faster.
>

There's still an outstanding issue though. Unless you generate the
provides/rdepends along with the monolithic package, any existing
rdepends on kernel modules break.

While that likely isn't something you hit in your use case, it would
be something that would have to be covered to make this part of core.

About 7 years ago, I was most of the way through a similar change,
that created on large package with all the rprovides generated (I'm
not sure I can track it down now, but I could have a look), since
doing an on target update with thousands of kernel module package is
very painful.

Cheers,

Bruce

> Set KERNEL_SPLIT_MODULES="0" for this.
>
> The default is one subpackage per module.
>
> Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
> ---
>  meta/classes/kernel-module-split.bbclass | 17 +++++++++++++----
>  1 file changed, 13 insertions(+), 4 deletions(-)
>
> diff --git a/meta/classes/kernel-module-split.bbclass b/meta/classes/kernel-module-split.bbclass
> index 6c1de4c992..a29c294810 100644
> --- a/meta/classes/kernel-module-split.bbclass
> +++ b/meta/classes/kernel-module-split.bbclass
> @@ -28,6 +28,7 @@ do_install:append() {
>         install -d ${D}${sysconfdir}/modules-load.d/ ${D}${sysconfdir}/modprobe.d/
>  }
>
> +KERNEL_SPLIT_MODULES ?= "1"
>  PACKAGESPLITFUNCS:prepend = "split_kernel_module_packages "
>
>  KERNEL_MODULES_META_PACKAGE ?= "${@ d.getVar("KERNEL_PACKAGE_NAME") or "kernel" }-modules"
> @@ -156,18 +157,26 @@ python split_kernel_module_packages () {
>      kernel_package_name = d.getVar("KERNEL_PACKAGE_NAME") or "kernel"
>      kernel_version = d.getVar("KERNEL_VERSION")
>
> +    metapkg = d.getVar('KERNEL_MODULES_META_PACKAGE')
> +    splitmods = d.getVar('KERNEL_SPLIT_MODULES')
> +    postinst = d.getVar('pkg_postinst:modules')
> +    postrm = d.getVar('pkg_postrm:modules')
> +
> +    if splitmods != '1':
> +        etcdir = d.getVar('sysconfdir')
> +        d.appendVar('FILES:' + metapkg, '%s/modules-load.d/ %s/modprobe.d/ %s/modules/' % (etcdir, etcdir, d.getVar("nonarch_base_libdir")))
> +        d.appendVar('pkg_postinst:%s' % metapkg, postinst)
> +        d.prependVar('pkg_postrm:%s' % metapkg, postrm);
> +        return
> +
>      module_regex = r'^(.*)\.k?o(?:\.(gz|xz|zst))?$'
>
>      module_pattern_prefix = d.getVar('KERNEL_MODULE_PACKAGE_PREFIX')
>      module_pattern_suffix = d.getVar('KERNEL_MODULE_PACKAGE_SUFFIX')
>      module_pattern = module_pattern_prefix + kernel_package_name + '-module-%s' + module_pattern_suffix
>
> -    postinst = d.getVar('pkg_postinst:modules')
> -    postrm = d.getVar('pkg_postrm:modules')
> -
>      modules = do_split_packages(d, root='${nonarch_base_libdir}/modules', file_regex=module_regex, output_pattern=module_pattern, description='%s kernel module', postinst=postinst, postrm=postrm, recursive=True, hook=frob_metadata, extra_depends='%s-%s' % (kernel_package_name, kernel_version))
>      if modules:
> -        metapkg = d.getVar('KERNEL_MODULES_META_PACKAGE')
>          d.appendVar('RDEPENDS:' + metapkg, ' '+' '.join(modules))
>
>      # If modules-load.d and modprobe.d are empty at this point, remove them to
> --
> 2.31.1
>
>
> 
>


-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Re: [OE-core] [PATCH 3/6] kernel-module-split.bbclass: Allow opt-out of split kernel modules

[Bruce Ashfield]

On Mon, Aug 23, 2021 at 8:55 AM Bruce Ashfield via
lists.openembedded.org
<bruce.ashfield=gmail.com@lists.openembedded.org> wrote:
>
> On Mon, Aug 23, 2021 at 8:23 AM Zoltan Boszormenyi via
> lists.openembedded.org <zboszor=pr.hu@lists.openembedded.org> wrote:
> >
> > From: Zoltán Böszörményi <zboszor@gmail.com>
> >
> > For some use cases, a monolithic kernel-modules package containing
> > all modules built from the kernel sources may be preferred.
> > For one, download time is shorter and installation time is faster.
> >
>
> There's still an outstanding issue though. Unless you generate the
> provides/rdepends along with the monolithic package, any existing
> rdepends on kernel modules break.

Ha.

Never mind, I see the next patch in the series.

It would be worthwhile to document that other changes handle that
detail in the commit log, that way if we are looking for issues in the
future, we know there is hope :D

Bruce

>
> While that likely isn't something you hit in your use case, it would
> be something that would have to be covered to make this part of core.
>
> About 7 years ago, I was most of the way through a similar change,
> that created on large package with all the rprovides generated (I'm
> not sure I can track it down now, but I could have a look), since
> doing an on target update with thousands of kernel module package is
> very painful.
>
> Cheers,
>
> Bruce
>
> > Set KERNEL_SPLIT_MODULES="0" for this.
> >
> > The default is one subpackage per module.
> >
> > Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
> > ---
> >  meta/classes/kernel-module-split.bbclass | 17 +++++++++++++----
> >  1 file changed, 13 insertions(+), 4 deletions(-)
> >
> > diff --git a/meta/classes/kernel-module-split.bbclass b/meta/classes/kernel-module-split.bbclass
> > index 6c1de4c992..a29c294810 100644
> > --- a/meta/classes/kernel-module-split.bbclass
> > +++ b/meta/classes/kernel-module-split.bbclass
> > @@ -28,6 +28,7 @@ do_install:append() {
> >         install -d ${D}${sysconfdir}/modules-load.d/ ${D}${sysconfdir}/modprobe.d/
> >  }
> >
> > +KERNEL_SPLIT_MODULES ?= "1"
> >  PACKAGESPLITFUNCS:prepend = "split_kernel_module_packages "
> >
> >  KERNEL_MODULES_META_PACKAGE ?= "${@ d.getVar("KERNEL_PACKAGE_NAME") or "kernel" }-modules"
> > @@ -156,18 +157,26 @@ python split_kernel_module_packages () {
> >      kernel_package_name = d.getVar("KERNEL_PACKAGE_NAME") or "kernel"
> >      kernel_version = d.getVar("KERNEL_VERSION")
> >
> > +    metapkg = d.getVar('KERNEL_MODULES_META_PACKAGE')
> > +    splitmods = d.getVar('KERNEL_SPLIT_MODULES')
> > +    postinst = d.getVar('pkg_postinst:modules')
> > +    postrm = d.getVar('pkg_postrm:modules')
> > +
> > +    if splitmods != '1':
> > +        etcdir = d.getVar('sysconfdir')
> > +        d.appendVar('FILES:' + metapkg, '%s/modules-load.d/ %s/modprobe.d/ %s/modules/' % (etcdir, etcdir, d.getVar("nonarch_base_libdir")))
> > +        d.appendVar('pkg_postinst:%s' % metapkg, postinst)
> > +        d.prependVar('pkg_postrm:%s' % metapkg, postrm);
> > +        return
> > +
> >      module_regex = r'^(.*)\.k?o(?:\.(gz|xz|zst))?$'
> >
> >      module_pattern_prefix = d.getVar('KERNEL_MODULE_PACKAGE_PREFIX')
> >      module_pattern_suffix = d.getVar('KERNEL_MODULE_PACKAGE_SUFFIX')
> >      module_pattern = module_pattern_prefix + kernel_package_name + '-module-%s' + module_pattern_suffix
> >
> > -    postinst = d.getVar('pkg_postinst:modules')
> > -    postrm = d.getVar('pkg_postrm:modules')
> > -
> >      modules = do_split_packages(d, root='${nonarch_base_libdir}/modules', file_regex=module_regex, output_pattern=module_pattern, description='%s kernel module', postinst=postinst, postrm=postrm, recursive=True, hook=frob_metadata, extra_depends='%s-%s' % (kernel_package_name, kernel_version))
> >      if modules:
> > -        metapkg = d.getVar('KERNEL_MODULES_META_PACKAGE')
> >          d.appendVar('RDEPENDS:' + metapkg, ' '+' '.join(modules))
> >
> >      # If modules-load.d and modprobe.d are empty at this point, remove them to
> > --
> > 2.31.1
> >
> >
> >
> >
>
>
> --
> - Thou shalt not follow the NULL pointer, for chaos and madness await
> thee at its end
> - "Use the force Harry" - Gandalf, Star Trek II
>
> 
>


-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Re: [OE-core] [PATCH 5/6] kernel.bbclass: Use full versions for inter-package dependencies

[Bruce Ashfield]

On Mon, Aug 23, 2021 at 8:23 AM Zoltan Boszormenyi via
lists.openembedded.org <zboszor=pr.hu@lists.openembedded.org> wrote:
>
> From: Zoltán Böszörményi <zboszor@gmail.com>
>
> Some Yocto users do use package repositories and sometimes the PR
> value is forgotten. Use full versions for inter-package dependencies
> for the kernel subpackages.

Can you expand this commit message ? What is the observable problem
with the full versions not being in the rdpends ?

I've run many different package feeds with different kernels and
modules, and haven't run into anything on this front, in my experience
the extended version in the package depends/provides causes issues
with some package managers .. so we'd need the testing for this
documented/provided so we could ensure that there are no hidden
issues.

Bruce


>
> Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
> ---
>  meta/classes/kernel.bbclass | 13 +++++++------
>  1 file changed, 7 insertions(+), 6 deletions(-)
>
> diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
> index 6dc5387a9b..9ec7daa17a 100644
> --- a/meta/classes/kernel.bbclass
> +++ b/meta/classes/kernel.bbclass
> @@ -91,17 +91,18 @@ python __anonymous () {
>      kname = d.getVar('KERNEL_PACKAGE_NAME') or "kernel"
>      imagedest = d.getVar('KERNEL_IMAGEDEST')
>
> +    fullver = d.getVar('EXTENDPKGV')
>      for type in types.split():
>          if bb.data.inherits_class('nopackages', d):
>              continue
>          typelower = type.lower()
>          d.appendVar('PACKAGES', ' %s-image-%s' % (kname, typelower))
>          d.setVar('FILES:' + kname + '-image-' + typelower, '/' + imagedest + '/' + type + '-${KERNEL_VERSION_NAME}' + ' /' + imagedest + '/' + type)
> -        d.appendVar('RDEPENDS:%s-image' % kname, ' %s-image-%s' % (kname, typelower))
> +        d.appendVar('RDEPENDS:%s-image' % kname, ' %s-image-%s (= %s)' % (kname, typelower, fullver))
>          splitmods = d.getVar("KERNEL_SPLIT_MODULES")
>          if splitmods != '1':
> -            d.appendVar('RDEPENDS:%s-image' % kname, ' %s-modules' % kname)
> -            d.appendVar('RDEPENDS:%s-image-%s' % (kname, typelower), ' %s-modules-${KERNEL_VERSION_PKG_NAME}' % kname)
> +            d.appendVar('RDEPENDS:%s-image' % kname, ' %s-modules (= %s)' % (kname, fullver))
> +            d.appendVar('RDEPENDS:%s-image-%s' % (kname, typelower), ' %s-modules-${KERNEL_VERSION_PKG_NAME} (= %s)' % (kname, fullver))
>              d.setVar('PKG:%s-modules' % kname, '%s-modules-${KERNEL_VERSION_PKG_NAME}' % kname)
>              d.appendVar('RPROVIDES:%s-modules' % kname, '%s-modules-${KERNEL_VERSION_PKG_NAME}' % kname)
>
> @@ -628,12 +629,12 @@ FILES:${KERNEL_PACKAGE_NAME}-image = ""
>  FILES:${KERNEL_PACKAGE_NAME}-dev = "/boot/System.map* /boot/Module.symvers* /boot/config* ${KERNEL_SRC_PATH} ${nonarch_base_libdir}/modules/${KERNEL_VERSION}/build"
>  FILES:${KERNEL_PACKAGE_NAME}-vmlinux = "/boot/vmlinux-${KERNEL_VERSION_NAME}"
>  FILES:${KERNEL_PACKAGE_NAME}-modules = ""
> -RDEPENDS:${KERNEL_PACKAGE_NAME} = "${KERNEL_PACKAGE_NAME}-base"
> +RDEPENDS:${KERNEL_PACKAGE_NAME} = "${KERNEL_PACKAGE_NAME}-base (= ${EXTENDPKGV})"
>  # Allow machines to override this dependency if kernel image files are
>  # not wanted in images as standard
> -RDEPENDS:${KERNEL_PACKAGE_NAME}-base ?= "${KERNEL_PACKAGE_NAME}-image"
> +RDEPENDS:${KERNEL_PACKAGE_NAME}-base ?= "${KERNEL_PACKAGE_NAME}-image (= ${EXTENDPKGV})"
>  PKG:${KERNEL_PACKAGE_NAME}-image = "${KERNEL_PACKAGE_NAME}-image-${@legitimize_package_name(d.getVar('KERNEL_VERSION'))}"
> -RDEPENDS:${KERNEL_PACKAGE_NAME}-image += "${@oe.utils.conditional('KERNEL_IMAGETYPE', 'vmlinux', '${KERNEL_PACKAGE_NAME}-vmlinux', '', d)}"
> +RDEPENDS:${KERNEL_PACKAGE_NAME}-image += "${@oe.utils.conditional('KERNEL_IMAGETYPE', 'vmlinux', '${KERNEL_PACKAGE_NAME}-vmlinux (= ${EXTENDPKGV})', '', d)}"
>  PKG:${KERNEL_PACKAGE_NAME}-base = "${KERNEL_PACKAGE_NAME}-${@legitimize_package_name(d.getVar('KERNEL_VERSION'))}"
>  RPROVIDES:${KERNEL_PACKAGE_NAME}-base += "${KERNEL_PACKAGE_NAME}-${KERNEL_VERSION}"
>  ALLOW_EMPTY:${KERNEL_PACKAGE_NAME} = "1"
> --
> 2.31.1
>
>
> 
>


-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Re: [OE-core] [PATCH 1/6] package_rpm.bbclass: Handle posttrans scriptlets

[Zoltan Boszormenyi]

It's documented at www.rpm.org, Red Hat and SuSE.

https://docs.fedoraproject.org/en-US/packaging-guidelines/Scriptlets/
https://en.opensuse.org/openSUSE:Packaging_scriptlet_snippets
https://rpm-packaging-guide.github.io/

It's available since RPM 4.4. RPM 4.16.1.3 is in Hardknott.

A short Google search showed that *maybe* dpkg also supports it,
or at least there are(were?) plans to implement it:
https://wiki.debian.org/i18n/TranslationDebsProposals

With postinst, the time when the scriptlet runs may depend
on the order of the packages in the upgrade transaction and
may run with the older binaries in place.

The posttrans scriptlet runs at the very end of the
install/upgrade transaction, just like the OPKG "intercept"
scripts.

I successfully tested it long ago with a kernel upgrade
that was running dracut on the new version, plus a dracut
upgrade in the same session. Dissecting the initramfs proved
that with postinst, the old dracut version generated it,
while with posttrans, the new one did.

Who would want to use it? Anyone who needs some finalizer
script to be run at the end of an RPM/DNF transaction.

I started on Angstrom and I know opkg doesn't implement this
and Yocto inherited it. This patch is basically a heads up
that there's another way to do the same thing. Maybe opkg
implements this some day, maybe not. Anyway, Yocto in general
will need more changes before the main package.bbclass can
start to use it for all package formats.

Zoltán

2021. 08. 23. 14:34 keltezéssel, Alexander Kanavin írta:
> This needs to be better documented and tested.
> 
> What does this posttrans thing really do?
> Who would want to use it?
> Can there be examples?
> Can the Postinst test in meta/lib/oeqa/selftest/cases/runtime_test.py be extended to 
> regression-check that it works?
> 
> Alex
> 
> On Mon, 23 Aug 2021 at 14:23, Zoltan Boszormenyi via lists.openembedded.org 
> <http://lists.openembedded.org> <zboszor=pr.hu@lists.openembedded.org 
> <mailto:pr.hu@lists.openembedded.org>> wrote:
> 
>     From: Zoltán Böszörményi <zboszor@gmail.com <mailto:zboszor@gmail.com>>
> 
>     The "posttrans" scriptlet is the RPM equivalent of the
>     OPKG "intercept script" concept and probably a cleaner one.
> 
>     "pretrans" also exists in RPM but there's no equivalent
>     for it in OPKG.
> 
>     Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com <mailto:zboszor@gmail.com>>
>     ---
>       meta/classes/package_rpm.bbclass | 30 +++++++++++++++++++++---------
>       1 file changed, 21 insertions(+), 9 deletions(-)
> 
>     diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass
>     index 88d861c0e7..5992d3fd06 100644
>     --- a/meta/classes/package_rpm.bbclass
>     +++ b/meta/classes/package_rpm.bbclass
>     @@ -308,10 +308,11 @@ python write_specfile () {
>           srcrconflicts  = ""
>           srcrobsoletes  = ""
> 
>     -    srcrpreinst  = []
>     -    srcrpostinst = []
>     -    srcrprerm    = []
>     -    srcrpostrm   = []
>     +    srcrpreinst   = []
>     +    srcrpostinst  = []
>     +    srcrprerm     = []
>     +    srcrpostrm    = []
>     +    srcrposttrans = []
> 
>           spec_preamble_top = []
>           spec_preamble_bottom = []
>     @@ -373,11 +374,11 @@ python write_specfile () {
>               splitrconflicts  = localdata.getVar('RCONFLICTS') or ""
>               splitrobsoletes  = ""
> 
>     -        splitrpreinst  = localdata.getVar('pkg_preinst')
>     -        splitrpostinst = localdata.getVar('pkg_postinst')
>     -        splitrprerm    = localdata.getVar('pkg_prerm')
>     -        splitrpostrm   = localdata.getVar('pkg_postrm')
>     -
>     +        splitrpreinst   = localdata.getVar('pkg_preinst')
>     +        splitrpostinst  = localdata.getVar('pkg_postinst')
>     +        splitrprerm     = localdata.getVar('pkg_prerm')
>     +        splitrpostrm    = localdata.getVar('pkg_postrm')
>     +        splitrposttrans = localdata.getVar('pkg_posttrans')
> 
>               if not perfiledeps:
>                   # Add in summary of per file dependencies
>     @@ -405,6 +406,7 @@ python write_specfile () {
>                   srcrpostinst   = splitrpostinst
>                   srcrprerm      = splitrprerm
>                   srcrpostrm     = splitrpostrm
>     +            srcrposttrans  = splitrposttrans
> 
>                   file_list = []
>                   walk_files(root, file_list, conffiles, dirfiles)
>     @@ -496,6 +498,11 @@ python write_specfile () {
>                   scriptvar = wrap_uninstall(splitrpostrm)
>                   spec_scriptlets_bottom.append(scriptvar)
>                   spec_scriptlets_bottom.append('')
>     +        if splitrposttrans:
>     +            spec_scriptlets_bottom.append('%%posttrans -n %s' % splitname)
>     +            spec_scriptlets_bottom.append('# %s - posttrans' % splitname)
>     +            spec_scriptlets_bottom.append(splitrposttrans)
>     +            spec_scriptlets_bottom.append('')
> 
>               # Now process files
>               file_list = []
>     @@ -590,6 +597,11 @@ python write_specfile () {
>               scriptvar = wrap_uninstall(srcrpostrm)
>               spec_scriptlets_top.append(scriptvar)
>               spec_scriptlets_top.append('')
>     +    if srcrposttrans:
>     +        spec_scriptlets_top.append('%posttrans')
>     +        spec_scriptlets_top.append('# %s - posttrans' % srcname)
>     +        spec_scriptlets_top.append(srcrposttrans)
>     +        spec_scriptlets_top.append('')
> 
>           # Write the SPEC file
>           specfile = open(outspecfile, 'w')
>     -- 
>     2.31.1
> 
> 
> 
> 
> 
> 
> 
> 

Re: [OE-core] [PATCH 3/6] kernel-module-split.bbclass: Allow opt-out of split kernel modules

[Zoltan Boszormenyi]

2021. 08. 23. 14:55 keltezéssel, Bruce Ashfield írta:
> On Mon, Aug 23, 2021 at 8:23 AM Zoltan Boszormenyi via
> lists.openembedded.org <zboszor=pr.hu@lists.openembedded.org> wrote:
>>
>> From: Zoltán Böszörményi <zboszor@gmail.com>
>>
>> For some use cases, a monolithic kernel-modules package containing
>> all modules built from the kernel sources may be preferred.
>> For one, download time is shorter and installation time is faster.
>>
> 
> There's still an outstanding issue though. Unless you generate the
> provides/rdepends along with the monolithic package, any existing
> rdepends on kernel modules break.

The next patch in the thread does it.

> 
> While that likely isn't something you hit in your use case, it would
> be something that would have to be covered to make this part of core.
> 
> About 7 years ago, I was most of the way through a similar change,
> that created on large package with all the rprovides generated (I'm
> not sure I can track it down now, but I could have a look), since
> doing an on target update with thousands of kernel module package is
> very painful.

Well, yes. In our defconfig, about 1500 kernel-module-* subpackages
were created and had to be upgraded at once. It was *very* painful
with over 50 minutes to download and upgrade the kernel.

> 
> Cheers,
> 
> Bruce
> 
>> Set KERNEL_SPLIT_MODULES="0" for this.
>>
>> The default is one subpackage per module.
>>
>> Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
>> ---
>>   meta/classes/kernel-module-split.bbclass | 17 +++++++++++++----
>>   1 file changed, 13 insertions(+), 4 deletions(-)
>>
>> diff --git a/meta/classes/kernel-module-split.bbclass b/meta/classes/kernel-module-split.bbclass
>> index 6c1de4c992..a29c294810 100644
>> --- a/meta/classes/kernel-module-split.bbclass
>> +++ b/meta/classes/kernel-module-split.bbclass
>> @@ -28,6 +28,7 @@ do_install:append() {
>>          install -d ${D}${sysconfdir}/modules-load.d/ ${D}${sysconfdir}/modprobe.d/
>>   }
>>
>> +KERNEL_SPLIT_MODULES ?= "1"
>>   PACKAGESPLITFUNCS:prepend = "split_kernel_module_packages "
>>
>>   KERNEL_MODULES_META_PACKAGE ?= "${@ d.getVar("KERNEL_PACKAGE_NAME") or "kernel" }-modules"
>> @@ -156,18 +157,26 @@ python split_kernel_module_packages () {
>>       kernel_package_name = d.getVar("KERNEL_PACKAGE_NAME") or "kernel"
>>       kernel_version = d.getVar("KERNEL_VERSION")
>>
>> +    metapkg = d.getVar('KERNEL_MODULES_META_PACKAGE')
>> +    splitmods = d.getVar('KERNEL_SPLIT_MODULES')
>> +    postinst = d.getVar('pkg_postinst:modules')
>> +    postrm = d.getVar('pkg_postrm:modules')
>> +
>> +    if splitmods != '1':
>> +        etcdir = d.getVar('sysconfdir')
>> +        d.appendVar('FILES:' + metapkg, '%s/modules-load.d/ %s/modprobe.d/ %s/modules/' % (etcdir, etcdir, d.getVar("nonarch_base_libdir")))
>> +        d.appendVar('pkg_postinst:%s' % metapkg, postinst)
>> +        d.prependVar('pkg_postrm:%s' % metapkg, postrm);
>> +        return
>> +
>>       module_regex = r'^(.*)\.k?o(?:\.(gz|xz|zst))?$'
>>
>>       module_pattern_prefix = d.getVar('KERNEL_MODULE_PACKAGE_PREFIX')
>>       module_pattern_suffix = d.getVar('KERNEL_MODULE_PACKAGE_SUFFIX')
>>       module_pattern = module_pattern_prefix + kernel_package_name + '-module-%s' + module_pattern_suffix
>>
>> -    postinst = d.getVar('pkg_postinst:modules')
>> -    postrm = d.getVar('pkg_postrm:modules')
>> -
>>       modules = do_split_packages(d, root='${nonarch_base_libdir}/modules', file_regex=module_regex, output_pattern=module_pattern, description='%s kernel module', postinst=postinst, postrm=postrm, recursive=True, hook=frob_metadata, extra_depends='%s-%s' % (kernel_package_name, kernel_version))
>>       if modules:
>> -        metapkg = d.getVar('KERNEL_MODULES_META_PACKAGE')
>>           d.appendVar('RDEPENDS:' + metapkg, ' '+' '.join(modules))
>>
>>       # If modules-load.d and modprobe.d are empty at this point, remove them to
>> --
>> 2.31.1
>>
>>
>>
>>
> 
> 
> 
> 
> 
> 

Re: [OE-core] [PATCH 1/6] package_rpm.bbclass: Handle posttrans scriptlets

[Richard Purdie]

On Mon, 2021-08-23 at 15:14 +0200, Zoltan Boszormenyi via lists.openembedded.org
wrote:
> It's documented at www.rpm.org, Red Hat and SuSE.
> 
> https://docs.fedoraproject.org/en-US/packaging-guidelines/Scriptlets/
> https://en.opensuse.org/openSUSE:Packaging_scriptlet_snippets
> https://rpm-packaging-guide.github.io/
> 
> It's available since RPM 4.4. RPM 4.16.1.3 is in Hardknott.
> 
> A short Google search showed that *maybe* dpkg also supports it,
> or at least there are(were?) plans to implement it:
> https://wiki.debian.org/i18n/TranslationDebsProposals
> 
> With postinst, the time when the scriptlet runs may depend
> on the order of the packages in the upgrade transaction and
> may run with the older binaries in place.
> 
> The posttrans scriptlet runs at the very end of the
> install/upgrade transaction, just like the OPKG "intercept"
> scripts.
> 
> I successfully tested it long ago with a kernel upgrade
> that was running dracut on the new version, plus a dracut
> upgrade in the same session. Dissecting the initramfs proved
> that with postinst, the old dracut version generated it,
> while with posttrans, the new one did.
> 
> Who would want to use it? Anyone who needs some finalizer
> script to be run at the end of an RPM/DNF transaction.
> 
> I started on Angstrom and I know opkg doesn't implement this
> and Yocto inherited it. This patch is basically a heads up
> that there's another way to do the same thing. Maybe opkg
> implements this some day, maybe not. Anyway, Yocto in general
> will need more changes before the main package.bbclass can
> start to use it for all package formats.
> 

There is a bigger issue here which is how can anyone use this without making
their recipe rpm specific? I'm very reluctant to add functionality like this
unless we have a way to emulate it or fall back on the other package formats.

The intercept pieces are used by both rpm and opkg and actually do something
slightly different, they stack common calls until the end of the transaction.
This means for example we can call "ldconfig" once rather than in every
postinst.

Cheers,

Richard

Re: [OE-core] [PATCH 3/6] kernel-module-split.bbclass: Allow opt-out of split kernel modules

[Bruce Ashfield]

On Mon, Aug 23, 2021 at 9:18 AM Böszörményi Zoltán <zboszor@pr.hu> wrote:
>
> 2021. 08. 23. 14:55 keltezéssel, Bruce Ashfield írta:
> > On Mon, Aug 23, 2021 at 8:23 AM Zoltan Boszormenyi via
> > lists.openembedded.org <zboszor=pr.hu@lists.openembedded.org> wrote:
> >>
> >> From: Zoltán Böszörményi <zboszor@gmail.com>
> >>
> >> For some use cases, a monolithic kernel-modules package containing
> >> all modules built from the kernel sources may be preferred.
> >> For one, download time is shorter and installation time is faster.
> >>
> >
> > There's still an outstanding issue though. Unless you generate the
> > provides/rdepends along with the monolithic package, any existing
> > rdepends on kernel modules break.
>
> The next patch in the thread does it.

That's what my follow up said ;)

See my second reply though, this gap in functionality created by this
commit, should be documented in the commit that it will be restored in
future ones.

Cheers,

Bruce

>
> >
> > While that likely isn't something you hit in your use case, it would
> > be something that would have to be covered to make this part of core.
> >
> > About 7 years ago, I was most of the way through a similar change,
> > that created on large package with all the rprovides generated (I'm
> > not sure I can track it down now, but I could have a look), since
> > doing an on target update with thousands of kernel module package is
> > very painful.
>
> Well, yes. In our defconfig, about 1500 kernel-module-* subpackages
> were created and had to be upgraded at once. It was *very* painful
> with over 50 minutes to download and upgrade the kernel.
>
> >
> > Cheers,
> >
> > Bruce
> >
> >> Set KERNEL_SPLIT_MODULES="0" for this.
> >>
> >> The default is one subpackage per module.
> >>
> >> Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
> >> ---
> >>   meta/classes/kernel-module-split.bbclass | 17 +++++++++++++----
> >>   1 file changed, 13 insertions(+), 4 deletions(-)
> >>
> >> diff --git a/meta/classes/kernel-module-split.bbclass b/meta/classes/kernel-module-split.bbclass
> >> index 6c1de4c992..a29c294810 100644
> >> --- a/meta/classes/kernel-module-split.bbclass
> >> +++ b/meta/classes/kernel-module-split.bbclass
> >> @@ -28,6 +28,7 @@ do_install:append() {
> >>          install -d ${D}${sysconfdir}/modules-load.d/ ${D}${sysconfdir}/modprobe.d/
> >>   }
> >>
> >> +KERNEL_SPLIT_MODULES ?= "1"
> >>   PACKAGESPLITFUNCS:prepend = "split_kernel_module_packages "
> >>
> >>   KERNEL_MODULES_META_PACKAGE ?= "${@ d.getVar("KERNEL_PACKAGE_NAME") or "kernel" }-modules"
> >> @@ -156,18 +157,26 @@ python split_kernel_module_packages () {
> >>       kernel_package_name = d.getVar("KERNEL_PACKAGE_NAME") or "kernel"
> >>       kernel_version = d.getVar("KERNEL_VERSION")
> >>
> >> +    metapkg = d.getVar('KERNEL_MODULES_META_PACKAGE')
> >> +    splitmods = d.getVar('KERNEL_SPLIT_MODULES')
> >> +    postinst = d.getVar('pkg_postinst:modules')
> >> +    postrm = d.getVar('pkg_postrm:modules')
> >> +
> >> +    if splitmods != '1':
> >> +        etcdir = d.getVar('sysconfdir')
> >> +        d.appendVar('FILES:' + metapkg, '%s/modules-load.d/ %s/modprobe.d/ %s/modules/' % (etcdir, etcdir, d.getVar("nonarch_base_libdir")))
> >> +        d.appendVar('pkg_postinst:%s' % metapkg, postinst)
> >> +        d.prependVar('pkg_postrm:%s' % metapkg, postrm);
> >> +        return
> >> +
> >>       module_regex = r'^(.*)\.k?o(?:\.(gz|xz|zst))?$'
> >>
> >>       module_pattern_prefix = d.getVar('KERNEL_MODULE_PACKAGE_PREFIX')
> >>       module_pattern_suffix = d.getVar('KERNEL_MODULE_PACKAGE_SUFFIX')
> >>       module_pattern = module_pattern_prefix + kernel_package_name + '-module-%s' + module_pattern_suffix
> >>
> >> -    postinst = d.getVar('pkg_postinst:modules')
> >> -    postrm = d.getVar('pkg_postrm:modules')
> >> -
> >>       modules = do_split_packages(d, root='${nonarch_base_libdir}/modules', file_regex=module_regex, output_pattern=module_pattern, description='%s kernel module', postinst=postinst, postrm=postrm, recursive=True, hook=frob_metadata, extra_depends='%s-%s' % (kernel_package_name, kernel_version))
> >>       if modules:
> >> -        metapkg = d.getVar('KERNEL_MODULES_META_PACKAGE')
> >>           d.appendVar('RDEPENDS:' + metapkg, ' '+' '.join(modules))
> >>
> >>       # If modules-load.d and modprobe.d are empty at this point, remove them to
> >> --
> >> 2.31.1
> >>
> >>
> >>
> >>
> >
> >
> >
> >
> > 
> >
>


-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Re: [OE-core] [PATCH 5/6] kernel.bbclass: Use full versions for inter-package dependencies

[Zoltan Boszormenyi]

2021. 08. 23. 15:03 keltezéssel, Bruce Ashfield írta:
> On Mon, Aug 23, 2021 at 8:23 AM Zoltan Boszormenyi via
> lists.openembedded.org <zboszor=pr.hu@lists.openembedded.org> wrote:
>>
>> From: Zoltán Böszörményi <zboszor@gmail.com>
>>
>> Some Yocto users do use package repositories and sometimes the PR
>> value is forgotten. Use full versions for inter-package dependencies
>> for the kernel subpackages.
> 
> Can you expand this commit message ?

I will send a new version with the expanded message.

> What is the observable problem
> with the full versions not being in the rdpends ?

Currently the kernel creates such subpackages:

kernel
kernel-x.y.z
kernel-image-x.y.z
kernel-image-bzimage-x.y.z
kernel-modules-x.y.z

and in the split modules case, a lot of

kernel-module-*-x.y.z

RDEPENDS currently only references the package names.

The issue I observe is this: I have enabled kernel module signing
but I don't provide a key, so the kernel will generate one at
every rebuild.

You get the idea what problem this can make if there's a
rebuild:

x.y.z-r0.0 -> x.y.z-r1.0 (in which case the PR value was duly changed)
or
x.y.z-r0.0 -> x.y.z-r0.1 (where not)

It's actually indifferent, but the same issue occurs if
only the kernel-image-bzimage is upgraded: the modules either
won't work or taint the kernel.

This scenario is currently valid in the package manager because
of the versionless RDEPENDS.

> 
> I've run many different package feeds with different kernels and
> modules, and haven't run into anything on this front, in my experience
> the extended version in the package depends/provides causes issues
> with some package managers .. so we'd need the testing for this
> documented/provided so we could ensure that there are no hidden
> issues.
> 
> Bruce
> 
> 
>>
>> Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
>> ---
>>   meta/classes/kernel.bbclass | 13 +++++++------
>>   1 file changed, 7 insertions(+), 6 deletions(-)
>>
>> diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
>> index 6dc5387a9b..9ec7daa17a 100644
>> --- a/meta/classes/kernel.bbclass
>> +++ b/meta/classes/kernel.bbclass
>> @@ -91,17 +91,18 @@ python __anonymous () {
>>       kname = d.getVar('KERNEL_PACKAGE_NAME') or "kernel"
>>       imagedest = d.getVar('KERNEL_IMAGEDEST')
>>
>> +    fullver = d.getVar('EXTENDPKGV')
>>       for type in types.split():
>>           if bb.data.inherits_class('nopackages', d):
>>               continue
>>           typelower = type.lower()
>>           d.appendVar('PACKAGES', ' %s-image-%s' % (kname, typelower))
>>           d.setVar('FILES:' + kname + '-image-' + typelower, '/' + imagedest + '/' + type + '-${KERNEL_VERSION_NAME}' + ' /' + imagedest + '/' + type)
>> -        d.appendVar('RDEPENDS:%s-image' % kname, ' %s-image-%s' % (kname, typelower))
>> +        d.appendVar('RDEPENDS:%s-image' % kname, ' %s-image-%s (= %s)' % (kname, typelower, fullver))
>>           splitmods = d.getVar("KERNEL_SPLIT_MODULES")
>>           if splitmods != '1':
>> -            d.appendVar('RDEPENDS:%s-image' % kname, ' %s-modules' % kname)
>> -            d.appendVar('RDEPENDS:%s-image-%s' % (kname, typelower), ' %s-modules-${KERNEL_VERSION_PKG_NAME}' % kname)
>> +            d.appendVar('RDEPENDS:%s-image' % kname, ' %s-modules (= %s)' % (kname, fullver))
>> +            d.appendVar('RDEPENDS:%s-image-%s' % (kname, typelower), ' %s-modules-${KERNEL_VERSION_PKG_NAME} (= %s)' % (kname, fullver))
>>               d.setVar('PKG:%s-modules' % kname, '%s-modules-${KERNEL_VERSION_PKG_NAME}' % kname)
>>               d.appendVar('RPROVIDES:%s-modules' % kname, '%s-modules-${KERNEL_VERSION_PKG_NAME}' % kname)
>>
>> @@ -628,12 +629,12 @@ FILES:${KERNEL_PACKAGE_NAME}-image = ""
>>   FILES:${KERNEL_PACKAGE_NAME}-dev = "/boot/System.map* /boot/Module.symvers* /boot/config* ${KERNEL_SRC_PATH} ${nonarch_base_libdir}/modules/${KERNEL_VERSION}/build"
>>   FILES:${KERNEL_PACKAGE_NAME}-vmlinux = "/boot/vmlinux-${KERNEL_VERSION_NAME}"
>>   FILES:${KERNEL_PACKAGE_NAME}-modules = ""
>> -RDEPENDS:${KERNEL_PACKAGE_NAME} = "${KERNEL_PACKAGE_NAME}-base"
>> +RDEPENDS:${KERNEL_PACKAGE_NAME} = "${KERNEL_PACKAGE_NAME}-base (= ${EXTENDPKGV})"
>>   # Allow machines to override this dependency if kernel image files are
>>   # not wanted in images as standard
>> -RDEPENDS:${KERNEL_PACKAGE_NAME}-base ?= "${KERNEL_PACKAGE_NAME}-image"
>> +RDEPENDS:${KERNEL_PACKAGE_NAME}-base ?= "${KERNEL_PACKAGE_NAME}-image (= ${EXTENDPKGV})"
>>   PKG:${KERNEL_PACKAGE_NAME}-image = "${KERNEL_PACKAGE_NAME}-image-${@legitimize_package_name(d.getVar('KERNEL_VERSION'))}"
>> -RDEPENDS:${KERNEL_PACKAGE_NAME}-image += "${@oe.utils.conditional('KERNEL_IMAGETYPE', 'vmlinux', '${KERNEL_PACKAGE_NAME}-vmlinux', '', d)}"
>> +RDEPENDS:${KERNEL_PACKAGE_NAME}-image += "${@oe.utils.conditional('KERNEL_IMAGETYPE', 'vmlinux', '${KERNEL_PACKAGE_NAME}-vmlinux (= ${EXTENDPKGV})', '', d)}"
>>   PKG:${KERNEL_PACKAGE_NAME}-base = "${KERNEL_PACKAGE_NAME}-${@legitimize_package_name(d.getVar('KERNEL_VERSION'))}"
>>   RPROVIDES:${KERNEL_PACKAGE_NAME}-base += "${KERNEL_PACKAGE_NAME}-${KERNEL_VERSION}"
>>   ALLOW_EMPTY:${KERNEL_PACKAGE_NAME} = "1"
>> --
>> 2.31.1
>>
>>
>> 
>>
> 
> 

Re: [OE-core] [PATCH 3/6] kernel-module-split.bbclass: Allow opt-out of split kernel modules

[Zoltan Boszormenyi]

2021. 08. 23. 15:27 keltezéssel, Bruce Ashfield írta:
> On Mon, Aug 23, 2021 at 9:18 AM Böszörményi Zoltán <zboszor@pr.hu> wrote:
>>
>> 2021. 08. 23. 14:55 keltezéssel, Bruce Ashfield írta:
>>> On Mon, Aug 23, 2021 at 8:23 AM Zoltan Boszormenyi via
>>> lists.openembedded.org <zboszor=pr.hu@lists.openembedded.org> wrote:
>>>>
>>>> From: Zoltán Böszörményi <zboszor@gmail.com>
>>>>
>>>> For some use cases, a monolithic kernel-modules package containing
>>>> all modules built from the kernel sources may be preferred.
>>>> For one, download time is shorter and installation time is faster.
>>>>
>>>
>>> There's still an outstanding issue though. Unless you generate the
>>> provides/rdepends along with the monolithic package, any existing
>>> rdepends on kernel modules break.
>>
>> The next patch in the thread does it.
> 
> That's what my follow up said ;)
> 
> See my second reply though, this gap in functionality created by this
> commit, should be documented in the commit that it will be restored in
> future ones.

I can squash the two commits so there's no gap.

> 
> Cheers,
> 
> Bruce
> 
>>
>>>
>>> While that likely isn't something you hit in your use case, it would
>>> be something that would have to be covered to make this part of core.
>>>
>>> About 7 years ago, I was most of the way through a similar change,
>>> that created on large package with all the rprovides generated (I'm
>>> not sure I can track it down now, but I could have a look), since
>>> doing an on target update with thousands of kernel module package is
>>> very painful.
>>
>> Well, yes. In our defconfig, about 1500 kernel-module-* subpackages
>> were created and had to be upgraded at once. It was *very* painful
>> with over 50 minutes to download and upgrade the kernel.
>>
>>>
>>> Cheers,
>>>
>>> Bruce
>>>
>>>> Set KERNEL_SPLIT_MODULES="0" for this.
>>>>
>>>> The default is one subpackage per module.
>>>>
>>>> Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
>>>> ---
>>>>    meta/classes/kernel-module-split.bbclass | 17 +++++++++++++----
>>>>    1 file changed, 13 insertions(+), 4 deletions(-)
>>>>
>>>> diff --git a/meta/classes/kernel-module-split.bbclass b/meta/classes/kernel-module-split.bbclass
>>>> index 6c1de4c992..a29c294810 100644
>>>> --- a/meta/classes/kernel-module-split.bbclass
>>>> +++ b/meta/classes/kernel-module-split.bbclass
>>>> @@ -28,6 +28,7 @@ do_install:append() {
>>>>           install -d ${D}${sysconfdir}/modules-load.d/ ${D}${sysconfdir}/modprobe.d/
>>>>    }
>>>>
>>>> +KERNEL_SPLIT_MODULES ?= "1"
>>>>    PACKAGESPLITFUNCS:prepend = "split_kernel_module_packages "
>>>>
>>>>    KERNEL_MODULES_META_PACKAGE ?= "${@ d.getVar("KERNEL_PACKAGE_NAME") or "kernel" }-modules"
>>>> @@ -156,18 +157,26 @@ python split_kernel_module_packages () {
>>>>        kernel_package_name = d.getVar("KERNEL_PACKAGE_NAME") or "kernel"
>>>>        kernel_version = d.getVar("KERNEL_VERSION")
>>>>
>>>> +    metapkg = d.getVar('KERNEL_MODULES_META_PACKAGE')
>>>> +    splitmods = d.getVar('KERNEL_SPLIT_MODULES')
>>>> +    postinst = d.getVar('pkg_postinst:modules')
>>>> +    postrm = d.getVar('pkg_postrm:modules')
>>>> +
>>>> +    if splitmods != '1':
>>>> +        etcdir = d.getVar('sysconfdir')
>>>> +        d.appendVar('FILES:' + metapkg, '%s/modules-load.d/ %s/modprobe.d/ %s/modules/' % (etcdir, etcdir, d.getVar("nonarch_base_libdir")))
>>>> +        d.appendVar('pkg_postinst:%s' % metapkg, postinst)
>>>> +        d.prependVar('pkg_postrm:%s' % metapkg, postrm);
>>>> +        return
>>>> +
>>>>        module_regex = r'^(.*)\.k?o(?:\.(gz|xz|zst))?$'
>>>>
>>>>        module_pattern_prefix = d.getVar('KERNEL_MODULE_PACKAGE_PREFIX')
>>>>        module_pattern_suffix = d.getVar('KERNEL_MODULE_PACKAGE_SUFFIX')
>>>>        module_pattern = module_pattern_prefix + kernel_package_name + '-module-%s' + module_pattern_suffix
>>>>
>>>> -    postinst = d.getVar('pkg_postinst:modules')
>>>> -    postrm = d.getVar('pkg_postrm:modules')
>>>> -
>>>>        modules = do_split_packages(d, root='${nonarch_base_libdir}/modules', file_regex=module_regex, output_pattern=module_pattern, description='%s kernel module', postinst=postinst, postrm=postrm, recursive=True, hook=frob_metadata, extra_depends='%s-%s' % (kernel_package_name, kernel_version))
>>>>        if modules:
>>>> -        metapkg = d.getVar('KERNEL_MODULES_META_PACKAGE')
>>>>            d.appendVar('RDEPENDS:' + metapkg, ' '+' '.join(modules))
>>>>
>>>>        # If modules-load.d and modprobe.d are empty at this point, remove them to
>>>> --
>>>> 2.31.1
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
> 
> 
> 
> 
> 
> 

Re: [OE-core] [PATCH 3/6] kernel-module-split.bbclass: Allow opt-out of split kernel modules

[Bruce Ashfield]

On Mon, Aug 23, 2021 at 9:31 AM Böszörményi Zoltán <zboszor@pr.hu> wrote:
>
> 2021. 08. 23. 15:27 keltezéssel, Bruce Ashfield írta:
> > On Mon, Aug 23, 2021 at 9:18 AM Böszörményi Zoltán <zboszor@pr.hu> wrote:
> >>
> >> 2021. 08. 23. 14:55 keltezéssel, Bruce Ashfield írta:
> >>> On Mon, Aug 23, 2021 at 8:23 AM Zoltan Boszormenyi via
> >>> lists.openembedded.org <zboszor=pr.hu@lists.openembedded.org> wrote:
> >>>>
> >>>> From: Zoltán Böszörményi <zboszor@gmail.com>
> >>>>
> >>>> For some use cases, a monolithic kernel-modules package containing
> >>>> all modules built from the kernel sources may be preferred.
> >>>> For one, download time is shorter and installation time is faster.
> >>>>
> >>>
> >>> There's still an outstanding issue though. Unless you generate the
> >>> provides/rdepends along with the monolithic package, any existing
> >>> rdepends on kernel modules break.
> >>
> >> The next patch in the thread does it.
> >
> > That's what my follow up said ;)
> >
> > See my second reply though, this gap in functionality created by this
> > commit, should be documented in the commit that it will be restored in
> > future ones.
>
> I can squash the two commits so there's no gap.

That's my preference (others may disagree), since they are related and
anyone that has poked around in this area may notice the issue (like I
did).

I don't normally bisect through oe-core, but it is nice to know that
if you were on this intermediate commit, that the dependencies would
stay valid/consistent.

Cheers,

Bruce

>
> >
> > Cheers,
> >
> > Bruce
> >
> >>
> >>>
> >>> While that likely isn't something you hit in your use case, it would
> >>> be something that would have to be covered to make this part of core.
> >>>
> >>> About 7 years ago, I was most of the way through a similar change,
> >>> that created on large package with all the rprovides generated (I'm
> >>> not sure I can track it down now, but I could have a look), since
> >>> doing an on target update with thousands of kernel module package is
> >>> very painful.
> >>
> >> Well, yes. In our defconfig, about 1500 kernel-module-* subpackages
> >> were created and had to be upgraded at once. It was *very* painful
> >> with over 50 minutes to download and upgrade the kernel.
> >>
> >>>
> >>> Cheers,
> >>>
> >>> Bruce
> >>>
> >>>> Set KERNEL_SPLIT_MODULES="0" for this.
> >>>>
> >>>> The default is one subpackage per module.
> >>>>
> >>>> Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
> >>>> ---
> >>>>    meta/classes/kernel-module-split.bbclass | 17 +++++++++++++----
> >>>>    1 file changed, 13 insertions(+), 4 deletions(-)
> >>>>
> >>>> diff --git a/meta/classes/kernel-module-split.bbclass b/meta/classes/kernel-module-split.bbclass
> >>>> index 6c1de4c992..a29c294810 100644
> >>>> --- a/meta/classes/kernel-module-split.bbclass
> >>>> +++ b/meta/classes/kernel-module-split.bbclass
> >>>> @@ -28,6 +28,7 @@ do_install:append() {
> >>>>           install -d ${D}${sysconfdir}/modules-load.d/ ${D}${sysconfdir}/modprobe.d/
> >>>>    }
> >>>>
> >>>> +KERNEL_SPLIT_MODULES ?= "1"
> >>>>    PACKAGESPLITFUNCS:prepend = "split_kernel_module_packages "
> >>>>
> >>>>    KERNEL_MODULES_META_PACKAGE ?= "${@ d.getVar("KERNEL_PACKAGE_NAME") or "kernel" }-modules"
> >>>> @@ -156,18 +157,26 @@ python split_kernel_module_packages () {
> >>>>        kernel_package_name = d.getVar("KERNEL_PACKAGE_NAME") or "kernel"
> >>>>        kernel_version = d.getVar("KERNEL_VERSION")
> >>>>
> >>>> +    metapkg = d.getVar('KERNEL_MODULES_META_PACKAGE')
> >>>> +    splitmods = d.getVar('KERNEL_SPLIT_MODULES')
> >>>> +    postinst = d.getVar('pkg_postinst:modules')
> >>>> +    postrm = d.getVar('pkg_postrm:modules')
> >>>> +
> >>>> +    if splitmods != '1':
> >>>> +        etcdir = d.getVar('sysconfdir')
> >>>> +        d.appendVar('FILES:' + metapkg, '%s/modules-load.d/ %s/modprobe.d/ %s/modules/' % (etcdir, etcdir, d.getVar("nonarch_base_libdir")))
> >>>> +        d.appendVar('pkg_postinst:%s' % metapkg, postinst)
> >>>> +        d.prependVar('pkg_postrm:%s' % metapkg, postrm);
> >>>> +        return
> >>>> +
> >>>>        module_regex = r'^(.*)\.k?o(?:\.(gz|xz|zst))?$'
> >>>>
> >>>>        module_pattern_prefix = d.getVar('KERNEL_MODULE_PACKAGE_PREFIX')
> >>>>        module_pattern_suffix = d.getVar('KERNEL_MODULE_PACKAGE_SUFFIX')
> >>>>        module_pattern = module_pattern_prefix + kernel_package_name + '-module-%s' + module_pattern_suffix
> >>>>
> >>>> -    postinst = d.getVar('pkg_postinst:modules')
> >>>> -    postrm = d.getVar('pkg_postrm:modules')
> >>>> -
> >>>>        modules = do_split_packages(d, root='${nonarch_base_libdir}/modules', file_regex=module_regex, output_pattern=module_pattern, description='%s kernel module', postinst=postinst, postrm=postrm, recursive=True, hook=frob_metadata, extra_depends='%s-%s' % (kernel_package_name, kernel_version))
> >>>>        if modules:
> >>>> -        metapkg = d.getVar('KERNEL_MODULES_META_PACKAGE')
> >>>>            d.appendVar('RDEPENDS:' + metapkg, ' '+' '.join(modules))
> >>>>
> >>>>        # If modules-load.d and modprobe.d are empty at this point, remove them to
> >>>> --
> >>>> 2.31.1
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> >
> >
> >
> >
> > 
> >
>


-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Re: [OE-core] [PATCH 5/6] kernel.bbclass: Use full versions for inter-package dependencies

[Bruce Ashfield]

On Mon, Aug 23, 2021 at 9:30 AM Böszörményi Zoltán <zboszor@pr.hu> wrote:
>
> 2021. 08. 23. 15:03 keltezéssel, Bruce Ashfield írta:
> > On Mon, Aug 23, 2021 at 8:23 AM Zoltan Boszormenyi via
> > lists.openembedded.org <zboszor=pr.hu@lists.openembedded.org> wrote:
> >>
> >> From: Zoltán Böszörményi <zboszor@gmail.com>
> >>
> >> Some Yocto users do use package repositories and sometimes the PR
> >> value is forgotten. Use full versions for inter-package dependencies
> >> for the kernel subpackages.
> >
> > Can you expand this commit message ?
>
> I will send a new version with the expanded message.
>
> > What is the observable problem
> > with the full versions not being in the rdpends ?
>
> Currently the kernel creates such subpackages:
>
> kernel
> kernel-x.y.z
> kernel-image-x.y.z
> kernel-image-bzimage-x.y.z
> kernel-modules-x.y.z
>
> and in the split modules case, a lot of
>
> kernel-module-*-x.y.z
>
> RDEPENDS currently only references the package names.
>
> The issue I observe is this: I have enabled kernel module signing
> but I don't provide a key, so the kernel will generate one at
> every rebuild.
>
> You get the idea what problem this can make if there's a
> rebuild:
>
> x.y.z-r0.0 -> x.y.z-r1.0 (in which case the PR value was duly changed)
> or
> x.y.z-r0.0 -> x.y.z-r0.1 (where not)
>
> It's actually indifferent, but the same issue occurs if
> only the kernel-image-bzimage is upgraded: the modules either
> won't work or taint the kernel.

Your second example is similar to the times I've seen some module
issues as well, and honestly, I can't recall if it was solved in a
similar way to this (I don't have access to my old repos from Wind
River any more).

But that does make sense to me, and the details you have in this
follow up, make a nice commit explanation.

While opkg may not handle the situation you describe as well as RPM,
we should make sure that the full version doesn't cause issues with
the package generation or standard image creation.

Cheers,

Bruce

>
> This scenario is currently valid in the package manager because
> of the versionless RDEPENDS.
>
> >
> > I've run many different package feeds with different kernels and
> > modules, and haven't run into anything on this front, in my experience
> > the extended version in the package depends/provides causes issues
> > with some package managers .. so we'd need the testing for this
> > documented/provided so we could ensure that there are no hidden
> > issues.
> >
> > Bruce
> >
> >
> >>
> >> Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
> >> ---
> >>   meta/classes/kernel.bbclass | 13 +++++++------
> >>   1 file changed, 7 insertions(+), 6 deletions(-)
> >>
> >> diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
> >> index 6dc5387a9b..9ec7daa17a 100644
> >> --- a/meta/classes/kernel.bbclass
> >> +++ b/meta/classes/kernel.bbclass
> >> @@ -91,17 +91,18 @@ python __anonymous () {
> >>       kname = d.getVar('KERNEL_PACKAGE_NAME') or "kernel"
> >>       imagedest = d.getVar('KERNEL_IMAGEDEST')
> >>
> >> +    fullver = d.getVar('EXTENDPKGV')
> >>       for type in types.split():
> >>           if bb.data.inherits_class('nopackages', d):
> >>               continue
> >>           typelower = type.lower()
> >>           d.appendVar('PACKAGES', ' %s-image-%s' % (kname, typelower))
> >>           d.setVar('FILES:' + kname + '-image-' + typelower, '/' + imagedest + '/' + type + '-${KERNEL_VERSION_NAME}' + ' /' + imagedest + '/' + type)
> >> -        d.appendVar('RDEPENDS:%s-image' % kname, ' %s-image-%s' % (kname, typelower))
> >> +        d.appendVar('RDEPENDS:%s-image' % kname, ' %s-image-%s (= %s)' % (kname, typelower, fullver))
> >>           splitmods = d.getVar("KERNEL_SPLIT_MODULES")
> >>           if splitmods != '1':
> >> -            d.appendVar('RDEPENDS:%s-image' % kname, ' %s-modules' % kname)
> >> -            d.appendVar('RDEPENDS:%s-image-%s' % (kname, typelower), ' %s-modules-${KERNEL_VERSION_PKG_NAME}' % kname)
> >> +            d.appendVar('RDEPENDS:%s-image' % kname, ' %s-modules (= %s)' % (kname, fullver))
> >> +            d.appendVar('RDEPENDS:%s-image-%s' % (kname, typelower), ' %s-modules-${KERNEL_VERSION_PKG_NAME} (= %s)' % (kname, fullver))
> >>               d.setVar('PKG:%s-modules' % kname, '%s-modules-${KERNEL_VERSION_PKG_NAME}' % kname)
> >>               d.appendVar('RPROVIDES:%s-modules' % kname, '%s-modules-${KERNEL_VERSION_PKG_NAME}' % kname)
> >>
> >> @@ -628,12 +629,12 @@ FILES:${KERNEL_PACKAGE_NAME}-image = ""
> >>   FILES:${KERNEL_PACKAGE_NAME}-dev = "/boot/System.map* /boot/Module.symvers* /boot/config* ${KERNEL_SRC_PATH} ${nonarch_base_libdir}/modules/${KERNEL_VERSION}/build"
> >>   FILES:${KERNEL_PACKAGE_NAME}-vmlinux = "/boot/vmlinux-${KERNEL_VERSION_NAME}"
> >>   FILES:${KERNEL_PACKAGE_NAME}-modules = ""
> >> -RDEPENDS:${KERNEL_PACKAGE_NAME} = "${KERNEL_PACKAGE_NAME}-base"
> >> +RDEPENDS:${KERNEL_PACKAGE_NAME} = "${KERNEL_PACKAGE_NAME}-base (= ${EXTENDPKGV})"
> >>   # Allow machines to override this dependency if kernel image files are
> >>   # not wanted in images as standard
> >> -RDEPENDS:${KERNEL_PACKAGE_NAME}-base ?= "${KERNEL_PACKAGE_NAME}-image"
> >> +RDEPENDS:${KERNEL_PACKAGE_NAME}-base ?= "${KERNEL_PACKAGE_NAME}-image (= ${EXTENDPKGV})"
> >>   PKG:${KERNEL_PACKAGE_NAME}-image = "${KERNEL_PACKAGE_NAME}-image-${@legitimize_package_name(d.getVar('KERNEL_VERSION'))}"
> >> -RDEPENDS:${KERNEL_PACKAGE_NAME}-image += "${@oe.utils.conditional('KERNEL_IMAGETYPE', 'vmlinux', '${KERNEL_PACKAGE_NAME}-vmlinux', '', d)}"
> >> +RDEPENDS:${KERNEL_PACKAGE_NAME}-image += "${@oe.utils.conditional('KERNEL_IMAGETYPE', 'vmlinux', '${KERNEL_PACKAGE_NAME}-vmlinux (= ${EXTENDPKGV})', '', d)}"
> >>   PKG:${KERNEL_PACKAGE_NAME}-base = "${KERNEL_PACKAGE_NAME}-${@legitimize_package_name(d.getVar('KERNEL_VERSION'))}"
> >>   RPROVIDES:${KERNEL_PACKAGE_NAME}-base += "${KERNEL_PACKAGE_NAME}-${KERNEL_VERSION}"
> >>   ALLOW_EMPTY:${KERNEL_PACKAGE_NAME} = "1"
> >> --
> >> 2.31.1
> >>
> >>
> >> 
> >>
> >
> >
>


-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Re: [OE-core] [PATCH 1/6] package_rpm.bbclass: Handle posttrans scriptlets

[Zoltan Boszormenyi]

2021. 08. 23. 15:23 keltezéssel, Richard Purdie írta:
> On Mon, 2021-08-23 at 15:14 +0200, Zoltan Boszormenyi via lists.openembedded.org
> wrote:
>> It's documented at www.rpm.org, Red Hat and SuSE.
>>
>> https://docs.fedoraproject.org/en-US/packaging-guidelines/Scriptlets/
>> https://en.opensuse.org/openSUSE:Packaging_scriptlet_snippets
>> https://rpm-packaging-guide.github.io/
>>
>> It's available since RPM 4.4. RPM 4.16.1.3 is in Hardknott.
>>
>> A short Google search showed that *maybe* dpkg also supports it,
>> or at least there are(were?) plans to implement it:
>> https://wiki.debian.org/i18n/TranslationDebsProposals
>>
>> With postinst, the time when the scriptlet runs may depend
>> on the order of the packages in the upgrade transaction and
>> may run with the older binaries in place.
>>
>> The posttrans scriptlet runs at the very end of the
>> install/upgrade transaction, just like the OPKG "intercept"
>> scripts.
>>
>> I successfully tested it long ago with a kernel upgrade
>> that was running dracut on the new version, plus a dracut
>> upgrade in the same session. Dissecting the initramfs proved
>> that with postinst, the old dracut version generated it,
>> while with posttrans, the new one did.
>>
>> Who would want to use it? Anyone who needs some finalizer
>> script to be run at the end of an RPM/DNF transaction.
>>
>> I started on Angstrom and I know opkg doesn't implement this
>> and Yocto inherited it. This patch is basically a heads up
>> that there's another way to do the same thing. Maybe opkg
>> implements this some day, maybe not. Anyway, Yocto in general
>> will need more changes before the main package.bbclass can
>> start to use it for all package formats.
>>
> 
> There is a bigger issue here which is how can anyone use this without making
> their recipe rpm specific? I'm very reluctant to add functionality like this
> unless we have a way to emulate it or fall back on the other package formats.

Indeed. This patch can be dropped.

Full disclosure: I don't use this anymore, since I implemented
another two bbclass recipes to generate the kernel's initramfs
either during the kernel build (and ship it in a subpackage) or
during do_rootfs and move it out of the way in the image postprocess
script so e.g. a squashfs image will have it separately.
They work well for my use limited cases:
* full OS installed on disk
* squashfs for PXE-based remote booted machines

FYI: the dracutsysrootdir feature in recent dracut versions is my work.

I just carried this patch for a couple of years now and I thought maybe
someone sees some value in it.

> 
> The intercept pieces are used by both rpm and opkg and actually do something
> slightly different, they stack common calls until the end of the transaction.
> This means for example we can call "ldconfig" once rather than in every
> postinst.
> 
> Cheers,
> 
> Richard
> 
> 
> 
> 
> 
> 
> 

Re: [OE-core] [PATCH 5/6] kernel.bbclass: Use full versions for inter-package dependencies

[Zoltan Boszormenyi]

2021. 08. 23. 15:38 keltezéssel, Bruce Ashfield írta:
> On Mon, Aug 23, 2021 at 9:30 AM Böszörményi Zoltán <zboszor@pr.hu> wrote:
>>
>> 2021. 08. 23. 15:03 keltezéssel, Bruce Ashfield írta:
>>> On Mon, Aug 23, 2021 at 8:23 AM Zoltan Boszormenyi via
>>> lists.openembedded.org <zboszor=pr.hu@lists.openembedded.org> wrote:
>>>>
>>>> From: Zoltán Böszörményi <zboszor@gmail.com>
>>>>
>>>> Some Yocto users do use package repositories and sometimes the PR
>>>> value is forgotten. Use full versions for inter-package dependencies
>>>> for the kernel subpackages.
>>>
>>> Can you expand this commit message ?
>>
>> I will send a new version with the expanded message.
>>
>>> What is the observable problem
>>> with the full versions not being in the rdpends ?
>>
>> Currently the kernel creates such subpackages:
>>
>> kernel
>> kernel-x.y.z
>> kernel-image-x.y.z
>> kernel-image-bzimage-x.y.z
>> kernel-modules-x.y.z
>>
>> and in the split modules case, a lot of
>>
>> kernel-module-*-x.y.z
>>
>> RDEPENDS currently only references the package names.
>>
>> The issue I observe is this: I have enabled kernel module signing
>> but I don't provide a key, so the kernel will generate one at
>> every rebuild.
>>
>> You get the idea what problem this can make if there's a
>> rebuild:
>>
>> x.y.z-r0.0 -> x.y.z-r1.0 (in which case the PR value was duly changed)
>> or
>> x.y.z-r0.0 -> x.y.z-r0.1 (where not)
>>
>> It's actually indifferent, but the same issue occurs if
>> only the kernel-image-bzimage is upgraded: the modules either
>> won't work or taint the kernel.
> 
> Your second example is similar to the times I've seen some module
> issues as well, and honestly, I can't recall if it was solved in a
> similar way to this (I don't have access to my old repos from Wind
> River any more).
> 
> But that does make sense to me, and the details you have in this
> follow up, make a nice commit explanation.
> 
> While opkg may not handle the situation you describe as well as RPM,
> we should make sure that the full version doesn't cause issues with
> the package generation or standard image creation.

You can bet it doesn't. I have the same patch in a Sumo-based distro
where we still use ipk and opkg and there's no issue in either image
generation or package upgrades. Only that there's no "installonly"
knowledge in opkg (unlike dnf) so old kernel packages have to be
manually cleaned up there.

> 
> Cheers,
> 
> Bruce
> 
>>
>> This scenario is currently valid in the package manager because
>> of the versionless RDEPENDS.
>>
>>>
>>> I've run many different package feeds with different kernels and
>>> modules, and haven't run into anything on this front, in my experience
>>> the extended version in the package depends/provides causes issues
>>> with some package managers .. so we'd need the testing for this
>>> documented/provided so we could ensure that there are no hidden
>>> issues.
>>>
>>> Bruce
>>>
>>>
>>>>
>>>> Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
>>>> ---
>>>>    meta/classes/kernel.bbclass | 13 +++++++------
>>>>    1 file changed, 7 insertions(+), 6 deletions(-)
>>>>
>>>> diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
>>>> index 6dc5387a9b..9ec7daa17a 100644
>>>> --- a/meta/classes/kernel.bbclass
>>>> +++ b/meta/classes/kernel.bbclass
>>>> @@ -91,17 +91,18 @@ python __anonymous () {
>>>>        kname = d.getVar('KERNEL_PACKAGE_NAME') or "kernel"
>>>>        imagedest = d.getVar('KERNEL_IMAGEDEST')
>>>>
>>>> +    fullver = d.getVar('EXTENDPKGV')
>>>>        for type in types.split():
>>>>            if bb.data.inherits_class('nopackages', d):
>>>>                continue
>>>>            typelower = type.lower()
>>>>            d.appendVar('PACKAGES', ' %s-image-%s' % (kname, typelower))
>>>>            d.setVar('FILES:' + kname + '-image-' + typelower, '/' + imagedest + '/' + type + '-${KERNEL_VERSION_NAME}' + ' /' + imagedest + '/' + type)
>>>> -        d.appendVar('RDEPENDS:%s-image' % kname, ' %s-image-%s' % (kname, typelower))
>>>> +        d.appendVar('RDEPENDS:%s-image' % kname, ' %s-image-%s (= %s)' % (kname, typelower, fullver))
>>>>            splitmods = d.getVar("KERNEL_SPLIT_MODULES")
>>>>            if splitmods != '1':
>>>> -            d.appendVar('RDEPENDS:%s-image' % kname, ' %s-modules' % kname)
>>>> -            d.appendVar('RDEPENDS:%s-image-%s' % (kname, typelower), ' %s-modules-${KERNEL_VERSION_PKG_NAME}' % kname)
>>>> +            d.appendVar('RDEPENDS:%s-image' % kname, ' %s-modules (= %s)' % (kname, fullver))
>>>> +            d.appendVar('RDEPENDS:%s-image-%s' % (kname, typelower), ' %s-modules-${KERNEL_VERSION_PKG_NAME} (= %s)' % (kname, fullver))
>>>>                d.setVar('PKG:%s-modules' % kname, '%s-modules-${KERNEL_VERSION_PKG_NAME}' % kname)
>>>>                d.appendVar('RPROVIDES:%s-modules' % kname, '%s-modules-${KERNEL_VERSION_PKG_NAME}' % kname)
>>>>
>>>> @@ -628,12 +629,12 @@ FILES:${KERNEL_PACKAGE_NAME}-image = ""
>>>>    FILES:${KERNEL_PACKAGE_NAME}-dev = "/boot/System.map* /boot/Module.symvers* /boot/config* ${KERNEL_SRC_PATH} ${nonarch_base_libdir}/modules/${KERNEL_VERSION}/build"
>>>>    FILES:${KERNEL_PACKAGE_NAME}-vmlinux = "/boot/vmlinux-${KERNEL_VERSION_NAME}"
>>>>    FILES:${KERNEL_PACKAGE_NAME}-modules = ""
>>>> -RDEPENDS:${KERNEL_PACKAGE_NAME} = "${KERNEL_PACKAGE_NAME}-base"
>>>> +RDEPENDS:${KERNEL_PACKAGE_NAME} = "${KERNEL_PACKAGE_NAME}-base (= ${EXTENDPKGV})"
>>>>    # Allow machines to override this dependency if kernel image files are
>>>>    # not wanted in images as standard
>>>> -RDEPENDS:${KERNEL_PACKAGE_NAME}-base ?= "${KERNEL_PACKAGE_NAME}-image"
>>>> +RDEPENDS:${KERNEL_PACKAGE_NAME}-base ?= "${KERNEL_PACKAGE_NAME}-image (= ${EXTENDPKGV})"
>>>>    PKG:${KERNEL_PACKAGE_NAME}-image = "${KERNEL_PACKAGE_NAME}-image-${@legitimize_package_name(d.getVar('KERNEL_VERSION'))}"
>>>> -RDEPENDS:${KERNEL_PACKAGE_NAME}-image += "${@oe.utils.conditional('KERNEL_IMAGETYPE', 'vmlinux', '${KERNEL_PACKAGE_NAME}-vmlinux', '', d)}"
>>>> +RDEPENDS:${KERNEL_PACKAGE_NAME}-image += "${@oe.utils.conditional('KERNEL_IMAGETYPE', 'vmlinux', '${KERNEL_PACKAGE_NAME}-vmlinux (= ${EXTENDPKGV})', '', d)}"
>>>>    PKG:${KERNEL_PACKAGE_NAME}-base = "${KERNEL_PACKAGE_NAME}-${@legitimize_package_name(d.getVar('KERNEL_VERSION'))}"
>>>>    RPROVIDES:${KERNEL_PACKAGE_NAME}-base += "${KERNEL_PACKAGE_NAME}-${KERNEL_VERSION}"
>>>>    ALLOW_EMPTY:${KERNEL_PACKAGE_NAME} = "1"
>>>> --
>>>> 2.31.1
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
> 
> 
> 
> 
> 
> 

Kernel related bbclass changes

[Zoltan Boszormenyi]

I have been carrying some forked bbclass recipes and I think
these can be beneficial to others, not to mention that I could
get rid of the private forks of these.

v2:
- drop the package_rpm.bbclass / posttrans patch
- squash the previous patches #3 and #4 together (patch #2 now)
- more verbose commit message for patch #3 (previously patch #5)

[PATCH v2 1/4] kernel-module-split.bbclass: Support zstd-compressed modules

[Zoltan Boszormenyi]

From: Zoltán Böszörményi <zboszor@gmail.com>

Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
---
 meta/classes/kernel-module-split.bbclass | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/meta/classes/kernel-module-split.bbclass b/meta/classes/kernel-module-split.bbclass
index b56dd4a9c7..6c1de4c992 100644
--- a/meta/classes/kernel-module-split.bbclass
+++ b/meta/classes/kernel-module-split.bbclass
@@ -44,17 +44,20 @@ python split_kernel_module_packages () {
     def extract_modinfo(file):
         import tempfile, subprocess
         tempfile.tempdir = d.getVar("WORKDIR")
-        compressed = re.match( r'.*\.([xg])z$', file)
+        compressed = re.match( r'.*\.(gz|xz|zst)$', file)
         tf = tempfile.mkstemp()
         tmpfile = tf[1]
         if compressed:
             tmpkofile = tmpfile + ".ko"
-            if compressed.group(1) == 'g':
+            if compressed.group(1) == 'gz':
                 cmd = "gunzip -dc %s > %s" % (file, tmpkofile)
                 subprocess.check_call(cmd, shell=True)
-            elif compressed.group(1) == 'x':
+            elif compressed.group(1) == 'xz':
                 cmd = "xz -dc %s > %s" % (file, tmpkofile)
                 subprocess.check_call(cmd, shell=True)
+            elif compressed.group(1) == 'zst':
+                cmd = "zstd -dc %s > %s" % (file, tmpkofile)
+                subprocess.check_call(cmd, shell=True)
             else:
                 msg = "Cannot decompress '%s'" % file
                 raise msg
@@ -153,7 +156,7 @@ python split_kernel_module_packages () {
     kernel_package_name = d.getVar("KERNEL_PACKAGE_NAME") or "kernel"
     kernel_version = d.getVar("KERNEL_VERSION")
 
-    module_regex = r'^(.*)\.k?o(?:\.[xg]z)?$'
+    module_regex = r'^(.*)\.k?o(?:\.(gz|xz|zst))?$'
 
     module_pattern_prefix = d.getVar('KERNEL_MODULE_PACKAGE_PREFIX')
     module_pattern_suffix = d.getVar('KERNEL_MODULE_PACKAGE_SUFFIX')
-- 
2.31.1

[PATCH v2 2/4] Allow opt-out of split kernel modules

[Zoltan Boszormenyi]

From: Zoltán Böszörményi <zboszor@gmail.com>

For some use cases, a monolithic kernel-modules package containing
all modules built from the kernel sources may be preferred.
For one, download time is shorter and installation time is faster.

Set KERNEL_SPLIT_MODULES="0" for this in. The default is one subpackage
per module.

Also, adapt kernel.bbclass to KERNEL_SPLIT_MODULES != "1" case
Extra RDEPENDS and other inter-package references are needed in
this case.

Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
---
 meta/classes/kernel-module-split.bbclass | 17 +++++++++++++----
 meta/classes/kernel.bbclass              |  7 +++++++
 2 files changed, 20 insertions(+), 4 deletions(-)

diff --git a/meta/classes/kernel-module-split.bbclass b/meta/classes/kernel-module-split.bbclass
index 6c1de4c992..a29c294810 100644
--- a/meta/classes/kernel-module-split.bbclass
+++ b/meta/classes/kernel-module-split.bbclass
@@ -28,6 +28,7 @@ do_install:append() {
 	install -d ${D}${sysconfdir}/modules-load.d/ ${D}${sysconfdir}/modprobe.d/
 }
 
+KERNEL_SPLIT_MODULES ?= "1"
 PACKAGESPLITFUNCS:prepend = "split_kernel_module_packages "
 
 KERNEL_MODULES_META_PACKAGE ?= "${@ d.getVar("KERNEL_PACKAGE_NAME") or "kernel" }-modules"
@@ -156,18 +157,26 @@ python split_kernel_module_packages () {
     kernel_package_name = d.getVar("KERNEL_PACKAGE_NAME") or "kernel"
     kernel_version = d.getVar("KERNEL_VERSION")
 
+    metapkg = d.getVar('KERNEL_MODULES_META_PACKAGE')
+    splitmods = d.getVar('KERNEL_SPLIT_MODULES')
+    postinst = d.getVar('pkg_postinst:modules')
+    postrm = d.getVar('pkg_postrm:modules')
+
+    if splitmods != '1':
+        etcdir = d.getVar('sysconfdir')
+        d.appendVar('FILES:' + metapkg, '%s/modules-load.d/ %s/modprobe.d/ %s/modules/' % (etcdir, etcdir, d.getVar("nonarch_base_libdir")))
+        d.appendVar('pkg_postinst:%s' % metapkg, postinst)
+        d.prependVar('pkg_postrm:%s' % metapkg, postrm);
+        return
+
     module_regex = r'^(.*)\.k?o(?:\.(gz|xz|zst))?$'
 
     module_pattern_prefix = d.getVar('KERNEL_MODULE_PACKAGE_PREFIX')
     module_pattern_suffix = d.getVar('KERNEL_MODULE_PACKAGE_SUFFIX')
     module_pattern = module_pattern_prefix + kernel_package_name + '-module-%s' + module_pattern_suffix
 
-    postinst = d.getVar('pkg_postinst:modules')
-    postrm = d.getVar('pkg_postrm:modules')
-
     modules = do_split_packages(d, root='${nonarch_base_libdir}/modules', file_regex=module_regex, output_pattern=module_pattern, description='%s kernel module', postinst=postinst, postrm=postrm, recursive=True, hook=frob_metadata, extra_depends='%s-%s' % (kernel_package_name, kernel_version))
     if modules:
-        metapkg = d.getVar('KERNEL_MODULES_META_PACKAGE')
         d.appendVar('RDEPENDS:' + metapkg, ' '+' '.join(modules))
 
     # If modules-load.d and modprobe.d are empty at this point, remove them to
diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
index 027e66eec7..6dc5387a9b 100644
--- a/meta/classes/kernel.bbclass
+++ b/meta/classes/kernel.bbclass
@@ -98,6 +98,13 @@ python __anonymous () {
         d.appendVar('PACKAGES', ' %s-image-%s' % (kname, typelower))
         d.setVar('FILES:' + kname + '-image-' + typelower, '/' + imagedest + '/' + type + '-${KERNEL_VERSION_NAME}' + ' /' + imagedest + '/' + type)
         d.appendVar('RDEPENDS:%s-image' % kname, ' %s-image-%s' % (kname, typelower))
+        splitmods = d.getVar("KERNEL_SPLIT_MODULES")
+        if splitmods != '1':
+            d.appendVar('RDEPENDS:%s-image' % kname, ' %s-modules' % kname)
+            d.appendVar('RDEPENDS:%s-image-%s' % (kname, typelower), ' %s-modules-${KERNEL_VERSION_PKG_NAME}' % kname)
+            d.setVar('PKG:%s-modules' % kname, '%s-modules-${KERNEL_VERSION_PKG_NAME}' % kname)
+            d.appendVar('RPROVIDES:%s-modules' % kname, '%s-modules-${KERNEL_VERSION_PKG_NAME}' % kname)
+
         d.setVar('PKG:%s-image-%s' % (kname,typelower), '%s-image-%s-${KERNEL_VERSION_PKG_NAME}' % (kname, typelower))
         d.setVar('ALLOW_EMPTY:%s-image-%s' % (kname, typelower), '1')
         d.setVar('pkg_postinst:%s-image-%s' % (kname,typelower), """set +e
-- 
2.31.1

[PATCH v2 3/4] kernel.bbclass: Use full versions for inter-package dependencies

[Zoltan Boszormenyi]

From: Zoltán Böszörményi <zboszor@gmail.com>

If the kernel configuration enables module signing but no key
is provided, then the kernel generates one during the kernel build.

The current runtime-dependency references (with only package names
without full versions) allow mixed package installations from different
rebuilds of the same kernel version.

This does create an issue if the kernel configuration enables module
signing but no key is provided, in which case the kernel generates
the signing key during the kernel build.

Tighten RDEPENDS with the full package version, i.e. use (= ${EXTENDPKGV})
markers for inter-package dependencies.

The kernel will pull in the kernel-modules subpackage of the same
exact version automatically if KERNEL_SPLIT_MODULES="0" is set.
Otherwise the situation is the same as with the old default with
one subpackage per kernel module where they have to be upgraded
manually.

Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
---
 meta/classes/kernel.bbclass | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)

diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
index 6dc5387a9b..9ec7daa17a 100644
--- a/meta/classes/kernel.bbclass
+++ b/meta/classes/kernel.bbclass
@@ -91,17 +91,18 @@ python __anonymous () {
     kname = d.getVar('KERNEL_PACKAGE_NAME') or "kernel"
     imagedest = d.getVar('KERNEL_IMAGEDEST')
 
+    fullver = d.getVar('EXTENDPKGV')
     for type in types.split():
         if bb.data.inherits_class('nopackages', d):
             continue
         typelower = type.lower()
         d.appendVar('PACKAGES', ' %s-image-%s' % (kname, typelower))
         d.setVar('FILES:' + kname + '-image-' + typelower, '/' + imagedest + '/' + type + '-${KERNEL_VERSION_NAME}' + ' /' + imagedest + '/' + type)
-        d.appendVar('RDEPENDS:%s-image' % kname, ' %s-image-%s' % (kname, typelower))
+        d.appendVar('RDEPENDS:%s-image' % kname, ' %s-image-%s (= %s)' % (kname, typelower, fullver))
         splitmods = d.getVar("KERNEL_SPLIT_MODULES")
         if splitmods != '1':
-            d.appendVar('RDEPENDS:%s-image' % kname, ' %s-modules' % kname)
-            d.appendVar('RDEPENDS:%s-image-%s' % (kname, typelower), ' %s-modules-${KERNEL_VERSION_PKG_NAME}' % kname)
+            d.appendVar('RDEPENDS:%s-image' % kname, ' %s-modules (= %s)' % (kname, fullver))
+            d.appendVar('RDEPENDS:%s-image-%s' % (kname, typelower), ' %s-modules-${KERNEL_VERSION_PKG_NAME} (= %s)' % (kname, fullver))
             d.setVar('PKG:%s-modules' % kname, '%s-modules-${KERNEL_VERSION_PKG_NAME}' % kname)
             d.appendVar('RPROVIDES:%s-modules' % kname, '%s-modules-${KERNEL_VERSION_PKG_NAME}' % kname)
 
@@ -628,12 +629,12 @@ FILES:${KERNEL_PACKAGE_NAME}-image = ""
 FILES:${KERNEL_PACKAGE_NAME}-dev = "/boot/System.map* /boot/Module.symvers* /boot/config* ${KERNEL_SRC_PATH} ${nonarch_base_libdir}/modules/${KERNEL_VERSION}/build"
 FILES:${KERNEL_PACKAGE_NAME}-vmlinux = "/boot/vmlinux-${KERNEL_VERSION_NAME}"
 FILES:${KERNEL_PACKAGE_NAME}-modules = ""
-RDEPENDS:${KERNEL_PACKAGE_NAME} = "${KERNEL_PACKAGE_NAME}-base"
+RDEPENDS:${KERNEL_PACKAGE_NAME} = "${KERNEL_PACKAGE_NAME}-base (= ${EXTENDPKGV})"
 # Allow machines to override this dependency if kernel image files are
 # not wanted in images as standard
-RDEPENDS:${KERNEL_PACKAGE_NAME}-base ?= "${KERNEL_PACKAGE_NAME}-image"
+RDEPENDS:${KERNEL_PACKAGE_NAME}-base ?= "${KERNEL_PACKAGE_NAME}-image (= ${EXTENDPKGV})"
 PKG:${KERNEL_PACKAGE_NAME}-image = "${KERNEL_PACKAGE_NAME}-image-${@legitimize_package_name(d.getVar('KERNEL_VERSION'))}"
-RDEPENDS:${KERNEL_PACKAGE_NAME}-image += "${@oe.utils.conditional('KERNEL_IMAGETYPE', 'vmlinux', '${KERNEL_PACKAGE_NAME}-vmlinux', '', d)}"
+RDEPENDS:${KERNEL_PACKAGE_NAME}-image += "${@oe.utils.conditional('KERNEL_IMAGETYPE', 'vmlinux', '${KERNEL_PACKAGE_NAME}-vmlinux (= ${EXTENDPKGV})', '', d)}"
 PKG:${KERNEL_PACKAGE_NAME}-base = "${KERNEL_PACKAGE_NAME}-${@legitimize_package_name(d.getVar('KERNEL_VERSION'))}"
 RPROVIDES:${KERNEL_PACKAGE_NAME}-base += "${KERNEL_PACKAGE_NAME}-${KERNEL_VERSION}"
 ALLOW_EMPTY:${KERNEL_PACKAGE_NAME} = "1"
-- 
2.31.1

[PATCH v2 4/4] Support zstd-compressed squashfs and cpio initramfs

[Zoltan Boszormenyi]

From: Zoltán Böszörményi <zboszor@gmail.com>

Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
---
 meta/classes/base.bbclass            |  4 ++++
 meta/classes/image_types.bbclass     |  6 ++++--
 meta/classes/kernel-fitimage.bbclass |  2 +-
 meta/classes/kernel.bbclass          | 10 ++++++++--
 4 files changed, 17 insertions(+), 5 deletions(-)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 11b65171d9..340ebe7d78 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -631,6 +631,10 @@ python () {
         if path.endswith('.lz4'):
             d.appendVarFlag('do_unpack', 'depends', ' lz4-native:do_populate_sysroot')
 
+        # *.zst should DEPEND on zstd-native for unpacking
+        elif path.endswith('.zst'):
+            d.appendVarFlag('do_unpack', 'depends', ' zstd-native:do_populate_sysroot')
+
         # *.lz should DEPEND on lzip-native for unpacking
         elif path.endswith('.lz'):
             d.appendVarFlag('do_unpack', 'depends', ' lzip-native:do_populate_sysroot')
diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index e0eb06c1e3..32d4cd4c76 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -107,6 +107,7 @@ IMAGE_CMD:squashfs = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${
 IMAGE_CMD:squashfs-xz = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-xz ${EXTRA_IMAGECMD} -noappend -comp xz"
 IMAGE_CMD:squashfs-lzo = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-lzo ${EXTRA_IMAGECMD} -noappend -comp lzo"
 IMAGE_CMD:squashfs-lz4 = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-lz4 ${EXTRA_IMAGECMD} -noappend -comp lz4"
+IMAGE_CMD:squashfs-zst = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-zst ${EXTRA_IMAGECMD} -noappend -comp zstd"
 
 IMAGE_CMD:erofs = "mkfs.erofs ${EXTRA_IMAGECMD} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.erofs ${IMAGE_ROOTFS}"
 IMAGE_CMD:erofs-lz4 = "mkfs.erofs -zlz4 ${EXTRA_IMAGECMD} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.erofs-lz4 ${IMAGE_ROOTFS}"
@@ -244,6 +245,7 @@ do_image_squashfs[depends] += "squashfs-tools-native:do_populate_sysroot"
 do_image_squashfs_xz[depends] += "squashfs-tools-native:do_populate_sysroot"
 do_image_squashfs_lzo[depends] += "squashfs-tools-native:do_populate_sysroot"
 do_image_squashfs_lz4[depends] += "squashfs-tools-native:do_populate_sysroot"
+do_image_squashfs_zst[depends] += "squashfs-tools-native:do_populate_sysroot"
 do_image_ubi[depends] += "mtd-utils-native:do_populate_sysroot"
 do_image_ubifs[depends] += "mtd-utils-native:do_populate_sysroot"
 do_image_multiubi[depends] += "mtd-utils-native:do_populate_sysroot"
@@ -262,10 +264,10 @@ IMAGE_TYPES = " \
     btrfs \
     iso \
     hddimg \
-    squashfs squashfs-xz squashfs-lzo squashfs-lz4 \
+    squashfs squashfs-xz squashfs-lzo squashfs-lz4 squashfs-zst \
     ubi ubifs multiubi \
     tar tar.gz tar.bz2 tar.xz tar.lz4 tar.zst \
-    cpio cpio.gz cpio.xz cpio.lzma cpio.lz4 \
+    cpio cpio.gz cpio.xz cpio.lzma cpio.lz4 cpio.zst \
     wic wic.gz wic.bz2 wic.lzma wic.zst \
     container \
     f2fs \
diff --git a/meta/classes/kernel-fitimage.bbclass b/meta/classes/kernel-fitimage.bbclass
index 2ef8f06b14..38e05153e3 100644
--- a/meta/classes/kernel-fitimage.bbclass
+++ b/meta/classes/kernel-fitimage.bbclass
@@ -576,7 +576,7 @@ fitimage_assemble() {
 	#
 	if [ "x${ramdiskcount}" = "x1" ] && [ "${INITRAMFS_IMAGE_BUNDLE}" != "1" ]; then
 		# Find and use the first initramfs image archive type we find
-		for img in cpio.lz4 cpio.lzo cpio.lzma cpio.xz cpio.gz ext2.gz cpio; do
+		for img in cpio.lz4 cpio.lzo cpio.lzma cpio.xz cpio.zst cpio.gz ext2.gz cpio; do
 			initramfs_path="${DEPLOY_DIR_IMAGE}/${INITRAMFS_IMAGE_NAME}.${img}"
 			echo "Using $initramfs_path"
 			if [ -e "${initramfs_path}" ]; then
diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
index 9ec7daa17a..ab60e060c6 100644
--- a/meta/classes/kernel.bbclass
+++ b/meta/classes/kernel.bbclass
@@ -9,6 +9,7 @@ PROVIDES += "${@ "virtual/kernel" if (d.getVar("KERNEL_PACKAGE_NAME") == "kernel
 DEPENDS += "virtual/${TARGET_PREFIX}binutils virtual/${TARGET_PREFIX}gcc kmod-native bc-native bison-native"
 DEPENDS += "${@bb.utils.contains("INITRAMFS_FSTYPES", "cpio.lzo", "lzop-native", "", d)}"
 DEPENDS += "${@bb.utils.contains("INITRAMFS_FSTYPES", "cpio.lz4", "lz4-native", "", d)}"
+DEPENDS += "${@bb.utils.contains("INITRAMFS_FSTYPES", "cpio.zst", "zstd-native", "", d)}"
 PACKAGE_WRITE_DEPS += "depmodwrapper-cross"
 
 do_deploy[depends] += "depmodwrapper-cross:do_populate_sysroot gzip-native:do_populate_sysroot"
@@ -237,7 +238,7 @@ copy_initramfs() {
 	mkdir -p ${B}/usr
 	# Find and use the first initramfs image archive type we find
 	rm -f ${B}/usr/${INITRAMFS_IMAGE_NAME}.cpio
-	for img in cpio cpio.gz cpio.lz4 cpio.lzo cpio.lzma cpio.xz; do
+	for img in cpio cpio.gz cpio.lz4 cpio.lzo cpio.lzma cpio.xz cpio.zst; do
 		if [ -e "${DEPLOY_DIR_IMAGE}/${INITRAMFS_IMAGE_NAME}.$img" ]; then
 			cp ${DEPLOY_DIR_IMAGE}/${INITRAMFS_IMAGE_NAME}.$img ${B}/usr/.
 			case $img in
@@ -266,12 +267,17 @@ copy_initramfs() {
 				xz -df ${B}/usr/${INITRAMFS_IMAGE_NAME}.$img
 				break
 				;;
+			*zst)
+				echo "zst decompressing image"
+				zstd -df ${B}/usr/${INITRAMFS_IMAGE_NAME}.$img
+				break
+				;;
 			esac
 			break
 		fi
 	done
 	# Verify that the above loop found a initramfs, fail otherwise
-	[ -f ${B}/usr/${INITRAMFS_IMAGE_NAME}.cpio ] && echo "Finished copy of initramfs into ./usr" || die "Could not find any ${DEPLOY_DIR_IMAGE}/${INITRAMFS_IMAGE_NAME}.cpio{.gz|.lz4|.lzo|.lzma|.xz) for bundling; INITRAMFS_IMAGE_NAME might be wrong."
+	[ -f ${B}/usr/${INITRAMFS_IMAGE_NAME}.cpio ] && echo "Finished copy of initramfs into ./usr" || die "Could not find any ${DEPLOY_DIR_IMAGE}/${INITRAMFS_IMAGE_NAME}.cpio{.gz|.lz4|.lzo|.lzma|.xz|.zst) for bundling; INITRAMFS_IMAGE_NAME might be wrong."
 }
 
 do_bundle_initramfs () {
-- 
2.31.1

Kernel related bbclass changes

[Zoltan Boszormenyi]

I have been carrying some forked bbclass recipes and I think
these can be beneficial to others, not to mention that I could
get rid of the private forks of these.

v2:
- drop the package_rpm.bbclass / posttrans patch
- squash the previous patches #3 and #4 together (patch #2 now)
- more verbose commit message for patch #3 (previously patch #5)

v3:
- fix commit message for patch #3 (stupid copy&paste&thinko)

[PATCH v3 1/4] kernel-module-split.bbclass: Support zstd-compressed modules

[Zoltan Boszormenyi]

From: Zoltán Böszörményi <zboszor@gmail.com>

Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
---
 meta/classes/kernel-module-split.bbclass | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/meta/classes/kernel-module-split.bbclass b/meta/classes/kernel-module-split.bbclass
index b56dd4a9c7..6c1de4c992 100644
--- a/meta/classes/kernel-module-split.bbclass
+++ b/meta/classes/kernel-module-split.bbclass
@@ -44,17 +44,20 @@ python split_kernel_module_packages () {
     def extract_modinfo(file):
         import tempfile, subprocess
         tempfile.tempdir = d.getVar("WORKDIR")
-        compressed = re.match( r'.*\.([xg])z$', file)
+        compressed = re.match( r'.*\.(gz|xz|zst)$', file)
         tf = tempfile.mkstemp()
         tmpfile = tf[1]
         if compressed:
             tmpkofile = tmpfile + ".ko"
-            if compressed.group(1) == 'g':
+            if compressed.group(1) == 'gz':
                 cmd = "gunzip -dc %s > %s" % (file, tmpkofile)
                 subprocess.check_call(cmd, shell=True)
-            elif compressed.group(1) == 'x':
+            elif compressed.group(1) == 'xz':
                 cmd = "xz -dc %s > %s" % (file, tmpkofile)
                 subprocess.check_call(cmd, shell=True)
+            elif compressed.group(1) == 'zst':
+                cmd = "zstd -dc %s > %s" % (file, tmpkofile)
+                subprocess.check_call(cmd, shell=True)
             else:
                 msg = "Cannot decompress '%s'" % file
                 raise msg
@@ -153,7 +156,7 @@ python split_kernel_module_packages () {
     kernel_package_name = d.getVar("KERNEL_PACKAGE_NAME") or "kernel"
     kernel_version = d.getVar("KERNEL_VERSION")
 
-    module_regex = r'^(.*)\.k?o(?:\.[xg]z)?$'
+    module_regex = r'^(.*)\.k?o(?:\.(gz|xz|zst))?$'
 
     module_pattern_prefix = d.getVar('KERNEL_MODULE_PACKAGE_PREFIX')
     module_pattern_suffix = d.getVar('KERNEL_MODULE_PACKAGE_SUFFIX')
-- 
2.31.1

[PATCH v3 2/4] Allow opt-out of split kernel modules

[Zoltan Boszormenyi]

From: Zoltán Böszörményi <zboszor@gmail.com>

For some use cases, a monolithic kernel-modules package containing
all modules built from the kernel sources may be preferred.
For one, download time is shorter and installation time is faster.

Set KERNEL_SPLIT_MODULES="0" for this in. The default is one subpackage
per module.

Also, adapt kernel.bbclass to KERNEL_SPLIT_MODULES != "1" case
Extra RDEPENDS and other inter-package references are needed in
this case.

Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
---
 meta/classes/kernel-module-split.bbclass | 17 +++++++++++++----
 meta/classes/kernel.bbclass              |  7 +++++++
 2 files changed, 20 insertions(+), 4 deletions(-)

diff --git a/meta/classes/kernel-module-split.bbclass b/meta/classes/kernel-module-split.bbclass
index 6c1de4c992..a29c294810 100644
--- a/meta/classes/kernel-module-split.bbclass
+++ b/meta/classes/kernel-module-split.bbclass
@@ -28,6 +28,7 @@ do_install:append() {
 	install -d ${D}${sysconfdir}/modules-load.d/ ${D}${sysconfdir}/modprobe.d/
 }
 
+KERNEL_SPLIT_MODULES ?= "1"
 PACKAGESPLITFUNCS:prepend = "split_kernel_module_packages "
 
 KERNEL_MODULES_META_PACKAGE ?= "${@ d.getVar("KERNEL_PACKAGE_NAME") or "kernel" }-modules"
@@ -156,18 +157,26 @@ python split_kernel_module_packages () {
     kernel_package_name = d.getVar("KERNEL_PACKAGE_NAME") or "kernel"
     kernel_version = d.getVar("KERNEL_VERSION")
 
+    metapkg = d.getVar('KERNEL_MODULES_META_PACKAGE')
+    splitmods = d.getVar('KERNEL_SPLIT_MODULES')
+    postinst = d.getVar('pkg_postinst:modules')
+    postrm = d.getVar('pkg_postrm:modules')
+
+    if splitmods != '1':
+        etcdir = d.getVar('sysconfdir')
+        d.appendVar('FILES:' + metapkg, '%s/modules-load.d/ %s/modprobe.d/ %s/modules/' % (etcdir, etcdir, d.getVar("nonarch_base_libdir")))
+        d.appendVar('pkg_postinst:%s' % metapkg, postinst)
+        d.prependVar('pkg_postrm:%s' % metapkg, postrm);
+        return
+
     module_regex = r'^(.*)\.k?o(?:\.(gz|xz|zst))?$'
 
     module_pattern_prefix = d.getVar('KERNEL_MODULE_PACKAGE_PREFIX')
     module_pattern_suffix = d.getVar('KERNEL_MODULE_PACKAGE_SUFFIX')
     module_pattern = module_pattern_prefix + kernel_package_name + '-module-%s' + module_pattern_suffix
 
-    postinst = d.getVar('pkg_postinst:modules')
-    postrm = d.getVar('pkg_postrm:modules')
-
     modules = do_split_packages(d, root='${nonarch_base_libdir}/modules', file_regex=module_regex, output_pattern=module_pattern, description='%s kernel module', postinst=postinst, postrm=postrm, recursive=True, hook=frob_metadata, extra_depends='%s-%s' % (kernel_package_name, kernel_version))
     if modules:
-        metapkg = d.getVar('KERNEL_MODULES_META_PACKAGE')
         d.appendVar('RDEPENDS:' + metapkg, ' '+' '.join(modules))
 
     # If modules-load.d and modprobe.d are empty at this point, remove them to
diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
index 027e66eec7..6dc5387a9b 100644
--- a/meta/classes/kernel.bbclass
+++ b/meta/classes/kernel.bbclass
@@ -98,6 +98,13 @@ python __anonymous () {
         d.appendVar('PACKAGES', ' %s-image-%s' % (kname, typelower))
         d.setVar('FILES:' + kname + '-image-' + typelower, '/' + imagedest + '/' + type + '-${KERNEL_VERSION_NAME}' + ' /' + imagedest + '/' + type)
         d.appendVar('RDEPENDS:%s-image' % kname, ' %s-image-%s' % (kname, typelower))
+        splitmods = d.getVar("KERNEL_SPLIT_MODULES")
+        if splitmods != '1':
+            d.appendVar('RDEPENDS:%s-image' % kname, ' %s-modules' % kname)
+            d.appendVar('RDEPENDS:%s-image-%s' % (kname, typelower), ' %s-modules-${KERNEL_VERSION_PKG_NAME}' % kname)
+            d.setVar('PKG:%s-modules' % kname, '%s-modules-${KERNEL_VERSION_PKG_NAME}' % kname)
+            d.appendVar('RPROVIDES:%s-modules' % kname, '%s-modules-${KERNEL_VERSION_PKG_NAME}' % kname)
+
         d.setVar('PKG:%s-image-%s' % (kname,typelower), '%s-image-%s-${KERNEL_VERSION_PKG_NAME}' % (kname, typelower))
         d.setVar('ALLOW_EMPTY:%s-image-%s' % (kname, typelower), '1')
         d.setVar('pkg_postinst:%s-image-%s' % (kname,typelower), """set +e
-- 
2.31.1

[PATCH v3 3/4] kernel.bbclass: Use full versions for inter-package dependencies

[Zoltan Boszormenyi]

From: Zoltán Böszörményi <zboszor@gmail.com>

If the kernel configuration enables module signing but no key
is provided, then the kernel generates one during the kernel build.

The current runtime-dependency references (with only package names
without full versions) allow mixed package installations from different
rebuilds of the same kernel version.

This creates an issue because then the modules either don't work
or taint the kernel.

Tighten RDEPENDS with the full package version, i.e. use (= ${EXTENDPKGV})
markers for inter-package dependencies.

The kernel will pull in the kernel-modules subpackage of the same
exact version automatically if KERNEL_SPLIT_MODULES="0" is set.
Otherwise the situation is the same as with the old default with
one subpackage per kernel module where they have to be upgraded
manually.

Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
---
 meta/classes/kernel.bbclass | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)

diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
index 6dc5387a9b..9ec7daa17a 100644
--- a/meta/classes/kernel.bbclass
+++ b/meta/classes/kernel.bbclass
@@ -91,17 +91,18 @@ python __anonymous () {
     kname = d.getVar('KERNEL_PACKAGE_NAME') or "kernel"
     imagedest = d.getVar('KERNEL_IMAGEDEST')
 
+    fullver = d.getVar('EXTENDPKGV')
     for type in types.split():
         if bb.data.inherits_class('nopackages', d):
             continue
         typelower = type.lower()
         d.appendVar('PACKAGES', ' %s-image-%s' % (kname, typelower))
         d.setVar('FILES:' + kname + '-image-' + typelower, '/' + imagedest + '/' + type + '-${KERNEL_VERSION_NAME}' + ' /' + imagedest + '/' + type)
-        d.appendVar('RDEPENDS:%s-image' % kname, ' %s-image-%s' % (kname, typelower))
+        d.appendVar('RDEPENDS:%s-image' % kname, ' %s-image-%s (= %s)' % (kname, typelower, fullver))
         splitmods = d.getVar("KERNEL_SPLIT_MODULES")
         if splitmods != '1':
-            d.appendVar('RDEPENDS:%s-image' % kname, ' %s-modules' % kname)
-            d.appendVar('RDEPENDS:%s-image-%s' % (kname, typelower), ' %s-modules-${KERNEL_VERSION_PKG_NAME}' % kname)
+            d.appendVar('RDEPENDS:%s-image' % kname, ' %s-modules (= %s)' % (kname, fullver))
+            d.appendVar('RDEPENDS:%s-image-%s' % (kname, typelower), ' %s-modules-${KERNEL_VERSION_PKG_NAME} (= %s)' % (kname, fullver))
             d.setVar('PKG:%s-modules' % kname, '%s-modules-${KERNEL_VERSION_PKG_NAME}' % kname)
             d.appendVar('RPROVIDES:%s-modules' % kname, '%s-modules-${KERNEL_VERSION_PKG_NAME}' % kname)
 
@@ -628,12 +629,12 @@ FILES:${KERNEL_PACKAGE_NAME}-image = ""
 FILES:${KERNEL_PACKAGE_NAME}-dev = "/boot/System.map* /boot/Module.symvers* /boot/config* ${KERNEL_SRC_PATH} ${nonarch_base_libdir}/modules/${KERNEL_VERSION}/build"
 FILES:${KERNEL_PACKAGE_NAME}-vmlinux = "/boot/vmlinux-${KERNEL_VERSION_NAME}"
 FILES:${KERNEL_PACKAGE_NAME}-modules = ""
-RDEPENDS:${KERNEL_PACKAGE_NAME} = "${KERNEL_PACKAGE_NAME}-base"
+RDEPENDS:${KERNEL_PACKAGE_NAME} = "${KERNEL_PACKAGE_NAME}-base (= ${EXTENDPKGV})"
 # Allow machines to override this dependency if kernel image files are
 # not wanted in images as standard
-RDEPENDS:${KERNEL_PACKAGE_NAME}-base ?= "${KERNEL_PACKAGE_NAME}-image"
+RDEPENDS:${KERNEL_PACKAGE_NAME}-base ?= "${KERNEL_PACKAGE_NAME}-image (= ${EXTENDPKGV})"
 PKG:${KERNEL_PACKAGE_NAME}-image = "${KERNEL_PACKAGE_NAME}-image-${@legitimize_package_name(d.getVar('KERNEL_VERSION'))}"
-RDEPENDS:${KERNEL_PACKAGE_NAME}-image += "${@oe.utils.conditional('KERNEL_IMAGETYPE', 'vmlinux', '${KERNEL_PACKAGE_NAME}-vmlinux', '', d)}"
+RDEPENDS:${KERNEL_PACKAGE_NAME}-image += "${@oe.utils.conditional('KERNEL_IMAGETYPE', 'vmlinux', '${KERNEL_PACKAGE_NAME}-vmlinux (= ${EXTENDPKGV})', '', d)}"
 PKG:${KERNEL_PACKAGE_NAME}-base = "${KERNEL_PACKAGE_NAME}-${@legitimize_package_name(d.getVar('KERNEL_VERSION'))}"
 RPROVIDES:${KERNEL_PACKAGE_NAME}-base += "${KERNEL_PACKAGE_NAME}-${KERNEL_VERSION}"
 ALLOW_EMPTY:${KERNEL_PACKAGE_NAME} = "1"
-- 
2.31.1

[PATCH v3 4/4] Support zstd-compressed squashfs and cpio initramfs

[Zoltan Boszormenyi]

From: Zoltán Böszörményi <zboszor@gmail.com>

Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
---
 meta/classes/base.bbclass            |  4 ++++
 meta/classes/image_types.bbclass     |  6 ++++--
 meta/classes/kernel-fitimage.bbclass |  2 +-
 meta/classes/kernel.bbclass          | 10 ++++++++--
 4 files changed, 17 insertions(+), 5 deletions(-)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 11b65171d9..340ebe7d78 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -631,6 +631,10 @@ python () {
         if path.endswith('.lz4'):
             d.appendVarFlag('do_unpack', 'depends', ' lz4-native:do_populate_sysroot')
 
+        # *.zst should DEPEND on zstd-native for unpacking
+        elif path.endswith('.zst'):
+            d.appendVarFlag('do_unpack', 'depends', ' zstd-native:do_populate_sysroot')
+
         # *.lz should DEPEND on lzip-native for unpacking
         elif path.endswith('.lz'):
             d.appendVarFlag('do_unpack', 'depends', ' lzip-native:do_populate_sysroot')
diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index e0eb06c1e3..32d4cd4c76 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -107,6 +107,7 @@ IMAGE_CMD:squashfs = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${
 IMAGE_CMD:squashfs-xz = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-xz ${EXTRA_IMAGECMD} -noappend -comp xz"
 IMAGE_CMD:squashfs-lzo = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-lzo ${EXTRA_IMAGECMD} -noappend -comp lzo"
 IMAGE_CMD:squashfs-lz4 = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-lz4 ${EXTRA_IMAGECMD} -noappend -comp lz4"
+IMAGE_CMD:squashfs-zst = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-zst ${EXTRA_IMAGECMD} -noappend -comp zstd"
 
 IMAGE_CMD:erofs = "mkfs.erofs ${EXTRA_IMAGECMD} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.erofs ${IMAGE_ROOTFS}"
 IMAGE_CMD:erofs-lz4 = "mkfs.erofs -zlz4 ${EXTRA_IMAGECMD} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.erofs-lz4 ${IMAGE_ROOTFS}"
@@ -244,6 +245,7 @@ do_image_squashfs[depends] += "squashfs-tools-native:do_populate_sysroot"
 do_image_squashfs_xz[depends] += "squashfs-tools-native:do_populate_sysroot"
 do_image_squashfs_lzo[depends] += "squashfs-tools-native:do_populate_sysroot"
 do_image_squashfs_lz4[depends] += "squashfs-tools-native:do_populate_sysroot"
+do_image_squashfs_zst[depends] += "squashfs-tools-native:do_populate_sysroot"
 do_image_ubi[depends] += "mtd-utils-native:do_populate_sysroot"
 do_image_ubifs[depends] += "mtd-utils-native:do_populate_sysroot"
 do_image_multiubi[depends] += "mtd-utils-native:do_populate_sysroot"
@@ -262,10 +264,10 @@ IMAGE_TYPES = " \
     btrfs \
     iso \
     hddimg \
-    squashfs squashfs-xz squashfs-lzo squashfs-lz4 \
+    squashfs squashfs-xz squashfs-lzo squashfs-lz4 squashfs-zst \
     ubi ubifs multiubi \
     tar tar.gz tar.bz2 tar.xz tar.lz4 tar.zst \
-    cpio cpio.gz cpio.xz cpio.lzma cpio.lz4 \
+    cpio cpio.gz cpio.xz cpio.lzma cpio.lz4 cpio.zst \
     wic wic.gz wic.bz2 wic.lzma wic.zst \
     container \
     f2fs \
diff --git a/meta/classes/kernel-fitimage.bbclass b/meta/classes/kernel-fitimage.bbclass
index 2ef8f06b14..38e05153e3 100644
--- a/meta/classes/kernel-fitimage.bbclass
+++ b/meta/classes/kernel-fitimage.bbclass
@@ -576,7 +576,7 @@ fitimage_assemble() {
 	#
 	if [ "x${ramdiskcount}" = "x1" ] && [ "${INITRAMFS_IMAGE_BUNDLE}" != "1" ]; then
 		# Find and use the first initramfs image archive type we find
-		for img in cpio.lz4 cpio.lzo cpio.lzma cpio.xz cpio.gz ext2.gz cpio; do
+		for img in cpio.lz4 cpio.lzo cpio.lzma cpio.xz cpio.zst cpio.gz ext2.gz cpio; do
 			initramfs_path="${DEPLOY_DIR_IMAGE}/${INITRAMFS_IMAGE_NAME}.${img}"
 			echo "Using $initramfs_path"
 			if [ -e "${initramfs_path}" ]; then
diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
index 9ec7daa17a..ab60e060c6 100644
--- a/meta/classes/kernel.bbclass
+++ b/meta/classes/kernel.bbclass
@@ -9,6 +9,7 @@ PROVIDES += "${@ "virtual/kernel" if (d.getVar("KERNEL_PACKAGE_NAME") == "kernel
 DEPENDS += "virtual/${TARGET_PREFIX}binutils virtual/${TARGET_PREFIX}gcc kmod-native bc-native bison-native"
 DEPENDS += "${@bb.utils.contains("INITRAMFS_FSTYPES", "cpio.lzo", "lzop-native", "", d)}"
 DEPENDS += "${@bb.utils.contains("INITRAMFS_FSTYPES", "cpio.lz4", "lz4-native", "", d)}"
+DEPENDS += "${@bb.utils.contains("INITRAMFS_FSTYPES", "cpio.zst", "zstd-native", "", d)}"
 PACKAGE_WRITE_DEPS += "depmodwrapper-cross"
 
 do_deploy[depends] += "depmodwrapper-cross:do_populate_sysroot gzip-native:do_populate_sysroot"
@@ -237,7 +238,7 @@ copy_initramfs() {
 	mkdir -p ${B}/usr
 	# Find and use the first initramfs image archive type we find
 	rm -f ${B}/usr/${INITRAMFS_IMAGE_NAME}.cpio
-	for img in cpio cpio.gz cpio.lz4 cpio.lzo cpio.lzma cpio.xz; do
+	for img in cpio cpio.gz cpio.lz4 cpio.lzo cpio.lzma cpio.xz cpio.zst; do
 		if [ -e "${DEPLOY_DIR_IMAGE}/${INITRAMFS_IMAGE_NAME}.$img" ]; then
 			cp ${DEPLOY_DIR_IMAGE}/${INITRAMFS_IMAGE_NAME}.$img ${B}/usr/.
 			case $img in
@@ -266,12 +267,17 @@ copy_initramfs() {
 				xz -df ${B}/usr/${INITRAMFS_IMAGE_NAME}.$img
 				break
 				;;
+			*zst)
+				echo "zst decompressing image"
+				zstd -df ${B}/usr/${INITRAMFS_IMAGE_NAME}.$img
+				break
+				;;
 			esac
 			break
 		fi
 	done
 	# Verify that the above loop found a initramfs, fail otherwise
-	[ -f ${B}/usr/${INITRAMFS_IMAGE_NAME}.cpio ] && echo "Finished copy of initramfs into ./usr" || die "Could not find any ${DEPLOY_DIR_IMAGE}/${INITRAMFS_IMAGE_NAME}.cpio{.gz|.lz4|.lzo|.lzma|.xz) for bundling; INITRAMFS_IMAGE_NAME might be wrong."
+	[ -f ${B}/usr/${INITRAMFS_IMAGE_NAME}.cpio ] && echo "Finished copy of initramfs into ./usr" || die "Could not find any ${DEPLOY_DIR_IMAGE}/${INITRAMFS_IMAGE_NAME}.cpio{.gz|.lz4|.lzo|.lzma|.xz|.zst) for bundling; INITRAMFS_IMAGE_NAME might be wrong."
 }
 
 do_bundle_initramfs () {
-- 
2.31.1

Re: [OE-core] [PATCH v3 4/4] Support zstd-compressed squashfs and cpio initramfs

[Richard Purdie]

On Mon, 2021-08-23 at 16:54 +0200, Zoltan Boszormenyi via lists.openembedded.org
wrote:
> From: Zoltán Böszörményi <zboszor@gmail.com>
> 
> Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
> ---
>  meta/classes/base.bbclass            |  4 ++++
>  meta/classes/image_types.bbclass     |  6 ++++--
>  meta/classes/kernel-fitimage.bbclass |  2 +-
>  meta/classes/kernel.bbclass          | 10 ++++++++--
>  4 files changed, 17 insertions(+), 5 deletions(-)
> 
> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
> index 11b65171d9..340ebe7d78 100644
> --- a/meta/classes/base.bbclass
> +++ b/meta/classes/base.bbclass
> @@ -631,6 +631,10 @@ python () {
>          if path.endswith('.lz4'):
>              d.appendVarFlag('do_unpack', 'depends', ' lz4-native:do_populate_sysroot')
>  
> +        # *.zst should DEPEND on zstd-native for unpacking
> +        elif path.endswith('.zst'):
> +            d.appendVarFlag('do_unpack', 'depends', ' zstd-native:do_populate_sysroot')
> +
>          # *.lz should DEPEND on lzip-native for unpacking
>          elif path.endswith('.lz'):
>              d.appendVarFlag('do_unpack', 'depends', ' lzip-native:do_populate_sysroot')
> diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
> index e0eb06c1e3..32d4cd4c76 100644
> --- a/meta/classes/image_types.bbclass
> +++ b/meta/classes/image_types.bbclass
> @@ -107,6 +107,7 @@ IMAGE_CMD:squashfs = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${
>  IMAGE_CMD:squashfs-xz = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-xz ${EXTRA_IMAGECMD} -noappend -comp xz"
>  IMAGE_CMD:squashfs-lzo = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-lzo ${EXTRA_IMAGECMD} -noappend -comp lzo"
>  IMAGE_CMD:squashfs-lz4 = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-lz4 ${EXTRA_IMAGECMD} -noappend -comp lz4"
> +IMAGE_CMD:squashfs-zst = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-zst ${EXTRA_IMAGECMD} -noappend -comp zstd"
>  
>  IMAGE_CMD:erofs = "mkfs.erofs ${EXTRA_IMAGECMD} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.erofs ${IMAGE_ROOTFS}"
>  IMAGE_CMD:erofs-lz4 = "mkfs.erofs -zlz4 ${EXTRA_IMAGECMD} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.erofs-lz4 ${IMAGE_ROOTFS}"
> @@ -244,6 +245,7 @@ do_image_squashfs[depends] += "squashfs-tools-native:do_populate_sysroot"
>  do_image_squashfs_xz[depends] += "squashfs-tools-native:do_populate_sysroot"
>  do_image_squashfs_lzo[depends] += "squashfs-tools-native:do_populate_sysroot"
>  do_image_squashfs_lz4[depends] += "squashfs-tools-native:do_populate_sysroot"
> +do_image_squashfs_zst[depends] += "squashfs-tools-native:do_populate_sysroot"
>  do_image_ubi[depends] += "mtd-utils-native:do_populate_sysroot"
>  do_image_ubifs[depends] += "mtd-utils-native:do_populate_sysroot"
>  do_image_multiubi[depends] += "mtd-utils-native:do_populate_sysroot"
> @@ -262,10 +264,10 @@ IMAGE_TYPES = " \
>      btrfs \
>      iso \
>      hddimg \
> -    squashfs squashfs-xz squashfs-lzo squashfs-lz4 \
> +    squashfs squashfs-xz squashfs-lzo squashfs-lz4 squashfs-zst \
>      ubi ubifs multiubi \
>      tar tar.gz tar.bz2 tar.xz tar.lz4 tar.zst \
> -    cpio cpio.gz cpio.xz cpio.lzma cpio.lz4 \
> +    cpio cpio.gz cpio.xz cpio.lzma cpio.lz4 cpio.zst \
>      wic wic.gz wic.bz2 wic.lzma wic.zst \
>      container \
>      f2fs \
> diff --git a/meta/classes/kernel-fitimage.bbclass b/meta/classes/kernel-fitimage.bbclass
> index 2ef8f06b14..38e05153e3 100644
> --- a/meta/classes/kernel-fitimage.bbclass
> +++ b/meta/classes/kernel-fitimage.bbclass
> @@ -576,7 +576,7 @@ fitimage_assemble() {
>  	#
>  	if [ "x${ramdiskcount}" = "x1" ] && [ "${INITRAMFS_IMAGE_BUNDLE}" != "1" ]; then
>  		# Find and use the first initramfs image archive type we find
> -		for img in cpio.lz4 cpio.lzo cpio.lzma cpio.xz cpio.gz ext2.gz cpio; do
> +		for img in cpio.lz4 cpio.lzo cpio.lzma cpio.xz cpio.zst cpio.gz ext2.gz cpio; do
>  			initramfs_path="${DEPLOY_DIR_IMAGE}/${INITRAMFS_IMAGE_NAME}.${img}"
>  			echo "Using $initramfs_path"
>  			if [ -e "${initramfs_path}" ]; then
> diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
> index 9ec7daa17a..ab60e060c6 100644
> --- a/meta/classes/kernel.bbclass
> +++ b/meta/classes/kernel.bbclass
> @@ -9,6 +9,7 @@ PROVIDES += "${@ "virtual/kernel" if (d.getVar("KERNEL_PACKAGE_NAME") == "kernel
>  DEPENDS += "virtual/${TARGET_PREFIX}binutils virtual/${TARGET_PREFIX}gcc kmod-native bc-native bison-native"
>  DEPENDS += "${@bb.utils.contains("INITRAMFS_FSTYPES", "cpio.lzo", "lzop-native", "", d)}"
>  DEPENDS += "${@bb.utils.contains("INITRAMFS_FSTYPES", "cpio.lz4", "lz4-native", "", d)}"
> +DEPENDS += "${@bb.utils.contains("INITRAMFS_FSTYPES", "cpio.zst", "zstd-native", "", d)}"
>  PACKAGE_WRITE_DEPS += "depmodwrapper-cross"
>  
>  do_deploy[depends] += "depmodwrapper-cross:do_populate_sysroot gzip-native:do_populate_sysroot"
> @@ -237,7 +238,7 @@ copy_initramfs() {
>  	mkdir -p ${B}/usr
>  	# Find and use the first initramfs image archive type we find
>  	rm -f ${B}/usr/${INITRAMFS_IMAGE_NAME}.cpio
> -	for img in cpio cpio.gz cpio.lz4 cpio.lzo cpio.lzma cpio.xz; do
> +	for img in cpio cpio.gz cpio.lz4 cpio.lzo cpio.lzma cpio.xz cpio.zst; do
>  		if [ -e "${DEPLOY_DIR_IMAGE}/${INITRAMFS_IMAGE_NAME}.$img" ]; then
>  			cp ${DEPLOY_DIR_IMAGE}/${INITRAMFS_IMAGE_NAME}.$img ${B}/usr/.
>  			case $img in
> @@ -266,12 +267,17 @@ copy_initramfs() {
>  				xz -df ${B}/usr/${INITRAMFS_IMAGE_NAME}.$img
>  				break
>  				;;
> +			*zst)
> +				echo "zst decompressing image"
> +				zstd -df ${B}/usr/${INITRAMFS_IMAGE_NAME}.$img
> +				break
> +				;;
>  			esac
>  			break
>  		fi
>  	done
>  	# Verify that the above loop found a initramfs, fail otherwise
> -	[ -f ${B}/usr/${INITRAMFS_IMAGE_NAME}.cpio ] && echo "Finished copy of initramfs into ./usr" || die "Could not find any ${DEPLOY_DIR_IMAGE}/${INITRAMFS_IMAGE_NAME}.cpio{.gz|.lz4|.lzo|.lzma|.xz) for bundling; INITRAMFS_IMAGE_NAME might be wrong."
> +	[ -f ${B}/usr/${INITRAMFS_IMAGE_NAME}.cpio ] && echo "Finished copy of initramfs into ./usr" || die "Could not find any ${DEPLOY_DIR_IMAGE}/${INITRAMFS_IMAGE_NAME}.cpio{.gz|.lz4|.lzo|.lzma|.xz|.zst) for bundling; INITRAMFS_IMAGE_NAME might be wrong."
>  }

Failed in testing:

https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/2467/steps/14/logs/stdio

"mksquashfs: Compressor "zstd" is not supported!"

Cheers,

Richard

Re: [OE-core] [PATCH v3 4/4] Support zstd-compressed squashfs and cpio initramfs

[Zoltan Boszormenyi]

Yeah, squashfs-tools needs PACKAGECONFIG = "zstd" by default.
I will send an updated series soon with this patch updated.

2021. 08. 26. 14:04 keltezéssel, Richard Purdie írta:
> On Mon, 2021-08-23 at 16:54 +0200, Zoltan Boszormenyi via lists.openembedded.org
> wrote:
>> From: Zoltán Böszörményi <zboszor@gmail.com>
>>
>> Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
>> ---
>>   meta/classes/base.bbclass            |  4 ++++
>>   meta/classes/image_types.bbclass     |  6 ++++--
>>   meta/classes/kernel-fitimage.bbclass |  2 +-
>>   meta/classes/kernel.bbclass          | 10 ++++++++--
>>   4 files changed, 17 insertions(+), 5 deletions(-)
>>
>> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
>> index 11b65171d9..340ebe7d78 100644
>> --- a/meta/classes/base.bbclass
>> +++ b/meta/classes/base.bbclass
>> @@ -631,6 +631,10 @@ python () {
>>           if path.endswith('.lz4'):
>>               d.appendVarFlag('do_unpack', 'depends', ' lz4-native:do_populate_sysroot')
>>   
>> +        # *.zst should DEPEND on zstd-native for unpacking
>> +        elif path.endswith('.zst'):
>> +            d.appendVarFlag('do_unpack', 'depends', ' zstd-native:do_populate_sysroot')
>> +
>>           # *.lz should DEPEND on lzip-native for unpacking
>>           elif path.endswith('.lz'):
>>               d.appendVarFlag('do_unpack', 'depends', ' lzip-native:do_populate_sysroot')
>> diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
>> index e0eb06c1e3..32d4cd4c76 100644
>> --- a/meta/classes/image_types.bbclass
>> +++ b/meta/classes/image_types.bbclass
>> @@ -107,6 +107,7 @@ IMAGE_CMD:squashfs = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${
>>   IMAGE_CMD:squashfs-xz = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-xz ${EXTRA_IMAGECMD} -noappend -comp xz"
>>   IMAGE_CMD:squashfs-lzo = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-lzo ${EXTRA_IMAGECMD} -noappend -comp lzo"
>>   IMAGE_CMD:squashfs-lz4 = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-lz4 ${EXTRA_IMAGECMD} -noappend -comp lz4"
>> +IMAGE_CMD:squashfs-zst = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-zst ${EXTRA_IMAGECMD} -noappend -comp zstd"
>>   
>>   IMAGE_CMD:erofs = "mkfs.erofs ${EXTRA_IMAGECMD} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.erofs ${IMAGE_ROOTFS}"
>>   IMAGE_CMD:erofs-lz4 = "mkfs.erofs -zlz4 ${EXTRA_IMAGECMD} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.erofs-lz4 ${IMAGE_ROOTFS}"
>> @@ -244,6 +245,7 @@ do_image_squashfs[depends] += "squashfs-tools-native:do_populate_sysroot"
>>   do_image_squashfs_xz[depends] += "squashfs-tools-native:do_populate_sysroot"
>>   do_image_squashfs_lzo[depends] += "squashfs-tools-native:do_populate_sysroot"
>>   do_image_squashfs_lz4[depends] += "squashfs-tools-native:do_populate_sysroot"
>> +do_image_squashfs_zst[depends] += "squashfs-tools-native:do_populate_sysroot"
>>   do_image_ubi[depends] += "mtd-utils-native:do_populate_sysroot"
>>   do_image_ubifs[depends] += "mtd-utils-native:do_populate_sysroot"
>>   do_image_multiubi[depends] += "mtd-utils-native:do_populate_sysroot"
>> @@ -262,10 +264,10 @@ IMAGE_TYPES = " \
>>       btrfs \
>>       iso \
>>       hddimg \
>> -    squashfs squashfs-xz squashfs-lzo squashfs-lz4 \
>> +    squashfs squashfs-xz squashfs-lzo squashfs-lz4 squashfs-zst \
>>       ubi ubifs multiubi \
>>       tar tar.gz tar.bz2 tar.xz tar.lz4 tar.zst \
>> -    cpio cpio.gz cpio.xz cpio.lzma cpio.lz4 \
>> +    cpio cpio.gz cpio.xz cpio.lzma cpio.lz4 cpio.zst \
>>       wic wic.gz wic.bz2 wic.lzma wic.zst \
>>       container \
>>       f2fs \
>> diff --git a/meta/classes/kernel-fitimage.bbclass b/meta/classes/kernel-fitimage.bbclass
>> index 2ef8f06b14..38e05153e3 100644
>> --- a/meta/classes/kernel-fitimage.bbclass
>> +++ b/meta/classes/kernel-fitimage.bbclass
>> @@ -576,7 +576,7 @@ fitimage_assemble() {
>>   	#
>>   	if [ "x${ramdiskcount}" = "x1" ] && [ "${INITRAMFS_IMAGE_BUNDLE}" != "1" ]; then
>>   		# Find and use the first initramfs image archive type we find
>> -		for img in cpio.lz4 cpio.lzo cpio.lzma cpio.xz cpio.gz ext2.gz cpio; do
>> +		for img in cpio.lz4 cpio.lzo cpio.lzma cpio.xz cpio.zst cpio.gz ext2.gz cpio; do
>>   			initramfs_path="${DEPLOY_DIR_IMAGE}/${INITRAMFS_IMAGE_NAME}.${img}"
>>   			echo "Using $initramfs_path"
>>   			if [ -e "${initramfs_path}" ]; then
>> diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
>> index 9ec7daa17a..ab60e060c6 100644
>> --- a/meta/classes/kernel.bbclass
>> +++ b/meta/classes/kernel.bbclass
>> @@ -9,6 +9,7 @@ PROVIDES += "${@ "virtual/kernel" if (d.getVar("KERNEL_PACKAGE_NAME") == "kernel
>>   DEPENDS += "virtual/${TARGET_PREFIX}binutils virtual/${TARGET_PREFIX}gcc kmod-native bc-native bison-native"
>>   DEPENDS += "${@bb.utils.contains("INITRAMFS_FSTYPES", "cpio.lzo", "lzop-native", "", d)}"
>>   DEPENDS += "${@bb.utils.contains("INITRAMFS_FSTYPES", "cpio.lz4", "lz4-native", "", d)}"
>> +DEPENDS += "${@bb.utils.contains("INITRAMFS_FSTYPES", "cpio.zst", "zstd-native", "", d)}"
>>   PACKAGE_WRITE_DEPS += "depmodwrapper-cross"
>>   
>>   do_deploy[depends] += "depmodwrapper-cross:do_populate_sysroot gzip-native:do_populate_sysroot"
>> @@ -237,7 +238,7 @@ copy_initramfs() {
>>   	mkdir -p ${B}/usr
>>   	# Find and use the first initramfs image archive type we find
>>   	rm -f ${B}/usr/${INITRAMFS_IMAGE_NAME}.cpio
>> -	for img in cpio cpio.gz cpio.lz4 cpio.lzo cpio.lzma cpio.xz; do
>> +	for img in cpio cpio.gz cpio.lz4 cpio.lzo cpio.lzma cpio.xz cpio.zst; do
>>   		if [ -e "${DEPLOY_DIR_IMAGE}/${INITRAMFS_IMAGE_NAME}.$img" ]; then
>>   			cp ${DEPLOY_DIR_IMAGE}/${INITRAMFS_IMAGE_NAME}.$img ${B}/usr/.
>>   			case $img in
>> @@ -266,12 +267,17 @@ copy_initramfs() {
>>   				xz -df ${B}/usr/${INITRAMFS_IMAGE_NAME}.$img
>>   				break
>>   				;;
>> +			*zst)
>> +				echo "zst decompressing image"
>> +				zstd -df ${B}/usr/${INITRAMFS_IMAGE_NAME}.$img
>> +				break
>> +				;;
>>   			esac
>>   			break
>>   		fi
>>   	done
>>   	# Verify that the above loop found a initramfs, fail otherwise
>> -	[ -f ${B}/usr/${INITRAMFS_IMAGE_NAME}.cpio ] && echo "Finished copy of initramfs into ./usr" || die "Could not find any ${DEPLOY_DIR_IMAGE}/${INITRAMFS_IMAGE_NAME}.cpio{.gz|.lz4|.lzo|.lzma|.xz) for bundling; INITRAMFS_IMAGE_NAME might be wrong."
>> +	[ -f ${B}/usr/${INITRAMFS_IMAGE_NAME}.cpio ] && echo "Finished copy of initramfs into ./usr" || die "Could not find any ${DEPLOY_DIR_IMAGE}/${INITRAMFS_IMAGE_NAME}.cpio{.gz|.lz4|.lzo|.lzma|.xz|.zst) for bundling; INITRAMFS_IMAGE_NAME might be wrong."
>>   }
> 
> Failed in testing:
> 
> https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/2467/steps/14/logs/stdio
> 
> "mksquashfs: Compressor "zstd" is not supported!"
> 
> Cheers,
> 
> Richard
> 
> 
> 
> 
> 

Kernel and image related bbclass changes

[Zoltan Boszormenyi]

I have been carrying some forked bbclass recipes and I think
these can be beneficial to others, not to mention that I could
get rid of the private forks of these.

v2:
- drop the package_rpm.bbclass / posttrans patch
- squash the previous patches #3 and #4 together (patch #2 now)
- more verbose commit message for patch #3 (previously patch #5)

v3:
- fix commit message for patch #3 (stupid copy&paste&thinko)

v4:
- enable PACKAGECONFIG=zstd for squashfs-tools in patch #4

[PATCH v4 1/4] kernel-module-split.bbclass: Support zstd-compressed modules

[Zoltan Boszormenyi]

From: Zoltán Böszörményi <zboszor@gmail.com>

Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
---
 meta/classes/kernel-module-split.bbclass | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/meta/classes/kernel-module-split.bbclass b/meta/classes/kernel-module-split.bbclass
index b56dd4a9c7..6c1de4c992 100644
--- a/meta/classes/kernel-module-split.bbclass
+++ b/meta/classes/kernel-module-split.bbclass
@@ -44,17 +44,20 @@ python split_kernel_module_packages () {
     def extract_modinfo(file):
         import tempfile, subprocess
         tempfile.tempdir = d.getVar("WORKDIR")
-        compressed = re.match( r'.*\.([xg])z$', file)
+        compressed = re.match( r'.*\.(gz|xz|zst)$', file)
         tf = tempfile.mkstemp()
         tmpfile = tf[1]
         if compressed:
             tmpkofile = tmpfile + ".ko"
-            if compressed.group(1) == 'g':
+            if compressed.group(1) == 'gz':
                 cmd = "gunzip -dc %s > %s" % (file, tmpkofile)
                 subprocess.check_call(cmd, shell=True)
-            elif compressed.group(1) == 'x':
+            elif compressed.group(1) == 'xz':
                 cmd = "xz -dc %s > %s" % (file, tmpkofile)
                 subprocess.check_call(cmd, shell=True)
+            elif compressed.group(1) == 'zst':
+                cmd = "zstd -dc %s > %s" % (file, tmpkofile)
+                subprocess.check_call(cmd, shell=True)
             else:
                 msg = "Cannot decompress '%s'" % file
                 raise msg
@@ -153,7 +156,7 @@ python split_kernel_module_packages () {
     kernel_package_name = d.getVar("KERNEL_PACKAGE_NAME") or "kernel"
     kernel_version = d.getVar("KERNEL_VERSION")
 
-    module_regex = r'^(.*)\.k?o(?:\.[xg]z)?$'
+    module_regex = r'^(.*)\.k?o(?:\.(gz|xz|zst))?$'
 
     module_pattern_prefix = d.getVar('KERNEL_MODULE_PACKAGE_PREFIX')
     module_pattern_suffix = d.getVar('KERNEL_MODULE_PACKAGE_SUFFIX')
-- 
2.31.1

[PATCH v4 2/4] Allow opt-out of split kernel modules

[Zoltan Boszormenyi]

From: Zoltán Böszörményi <zboszor@gmail.com>

For some use cases, a monolithic kernel-modules package containing
all modules built from the kernel sources may be preferred.
For one, download time is shorter and installation time is faster.

Set KERNEL_SPLIT_MODULES="0" for this in. The default is one subpackage
per module.

Also, adapt kernel.bbclass to KERNEL_SPLIT_MODULES != "1" case
Extra RDEPENDS and other inter-package references are needed in
this case.

Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
---
 meta/classes/kernel-module-split.bbclass | 17 +++++++++++++----
 meta/classes/kernel.bbclass              |  7 +++++++
 2 files changed, 20 insertions(+), 4 deletions(-)

diff --git a/meta/classes/kernel-module-split.bbclass b/meta/classes/kernel-module-split.bbclass
index 6c1de4c992..a29c294810 100644
--- a/meta/classes/kernel-module-split.bbclass
+++ b/meta/classes/kernel-module-split.bbclass
@@ -28,6 +28,7 @@ do_install:append() {
 	install -d ${D}${sysconfdir}/modules-load.d/ ${D}${sysconfdir}/modprobe.d/
 }
 
+KERNEL_SPLIT_MODULES ?= "1"
 PACKAGESPLITFUNCS:prepend = "split_kernel_module_packages "
 
 KERNEL_MODULES_META_PACKAGE ?= "${@ d.getVar("KERNEL_PACKAGE_NAME") or "kernel" }-modules"
@@ -156,18 +157,26 @@ python split_kernel_module_packages () {
     kernel_package_name = d.getVar("KERNEL_PACKAGE_NAME") or "kernel"
     kernel_version = d.getVar("KERNEL_VERSION")
 
+    metapkg = d.getVar('KERNEL_MODULES_META_PACKAGE')
+    splitmods = d.getVar('KERNEL_SPLIT_MODULES')
+    postinst = d.getVar('pkg_postinst:modules')
+    postrm = d.getVar('pkg_postrm:modules')
+
+    if splitmods != '1':
+        etcdir = d.getVar('sysconfdir')
+        d.appendVar('FILES:' + metapkg, '%s/modules-load.d/ %s/modprobe.d/ %s/modules/' % (etcdir, etcdir, d.getVar("nonarch_base_libdir")))
+        d.appendVar('pkg_postinst:%s' % metapkg, postinst)
+        d.prependVar('pkg_postrm:%s' % metapkg, postrm);
+        return
+
     module_regex = r'^(.*)\.k?o(?:\.(gz|xz|zst))?$'
 
     module_pattern_prefix = d.getVar('KERNEL_MODULE_PACKAGE_PREFIX')
     module_pattern_suffix = d.getVar('KERNEL_MODULE_PACKAGE_SUFFIX')
     module_pattern = module_pattern_prefix + kernel_package_name + '-module-%s' + module_pattern_suffix
 
-    postinst = d.getVar('pkg_postinst:modules')
-    postrm = d.getVar('pkg_postrm:modules')
-
     modules = do_split_packages(d, root='${nonarch_base_libdir}/modules', file_regex=module_regex, output_pattern=module_pattern, description='%s kernel module', postinst=postinst, postrm=postrm, recursive=True, hook=frob_metadata, extra_depends='%s-%s' % (kernel_package_name, kernel_version))
     if modules:
-        metapkg = d.getVar('KERNEL_MODULES_META_PACKAGE')
         d.appendVar('RDEPENDS:' + metapkg, ' '+' '.join(modules))
 
     # If modules-load.d and modprobe.d are empty at this point, remove them to
diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
index 027e66eec7..6dc5387a9b 100644
--- a/meta/classes/kernel.bbclass
+++ b/meta/classes/kernel.bbclass
@@ -98,6 +98,13 @@ python __anonymous () {
         d.appendVar('PACKAGES', ' %s-image-%s' % (kname, typelower))
         d.setVar('FILES:' + kname + '-image-' + typelower, '/' + imagedest + '/' + type + '-${KERNEL_VERSION_NAME}' + ' /' + imagedest + '/' + type)
         d.appendVar('RDEPENDS:%s-image' % kname, ' %s-image-%s' % (kname, typelower))
+        splitmods = d.getVar("KERNEL_SPLIT_MODULES")
+        if splitmods != '1':
+            d.appendVar('RDEPENDS:%s-image' % kname, ' %s-modules' % kname)
+            d.appendVar('RDEPENDS:%s-image-%s' % (kname, typelower), ' %s-modules-${KERNEL_VERSION_PKG_NAME}' % kname)
+            d.setVar('PKG:%s-modules' % kname, '%s-modules-${KERNEL_VERSION_PKG_NAME}' % kname)
+            d.appendVar('RPROVIDES:%s-modules' % kname, '%s-modules-${KERNEL_VERSION_PKG_NAME}' % kname)
+
         d.setVar('PKG:%s-image-%s' % (kname,typelower), '%s-image-%s-${KERNEL_VERSION_PKG_NAME}' % (kname, typelower))
         d.setVar('ALLOW_EMPTY:%s-image-%s' % (kname, typelower), '1')
         d.setVar('pkg_postinst:%s-image-%s' % (kname,typelower), """set +e
-- 
2.31.1

[PATCH v4 3/4] kernel.bbclass: Use full versions for inter-package dependencies

[Zoltan Boszormenyi]

From: Zoltán Böszörményi <zboszor@gmail.com>

If the kernel configuration enables module signing but no key
is provided, then the kernel generates one during the kernel build.

The current runtime-dependency references (with only package names
without full versions) allow mixed package installations from different
rebuilds of the same kernel version.

This creates an issue because then the modules either don't work
or taint the kernel.

Tighten RDEPENDS with the full package version, i.e. use (= ${EXTENDPKGV})
markers for inter-package dependencies.

The kernel will pull in the kernel-modules subpackage of the same
exact version automatically if KERNEL_SPLIT_MODULES="0" is set.
Otherwise the situation is the same as with the old default with
one subpackage per kernel module where they have to be upgraded
manually.

Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
---
 meta/classes/kernel.bbclass | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)

diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
index 6dc5387a9b..9ec7daa17a 100644
--- a/meta/classes/kernel.bbclass
+++ b/meta/classes/kernel.bbclass
@@ -91,17 +91,18 @@ python __anonymous () {
     kname = d.getVar('KERNEL_PACKAGE_NAME') or "kernel"
     imagedest = d.getVar('KERNEL_IMAGEDEST')
 
+    fullver = d.getVar('EXTENDPKGV')
     for type in types.split():
         if bb.data.inherits_class('nopackages', d):
             continue
         typelower = type.lower()
         d.appendVar('PACKAGES', ' %s-image-%s' % (kname, typelower))
         d.setVar('FILES:' + kname + '-image-' + typelower, '/' + imagedest + '/' + type + '-${KERNEL_VERSION_NAME}' + ' /' + imagedest + '/' + type)
-        d.appendVar('RDEPENDS:%s-image' % kname, ' %s-image-%s' % (kname, typelower))
+        d.appendVar('RDEPENDS:%s-image' % kname, ' %s-image-%s (= %s)' % (kname, typelower, fullver))
         splitmods = d.getVar("KERNEL_SPLIT_MODULES")
         if splitmods != '1':
-            d.appendVar('RDEPENDS:%s-image' % kname, ' %s-modules' % kname)
-            d.appendVar('RDEPENDS:%s-image-%s' % (kname, typelower), ' %s-modules-${KERNEL_VERSION_PKG_NAME}' % kname)
+            d.appendVar('RDEPENDS:%s-image' % kname, ' %s-modules (= %s)' % (kname, fullver))
+            d.appendVar('RDEPENDS:%s-image-%s' % (kname, typelower), ' %s-modules-${KERNEL_VERSION_PKG_NAME} (= %s)' % (kname, fullver))
             d.setVar('PKG:%s-modules' % kname, '%s-modules-${KERNEL_VERSION_PKG_NAME}' % kname)
             d.appendVar('RPROVIDES:%s-modules' % kname, '%s-modules-${KERNEL_VERSION_PKG_NAME}' % kname)
 
@@ -628,12 +629,12 @@ FILES:${KERNEL_PACKAGE_NAME}-image = ""
 FILES:${KERNEL_PACKAGE_NAME}-dev = "/boot/System.map* /boot/Module.symvers* /boot/config* ${KERNEL_SRC_PATH} ${nonarch_base_libdir}/modules/${KERNEL_VERSION}/build"
 FILES:${KERNEL_PACKAGE_NAME}-vmlinux = "/boot/vmlinux-${KERNEL_VERSION_NAME}"
 FILES:${KERNEL_PACKAGE_NAME}-modules = ""
-RDEPENDS:${KERNEL_PACKAGE_NAME} = "${KERNEL_PACKAGE_NAME}-base"
+RDEPENDS:${KERNEL_PACKAGE_NAME} = "${KERNEL_PACKAGE_NAME}-base (= ${EXTENDPKGV})"
 # Allow machines to override this dependency if kernel image files are
 # not wanted in images as standard
-RDEPENDS:${KERNEL_PACKAGE_NAME}-base ?= "${KERNEL_PACKAGE_NAME}-image"
+RDEPENDS:${KERNEL_PACKAGE_NAME}-base ?= "${KERNEL_PACKAGE_NAME}-image (= ${EXTENDPKGV})"
 PKG:${KERNEL_PACKAGE_NAME}-image = "${KERNEL_PACKAGE_NAME}-image-${@legitimize_package_name(d.getVar('KERNEL_VERSION'))}"
-RDEPENDS:${KERNEL_PACKAGE_NAME}-image += "${@oe.utils.conditional('KERNEL_IMAGETYPE', 'vmlinux', '${KERNEL_PACKAGE_NAME}-vmlinux', '', d)}"
+RDEPENDS:${KERNEL_PACKAGE_NAME}-image += "${@oe.utils.conditional('KERNEL_IMAGETYPE', 'vmlinux', '${KERNEL_PACKAGE_NAME}-vmlinux (= ${EXTENDPKGV})', '', d)}"
 PKG:${KERNEL_PACKAGE_NAME}-base = "${KERNEL_PACKAGE_NAME}-${@legitimize_package_name(d.getVar('KERNEL_VERSION'))}"
 RPROVIDES:${KERNEL_PACKAGE_NAME}-base += "${KERNEL_PACKAGE_NAME}-${KERNEL_VERSION}"
 ALLOW_EMPTY:${KERNEL_PACKAGE_NAME} = "1"
-- 
2.31.1

[PATCH v4 4/4] Support zstd-compressed squashfs and cpio initramfs

[Zoltan Boszormenyi]

From: Zoltán Böszörményi <zboszor@gmail.com>

Enable zstd PACKAGECONFIG knob for squashfs-tools.

Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
---
 meta/classes/base.bbclass                              |  4 ++++
 meta/classes/image_types.bbclass                       |  6 ++++--
 meta/classes/kernel-fitimage.bbclass                   |  2 +-
 meta/classes/kernel.bbclass                            | 10 ++++++++--
 .../squashfs-tools/squashfs-tools_git.bb               |  2 +-
 5 files changed, 18 insertions(+), 6 deletions(-)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 11b65171d9..340ebe7d78 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -631,6 +631,10 @@ python () {
         if path.endswith('.lz4'):
             d.appendVarFlag('do_unpack', 'depends', ' lz4-native:do_populate_sysroot')
 
+        # *.zst should DEPEND on zstd-native for unpacking
+        elif path.endswith('.zst'):
+            d.appendVarFlag('do_unpack', 'depends', ' zstd-native:do_populate_sysroot')
+
         # *.lz should DEPEND on lzip-native for unpacking
         elif path.endswith('.lz'):
             d.appendVarFlag('do_unpack', 'depends', ' lzip-native:do_populate_sysroot')
diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index e0eb06c1e3..32d4cd4c76 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -107,6 +107,7 @@ IMAGE_CMD:squashfs = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${
 IMAGE_CMD:squashfs-xz = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-xz ${EXTRA_IMAGECMD} -noappend -comp xz"
 IMAGE_CMD:squashfs-lzo = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-lzo ${EXTRA_IMAGECMD} -noappend -comp lzo"
 IMAGE_CMD:squashfs-lz4 = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-lz4 ${EXTRA_IMAGECMD} -noappend -comp lz4"
+IMAGE_CMD:squashfs-zst = "mksquashfs ${IMAGE_ROOTFS} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.squashfs-zst ${EXTRA_IMAGECMD} -noappend -comp zstd"
 
 IMAGE_CMD:erofs = "mkfs.erofs ${EXTRA_IMAGECMD} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.erofs ${IMAGE_ROOTFS}"
 IMAGE_CMD:erofs-lz4 = "mkfs.erofs -zlz4 ${EXTRA_IMAGECMD} ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.erofs-lz4 ${IMAGE_ROOTFS}"
@@ -244,6 +245,7 @@ do_image_squashfs[depends] += "squashfs-tools-native:do_populate_sysroot"
 do_image_squashfs_xz[depends] += "squashfs-tools-native:do_populate_sysroot"
 do_image_squashfs_lzo[depends] += "squashfs-tools-native:do_populate_sysroot"
 do_image_squashfs_lz4[depends] += "squashfs-tools-native:do_populate_sysroot"
+do_image_squashfs_zst[depends] += "squashfs-tools-native:do_populate_sysroot"
 do_image_ubi[depends] += "mtd-utils-native:do_populate_sysroot"
 do_image_ubifs[depends] += "mtd-utils-native:do_populate_sysroot"
 do_image_multiubi[depends] += "mtd-utils-native:do_populate_sysroot"
@@ -262,10 +264,10 @@ IMAGE_TYPES = " \
     btrfs \
     iso \
     hddimg \
-    squashfs squashfs-xz squashfs-lzo squashfs-lz4 \
+    squashfs squashfs-xz squashfs-lzo squashfs-lz4 squashfs-zst \
     ubi ubifs multiubi \
     tar tar.gz tar.bz2 tar.xz tar.lz4 tar.zst \
-    cpio cpio.gz cpio.xz cpio.lzma cpio.lz4 \
+    cpio cpio.gz cpio.xz cpio.lzma cpio.lz4 cpio.zst \
     wic wic.gz wic.bz2 wic.lzma wic.zst \
     container \
     f2fs \
diff --git a/meta/classes/kernel-fitimage.bbclass b/meta/classes/kernel-fitimage.bbclass
index 2ef8f06b14..38e05153e3 100644
--- a/meta/classes/kernel-fitimage.bbclass
+++ b/meta/classes/kernel-fitimage.bbclass
@@ -576,7 +576,7 @@ fitimage_assemble() {
 	#
 	if [ "x${ramdiskcount}" = "x1" ] && [ "${INITRAMFS_IMAGE_BUNDLE}" != "1" ]; then
 		# Find and use the first initramfs image archive type we find
-		for img in cpio.lz4 cpio.lzo cpio.lzma cpio.xz cpio.gz ext2.gz cpio; do
+		for img in cpio.lz4 cpio.lzo cpio.lzma cpio.xz cpio.zst cpio.gz ext2.gz cpio; do
 			initramfs_path="${DEPLOY_DIR_IMAGE}/${INITRAMFS_IMAGE_NAME}.${img}"
 			echo "Using $initramfs_path"
 			if [ -e "${initramfs_path}" ]; then
diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
index 9ec7daa17a..ab60e060c6 100644
--- a/meta/classes/kernel.bbclass
+++ b/meta/classes/kernel.bbclass
@@ -9,6 +9,7 @@ PROVIDES += "${@ "virtual/kernel" if (d.getVar("KERNEL_PACKAGE_NAME") == "kernel
 DEPENDS += "virtual/${TARGET_PREFIX}binutils virtual/${TARGET_PREFIX}gcc kmod-native bc-native bison-native"
 DEPENDS += "${@bb.utils.contains("INITRAMFS_FSTYPES", "cpio.lzo", "lzop-native", "", d)}"
 DEPENDS += "${@bb.utils.contains("INITRAMFS_FSTYPES", "cpio.lz4", "lz4-native", "", d)}"
+DEPENDS += "${@bb.utils.contains("INITRAMFS_FSTYPES", "cpio.zst", "zstd-native", "", d)}"
 PACKAGE_WRITE_DEPS += "depmodwrapper-cross"
 
 do_deploy[depends] += "depmodwrapper-cross:do_populate_sysroot gzip-native:do_populate_sysroot"
@@ -237,7 +238,7 @@ copy_initramfs() {
 	mkdir -p ${B}/usr
 	# Find and use the first initramfs image archive type we find
 	rm -f ${B}/usr/${INITRAMFS_IMAGE_NAME}.cpio
-	for img in cpio cpio.gz cpio.lz4 cpio.lzo cpio.lzma cpio.xz; do
+	for img in cpio cpio.gz cpio.lz4 cpio.lzo cpio.lzma cpio.xz cpio.zst; do
 		if [ -e "${DEPLOY_DIR_IMAGE}/${INITRAMFS_IMAGE_NAME}.$img" ]; then
 			cp ${DEPLOY_DIR_IMAGE}/${INITRAMFS_IMAGE_NAME}.$img ${B}/usr/.
 			case $img in
@@ -266,12 +267,17 @@ copy_initramfs() {
 				xz -df ${B}/usr/${INITRAMFS_IMAGE_NAME}.$img
 				break
 				;;
+			*zst)
+				echo "zst decompressing image"
+				zstd -df ${B}/usr/${INITRAMFS_IMAGE_NAME}.$img
+				break
+				;;
 			esac
 			break
 		fi
 	done
 	# Verify that the above loop found a initramfs, fail otherwise
-	[ -f ${B}/usr/${INITRAMFS_IMAGE_NAME}.cpio ] && echo "Finished copy of initramfs into ./usr" || die "Could not find any ${DEPLOY_DIR_IMAGE}/${INITRAMFS_IMAGE_NAME}.cpio{.gz|.lz4|.lzo|.lzma|.xz) for bundling; INITRAMFS_IMAGE_NAME might be wrong."
+	[ -f ${B}/usr/${INITRAMFS_IMAGE_NAME}.cpio ] && echo "Finished copy of initramfs into ./usr" || die "Could not find any ${DEPLOY_DIR_IMAGE}/${INITRAMFS_IMAGE_NAME}.cpio{.gz|.lz4|.lzo|.lzma|.xz|.zst) for bundling; INITRAMFS_IMAGE_NAME might be wrong."
 }
 
 do_bundle_initramfs () {
diff --git a/meta/recipes-devtools/squashfs-tools/squashfs-tools_git.bb b/meta/recipes-devtools/squashfs-tools/squashfs-tools_git.bb
index 2b9a765bb2..c78f446711 100644
--- a/meta/recipes-devtools/squashfs-tools/squashfs-tools_git.bb
+++ b/meta/recipes-devtools/squashfs-tools/squashfs-tools_git.bb
@@ -17,7 +17,7 @@ S = "${WORKDIR}/git/squashfs-tools"
 
 EXTRA_OEMAKE = "${PACKAGECONFIG_CONFARGS}"
 
-PACKAGECONFIG ??= "gzip xz lzo lz4 lzma xattr reproducible"
+PACKAGECONFIG ??= "gzip xz lzo lz4 lzma xattr zstd reproducible"
 PACKAGECONFIG[gzip] = "GZIP_SUPPORT=1,GZIP_SUPPORT=0,zlib"
 PACKAGECONFIG[xz] = "XZ_SUPPORT=1,XZ_SUPPORT=0,xz"
 PACKAGECONFIG[lzo] = "LZO_SUPPORT=1,LZO_SUPPORT=0,lzo"
-- 
2.31.1

Re: [OE-core] [PATCH v4 3/4] kernel.bbclass: Use full versions for inter-package dependencies

[Andrey Zhizhikin]

Hello Zoltan,

On Fri, Aug 27, 2021 at 9:37 AM Zoltan Boszormenyi via
lists.openembedded.org <zboszor=pr.hu@lists.openembedded.org> wrote:
>
> From: Zoltán Böszörményi <zboszor@gmail.com>
>
> If the kernel configuration enables module signing but no key
> is provided, then the kernel generates one during the kernel build.
>
> The current runtime-dependency references (with only package names
> without full versions) allow mixed package installations from different
> rebuilds of the same kernel version.
>
> This creates an issue because then the modules either don't work
> or taint the kernel.
>
> Tighten RDEPENDS with the full package version, i.e. use (= ${EXTENDPKGV})
> markers for inter-package dependencies.
>
> The kernel will pull in the kernel-modules subpackage of the same
> exact version automatically if KERNEL_SPLIT_MODULES="0" is set.
> Otherwise the situation is the same as with the old default with
> one subpackage per kernel module where they have to be upgraded
> manually.
>
> Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
> ---
>  meta/classes/kernel.bbclass | 13 +++++++------
>  1 file changed, 7 insertions(+), 6 deletions(-)
>

I'm seeing errors during the do_rootfs() with this patch applied,
there are few messages like this:

 * Solver encountered 1 problem(s):
 * Problem 1/1:
 *   - package kernel-module-libchacha-5.13.13+g91381833a4e2-5.13.13+git0+91381833a4-r0.imx8mp_lpddr4_evk
requires kernel-5.13.13+g91381833a4e2, but none of the providers can
be installed
 *   - package kernel-modules-5.13.13+git0+91381833a4-r0.imx8mp_lpddr4_evk
requires kernel-module-libchacha-5.13.13+g91381833a4e2, but none of
the providers can be installed
 *   - package kernel-5.13.13+g91381833a4e2-5.13.13+git0+91381833a4-r0.imx8mp_lpddr4_evk
requires kernel-image-5.13.13+g91381833a4e2 =
5.13.13+git0+91381833a4-r0, but none of the providers can be installed
 *   - conflicting requests
 *   - nothing provides kernel-image-image-5.13.13+g91381833a4e2 =
5.13.13+gitAUTOINC+91381833a4-r0 needed by
kernel-image-5.13.13+g91381833a4e2-5.13.13+git0+91381833a4-r0.imx8mp_lpddr4_evk
 *
 * Solution 1:
 *   - do not ask to install a package providing kernel-modules

Reverting this commit does solve it, but I would like to understand
more on what should be done in order to use this new mechanism you've
introduced. Do you have any suggestion on what should be adapted in
the BSP layer to have those RDEPENDS properly resolved?

Package manager is set for me to IPK.

Setting KERNEL_SPLIT_MODULES="1" does not solve the issue.

-- 
Regards,
Andrey.

Re: [OE-core] [PATCH v4 3/4] kernel.bbclass: Use full versions for inter-package dependencies

[Zoltan Boszormenyi]

2021. 08. 30. 11:30 keltezéssel, Andrey Zhizhikin írta:
> Hello Zoltan,
>
> On Fri, Aug 27, 2021 at 9:37 AM Zoltan Boszormenyi via
> lists.openembedded.org <zboszor=pr.hu@lists.openembedded.org> wrote:
>> From: Zoltán Böszörményi <zboszor@gmail.com>
>>
>> If the kernel configuration enables module signing but no key
>> is provided, then the kernel generates one during the kernel build.
>>
>> The current runtime-dependency references (with only package names
>> without full versions) allow mixed package installations from different
>> rebuilds of the same kernel version.
>>
>> This creates an issue because then the modules either don't work
>> or taint the kernel.
>>
>> Tighten RDEPENDS with the full package version, i.e. use (= ${EXTENDPKGV})
>> markers for inter-package dependencies.
>>
>> The kernel will pull in the kernel-modules subpackage of the same
>> exact version automatically if KERNEL_SPLIT_MODULES="0" is set.
>> Otherwise the situation is the same as with the old default with
>> one subpackage per kernel module where they have to be upgraded
>> manually.
>>
>> Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
>> ---
>>    meta/classes/kernel.bbclass | 13 +++++++------
>>    1 file changed, 7 insertions(+), 6 deletions(-)
>>
> I'm seeing errors during the do_rootfs() with this patch applied,
> there are few messages like this:
>
>    * Solver encountered 1 problem(s):
>    * Problem 1/1:
>    *   - package kernel-module-libchacha-5.13.13+g91381833a4e2-5.13.13+git0+91381833a4-r0.imx8mp_lpddr4_evk
> requires kernel-5.13.13+g91381833a4e2, but none of the providers can
> be installed
>    *   - package kernel-modules-5.13.13+git0+91381833a4-r0.imx8mp_lpddr4_evk
> requires kernel-module-libchacha-5.13.13+g91381833a4e2, but none of
> the providers can be installed
>    *   - package kernel-5.13.13+g91381833a4e2-5.13.13+git0+91381833a4-r0.imx8mp_lpddr4_evk
> requires kernel-image-5.13.13+g91381833a4e2 =
> 5.13.13+git0+91381833a4-r0, but none of the providers can be installed
>    *   - conflicting requests
>    *   - nothing provides kernel-image-image-5.13.13+g91381833a4e2 =
> 5.13.13+gitAUTOINC+91381833a4-r0 needed by

This seems to be the problem.
Is there a "kernel-image-image-5.13.13" built from your kernel recipe?
For me, a kernel-image-bzimage-x.y.z is built.

> kernel-image-5.13.13+g91381833a4e2-5.13.13+git0+91381833a4-r0.imx8mp_lpddr4_evk
>    *
>    * Solution 1:
>    *   - do not ask to install a package providing kernel-modules
>
> Reverting this commit does solve it, but I would like to understand
> more on what should be done in order to use this new mechanism you've
> introduced. Do you have any suggestion on what should be adapted in
> the BSP layer to have those RDEPENDS properly resolved?
>
> Package manager is set for me to IPK.
>
> Setting KERNEL_SPLIT_MODULES="1" does not solve the issue.

This is the default.
But setting it to "0" won't change whether the
kernel-image-image-x.y.z subpackage gets built or not.

Re: [OE-core] [PATCH v4 3/4] kernel.bbclass: Use full versions for inter-package dependencies

[Andrey Zhizhikin]

On Mon, Aug 30, 2021 at 12:06 PM Böszörményi Zoltán <zboszor@gmail.com> wrote:
>
> 2021. 08. 30. 11:30 keltezéssel, Andrey Zhizhikin írta:
> > Hello Zoltan,
> >
> > On Fri, Aug 27, 2021 at 9:37 AM Zoltan Boszormenyi via
> > lists.openembedded.org <zboszor=pr.hu@lists.openembedded.org> wrote:
> >> From: Zoltán Böszörményi <zboszor@gmail.com>
> >>
> >> If the kernel configuration enables module signing but no key
> >> is provided, then the kernel generates one during the kernel build.
> >>
> >> The current runtime-dependency references (with only package names
> >> without full versions) allow mixed package installations from different
> >> rebuilds of the same kernel version.
> >>
> >> This creates an issue because then the modules either don't work
> >> or taint the kernel.
> >>
> >> Tighten RDEPENDS with the full package version, i.e. use (= ${EXTENDPKGV})
> >> markers for inter-package dependencies.
> >>
> >> The kernel will pull in the kernel-modules subpackage of the same
> >> exact version automatically if KERNEL_SPLIT_MODULES="0" is set.
> >> Otherwise the situation is the same as with the old default with
> >> one subpackage per kernel module where they have to be upgraded
> >> manually.
> >>
> >> Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
> >> ---
> >>   meta/classes/kernel.bbclass | 13 +++++++------
> >>   1 file changed, 7 insertions(+), 6 deletions(-)
> >>
> > I'm seeing errors during the do_rootfs() with this patch applied,
> > there are few messages like this:
> >
> >   * Solver encountered 1 problem(s):
> >   * Problem 1/1:
> >   *   - package kernel-module-libchacha-5.13.13+g91381833a4e2-5.13.13+git0+91381833a4-r0.imx8mp_lpddr4_evk
> > requires kernel-5.13.13+g91381833a4e2, but none of the providers can
> > be installed
> >   *   - package kernel-modules-5.13.13+git0+91381833a4-r0.imx8mp_lpddr4_evk
> > requires kernel-module-libchacha-5.13.13+g91381833a4e2, but none of
> > the providers can be installed
> >   *   - package kernel-5.13.13+g91381833a4e2-5.13.13+git0+91381833a4-r0.imx8mp_lpddr4_evk
> > requires kernel-image-5.13.13+g91381833a4e2 =
> > 5.13.13+git0+91381833a4-r0, but none of the providers can be installed
> >   *   - conflicting requests
> >   *   - nothing provides kernel-image-image-5.13.13+g91381833a4e2 =
> > 5.13.13+gitAUTOINC+91381833a4-r0 needed by
>
> This seems to be the problem.
> Is there a "kernel-image-image-5.13.13" built from your kernel recipe?

Yes, it is produced. But for some reasons opkg cannot resolve it
during the do_rootfs(), which is quite odd.

> For me, a kernel-image-bzimage-x.y.z is built.
>
> > kernel-image-5.13.13+g91381833a4e2-5.13.13+git0+91381833a4-r0.imx8mp_lpddr4_evk
> >   *
> >   * Solution 1:
> >   *   - do not ask to install a package providing kernel-modules
> >
> > Reverting this commit does solve it, but I would like to understand
> > more on what should be done in order to use this new mechanism you've
> > introduced. Do you have any suggestion on what should be adapted in
> > the BSP layer to have those RDEPENDS properly resolved?
> >
> > Package manager is set for me to IPK.
> >
> > Setting KERNEL_SPLIT_MODULES="1" does not solve the issue.
>
> This is the default.
> But setting it to "0" won't change whether the
> kernel-image-image-x.y.z subpackage gets built or not.
>


-- 
Regards,
Andrey.

Re: [OE-core] [PATCH v4 3/4] kernel.bbclass: Use full versions for inter-package dependencies

[Jon Mason]

On Mon, Aug 30, 2021 at 6:26 AM Andrey Zhizhikin <andrey.z@gmail.com> wrote:
>
> On Mon, Aug 30, 2021 at 12:06 PM Böszörményi Zoltán <zboszor@gmail.com> wrote:
> >
> > 2021. 08. 30. 11:30 keltezéssel, Andrey Zhizhikin írta:
> > > Hello Zoltan,
> > >
> > > On Fri, Aug 27, 2021 at 9:37 AM Zoltan Boszormenyi via
> > > lists.openembedded.org <zboszor=pr.hu@lists.openembedded.org> wrote:
> > >> From: Zoltán Böszörményi <zboszor@gmail.com>
> > >>
> > >> If the kernel configuration enables module signing but no key
> > >> is provided, then the kernel generates one during the kernel build.
> > >>
> > >> The current runtime-dependency references (with only package names
> > >> without full versions) allow mixed package installations from different
> > >> rebuilds of the same kernel version.
> > >>
> > >> This creates an issue because then the modules either don't work
> > >> or taint the kernel.
> > >>
> > >> Tighten RDEPENDS with the full package version, i.e. use (= ${EXTENDPKGV})
> > >> markers for inter-package dependencies.
> > >>
> > >> The kernel will pull in the kernel-modules subpackage of the same
> > >> exact version automatically if KERNEL_SPLIT_MODULES="0" is set.
> > >> Otherwise the situation is the same as with the old default with
> > >> one subpackage per kernel module where they have to be upgraded
> > >> manually.
> > >>
> > >> Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
> > >> ---
> > >>   meta/classes/kernel.bbclass | 13 +++++++------
> > >>   1 file changed, 7 insertions(+), 6 deletions(-)
> > >>
> > > I'm seeing errors during the do_rootfs() with this patch applied,
> > > there are few messages like this:
> > >
> > >   * Solver encountered 1 problem(s):
> > >   * Problem 1/1:
> > >   *   - package kernel-module-libchacha-5.13.13+g91381833a4e2-5.13.13+git0+91381833a4-r0.imx8mp_lpddr4_evk
> > > requires kernel-5.13.13+g91381833a4e2, but none of the providers can
> > > be installed
> > >   *   - package kernel-modules-5.13.13+git0+91381833a4-r0.imx8mp_lpddr4_evk
> > > requires kernel-module-libchacha-5.13.13+g91381833a4e2, but none of
> > > the providers can be installed
> > >   *   - package kernel-5.13.13+g91381833a4e2-5.13.13+git0+91381833a4-r0.imx8mp_lpddr4_evk
> > > requires kernel-image-5.13.13+g91381833a4e2 =
> > > 5.13.13+git0+91381833a4-r0, but none of the providers can be installed
> > >   *   - conflicting requests
> > >   *   - nothing provides kernel-image-image-5.13.13+g91381833a4e2 =
> > > 5.13.13+gitAUTOINC+91381833a4-r0 needed by
> >
> > This seems to be the problem.
> > Is there a "kernel-image-image-5.13.13" built from your kernel recipe?
>
> Yes, it is produced. But for some reasons opkg cannot resolve it
> during the do_rootfs(), which is quite odd.

I'm seeing the same issue.  All of the BSPs that I set the kernel to
not be 5.13 (i.e., 5.10, 5.4, etc) fail.  For example,
https://gitlab.com/jonmason00/meta-arm/-/jobs/1544819828
If I set the PACKAGE_CLASS to be rpm instead of ipk, everything works
as expected.  So, there must be some difference in the dep calculation
in ipk.


Thanks,
Jon

> > For me, a kernel-image-bzimage-x.y.z is built.
> >
> > > kernel-image-5.13.13+g91381833a4e2-5.13.13+git0+91381833a4-r0.imx8mp_lpddr4_evk
> > >   *
> > >   * Solution 1:
> > >   *   - do not ask to install a package providing kernel-modules
> > >
> > > Reverting this commit does solve it, but I would like to understand
> > > more on what should be done in order to use this new mechanism you've
> > > introduced. Do you have any suggestion on what should be adapted in
> > > the BSP layer to have those RDEPENDS properly resolved?
> > >
> > > Package manager is set for me to IPK.
> > >
> > > Setting KERNEL_SPLIT_MODULES="1" does not solve the issue.
> >
> > This is the default.
> > But setting it to "0" won't change whether the
> > kernel-image-image-x.y.z subpackage gets built or not.
> >
>
>
> --
> Regards,
> Andrey.
>
> 
>

Re: [OE-core] [PATCH v4 3/4] kernel.bbclass: Use full versions for inter-package dependencies

[Zoltan Boszormenyi]

2021. 08. 30. 21:51 keltezéssel, Jon Mason írta:
> On Mon, Aug 30, 2021 at 6:26 AM Andrey Zhizhikin <andrey.z@gmail.com> wrote:
>>
>> On Mon, Aug 30, 2021 at 12:06 PM Böszörményi Zoltán <zboszor@gmail.com> wrote:
>>>
>>> 2021. 08. 30. 11:30 keltezéssel, Andrey Zhizhikin írta:
>>>> Hello Zoltan,
>>>>
>>>> On Fri, Aug 27, 2021 at 9:37 AM Zoltan Boszormenyi via
>>>> lists.openembedded.org <zboszor=pr.hu@lists.openembedded.org> wrote:
>>>>> From: Zoltán Böszörményi <zboszor@gmail.com>
>>>>>
>>>>> If the kernel configuration enables module signing but no key
>>>>> is provided, then the kernel generates one during the kernel build.
>>>>>
>>>>> The current runtime-dependency references (with only package names
>>>>> without full versions) allow mixed package installations from different
>>>>> rebuilds of the same kernel version.
>>>>>
>>>>> This creates an issue because then the modules either don't work
>>>>> or taint the kernel.
>>>>>
>>>>> Tighten RDEPENDS with the full package version, i.e. use (= ${EXTENDPKGV})
>>>>> markers for inter-package dependencies.
>>>>>
>>>>> The kernel will pull in the kernel-modules subpackage of the same
>>>>> exact version automatically if KERNEL_SPLIT_MODULES="0" is set.
>>>>> Otherwise the situation is the same as with the old default with
>>>>> one subpackage per kernel module where they have to be upgraded
>>>>> manually.
>>>>>
>>>>> Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
>>>>> ---
>>>>>    meta/classes/kernel.bbclass | 13 +++++++------
>>>>>    1 file changed, 7 insertions(+), 6 deletions(-)
>>>>>
>>>> I'm seeing errors during the do_rootfs() with this patch applied,
>>>> there are few messages like this:
>>>>
>>>>    * Solver encountered 1 problem(s):
>>>>    * Problem 1/1:
>>>>    *   - package kernel-module-libchacha-5.13.13+g91381833a4e2-5.13.13+git0+91381833a4-r0.imx8mp_lpddr4_evk
>>>> requires kernel-5.13.13+g91381833a4e2, but none of the providers can
>>>> be installed
>>>>    *   - package kernel-modules-5.13.13+git0+91381833a4-r0.imx8mp_lpddr4_evk
>>>> requires kernel-module-libchacha-5.13.13+g91381833a4e2, but none of
>>>> the providers can be installed
>>>>    *   - package kernel-5.13.13+g91381833a4e2-5.13.13+git0+91381833a4-r0.imx8mp_lpddr4_evk
>>>> requires kernel-image-5.13.13+g91381833a4e2 =
>>>> 5.13.13+git0+91381833a4-r0, but none of the providers can be installed
>>>>    *   - conflicting requests
>>>>    *   - nothing provides kernel-image-image-5.13.13+g91381833a4e2 =
>>>> 5.13.13+gitAUTOINC+91381833a4-r0 needed by
>>>
>>> This seems to be the problem.
>>> Is there a "kernel-image-image-5.13.13" built from your kernel recipe?
>>
>> Yes, it is produced. But for some reasons opkg cannot resolve it
>> during the do_rootfs(), which is quite odd.
> 
> I'm seeing the same issue.  All of the BSPs that I set the kernel to
> not be 5.13 (i.e., 5.10, 5.4, etc) fail.  For example,
> https://gitlab.com/jonmason00/meta-arm/-/jobs/1544819828
> If I set the PACKAGE_CLASS to be rpm instead of ipk, everything works
> as expected.  So, there must be some difference in the dep calculation
> in ipk.

Interesting.

Can you both please try setting KERNEL_SPLIT_MODULES="0" in your kernel recipe?
In my testing, it works for both ipk and rpm properly.

I am thinking that probably the full version dependency should
only be used in the KERNEL_SPLIT_MODULES="0" case.
This can be easily tested, unlike the packaging method.


> 
> 
> Thanks,
> Jon
> 
>>> For me, a kernel-image-bzimage-x.y.z is built.
>>>
>>>> kernel-image-5.13.13+g91381833a4e2-5.13.13+git0+91381833a4-r0.imx8mp_lpddr4_evk
>>>>    *
>>>>    * Solution 1:
>>>>    *   - do not ask to install a package providing kernel-modules
>>>>
>>>> Reverting this commit does solve it, but I would like to understand
>>>> more on what should be done in order to use this new mechanism you've
>>>> introduced. Do you have any suggestion on what should be adapted in
>>>> the BSP layer to have those RDEPENDS properly resolved?
>>>>
>>>> Package manager is set for me to IPK.
>>>>
>>>> Setting KERNEL_SPLIT_MODULES="1" does not solve the issue.
>>>
>>> This is the default.
>>> But setting it to "0" won't change whether the
>>> kernel-image-image-x.y.z subpackage gets built or not.
>>>
>>
>>
>> --
>> Regards,
>> Andrey.
>>
>>
>>
>>
>>
>> 
>>

Re: [OE-core] [PATCH v4 3/4] kernel.bbclass: Use full versions for inter-package dependencies

[Richard Purdie]

On Tue, 2021-08-31 at 06:28 +0200, Zoltan Boszormenyi via lists.openembedded.org
wrote:
> 2021. 08. 30. 21:51 keltezéssel, Jon Mason írta:
> > On Mon, Aug 30, 2021 at 6:26 AM Andrey Zhizhikin <andrey.z@gmail.com> wrote:
> > > 
> > > On Mon, Aug 30, 2021 at 12:06 PM Böszörményi Zoltán <zboszor@gmail.com> wrote:
> > > > 
> > > > 2021. 08. 30. 11:30 keltezéssel, Andrey Zhizhikin írta:
> > > > > Hello Zoltan,
> > > > > 
> > > > > On Fri, Aug 27, 2021 at 9:37 AM Zoltan Boszormenyi via
> > > > > lists.openembedded.org <zboszor=pr.hu@lists.openembedded.org> wrote:
> > > > > > From: Zoltán Böszörményi <zboszor@gmail.com>
> > > > > > 
> > > > > > If the kernel configuration enables module signing but no key
> > > > > > is provided, then the kernel generates one during the kernel build.
> > > > > > 
> > > > > > The current runtime-dependency references (with only package names
> > > > > > without full versions) allow mixed package installations from different
> > > > > > rebuilds of the same kernel version.
> > > > > > 
> > > > > > This creates an issue because then the modules either don't work
> > > > > > or taint the kernel.
> > > > > > 
> > > > > > Tighten RDEPENDS with the full package version, i.e. use (= ${EXTENDPKGV})
> > > > > > markers for inter-package dependencies.
> > > > > > 
> > > > > > The kernel will pull in the kernel-modules subpackage of the same
> > > > > > exact version automatically if KERNEL_SPLIT_MODULES="0" is set.
> > > > > > Otherwise the situation is the same as with the old default with
> > > > > > one subpackage per kernel module where they have to be upgraded
> > > > > > manually.
> > > > > > 
> > > > > > Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
> > > > > > ---
> > > > > >    meta/classes/kernel.bbclass | 13 +++++++------
> > > > > >    1 file changed, 7 insertions(+), 6 deletions(-)
> > > > > > 
> > > > > I'm seeing errors during the do_rootfs() with this patch applied,
> > > > > there are few messages like this:
> > > > > 
> > > > >    * Solver encountered 1 problem(s):
> > > > >    * Problem 1/1:
> > > > >    *   - package kernel-module-libchacha-5.13.13+g91381833a4e2-5.13.13+git0+91381833a4-r0.imx8mp_lpddr4_evk
> > > > > requires kernel-5.13.13+g91381833a4e2, but none of the providers can
> > > > > be installed
> > > > >    *   - package kernel-modules-5.13.13+git0+91381833a4-r0.imx8mp_lpddr4_evk
> > > > > requires kernel-module-libchacha-5.13.13+g91381833a4e2, but none of
> > > > > the providers can be installed
> > > > >    *   - package kernel-5.13.13+g91381833a4e2-5.13.13+git0+91381833a4-r0.imx8mp_lpddr4_evk
> > > > > requires kernel-image-5.13.13+g91381833a4e2 =
> > > > > 5.13.13+git0+91381833a4-r0, but none of the providers can be installed
> > > > >    *   - conflicting requests
> > > > >    *   - nothing provides kernel-image-image-5.13.13+g91381833a4e2 =
> > > > > 5.13.13+gitAUTOINC+91381833a4-r0 needed by
> > > > 
> > > > This seems to be the problem.
> > > > Is there a "kernel-image-image-5.13.13" built from your kernel recipe?
> > > 
> > > Yes, it is produced. But for some reasons opkg cannot resolve it
> > > during the do_rootfs(), which is quite odd.
> > 
> > I'm seeing the same issue.  All of the BSPs that I set the kernel to
> > not be 5.13 (i.e., 5.10, 5.4, etc) fail.  For example,
> > https://gitlab.com/jonmason00/meta-arm/-/jobs/1544819828
> > If I set the PACKAGE_CLASS to be rpm instead of ipk, everything works
> > as expected.  So, there must be some difference in the dep calculation
> > in ipk.
> 
> Interesting.
> 
> Can you both please try setting KERNEL_SPLIT_MODULES="0" in your kernel recipe?
> In my testing, it works for both ipk and rpm properly.
> 
> I am thinking that probably the full version dependency should
> only be used in the KERNEL_SPLIT_MODULES="0" case.
> This can be easily tested, unlike the packaging method.

Ross noticed it was failing with unexpanded AUTOINC references. That made me
realise that using the unexpanded version of the variable in these cases might
help. I've sent out that patch for review/testing.

Cheers,

Richard

Re: [OE-core] [PATCH v4 3/4] kernel.bbclass: Use full versions for inter-package dependencies

[Andrey Zhizhikin]

Hello Richard,

On Tue, Aug 31, 2021 at 3:32 PM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Tue, 2021-08-31 at 06:28 +0200, Zoltan Boszormenyi via lists.openembedded.org
> wrote:
> > 2021. 08. 30. 21:51 keltezéssel, Jon Mason írta:
> > > On Mon, Aug 30, 2021 at 6:26 AM Andrey Zhizhikin <andrey.z@gmail.com> wrote:
> > > >
> > > > On Mon, Aug 30, 2021 at 12:06 PM Böszörményi Zoltán <zboszor@gmail.com> wrote:
> > > > >
> > > > > 2021. 08. 30. 11:30 keltezéssel, Andrey Zhizhikin írta:
> > > > > > Hello Zoltan,
> > > > > >
> > > > > > On Fri, Aug 27, 2021 at 9:37 AM Zoltan Boszormenyi via
> > > > > > lists.openembedded.org <zboszor=pr.hu@lists.openembedded.org> wrote:
> > > > > > > From: Zoltán Böszörményi <zboszor@gmail.com>
> > > > > > >
> > > > > > > If the kernel configuration enables module signing but no key
> > > > > > > is provided, then the kernel generates one during the kernel build.
> > > > > > >
> > > > > > > The current runtime-dependency references (with only package names
> > > > > > > without full versions) allow mixed package installations from different
> > > > > > > rebuilds of the same kernel version.
> > > > > > >
> > > > > > > This creates an issue because then the modules either don't work
> > > > > > > or taint the kernel.
> > > > > > >
> > > > > > > Tighten RDEPENDS with the full package version, i.e. use (= ${EXTENDPKGV})
> > > > > > > markers for inter-package dependencies.
> > > > > > >
> > > > > > > The kernel will pull in the kernel-modules subpackage of the same
> > > > > > > exact version automatically if KERNEL_SPLIT_MODULES="0" is set.
> > > > > > > Otherwise the situation is the same as with the old default with
> > > > > > > one subpackage per kernel module where they have to be upgraded
> > > > > > > manually.
> > > > > > >
> > > > > > > Signed-off-by: Zoltán Böszörményi <zboszor@gmail.com>
> > > > > > > ---
> > > > > > >    meta/classes/kernel.bbclass | 13 +++++++------
> > > > > > >    1 file changed, 7 insertions(+), 6 deletions(-)
> > > > > > >
> > > > > > I'm seeing errors during the do_rootfs() with this patch applied,
> > > > > > there are few messages like this:
> > > > > >
> > > > > >    * Solver encountered 1 problem(s):
> > > > > >    * Problem 1/1:
> > > > > >    *   - package kernel-module-libchacha-5.13.13+g91381833a4e2-5.13.13+git0+91381833a4-r0.imx8mp_lpddr4_evk
> > > > > > requires kernel-5.13.13+g91381833a4e2, but none of the providers can
> > > > > > be installed
> > > > > >    *   - package kernel-modules-5.13.13+git0+91381833a4-r0.imx8mp_lpddr4_evk
> > > > > > requires kernel-module-libchacha-5.13.13+g91381833a4e2, but none of
> > > > > > the providers can be installed
> > > > > >    *   - package kernel-5.13.13+g91381833a4e2-5.13.13+git0+91381833a4-r0.imx8mp_lpddr4_evk
> > > > > > requires kernel-image-5.13.13+g91381833a4e2 =
> > > > > > 5.13.13+git0+91381833a4-r0, but none of the providers can be installed
> > > > > >    *   - conflicting requests
> > > > > >    *   - nothing provides kernel-image-image-5.13.13+g91381833a4e2 =
> > > > > > 5.13.13+gitAUTOINC+91381833a4-r0 needed by
> > > > >
> > > > > This seems to be the problem.
> > > > > Is there a "kernel-image-image-5.13.13" built from your kernel recipe?
> > > >
> > > > Yes, it is produced. But for some reasons opkg cannot resolve it
> > > > during the do_rootfs(), which is quite odd.
> > >
> > > I'm seeing the same issue.  All of the BSPs that I set the kernel to
> > > not be 5.13 (i.e., 5.10, 5.4, etc) fail.  For example,
> > > https://gitlab.com/jonmason00/meta-arm/-/jobs/1544819828
> > > If I set the PACKAGE_CLASS to be rpm instead of ipk, everything works
> > > as expected.  So, there must be some difference in the dep calculation
> > > in ipk.
> >
> > Interesting.
> >
> > Can you both please try setting KERNEL_SPLIT_MODULES="0" in your kernel recipe?
> > In my testing, it works for both ipk and rpm properly.
> >
> > I am thinking that probably the full version dependency should
> > only be used in the KERNEL_SPLIT_MODULES="0" case.
> > This can be easily tested, unlike the packaging method.
>
> Ross noticed it was failing with unexpanded AUTOINC references. That made me
> realise that using the unexpanded version of the variable in these cases might
> help. I've sent out that patch for review/testing.

This did solve the build issue! Thanks for narrowing it down!

>
> Cheers,
>
> Richard
>


-- 
Regards,
Andrey.