[RFC WIP][PATCH v2] add two zram IMAGE_FEATUREs

[RFC WIP][PATCH v2] add two zram IMAGE_FEATUREs

[Trevor Woerner]

The zram kernel module creates RAM-based block devices named /dev/zram<id>
(<id> = 0, 1, ...). Pages written to these disks are compressed and stored
in memory itself. These disks allow very fast I/O and compression provides
good amounts of memory savings. Some of the use cases include /tmp storage,
use as swap disks, various caches under /var and maybe many more.

If the user wants to switch from using uncompressed tmpfs to compressed
zram-based in-RAM temporary filesystems then add "zram-tmpfs" to
IMAGE_FEATURES. Currently the base install uses tmpfs filesystems for /run and
/var/volatile. By default the /run filesystem will be 10MiB in size and the
/var/volatile (which holds /tmp, among others) will be 30MiB in size. The
following variables can be tweaked to adjust these sizes:
	ZRAM_RUN_SIZE
	ZRAM_TMP_SIZE

If the user wants to enable an in-RAM, compressed, zram-backed SWAP for
their system, simply add "zram-swap" to IMAGE_FEATURES. The size of the swap
will be based on a percentage of the total system RAM which the user can
adjust by modifying the following variable:
	ZRAM_SWAP_PERCENT

By default ZRAM_SWAP_PERCENT is set to "100" which means the initial SWAP size
will be the same size as the total system RAM.

The kernel provides several compression algorithms from which the user can
choose. By default the "lzo-rle" compression is used, but the user can modify
the following variable to choose from any one of the other available
compression algorithms:
	ZRAM_ALGORITHM

Currently the set of available compression algorithms include:
	lzo
	lzo-rle
	lz4
	lz4hc
	842
	zstd

!!!NOTE!!!
this is a WIP patch submitted for RFC
this patch only works, currently, with sysvinit
but I wanted to get feedback on the general approach before tackling systemd

Signed-off-by: Trevor Woerner <twoerner@gmail.com>
---
NOTE:
- this patch relies on there being a zram.scc available in the
  yocto-kernel-cache repository, a patch for which was submitted moments ago
  https://lists.yoctoproject.org/g/linux-yocto/message/10293

changes in v2
- move the check of the algorithm outside the check for zram-tmpfs since
  the algorithm applies to both zram-tmpfs and zram-swap
- add checks on the /var/volatile and /run sizes to make sure they're
  greater than 0
---
 meta/classes/core-image.bbclass               |  4 ++
 meta/classes/image.bbclass                    |  2 +-
 meta/classes/rootfs-postcommands.bbclass      | 35 +++++++++++++++
 .../initscripts/initscripts-1.0/mountall.sh   | 27 +++++++++++-
 .../packagegroup-core-zram-swap.bb            | 10 +++++
 .../packagegroup-core-zram-tmpfs.bb           |  9 ++++
 .../sysvinit/sysvinit/rcS-default             |  8 ++++
 meta/recipes-core/sysvinit/sysvinit_2.99.bb   | 44 ++++++++++++++++++-
 meta/recipes-kernel/linux/linux-yocto.inc     |  2 +
 9 files changed, 137 insertions(+), 4 deletions(-)
 create mode 100644 meta/recipes-core/packagegroups/packagegroup-core-zram-swap.bb
 create mode 100644 meta/recipes-core/packagegroups/packagegroup-core-zram-tmpfs.bb

diff --git a/meta/classes/core-image.bbclass b/meta/classes/core-image.bbclass
index 84fd3eeb38..ed6184d11c 100644
--- a/meta/classes/core-image.bbclass
+++ b/meta/classes/core-image.bbclass
@@ -39,6 +39,8 @@
 # - read-only-rootfs    - tweaks an image to support read-only rootfs
 # - stateless-rootfs    - systemctl-native not run, image populated by systemd at runtime
 # - splash              - bootup splash screen
+# - zram-tmpfs          - use a zram disk for temporary filesystems instead of tmpfs
+# - zram-swap           - use a zram-backed swap
 #
 FEATURE_PACKAGES_weston = "packagegroup-core-weston"
 FEATURE_PACKAGES_x11 = "packagegroup-core-x11"
@@ -53,6 +55,8 @@ FEATURE_PACKAGES_nfs-server = "packagegroup-core-nfs-server"
 FEATURE_PACKAGES_nfs-client = "packagegroup-core-nfs-client"
 FEATURE_PACKAGES_ssh-server-dropbear = "packagegroup-core-ssh-dropbear"
 FEATURE_PACKAGES_ssh-server-openssh = "packagegroup-core-ssh-openssh"
+FEATURE_PACKAGES_zram-tmpfs = "packagegroup-core-zram-tmpfs"
+FEATURE_PACKAGES_zram-swap = "packagegroup-core-zram-swap"
 FEATURE_PACKAGES_hwcodecs = "${MACHINE_HWCODECS}"
 
 
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index d76895178f..57178baace 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -33,7 +33,7 @@ INHIBIT_DEFAULT_DEPS = "1"
 # IMAGE_FEATURES may contain any available package group
 IMAGE_FEATURES ?= ""
 IMAGE_FEATURES[type] = "list"
-IMAGE_FEATURES[validitems] += "debug-tweaks read-only-rootfs read-only-rootfs-delayed-postinsts stateless-rootfs empty-root-password allow-empty-password allow-root-login post-install-logging"
+IMAGE_FEATURES[validitems] += "debug-tweaks read-only-rootfs read-only-rootfs-delayed-postinsts stateless-rootfs empty-root-password allow-empty-password allow-root-login post-install-logging zram-tmpfs zram-swap"
 
 # Generate companion debugfs?
 IMAGE_GEN_DEBUGFS ?= "0"
diff --git a/meta/classes/rootfs-postcommands.bbclass b/meta/classes/rootfs-postcommands.bbclass
index c5746eba13..a1e00aeebc 100644
--- a/meta/classes/rootfs-postcommands.bbclass
+++ b/meta/classes/rootfs-postcommands.bbclass
@@ -17,6 +17,12 @@ ROOTFS_POSTPROCESS_COMMAND += "rootfs_update_timestamp; "
 # Tweak the mount options for rootfs in /etc/fstab if read-only-rootfs is enabled
 ROOTFS_POSTPROCESS_COMMAND += '${@bb.utils.contains("IMAGE_FEATURES", "read-only-rootfs", "read_only_rootfs_hook; ", "",d)}'
 
+# Use zram for temporary filesystems instead of tmpfs
+ROOTFS_POSTPROCESS_COMMAND += '${@bb.utils.contains("IMAGE_FEATURES", "zram-tmpfs", "zram_tmpfs_hook; ", "",d)}'
+
+# Use a zram-backed swap
+ROOTFS_POSTPROCESS_COMMAND += '${@bb.utils.contains("IMAGE_FEATURES", "zram-swap", "zram_swap_hook; ", "",d)}'
+
 # We also need to do the same for the kernel boot parameters,
 # otherwise kernel or initramfs end up mounting the rootfs read/write
 # (the default) if supported by the underlying storage.
@@ -143,6 +149,35 @@ read_only_rootfs_hook () {
 	fi
 }
 
+#
+# A hook function to support zram-tmpfs IMAGE_FEATURES
+#
+zram_tmpfs_hook () {
+	# Remove tmpfs entries from fstab
+	if [ -f ${IMAGE_ROOTFS}/etc/fstab ]; then
+		sed -n -i -e '/tmpfs/!p' ${IMAGE_ROOTFS}/etc/fstab
+	fi
+
+	# Enable zram when sysvinit is used
+	if ${@bb.utils.contains("DISTRO_FEATURES", "sysvinit", "true", "false", d)}; then
+		if [ -e ${IMAGE_ROOTFS}/etc/default/rcS ]; then
+			sed -i 's/USE_ZRAM_TMPFS=no/USE_ZRAM_TMPFS=yes/' ${IMAGE_ROOTFS}/etc/default/rcS
+		fi
+	fi
+}
+
+#
+# A hook function to support using a zram-backed swap
+#
+zram_swap_hook () {
+	# Enable zram when sysvinit is used
+	if ${@bb.utils.contains("DISTRO_FEATURES", "sysvinit", "true", "false", d)}; then
+		if [ -e ${IMAGE_ROOTFS}/etc/default/rcS ]; then
+			sed -i 's/USE_ZRAM_SWAP=no/USE_ZRAM_SWAP=yes/' ${IMAGE_ROOTFS}/etc/default/rcS
+		fi
+	fi
+}
+
 #
 # This function is intended to disallow empty root password if 'debug-tweaks' is not in IMAGE_FEATURES.
 #
diff --git a/meta/recipes-core/initscripts/initscripts-1.0/mountall.sh b/meta/recipes-core/initscripts/initscripts-1.0/mountall.sh
index 2839d57cbe..996709bce6 100755
--- a/meta/recipes-core/initscripts/initscripts-1.0/mountall.sh
+++ b/meta/recipes-core/initscripts/initscripts-1.0/mountall.sh
@@ -35,11 +35,36 @@ if [ ! -p "$INITCTL" ] && [ "${INIT_SYSTEM}" = "sysvinit" ]; then
 		[ -n "$PID" ] && kill -s USR1 "$PID"
 fi
 
+# If the user wants to use zram tmpfs
+# create and mount those now
+if [ "$USE_ZRAM_TMPFS"z = "yes"z ]; then
+	ZDEVICE=$(zramctl -f -s ${ZRAM_TMP_SIZE}MiB -a ${ZRAM_ALGORITHM})
+	if [ $? -eq 0 ]; then
+		mkfs.ext2 $ZDEVICE
+		mount $ZDEVICE /var/volatile
+	fi
+	ZDEVICE=$(zramctl -f -s ${ZRAM_RUN_SIZE}MiB -a ${ZRAM_ALGORITHM})
+	if [ $? -eq 0 ]; then
+		mkfs.ext2 $ZDEVICE
+		mount $ZDEVICE /run
+	fi
+fi
+
 #
 # Execute swapon command again, in case we want to swap to
 # a file on a now mounted filesystem.
 #
-[ -x /sbin/swapon ] && swapon -a
+if [ "$USE_ZRAM_SWAP"z = "yes"z ]; then
+	MEMTOTAL=$(grep MemTotal /proc/meminfo | awk ' { print $2 } ')
+	MEMZRAM=$((${MEMTOTAL}*${ZRAM_SWAP_PERCENT}/100))
+	ZDEVICE=$(zramctl -f -s ${MEMZRAM}KiB -a ${ZRAM_ALGORITHM})
+	if [ $? -eq 0 ]; then
+		mkswap -L "zram-swap" $ZDEVICE
+		swapon -p 100 $ZDEVICE
+	fi
+else
+	[ -x /sbin/swapon ] && swapon -a
+fi
 
 : exit 0
 
diff --git a/meta/recipes-core/packagegroups/packagegroup-core-zram-swap.bb b/meta/recipes-core/packagegroups/packagegroup-core-zram-swap.bb
new file mode 100644
index 0000000000..fa6da61b47
--- /dev/null
+++ b/meta/recipes-core/packagegroups/packagegroup-core-zram-swap.bb
@@ -0,0 +1,10 @@
+SUMMARY = "Use zram-backed swap"
+PR = "r1"
+
+inherit packagegroup
+
+RDEPENDS:${PN} = " \
+	util-linux-zramctl \
+	util-linux-swaponoff \
+	util-linux-mkswap \
+	"
diff --git a/meta/recipes-core/packagegroups/packagegroup-core-zram-tmpfs.bb b/meta/recipes-core/packagegroups/packagegroup-core-zram-tmpfs.bb
new file mode 100644
index 0000000000..84b6642340
--- /dev/null
+++ b/meta/recipes-core/packagegroups/packagegroup-core-zram-tmpfs.bb
@@ -0,0 +1,9 @@
+SUMMARY = "Use zram for temporary directories instead of tmpfs"
+PR = "r1"
+
+inherit packagegroup
+
+RDEPENDS:${PN} = " \
+	util-linux-zramctl \
+	e2fsprogs-mke2fs \
+	"
diff --git a/meta/recipes-core/sysvinit/sysvinit/rcS-default b/meta/recipes-core/sysvinit/sysvinit/rcS-default
index f7c4a2f841..91739183d9 100644
--- a/meta/recipes-core/sysvinit/sysvinit/rcS-default
+++ b/meta/recipes-core/sysvinit/sysvinit/rcS-default
@@ -34,3 +34,11 @@ INIT_SYSTEM=sysvinit
 PSPLASH_FIFO_DIR=/mnt
 # psplash textual updates knob
 PSPLASH_TEXT_UPDATES=#PSPLASH_TEXT#
+# zram
+# zram can be used for swap or tmp filesystems (or both)
+ZRAM_ALGORITHM=#ZRAM_ALGORITHM#
+USE_ZRAM_SWAP=#USE_ZRAM_SWAP#
+ZRAM_SWAP_PERCENT=#ZRAM_SWAP_PERCENT#
+USE_ZRAM_TMPFS=#USE_ZRAM_TMPFS#
+ZRAM_TMP_SIZE=#ZRAM_TMP_SIZE#
+ZRAM_RUN_SIZE=#ZRAM_RUN_SIZE#
diff --git a/meta/recipes-core/sysvinit/sysvinit_2.99.bb b/meta/recipes-core/sysvinit/sysvinit_2.99.bb
index 9ba9652f94..db66c97eb1 100644
--- a/meta/recipes-core/sysvinit/sysvinit_2.99.bb
+++ b/meta/recipes-core/sysvinit/sysvinit_2.99.bb
@@ -83,6 +83,40 @@ EXTRA_OEMAKE += "'base_bindir=${base_bindir}' \
 		 'mandir=${mandir}' \
                  MNTPOINT=yes"
 
+# zram defaults (if enabled in IMAGE_FEATURES)
+# choose one of: lzo lzo-rle lz4 lz4hc 842 zstd (depending on your kernel config)
+ZRAM_ALGORITHM ?= "lzo-rle"
+# swap size as a percentage of total RAM
+ZRAM_SWAP_PERCENT ?= "100"
+# size of /tmp in MiB
+ZRAM_TMP_SIZE ?= "30"
+# size of /run in MiB
+ZRAM_RUN_SIZE ?= "10"
+python __anonymous() {
+    imagefeatures = (d.getVar("IMAGE_FEATURES") or "")
+    valid_algorithms = ['lzo', 'lzo-rle', 'lz4', 'lz4hc', '842', 'zstd']
+    algorithm = (d.getVar("ZRAM_ALGORITHM") or "")
+    if algorithm not in valid_algorithms:
+        bb.error("ZRAM_ALGORITHM must be one of: %s" % valid_algorithms)
+
+    d.setVar("USE_ZRAM_SWAP", "no")
+    if 'zram-swap' in imagefeatures:
+        swappercent = (d.getVar("ZRAM_SWAP_PERCENT") or "")
+        if int(swappercent) <= 0:
+            bb.error("ZRAM_SWAP_PERCENT must be greater than 0")
+        d.setVar("USE_ZRAM_SWAP", "yes")
+
+    d.setVar("USE_ZRAM_TMPFS", "no")
+    if 'zram-tmpfs' in imagefeatures:
+        d.setVar("USE_ZRAM_TMPFS", "yes")
+        tmpsize = (d.getVar("ZRAM_TMP_SIZE") or "")
+        if int(tmpsize) <= 0:
+            bb.error("ZRAM_TMP_SIZE must be greater than 0")
+        tmpsize = (d.getVar("ZRAM_RUN_SIZE") or "")
+        if int(tmpsize) <= 0:
+            bb.error("ZRAM_RUN_SIZE must be greater than 0")
+}
+
 do_install () {
 	oe_runmake 'ROOT=${D}' install
 
@@ -93,8 +127,14 @@ do_install () {
 		install -d ${D}${sysconfdir}/rc$level.d
 	done
 
-	sed -e \
-		's:#PSPLASH_TEXT#:${@bb.utils.contains("PACKAGECONFIG","psplash-text-updates","yes","no", d)}:g' \
+	sed \
+		-e 's:#PSPLASH_TEXT#:${@bb.utils.contains("PACKAGECONFIG","psplash-text-updates","yes","no", d)}:g' \
+		-e 's:#ZRAM_ALGORITHM#:${ZRAM_ALGORITHM}:g' \
+		-e 's:#USE_ZRAM_SWAP#:${USE_ZRAM_SWAP}:g' \
+		-e 's:#ZRAM_SWAP_PERCENT#:${ZRAM_SWAP_PERCENT}:g' \
+		-e 's:#USE_ZRAM_TMPFS#:${USE_ZRAM_TMPFS}:g' \
+		-e 's:#ZRAM_TMP_SIZE#:${ZRAM_TMP_SIZE}:g' \
+		-e 's:#ZRAM_RUN_SIZE#:${ZRAM_RUN_SIZE}:g' \
 		${WORKDIR}/rcS-default > ${D}${sysconfdir}/default/rcS
 	chmod 0644 ${D}${sysconfdir}/default/rcS
 	install -m 0755    ${WORKDIR}/rc		${D}${sysconfdir}/init.d
diff --git a/meta/recipes-kernel/linux/linux-yocto.inc b/meta/recipes-kernel/linux/linux-yocto.inc
index 331727d62c..66aa8df4a3 100644
--- a/meta/recipes-kernel/linux/linux-yocto.inc
+++ b/meta/recipes-kernel/linux/linux-yocto.inc
@@ -35,6 +35,8 @@ KERNEL_FEATURES:append:qemuall=" features/debug/printk.scc"
 
 KERNEL_FEATURES:append = " ${@bb.utils.contains('MACHINE_FEATURES', 'numa', 'features/numa/numa.scc', '', d)}"
 KERNEL_FEATURES:append = " ${@bb.utils.contains('MACHINE_FEATURES', 'vfat', 'cfg/fs/vfat.scc', '', d)}"
+KERNEL_FEATURES:append = " ${@bb.utils.contains('IMAGE_FEATURES', 'zram-tmpfs', 'features/zram/zram.scc', '', d)}"
+KERNEL_FEATURES:append = " ${@bb.utils.contains('IMAGE_FEATURES', 'zram-swap', 'features/zram/zram.scc', '', d)}"
 
 # A KMACHINE is the mapping of a yocto $MACHINE to what is built
 # by the kernel. This is typically the branch that should be built,
-- 
2.30.0.rc0

Re: [OE-core] [RFC WIP][PATCH v2] add two zram IMAGE_FEATUREs

[Sinan Kaya]

On 8/18/2021 11:36 PM, Trevor Woerner wrote:
> this is a WIP patch submitted for RFC
> this patch only works, currently, with sysvinit
> but I wanted to get feedback on the general approach before tackling systemd

How about using/extending the zram package in meta-openembedded?
Would that be an option?

Re: [OE-core] [RFC WIP][PATCH v2] add two zram IMAGE_FEATUREs

[Trevor Woerner]

On Wed 2021-08-18 @ 11:37:36 PM, Sinan Kaya wrote:
> On 8/18/2021 11:36 PM, Trevor Woerner wrote:
> > this is a WIP patch submitted for RFC
> > this patch only works, currently, with sysvinit
> > but I wanted to get feedback on the general approach before tackling systemd
> 
> How about using/extending the zram package in meta-openembedded?
> Would that be an option?

When I found the zram package in meta-openembedded I had thought that my work
was done. I figured I'd add it to my image and move on. Unfortunately the zram
package in meta-openembedded is woefully inadequate :-(

In fact, if my patch gets accepted into oe-core, my next patch would be to
remove this package from meta-openembedded ;-)

So far I have only considered the case where the user wants to use zram with
sysvinit. I haven't looked at the systemd case yet, but if it has as many
holes as the sysvinit case…

If you look at the package in meta-openembedded you'll see that it:
- tries to do its work using start/stop initscripts
- doesn't even try to handle the case where the user wants to use zram instead
  of tmpfs, and that's not a coincidence

A running system will happily incorporate added SWAP anytime that SWAP is
added. Simply prepare the partition for SWAP as appropriate then "swapon" and
everything is happy. Substituting zram for tmpfs, however is a lot more
finicky. You can't simply wait until after tmpfs has been used for your
various tmp partitions then mount new partitions on top of the existing
/var/volatile and /run. This will cause your logger to get upset, and it will
cause pretty much every currently-running program to not be very happy.
Programs that are started and put their pids in /run and various things in
/tmp don't usually react favourably when those filesystems have something else
mounted on top of them during their execution.

Therefore in order to properly switch from tmpfs to zram for tmp filesystems:
- /etc/fstab has to be modified to not try to mount tmpfs things to
  /var/volatile and /run
- the place where "mount -a" is called has to be modified to setup, format,
  and mount zram-based disks. specifying zram disks is not something you can
  simply tweak /etc/fstab to do. according to the kernel documentation there
  is no
  	zram  /var/volatile zram,ext2 defaults 0 2
  line that can be added to /etc/fstab to get zram-based filesystems to mount
  automatically via "mount -a"

Because this change is much more invasive than simply running a script at a
given runlevel I came to the conclusion that an IMAGE_FEATURE would be a
better fit for this sort of feature.

I also took the opportunity to parameterize various parts of the configuration
to make it easy for the user to either accept the sane defaults, or easily
tweak them to their liking. I added checks to these parameters so that they're
valid/sane. I added the feature in a way that any required packages are added
to the image automatically without the user having to do anything. I also tied
into the kernel's "kmeta config" system so that simply enabling this
IMAGE_FEATURE will also tweak the kernel config to add the necessary options
(again, without user intervention).

Also, if you look at the existing sysvinit script in meta-openembedded
you'll notice that it jumps through some hoops trying to figure out how many
cores/cpus are on the system, then it uses that number as a basis for how many
SWAPs to create. You can tell this is a very old script because, according
to the kernel documentation, this is something that was required a long time
ago, but has long since is no longer required. Simply create one SWAP or one
zram-backed /tmp mount and the number of "streams" is automatically handled
internally in the kernel:

	root@rock-pi-e:~# zramctl
	NAME       ALGORITHM DISKSIZE  DATA COMPR TOTAL STREAMS MOUNTPOINT
	/dev/zram2 lz4         975.6M    4K   73B   12K       4 [SWAP]
	/dev/zram1 lz4            10M  400K 16.1K  240K       4 /run
	/dev/zram0 lz4            30M   68K  1.6K   68K       4 /var/volatile

So, in conclusion, the stuff on meta-openembmedded's zram package is:
- old
- has very few knobs for the user to tweak
- doesn't handle the zram-tmpfs case
- doesn't update your kernel config automatically

(at least for the sysvinit portion)

Re: [OE-core] [RFC WIP][PATCH v2] add two zram IMAGE_FEATUREs

[Sinan Kaya]

On 8/19/2021 12:04 AM, Trevor Woerner wrote:
> root@rock-pi-e:~# zramctl
> 	NAME       ALGORITHM DISKSIZE  DATA COMPR TOTAL STREAMS MOUNTPOINT
> 	/dev/zram2 lz4         975.6M    4K   73B   12K       4 [SWAP]
> 	/dev/zram1 lz4            10M  400K 16.1K  240K       4 /run
> 	/dev/zram0 lz4            30M   68K  1.6K   68K       4 /var/volatile
> 
> So, in conclusion, the stuff on meta-openembmedded's zram package is:
> - old
> - has very few knobs for the user to tweak
> - doesn't handle the zram-tmpfs case
> - doesn't update your kernel config automatically

I see your point. zram package is focused on swap partitions only.
You seem to go beyond swap partitions and tap into tmpfs next.