[PATCH v3 0/8] Pull request (cover letter only)

[PATCH v3 0/8] Pull request (cover letter only)

[Vyacheslav Yurkov]

This is a V3 of overlayfs-etc image feature implementation, that allows
to setup the whole /etc under overlayfs. Please review and merge if you
find it OK

The following changes since commit 1a6c2a7345199d77ad5aeac8ad337ed80a8aa39b:

  build-appliance-image: Update to master head revision (2021-12-09 13:56:07 +0000)

are available in the Git repository at:

  git://github.com/UVV-gh/openembedded-core feature/overlayfs-etc
  https://github.com/UVV-gh/openembedded-core/tree/feature/overlayfs-etc

Vyacheslav Yurkov (8):
  files: add overlayfs-etc-preinit.sh.in
  overlayfs-etc: mount etc as overlayfs
  wic: image for overlayfs-etc tests
  image: add overlayfs-etc image feature
  oeqa/selftest: overlayfs helper function
  oeqa/selftest: unit tests for overlayfs-etc
  overlayfs: update notes on /etc
  overlayfs: move templates to files directory

 meta-selftest/wic/overlayfs_etc.wks.in       |   4 +
 meta/classes/image.bbclass                   |   3 +-
 meta/classes/overlayfs-etc.bbclass           |  76 +++++++
 meta/classes/overlayfs.bbclass               |  53 +----
 meta/files/overlayfs-all-overlays.service.in |  12 ++
 meta/files/overlayfs-create-dirs.service.in  |  14 ++
 meta/files/overlayfs-etc-preinit.sh.in       |  29 +++
 meta/files/overlayfs-unit.mount.in           |  13 ++
 meta/lib/oeqa/selftest/cases/overlayfs.py    | 216 +++++++++++++++++--
 9 files changed, 361 insertions(+), 59 deletions(-)
 create mode 100644 meta-selftest/wic/overlayfs_etc.wks.in
 create mode 100644 meta/classes/overlayfs-etc.bbclass
 create mode 100644 meta/files/overlayfs-all-overlays.service.in
 create mode 100644 meta/files/overlayfs-create-dirs.service.in
 create mode 100644 meta/files/overlayfs-etc-preinit.sh.in
 create mode 100644 meta/files/overlayfs-unit.mount.in

-- 
2.28.0

[PATCH 1/8] files: add overlayfs-etc-preinit.sh.in

[Vyacheslav Yurkov]

A template init script for overlayfs-etc class

Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com>
---
 meta/files/overlayfs-etc-preinit.sh.in | 29 ++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)
 create mode 100644 meta/files/overlayfs-etc-preinit.sh.in

diff --git a/meta/files/overlayfs-etc-preinit.sh.in b/meta/files/overlayfs-etc-preinit.sh.in
new file mode 100644
index 0000000000..2ebb6c9224
--- /dev/null
+++ b/meta/files/overlayfs-etc-preinit.sh.in
@@ -0,0 +1,29 @@
+#!/bin/sh
+
+echo "PREINIT: Start"
+
+PATH=/sbin:/bin:/usr/sbin:/usr/bin
+mount -o remount,rw /
+
+mkdir -p /proc
+mkdir -p /sys
+mkdir -p /run
+mkdir -p /var/run
+
+mount -t proc proc /proc
+mount -t sysfs sysfs /sys
+
+[ -z "$CONSOLE" ] && CONSOLE="/dev/console"
+
+mkdir -p {OVERLAYFS_ETC_MOUNT_POINT}
+if mount -n -t {OVERLAYFS_ETC_FSTYPE} -o {OVERLAYFS_ETC_MOUNT_OPTIONS} {OVERLAYFS_ETC_DEVICE} {OVERLAYFS_ETC_MOUNT_POINT}
+then
+    mkdir -p {OVERLAYFS_ETC_MOUNT_POINT}/overlay-etc/upper
+    mkdir -p {OVERLAYFS_ETC_MOUNT_POINT}/overlay-etc/work
+    mount -n -t overlay -o upperdir={OVERLAYFS_ETC_MOUNT_POINT}/overlay-etc/upper,lowerdir=/etc,workdir={OVERLAYFS_ETC_MOUNT_POINT}/overlay-etc/work {OVERLAYFS_ETC_MOUNT_POINT}/overlay-etc/upper /etc || echo "PREINIT: Mounting etc-overlay failed!"
+else
+    echo "PREINIT: Mounting </data> failed!"
+fi
+
+echo "PREINIT: done; starting </sbin/init>"
+exec {SBIN_INIT_NAME}
-- 
2.28.0

[PATCH 2/8] overlayfs-etc: mount etc as overlayfs

[Vyacheslav Yurkov]

This class provides an image feature that mounts /etc as an overlayfs
file system. This is an extension for existing overlayfs class, which
doesn't support /etc

Signed-off-by: Alfred Schapansky <alfred.schapansky@avantys.de>
Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com>
---
 meta/classes/overlayfs-etc.bbclass | 76 ++++++++++++++++++++++++++++++
 1 file changed, 76 insertions(+)
 create mode 100644 meta/classes/overlayfs-etc.bbclass

diff --git a/meta/classes/overlayfs-etc.bbclass b/meta/classes/overlayfs-etc.bbclass
new file mode 100644
index 0000000000..4ced07ba11
--- /dev/null
+++ b/meta/classes/overlayfs-etc.bbclass
@@ -0,0 +1,76 @@
+# Class for setting up /etc in overlayfs
+#
+# In order to have /etc directory in overlayfs a special handling at early boot stage is required
+# The idea is to supply a custom init script that mounts /etc before launching actual init program,
+# because the latter already requires /etc to be mounted
+#
+# The configuration must be machine specific. You should at least set these three variables:
+#   OVERLAYFS_ETC_MOUNT_POINT ?= "/data"
+#   OVERLAYFS_ETC_FSTYPE ?= "ext4"
+#   OVERLAYFS_ETC_DEVICE ?= "/dev/mmcblk0p2"
+#
+# To control more mount options you should consider setting mount options:
+#   OVERLAYFS_ETC_MOUNT_OPTIONS ?= "defaults"
+#
+# The class provides two options for /sbin/init generation
+# 1. Default option is to rename original /sbin/init to /sbin/init.orig and place generated init under
+#    original name, i.e. /sbin/init. It has an advantage that you won't need to change any kernel
+#    parameters in order to make it work, but it poses a restriction that package-management can't
+#    be used, becaause updating init manager would remove generated script
+# 2. If you are would like to keep original init as is, you can set
+#    OVERLAYFS_ETC_USE_ORIG_INIT_NAME = "0"
+#    Then generated init will be named /sbin/preinit and you would need to extend you kernel parameters
+#    manually in your bootloader configuration.
+#
+# Regardless which mode you choose, update and migration strategy of configuration files under /etc
+# overlay is out of scope of this class
+
+ROOTFS_POSTPROCESS_COMMAND += '${@bb.utils.contains("IMAGE_FEATURES", "overlayfs-etc", "create_overlayfs_etc_preinit;", "", d)}'
+IMAGE_FEATURES_CONFLICTS_overlayfs-etc = "package-management"
+
+OVERLAYFS_ETC_MOUNT_POINT ??= ""
+OVERLAYFS_ETC_FSTYPE ??= ""
+OVERLAYFS_ETC_DEVICE ??= ""
+OVERLAYFS_ETC_USE_ORIG_INIT_NAME ??= "1"
+OVERLAYFS_ETC_MOUNT_OPTIONS ??= "defaults"
+OVERLAYFS_ETC_INIT_TEMPLATE ??= "${COREBASE}/meta/files/overlayfs-etc-preinit.sh.in"
+
+python create_overlayfs_etc_preinit() {
+    overlayEtcMountPoint = d.getVar("OVERLAYFS_ETC_MOUNT_POINT")
+    overlayEtcFsType = d.getVar("OVERLAYFS_ETC_FSTYPE")
+    overlayEtcDevice = d.getVar("OVERLAYFS_ETC_DEVICE")
+
+    if not overlayEtcMountPoint:
+        bb.fatal("OVERLAYFS_ETC_MOUNT_POINT must be set in your MACHINE configuration")
+    if not overlayEtcDevice:
+        bb.fatal("OVERLAYFS_ETC_DEVICE must be set in your MACHINE configuration")
+    if not overlayEtcFsType:
+        bb.fatal("OVERLAYFS_ETC_FSTYPE should contain a valid file system type on {0}".format(overlayEtcDevice))
+
+    with open(d.getVar("OVERLAYFS_ETC_INIT_TEMPLATE"), "r") as f:
+        PreinitTemplate = f.read()
+
+    useOrigInit = oe.types.boolean(d.getVar('OVERLAYFS_ETC_USE_ORIG_INIT_NAME'))
+    preinitPath = oe.path.join(d.getVar("IMAGE_ROOTFS"), d.getVar("base_sbindir"), "preinit")
+    initBaseName = oe.path.join(d.getVar("base_sbindir"), "init")
+    origInitNameSuffix = ".orig"
+
+    args = {
+        'OVERLAYFS_ETC_MOUNT_POINT': overlayEtcMountPoint,
+        'OVERLAYFS_ETC_MOUNT_OPTIONS': d.getVar('OVERLAYFS_ETC_MOUNT_OPTIONS'),
+        'OVERLAYFS_ETC_FSTYPE': overlayEtcFsType,
+        'OVERLAYFS_ETC_DEVICE': overlayEtcDevice,
+        'SBIN_INIT_NAME': initBaseName + origInitNameSuffix if useOrigInit else initBaseName
+    }
+
+    if useOrigInit:
+        # rename original /sbin/init
+        origInit = oe.path.join(d.getVar("IMAGE_ROOTFS"), initBaseName)
+        bb.debug(1, "rootfs path %s, init path %s, test %s" % (d.getVar('IMAGE_ROOTFS'), origInit, d.getVar("IMAGE_ROOTFS")))
+        bb.utils.rename(origInit, origInit + origInitNameSuffix)
+        preinitPath = origInit
+
+    with open(preinitPath, 'w') as f:
+        f.write(PreinitTemplate.format(**args))
+    os.chmod(preinitPath, 0o755)
+}
-- 
2.28.0

[PATCH 3/8] wic: image for overlayfs-etc tests

[Vyacheslav Yurkov]

Introduce wic image for overlayfs-etc tests with a dedicated /data
partition and configurable kernel parameters

Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com>
---
 meta-selftest/wic/overlayfs_etc.wks.in | 4 ++++
 1 file changed, 4 insertions(+)
 create mode 100644 meta-selftest/wic/overlayfs_etc.wks.in

diff --git a/meta-selftest/wic/overlayfs_etc.wks.in b/meta-selftest/wic/overlayfs_etc.wks.in
new file mode 100644
index 0000000000..1e1e5836e7
--- /dev/null
+++ b/meta-selftest/wic/overlayfs_etc.wks.in
@@ -0,0 +1,4 @@
+part /boot --active --source bootimg-biosplusefi --ondisk sda --sourceparams="loader=grub-efi" --align 1024
+part / --source rootfs --ondisk sda --fstype=ext4 --use-uuid --align 1024
+part --ondisk sda --fstype=ext4 --size=5 --align 1024
+bootloader --ptable gpt --timeout=1 --append="rootfstype=ext4 console=ttyS0,115200 console=tty0 ${OVERLAYFS_INIT_OPTION}"
-- 
2.28.0

[PATCH 4/8] image: add overlayfs-etc image feature

[Vyacheslav Yurkov]

Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com>
---
 meta/classes/image.bbclass | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 8a46b4852c..2b0ce4a988 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -15,6 +15,7 @@ IMGCLASSES += "${@bb.utils.contains('IMAGE_FSTYPES', 'container', 'image-contain
 IMGCLASSES += "image_types_wic"
 IMGCLASSES += "rootfs-postcommands"
 IMGCLASSES += "image-postinst-intercepts"
+IMGCLASSES += "overlayfs-etc"
 inherit ${IMGCLASSES}
 
 TOOLCHAIN_TARGET_TASK += "${PACKAGE_INSTALL}"
@@ -33,7 +34,7 @@ INHIBIT_DEFAULT_DEPS = "1"
 # IMAGE_FEATURES may contain any available package group
 IMAGE_FEATURES ?= ""
 IMAGE_FEATURES[type] = "list"
-IMAGE_FEATURES[validitems] += "debug-tweaks read-only-rootfs read-only-rootfs-delayed-postinsts stateless-rootfs empty-root-password allow-empty-password allow-root-login post-install-logging"
+IMAGE_FEATURES[validitems] += "debug-tweaks read-only-rootfs read-only-rootfs-delayed-postinsts stateless-rootfs empty-root-password allow-empty-password allow-root-login post-install-logging overlayfs-etc"
 
 # Generate companion debugfs?
 IMAGE_GEN_DEBUGFS ?= "0"
-- 
2.28.0

[PATCH 5/8] oeqa/selftest: overlayfs helper function

[Vyacheslav Yurkov]

Move helper functions out of class scope so they can be used in other tests

Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com>
---
 meta/lib/oeqa/selftest/cases/overlayfs.py | 32 +++++++++++------------
 1 file changed, 16 insertions(+), 16 deletions(-)

diff --git a/meta/lib/oeqa/selftest/cases/overlayfs.py b/meta/lib/oeqa/selftest/cases/overlayfs.py
index 84242a1605..43415778ce 100644
--- a/meta/lib/oeqa/selftest/cases/overlayfs.py
+++ b/meta/lib/oeqa/selftest/cases/overlayfs.py
@@ -5,16 +5,16 @@
 from oeqa.selftest.case import OESelftestTestCase
 from oeqa.utils.commands import runCmd, bitbake, get_bb_var, runqemu
 
-class OverlayFSTests(OESelftestTestCase):
-    """Overlayfs class usage tests"""
+def getline_qemu(out, line):
+    for l in out.split('\n'):
+        if line in l:
+            return l
 
-    def getline_qemu(self, out, line):
-        for l in out.split('\n'):
-            if line in l:
-                return l
+def getline(res, line):
+    return getline_qemu(res.output, line)
 
-    def getline(self, res, line):
-        return self.getline_qemu(res.output, line)
+class OverlayFSTests(OESelftestTestCase):
+    """Overlayfs class usage tests"""
 
     def add_overlay_conf_to_machine(self):
         machine_inc = """
@@ -40,7 +40,7 @@ inherit overlayfs
         self.write_recipeinc('overlayfs-user', overlayfs_recipe_append)
 
         res = bitbake('core-image-minimal', ignore_status=True)
-        line = self.getline(res, "overlayfs-user was skipped: missing required distro features")
+        line = getline(res, "overlayfs-user was skipped: missing required distro features")
         self.assertTrue("overlayfs" in res.output, msg=res.output)
         self.assertTrue("systemd" in res.output, msg=res.output)
         self.assertTrue("ERROR: Required build target 'core-image-minimal' has no buildable providers." in res.output, msg=res.output)
@@ -61,9 +61,9 @@ DISTRO_FEATURES += "systemd overlayfs"
         self.add_overlay_conf_to_machine()
 
         res = bitbake('core-image-minimal', ignore_status=True)
-        line = self.getline(res, "Unit name mnt-overlay.mount not found in systemd unit directories")
+        line = getline(res, "Unit name mnt-overlay.mount not found in systemd unit directories")
         self.assertTrue(line and line.startswith("WARNING:"), msg=res.output)
-        line = self.getline(res, "Not all mount units are installed by the BSP")
+        line = getline(res, "Not all mount units are installed by the BSP")
         self.assertTrue(line and line.startswith("ERROR:"), msg=res.output)
 
     def test_mount_unit_not_set(self):
@@ -81,7 +81,7 @@ DISTRO_FEATURES += "systemd overlayfs"
         self.write_config(config)
 
         res = bitbake('core-image-minimal', ignore_status=True)
-        line = self.getline(res, "A recipe uses overlayfs class but there is no OVERLAYFS_MOUNT_POINT set in your MACHINE configuration")
+        line = getline(res, "A recipe uses overlayfs class but there is no OVERLAYFS_MOUNT_POINT set in your MACHINE configuration")
         self.assertTrue(line and line.startswith("Parsing recipes...ERROR:"), msg=res.output)
 
     def test_wrong_mount_unit_set(self):
@@ -104,7 +104,7 @@ OVERLAYFS_MOUNT_POINT[usr-share-overlay] = "/usr/share/overlay"
         self.set_machine_config(wrong_machine_config)
 
         res = bitbake('core-image-minimal', ignore_status=True)
-        line = self.getline(res, "Missing required mount point for OVERLAYFS_MOUNT_POINT[mnt-overlay] in your MACHINE configuration")
+        line = getline(res, "Missing required mount point for OVERLAYFS_MOUNT_POINT[mnt-overlay] in your MACHINE configuration")
         self.assertTrue(line and line.startswith("Parsing recipes...ERROR:"), msg=res.output)
 
     def test_correct_image(self):
@@ -201,11 +201,11 @@ EOT
             # /usr/share/my-application as an overlay (see overlayfs-user recipe)
             status, output = qemu.run_serial("/bin/mount -t tmpfs,overlay")
 
-            line = self.getline_qemu(output, "on /mnt/overlay")
+            line = getline_qemu(output, "on /mnt/overlay")
             self.assertTrue(line and line.startswith("tmpfs"), msg=output)
 
-            line = self.getline_qemu(output, "upperdir=/mnt/overlay/upper/usr/share/my-application")
+            line = getline_qemu(output, "upperdir=/mnt/overlay/upper/usr/share/my-application")
             self.assertTrue(line and line.startswith("overlay"), msg=output)
 
-            line = self.getline_qemu(output, "upperdir=/mnt/overlay/upper/usr/share/another-overlay-mount")
+            line = getline_qemu(output, "upperdir=/mnt/overlay/upper/usr/share/another-overlay-mount")
             self.assertTrue(line and line.startswith("overlay"), msg=output)
-- 
2.28.0

[PATCH 6/8] oeqa/selftest: unit tests for overlayfs-etc

[Vyacheslav Yurkov]

Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com>
---
 meta/lib/oeqa/selftest/cases/overlayfs.py | 184 ++++++++++++++++++++++
 1 file changed, 184 insertions(+)

diff --git a/meta/lib/oeqa/selftest/cases/overlayfs.py b/meta/lib/oeqa/selftest/cases/overlayfs.py
index 43415778ce..82007fade7 100644
--- a/meta/lib/oeqa/selftest/cases/overlayfs.py
+++ b/meta/lib/oeqa/selftest/cases/overlayfs.py
@@ -209,3 +209,187 @@ EOT
 
             line = getline_qemu(output, "upperdir=/mnt/overlay/upper/usr/share/another-overlay-mount")
             self.assertTrue(line and line.startswith("overlay"), msg=output)
+
+class OverlayFSEtcRunTimeTests(OESelftestTestCase):
+    """overlayfs-etc class tests"""
+
+    def test_all_required_variables_set(self):
+        """
+        Summary:   Check that required variables are set
+        Expected:  Fail when any of required variables is missing
+        Author:    Vyacheslav Yurkov <uvv.mail@gmail.com>
+        """
+
+        configBase = """
+DISTRO_FEATURES += "systemd"
+
+# Use systemd as init manager
+VIRTUAL-RUNTIME_init_manager = "systemd"
+
+# enable overlayfs in the kernel
+KERNEL_EXTRA_FEATURES:append = " features/overlayfs/overlayfs.scc"
+
+# Image configuration for overlayfs-etc
+EXTRA_IMAGE_FEATURES += "overlayfs-etc"
+IMAGE_FEATURES:remove = "package-management"
+"""
+        configMountPoint = """
+OVERLAYFS_ETC_MOUNT_POINT = "/data"
+"""
+        configDevice = """
+OVERLAYFS_ETC_DEVICE = "/dev/mmcblk0p1"
+"""
+
+        self.write_config(configBase)
+        res = bitbake('core-image-minimal', ignore_status=True)
+        line = getline(res, "OVERLAYFS_ETC_MOUNT_POINT must be set in your MACHINE configuration")
+        self.assertTrue(line, msg=res.output)
+
+        self.append_config(configMountPoint)
+        res = bitbake('core-image-minimal', ignore_status=True)
+        line = getline(res, "OVERLAYFS_ETC_DEVICE must be set in your MACHINE configuration")
+        self.assertTrue(line, msg=res.output)
+
+        self.append_config(configDevice)
+        res = bitbake('core-image-minimal', ignore_status=True)
+        line = getline(res, "OVERLAYFS_ETC_FSTYPE should contain a valid file system type on /dev/mmcblk0p1")
+        self.assertTrue(line, msg=res.output)
+
+    def test_image_feature_conflict(self):
+        """
+        Summary:   Overlayfs-etc is not allowed to be used with package-management
+        Expected:  Feature conflict
+        Author:    Vyacheslav Yurkov <uvv.mail@gmail.com>
+        """
+
+        config = """
+DISTRO_FEATURES += "systemd"
+
+# Use systemd as init manager
+VIRTUAL-RUNTIME_init_manager = "systemd"
+
+# enable overlayfs in the kernel
+KERNEL_EXTRA_FEATURES:append = " features/overlayfs/overlayfs.scc"
+EXTRA_IMAGE_FEATURES += "overlayfs-etc"
+EXTRA_IMAGE_FEATURES += "package-management"
+"""
+
+        self.write_config(config)
+
+        res = bitbake('core-image-minimal', ignore_status=True)
+        line = getline(res, "contains conflicting IMAGE_FEATURES")
+        self.assertTrue("overlayfs-etc" in res.output, msg=res.output)
+        self.assertTrue("package-management" in res.output, msg=res.output)
+
+    def test_image_feature_is_missing_class_included(self):
+        configAppend = """
+INHERIT += "overlayfs-etc"
+"""
+        self.run_check_image_feature(configAppend)
+
+    def test_image_feature_is_missing(self):
+        self.run_check_image_feature()
+
+    def run_check_image_feature(self, appendToConfig=""):
+        """
+        Summary:   Overlayfs-etc class is not applied when image feature is not set
+                   even if we inherit it directly,
+        Expected:  Image is created successfully but /etc is not an overlay
+        Author:    Vyacheslav Yurkov <uvv.mail@gmail.com>
+        """
+
+        config = f"""
+DISTRO_FEATURES += "systemd"
+
+# Use systemd as init manager
+VIRTUAL-RUNTIME_init_manager = "systemd"
+
+# enable overlayfs in the kernel
+KERNEL_EXTRA_FEATURES:append = " features/overlayfs/overlayfs.scc"
+
+IMAGE_FSTYPES += "wic"
+WKS_FILE = "overlayfs_etc.wks.in"
+
+EXTRA_IMAGE_FEATURES += "read-only-rootfs"
+# Image configuration for overlayfs-etc
+OVERLAYFS_ETC_MOUNT_POINT = "/data"
+OVERLAYFS_ETC_DEVICE = "/dev/sda3"
+{appendToConfig}
+"""
+
+        self.write_config(config)
+
+        bitbake('core-image-minimal')
+
+        with runqemu('core-image-minimal', image_fstype='wic') as qemu:
+            status, output = qemu.run_serial("/bin/mount")
+
+            line = getline_qemu(output, "upperdir=/data/overlay-etc/upper")
+            self.assertFalse(line, msg=output)
+
+    def test_sbin_init_preinit(self):
+        self.run_sbin_init(False)
+
+    def test_sbin_init_original(self):
+        self.run_sbin_init(True)
+
+    def run_sbin_init(self, origInit):
+        """
+        Summary:   Confirm we can replace original init and mount overlay on top of /etc
+        Expected:  Image is created successfully and /etc is mounted as an overlay
+        Author:    Vyacheslav Yurkov <uvv.mail@gmail.com>
+        """
+
+        config = """
+DISTRO_FEATURES += "systemd"
+
+# Use systemd as init manager
+VIRTUAL-RUNTIME_init_manager = "systemd"
+
+# enable overlayfs in the kernel
+KERNEL_EXTRA_FEATURES:append = " features/overlayfs/overlayfs.scc"
+
+IMAGE_FSTYPES += "wic"
+OVERLAYFS_INIT_OPTION = "{OVERLAYFS_INIT_OPTION}"
+WKS_FILE = "overlayfs_etc.wks.in"
+
+EXTRA_IMAGE_FEATURES += "read-only-rootfs"
+# Image configuration for overlayfs-etc
+EXTRA_IMAGE_FEATURES += "overlayfs-etc"
+IMAGE_FEATURES:remove = "package-management"
+OVERLAYFS_ETC_MOUNT_POINT = "/data"
+OVERLAYFS_ETC_FSTYPE = "ext4"
+OVERLAYFS_ETC_DEVICE = "/dev/sda3"
+OVERLAYFS_ETC_USE_ORIG_INIT_NAME = "{OVERLAYFS_ETC_USE_ORIG_INIT_NAME}"
+"""
+
+        args = {
+            'OVERLAYFS_INIT_OPTION': "" if origInit else "init=/sbin/preinit",
+            'OVERLAYFS_ETC_USE_ORIG_INIT_NAME': int(origInit == True)
+        }
+
+        self.write_config(config.format(**args))
+
+        bitbake('core-image-minimal')
+        testFile = "/etc/my-test-data"
+
+        with runqemu('core-image-minimal', image_fstype='wic', discard_writes=False) as qemu:
+            status, output = qemu.run_serial("/bin/mount")
+
+            line = getline_qemu(output, "/dev/sda3")
+            self.assertTrue("/data" in output, msg=output)
+
+            line = getline_qemu(output, "upperdir=/data/overlay-etc/upper")
+            self.assertTrue(line and line.startswith("/data/overlay-etc/upper on /etc type overlay"), msg=output)
+
+            status, output = qemu.run_serial("touch " + testFile)
+            status, output = qemu.run_serial("sync")
+            status, output = qemu.run_serial("ls -1 " + testFile)
+            line = getline_qemu(output, testFile)
+            self.assertTrue(line and line.startswith(testFile), msg=output)
+
+        # Check that file exists in /etc after reboot
+        with runqemu('core-image-minimal', image_fstype='wic') as qemu:
+            status, output = qemu.run_serial("ls -1 " + testFile)
+            line = getline_qemu(output, testFile)
+            self.assertTrue(line and line.startswith(testFile), msg=output)
-- 
2.28.0

[PATCH 7/8] overlayfs: update notes on /etc

[Vyacheslav Yurkov]

Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com>
---
 meta/classes/overlayfs.bbclass | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/classes/overlayfs.bbclass b/meta/classes/overlayfs.bbclass
index 3c0f4dc882..f1b8086ea8 100644
--- a/meta/classes/overlayfs.bbclass
+++ b/meta/classes/overlayfs.bbclass
@@ -31,6 +31,7 @@
 #   OVERLAYFS_WRITABLE_PATHS[mnt-overlay] = "/usr/share/another-application"
 #
 # Note: the class does not support /etc directory itself, because systemd depends on it
+# For /etc directory use overlayfs-etc class
 
 REQUIRED_DISTRO_FEATURES += "systemd overlayfs"
 
-- 
2.28.0

[PATCH 8/8] overlayfs: move templates to files directory

[Vyacheslav Yurkov]

Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com>
---
 meta/classes/overlayfs.bbclass               | 52 ++++----------------
 meta/files/overlayfs-all-overlays.service.in | 12 +++++
 meta/files/overlayfs-create-dirs.service.in  | 14 ++++++
 meta/files/overlayfs-unit.mount.in           | 13 +++++
 4 files changed, 49 insertions(+), 42 deletions(-)
 create mode 100644 meta/files/overlayfs-all-overlays.service.in
 create mode 100644 meta/files/overlayfs-create-dirs.service.in
 create mode 100644 meta/files/overlayfs-unit.mount.in

diff --git a/meta/classes/overlayfs.bbclass b/meta/classes/overlayfs.bbclass
index f1b8086ea8..4a860f7308 100644
--- a/meta/classes/overlayfs.bbclass
+++ b/meta/classes/overlayfs.bbclass
@@ -37,51 +37,19 @@ REQUIRED_DISTRO_FEATURES += "systemd overlayfs"
 
 inherit systemd features_check
 
+OVERLAYFS_CREATE_DIRS_TEMPLATE ??= "${COREBASE}/meta/files/overlayfs-create-dirs.service.in"
+OVERLAYFS_MOUNT_UNIT_TEMPLATE ??= "${COREBASE}/meta/files/overlayfs-unit.mount.in"
+OVERLAYFS_ALL_OVERLAYS_TEMPLATE ??= "${COREBASE}/meta/files/overlayfs-all-overlays.service.in"
+
 python do_create_overlayfs_units() {
     from oe.overlayfs import mountUnitName
 
-    CreateDirsUnitTemplate = """[Unit]
-Description=Overlayfs directories setup
-Requires={DATA_MOUNT_UNIT}
-After={DATA_MOUNT_UNIT}
-DefaultDependencies=no
-
-[Service]
-Type=oneshot
-ExecStart=mkdir -p {DATA_MOUNT_POINT}/workdir{LOWERDIR} && mkdir -p {DATA_MOUNT_POINT}/upper{LOWERDIR}
-RemainAfterExit=true
-StandardOutput=journal
-
-[Install]
-WantedBy=multi-user.target
-"""
-    MountUnitTemplate = """[Unit]
-Description=Overlayfs mount unit
-Requires={CREATE_DIRS_SERVICE}
-After={CREATE_DIRS_SERVICE}
-
-[Mount]
-What=overlay
-Where={LOWERDIR}
-Type=overlay
-Options=lowerdir={LOWERDIR},upperdir={DATA_MOUNT_POINT}/upper{LOWERDIR},workdir={DATA_MOUNT_POINT}/workdir{LOWERDIR}
-
-[Install]
-WantedBy=multi-user.target
-"""
-    AllOverlaysTemplate = """[Unit]
-Description=Groups all overlays required by {PN} in one unit
-After={ALL_OVERLAYFS_UNITS}
-Requires={ALL_OVERLAYFS_UNITS}
-
-[Service]
-Type=oneshot
-ExecStart=/bin/true
-RemainAfterExit=true
-
-[Install]
-WantedBy=local-fs.target
-"""
+    with open(d.getVar("OVERLAYFS_CREATE_DIRS_TEMPLATE"), "r") as f:
+        CreateDirsUnitTemplate = f.read()
+    with open(d.getVar("OVERLAYFS_MOUNT_UNIT_TEMPLATE"), "r") as f:
+        MountUnitTemplate = f.read()
+    with open(d.getVar("OVERLAYFS_ALL_OVERLAYS_TEMPLATE"), "r") as f:
+        AllOverlaysTemplate = f.read()
 
     def prepareUnits(data, lower):
         from oe.overlayfs import helperUnitName
diff --git a/meta/files/overlayfs-all-overlays.service.in b/meta/files/overlayfs-all-overlays.service.in
new file mode 100644
index 0000000000..74ee4e90ae
--- /dev/null
+++ b/meta/files/overlayfs-all-overlays.service.in
@@ -0,0 +1,12 @@
+[Unit]
+Description=Groups all overlays required by {PN} in one unit
+After={ALL_OVERLAYFS_UNITS}
+Requires={ALL_OVERLAYFS_UNITS}
+
+[Service]
+Type=oneshot
+ExecStart=/bin/true
+RemainAfterExit=true
+
+[Install]
+WantedBy=local-fs.target
diff --git a/meta/files/overlayfs-create-dirs.service.in b/meta/files/overlayfs-create-dirs.service.in
new file mode 100644
index 0000000000..17204145f2
--- /dev/null
+++ b/meta/files/overlayfs-create-dirs.service.in
@@ -0,0 +1,14 @@
+[Unit]
+Description=Overlayfs directories setup
+Requires={DATA_MOUNT_UNIT}
+After={DATA_MOUNT_UNIT}
+DefaultDependencies=no
+
+[Service]
+Type=oneshot
+ExecStart=mkdir -p {DATA_MOUNT_POINT}/workdir{LOWERDIR} && mkdir -p {DATA_MOUNT_POINT}/upper{LOWERDIR}
+RemainAfterExit=true
+StandardOutput=journal
+
+[Install]
+WantedBy=multi-user.target
diff --git a/meta/files/overlayfs-unit.mount.in b/meta/files/overlayfs-unit.mount.in
new file mode 100644
index 0000000000..1d33b7e39c
--- /dev/null
+++ b/meta/files/overlayfs-unit.mount.in
@@ -0,0 +1,13 @@
+[Unit]
+Description=Overlayfs mount unit
+Requires={CREATE_DIRS_SERVICE}
+After={CREATE_DIRS_SERVICE}
+
+[Mount]
+What=overlay
+Where={LOWERDIR}
+Type=overlay
+Options=lowerdir={LOWERDIR},upperdir={DATA_MOUNT_POINT}/upper{LOWERDIR},workdir={DATA_MOUNT_POINT}/workdir{LOWERDIR}
+
+[Install]
+WantedBy=multi-user.target
-- 
2.28.0

[PATCH 6/8] oeqa/selftest: unit tests for overlayfs-etc

[Vyacheslav Yurkov]

Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com>
---
 meta/lib/oeqa/selftest/cases/overlayfs.py | 184 ++++++++++++++++++++++
 1 file changed, 184 insertions(+)

diff --git a/meta/lib/oeqa/selftest/cases/overlayfs.py b/meta/lib/oeqa/selftest/cases/overlayfs.py
index 43415778ce..82007fade7 100644
--- a/meta/lib/oeqa/selftest/cases/overlayfs.py
+++ b/meta/lib/oeqa/selftest/cases/overlayfs.py
@@ -209,3 +209,187 @@ EOT
 
             line = getline_qemu(output, "upperdir=/mnt/overlay/upper/usr/share/another-overlay-mount")
             self.assertTrue(line and line.startswith("overlay"), msg=output)
+
+class OverlayFSEtcRunTimeTests(OESelftestTestCase):
+    """overlayfs-etc class tests"""
+
+    def test_all_required_variables_set(self):
+        """
+        Summary:   Check that required variables are set
+        Expected:  Fail when any of required variables is missing
+        Author:    Vyacheslav Yurkov <uvv.mail@gmail.com>
+        """
+
+        configBase = """
+DISTRO_FEATURES += "systemd"
+
+# Use systemd as init manager
+VIRTUAL-RUNTIME_init_manager = "systemd"
+
+# enable overlayfs in the kernel
+KERNEL_EXTRA_FEATURES:append = " features/overlayfs/overlayfs.scc"
+
+# Image configuration for overlayfs-etc
+EXTRA_IMAGE_FEATURES += "overlayfs-etc"
+IMAGE_FEATURES:remove = "package-management"
+"""
+        configMountPoint = """
+OVERLAYFS_ETC_MOUNT_POINT = "/data"
+"""
+        configDevice = """
+OVERLAYFS_ETC_DEVICE = "/dev/mmcblk0p1"
+"""
+
+        self.write_config(configBase)
+        res = bitbake('core-image-minimal', ignore_status=True)
+        line = getline(res, "OVERLAYFS_ETC_MOUNT_POINT must be set in your MACHINE configuration")
+        self.assertTrue(line, msg=res.output)
+
+        self.append_config(configMountPoint)
+        res = bitbake('core-image-minimal', ignore_status=True)
+        line = getline(res, "OVERLAYFS_ETC_DEVICE must be set in your MACHINE configuration")
+        self.assertTrue(line, msg=res.output)
+
+        self.append_config(configDevice)
+        res = bitbake('core-image-minimal', ignore_status=True)
+        line = getline(res, "OVERLAYFS_ETC_FSTYPE should contain a valid file system type on /dev/mmcblk0p1")
+        self.assertTrue(line, msg=res.output)
+
+    def test_image_feature_conflict(self):
+        """
+        Summary:   Overlayfs-etc is not allowed to be used with package-management
+        Expected:  Feature conflict
+        Author:    Vyacheslav Yurkov <uvv.mail@gmail.com>
+        """
+
+        config = """
+DISTRO_FEATURES += "systemd"
+
+# Use systemd as init manager
+VIRTUAL-RUNTIME_init_manager = "systemd"
+
+# enable overlayfs in the kernel
+KERNEL_EXTRA_FEATURES:append = " features/overlayfs/overlayfs.scc"
+EXTRA_IMAGE_FEATURES += "overlayfs-etc"
+EXTRA_IMAGE_FEATURES += "package-management"
+"""
+
+        self.write_config(config)
+
+        res = bitbake('core-image-minimal', ignore_status=True)
+        line = getline(res, "contains conflicting IMAGE_FEATURES")
+        self.assertTrue("overlayfs-etc" in res.output, msg=res.output)
+        self.assertTrue("package-management" in res.output, msg=res.output)
+
+    def test_image_feature_is_missing_class_included(self):
+        configAppend = """
+INHERIT += "overlayfs-etc"
+"""
+        self.run_check_image_feature(configAppend)
+
+    def test_image_feature_is_missing(self):
+        self.run_check_image_feature()
+
+    def run_check_image_feature(self, appendToConfig=""):
+        """
+        Summary:   Overlayfs-etc class is not applied when image feature is not set
+                   even if we inherit it directly,
+        Expected:  Image is created successfully but /etc is not an overlay
+        Author:    Vyacheslav Yurkov <uvv.mail@gmail.com>
+        """
+
+        config = f"""
+DISTRO_FEATURES += "systemd"
+
+# Use systemd as init manager
+VIRTUAL-RUNTIME_init_manager = "systemd"
+
+# enable overlayfs in the kernel
+KERNEL_EXTRA_FEATURES:append = " features/overlayfs/overlayfs.scc"
+
+IMAGE_FSTYPES += "wic"
+WKS_FILE = "overlayfs_etc.wks.in"
+
+EXTRA_IMAGE_FEATURES += "read-only-rootfs"
+# Image configuration for overlayfs-etc
+OVERLAYFS_ETC_MOUNT_POINT = "/data"
+OVERLAYFS_ETC_DEVICE = "/dev/sda3"
+{appendToConfig}
+"""
+
+        self.write_config(config)
+
+        bitbake('core-image-minimal')
+
+        with runqemu('core-image-minimal', image_fstype='wic') as qemu:
+            status, output = qemu.run_serial("/bin/mount")
+
+            line = getline_qemu(output, "upperdir=/data/overlay-etc/upper")
+            self.assertFalse(line, msg=output)
+
+    def test_sbin_init_preinit(self):
+        self.run_sbin_init(False)
+
+    def test_sbin_init_original(self):
+        self.run_sbin_init(True)
+
+    def run_sbin_init(self, origInit):
+        """
+        Summary:   Confirm we can replace original init and mount overlay on top of /etc
+        Expected:  Image is created successfully and /etc is mounted as an overlay
+        Author:    Vyacheslav Yurkov <uvv.mail@gmail.com>
+        """
+
+        config = """
+DISTRO_FEATURES += "systemd"
+
+# Use systemd as init manager
+VIRTUAL-RUNTIME_init_manager = "systemd"
+
+# enable overlayfs in the kernel
+KERNEL_EXTRA_FEATURES:append = " features/overlayfs/overlayfs.scc"
+
+IMAGE_FSTYPES += "wic"
+OVERLAYFS_INIT_OPTION = "{OVERLAYFS_INIT_OPTION}"
+WKS_FILE = "overlayfs_etc.wks.in"
+
+EXTRA_IMAGE_FEATURES += "read-only-rootfs"
+# Image configuration for overlayfs-etc
+EXTRA_IMAGE_FEATURES += "overlayfs-etc"
+IMAGE_FEATURES:remove = "package-management"
+OVERLAYFS_ETC_MOUNT_POINT = "/data"
+OVERLAYFS_ETC_FSTYPE = "ext4"
+OVERLAYFS_ETC_DEVICE = "/dev/sda3"
+OVERLAYFS_ETC_USE_ORIG_INIT_NAME = "{OVERLAYFS_ETC_USE_ORIG_INIT_NAME}"
+"""
+
+        args = {
+            'OVERLAYFS_INIT_OPTION': "" if origInit else "init=/sbin/preinit",
+            'OVERLAYFS_ETC_USE_ORIG_INIT_NAME': int(origInit == True)
+        }
+
+        self.write_config(config.format(**args))
+
+        bitbake('core-image-minimal')
+        testFile = "/etc/my-test-data"
+
+        with runqemu('core-image-minimal', image_fstype='wic', discard_writes=False) as qemu:
+            status, output = qemu.run_serial("/bin/mount")
+
+            line = getline_qemu(output, "/dev/sda3")
+            self.assertTrue("/data" in output, msg=output)
+
+            line = getline_qemu(output, "upperdir=/data/overlay-etc/upper")
+            self.assertTrue(line and line.startswith("/data/overlay-etc/upper on /etc type overlay"), msg=output)
+
+            status, output = qemu.run_serial("touch " + testFile)
+            status, output = qemu.run_serial("sync")
+            status, output = qemu.run_serial("ls -1 " + testFile)
+            line = getline_qemu(output, testFile)
+            self.assertTrue(line and line.startswith(testFile), msg=output)
+
+        # Check that file exists in /etc after reboot
+        with runqemu('core-image-minimal', image_fstype='wic') as qemu:
+            status, output = qemu.run_serial("ls -1 " + testFile)
+            line = getline_qemu(output, testFile)
+            self.assertTrue(line and line.startswith(testFile), msg=output)
-- 
2.28.0