Re: [OE-core] [PATCH] overlayfs.bbclass: generate overlayfs mount units

["Bruce Ashfield" <bruce.ashfield@gmail.com>]

On Thu, Jun 3, 2021 at 10:21 AM Vyacheslav Yurkov <uvv.mail@gmail.com> wrote:
>
> It's often desired in Embedded System design to have a read-only rootfs.
> But a lot of different applications might want to have a read-write access
> to some parts of a filesystem. It can be especially useful when your update
> mechanism overwrites the whole rootfs, but you want your application data
> to be preserved between updates. This class provides a way to achieve that
> by means of overlayfs and at the same time keeping the base rootfs read-only.

Hi Vyacheslav,

This looks like it could be quite useful, but a few comments came to
mind on a really quick scan.

Since this is systemd based, it should probably have
REQUIRED_DISTRO_FEATURES = "systemd", to ensure that the rest of the
system and packages are consistently enabled.

As part of getting this into core, it needs some sort of test cases /
test suite (and one that covers the different modes you describe in
the class) .. so something like a test image (a user) and the tests
themselves.

It really also needs a maintainer entry, since otherwise the
maintenance work falls to the same already overloaded folks.

Since this is intended to be used on a per-recipe basis (at least that
is my understanding from reading it), what happens if two recipes
create conflicting overlay mounts ? or if the mentioned data.mount is
not present in the image ?

Bruce

>
> Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com>
> ---
>  meta/classes/overlayfs.bbclass | 132 +++++++++++++++++++++++++++++++++
>  1 file changed, 132 insertions(+)
>  create mode 100644 meta/classes/overlayfs.bbclass
>
> diff --git a/meta/classes/overlayfs.bbclass b/meta/classes/overlayfs.bbclass
> new file mode 100644
> index 0000000000..7fac3e696e
> --- /dev/null
> +++ b/meta/classes/overlayfs.bbclass
> @@ -0,0 +1,132 @@
> +# Class for generation of overlayfs mount units
> +#
> +# It's often desired in Embedded System design to have a read-only rootfs.
> +# But a lot of different applications might want to have a read-write access to
> +# some parts of a filesystem. It can be especially useful when your update mechanism
> +# overwrites the whole rootfs, but you want your application data to be preserved
> +# between updates. This class provides a way to achieve that by means
> +# of overlayfs and at the same time keeping the base rootfs read-only.
> +#
> +# Usage example.
> +#
> +# Set a mount point for a partition overlayfs is going to use as upper layer
> +# in your machine configuration. Underlying file system can be anything that
> +# is supported by overlayfs
> +#
> +#   OVERLAYFS_MOUNT_POINT[data] ?= "/data"
> +#
> +# The class assumes you have a data.mount systemd unit defined elsewhere in your
> +# BSP and installed to the image.
> +#
> +# Then you can specify writable directories on a recipe base
> +#
> +#   OVERLAYFS_WRITABLE_PATHS[data] = "/usr/share/my-custom-application"
> +#
> +# To support several mount points you can use a different variable flag. Assume we
> +# what to have a writable location on the file system, but not interested where the data
> +# survive a reboot. The we could have a mnt-overlay.mount unit for a tmpfs file system:
> +#
> +#   OVERLAYFS_MOUNT_POINT[mnt-overlay] = "/mnt/overlay"
> +#   OVERLAYFS_WRITABLE_PATHS[mnt-overlay] = "/usr/share/another-application"
> +
> +OVERLAYFS_WRITABLE_PATHS[data] ?= ""
> +
> +inherit systemd
> +
> +def strForBash(s):
> +    return s.replace('\\', '\\\\')
> +
> +def unitFileList(d):
> +    fileList = []
> +    overlayMountPoints = d.getVarFlags("OVERLAYFS_MOUNT_POINT")
> +    for mountPoint in overlayMountPoints:
> +        for path in d.getVarFlag('OVERLAYFS_WRITABLE_PATHS', mountPoint).split():
> +            fileList.append(mountUnitName(path))
> +            fileList.append(helperUnitName(path))
> +
> +    return fileList
> +
> +# this function is based on https://github.com/systemd/systemd/blob/main/src/basic/unit-name.c
> +def escapeSystemdUnitName(path):
> +    escapeMap = {
> +        '/': '-',
> +        '-': "\\x2d",
> +        '\\': "\\x5d"
> +    }
> +    return "".join([escapeMap.get(c, c) for c in path.strip('/')])
> +
> +def mountUnitName(unit):
> +    return escapeSystemdUnitName(unit) + '.mount'
> +
> +def helperUnitName(unit):
> +    return escapeSystemdUnitName(unit) + '-create-upper-dir.service'
> +
> +python do_create_overlayfs_units() {
> +    CreateDirsUnitTemplate = """[Unit]
> +Description=Overlayfs directories setup
> +Requires={DATA_MOUNT_UNIT}
> +After={DATA_MOUNT_UNIT}
> +DefaultDependencies=no
> +
> +[Service]
> +Type=oneshot
> +ExecStart=mkdir -p {DATA_MOUNT_POINT}/workdir{LOWERDIR} && mkdir -p {DATA_MOUNT_POINT}/upper{LOWERDIR}
> +RemainAfterExit=true
> +StandardOutput=journal
> +
> +[Install]
> +WantedBy=multi-user.target
> +"""
> +    MountUnitTemplate = """[Unit]
> +Description=Overlayfs mount unit
> +Requires={CREATE_DIRS_SERVICE}
> +After={CREATE_DIRS_SERVICE}
> +
> +[Mount]
> +What=overlay
> +Where={LOWERDIR}
> +Type=overlay
> +Options=lowerdir={LOWERDIR},upperdir={DATA_MOUNT_POINT}/upper{LOWERDIR},workdir={DATA_MOUNT_POINT}/workdir{LOWERDIR}
> +
> +[Install]
> +WantedBy=multi-user.target
> +"""
> +
> +    def prepareUnits(data, lower):
> +        args = {
> +            'DATA_MOUNT_POINT': data,
> +            'DATA_MOUNT_UNIT': mountUnitName(data),
> +            'CREATE_DIRS_SERVICE': helperUnitName(lower),
> +            'LOWERDIR': lower,
> +        }
> +
> +        with open(os.path.join(d.getVar('WORKDIR'), mountUnitName(lower)), 'w') as f:
> +            f.write(MountUnitTemplate.format(**args))
> +
> +        with open(os.path.join(d.getVar('WORKDIR'), helperUnitName(lower)), 'w') as f:
> +            f.write(CreateDirsUnitTemplate.format(**args))
> +
> +    overlayMountPoints = d.getVarFlags("OVERLAYFS_MOUNT_POINT")
> +    for mountPoint in overlayMountPoints:
> +        for lower in d.getVarFlag('OVERLAYFS_WRITABLE_PATHS', mountPoint).split():
> +            prepareUnits(d.getVarFlag('OVERLAYFS_MOUNT_POINT', mountPoint), lower)
> +}
> +
> +# we need to generate file names early during parsing stage
> +python () {
> +    unitList = unitFileList(d)
> +    for unit in unitList:
> +        d.appendVar('SYSTEMD_SERVICE_' + d.getVar('PN'), ' ' + unit);
> +        d.appendVar('FILES_' + d.getVar('PN'), strForBash(unit))
> +
> +    d.setVar('OVERLAYFS_UNIT_LIST', ' '.join([strForBash(s) for s in unitList]))
> +}
> +
> +do_install_append() {
> +    install -d ${D}${systemd_system_unitdir}
> +    for unit in ${OVERLAYFS_UNIT_LIST}; do
> +        install -m 0444 ${WORKDIR}/${unit} ${D}${systemd_system_unitdir}
> +    done
> +}
> +
> +addtask create_overlayfs_units before do_install
> --
> 2.28.0
>
>
> 
>


-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II