Re: [OE-core] [PATCH] overlayfs.bbclass: generate overlayfs mount units

["Bruce Ashfield" <bruce.ashfield@gmail.com>]

On Fri, Jun 4, 2021 at 1:22 PM Vyacheslav Yurkov <uvv.mail@gmail.com> wrote:
>
> Hi Bruce,
> Thanks for a quick feedback.
>
> > Hi Vyacheslav,
> >
> > This looks like it could be quite useful, but a few comments came to
> > mind on a really quick scan.
> >
> > Since this is systemd based, it should probably have
> > REQUIRED_DISTRO_FEATURES = "systemd", to ensure that the rest of the
> > system and packages are consistently enabled.
>
> Will include it in my v2.
>
> > As part of getting this into core, it needs some sort of test cases /
> > test suite (and one that covers the different modes you describe in
> > the class) .. so something like a test image (a user) and the tests
> > themselves.
>
> Could you please give me a few hints as to where current tests are
> stored and how to run them? Is it meta-selftest layer?

Richard can guide you better on this front than I can, but the tests
can be in a number of forms. ptests (i.e. if you create the reference
image that uses it, that recipe could have a ptest), and there are also
the lib/oeqa/selftest/ tests.

How they are fully hooked in, so they can be tested with core, I haven't
looked in a while, and might steer you wrong, So my recommendations
are to checkout those components, and follow what is on the wiki and
in the docs (https://wiki.yoctoproject.org/wiki/Oe-selftest),
https://docs.yoctoproject.org/test-manual/intro.html, etc

>
> > It really also needs a maintainer entry, since otherwise the
> > maintenance work falls to the same already overloaded folks.
>
> You mean just an entry like:
> MAINTAINER = "Vyacheslav Yurkov <uvv.mail@gmail.com>

Yep.

>
> in the class itself?
> Because meta/conf/distro/include/maintainers.inc mentions only recipe
> maintainers.

Richard could advise better on this, but yes, even if this isn't a recipe
we'd want someone to look after it if it merges to core. So if we are
breaking into a new category here, that is probably ok.

>
> > Since this is intended to be used on a per-recipe basis (at least that
> > is my understanding from reading it), what happens if two recipes
> > create conflicting overlay mounts ? or if the mentioned data.mount is
> > not present in the image ?
>
> These are good questions. I guess if two recipes would try to use the
> same mounts, then do_rootfs should simply fail the same way when two
> recipes install the same file.

Yes, the image would definitely conflict during package install in that
case. It wasn't entirely clear to me that they'd have to have exactly the
same file.

> As for handling a mount.unit, that has to be done on an image level, I
> guess. The other question is how to do that.. Perhaps a QA-hook in
> ROOTFS_POSTPROCESS_COMMAND, which checks the file existence (not
> validity though!)?

That might be feasible, assuming that it doesn't always run, or if it
always run, that it doesn't take much time.

Cheers,

Bruce

>
> Vyacheslav
>
> > Bruce
> >
> >> Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com>
> >> ---
> >>   meta/classes/overlayfs.bbclass | 132 +++++++++++++++++++++++++++++++++
> >>   1 file changed, 132 insertions(+)
> >>   create mode 100644 meta/classes/overlayfs.bbclass
> >>
> >> diff --git a/meta/classes/overlayfs.bbclass b/meta/classes/overlayfs.bbclass
> >> new file mode 100644
> >> index 0000000000..7fac3e696e
> >> --- /dev/null
> >> +++ b/meta/classes/overlayfs.bbclass
> >> @@ -0,0 +1,132 @@
> >> +# Class for generation of overlayfs mount units
> >> +#
> >> +# It's often desired in Embedded System design to have a read-only rootfs.
> >> +# But a lot of different applications might want to have a read-write access to
> >> +# some parts of a filesystem. It can be especially useful when your update mechanism
> >> +# overwrites the whole rootfs, but you want your application data to be preserved
> >> +# between updates. This class provides a way to achieve that by means
> >> +# of overlayfs and at the same time keeping the base rootfs read-only.
> >> +#
> >> +# Usage example.
> >> +#
> >> +# Set a mount point for a partition overlayfs is going to use as upper layer
> >> +# in your machine configuration. Underlying file system can be anything that
> >> +# is supported by overlayfs
> >> +#
> >> +#   OVERLAYFS_MOUNT_POINT[data] ?= "/data"
> >> +#
> >> +# The class assumes you have a data.mount systemd unit defined elsewhere in your
> >> +# BSP and installed to the image.
> >> +#
> >> +# Then you can specify writable directories on a recipe base
> >> +#
> >> +#   OVERLAYFS_WRITABLE_PATHS[data] = "/usr/share/my-custom-application"
> >> +#
> >> +# To support several mount points you can use a different variable flag. Assume we
> >> +# what to have a writable location on the file system, but not interested where the data
> >> +# survive a reboot. The we could have a mnt-overlay.mount unit for a tmpfs file system:
> >> +#
> >> +#   OVERLAYFS_MOUNT_POINT[mnt-overlay] = "/mnt/overlay"
> >> +#   OVERLAYFS_WRITABLE_PATHS[mnt-overlay] = "/usr/share/another-application"
> >> +
> >> +OVERLAYFS_WRITABLE_PATHS[data] ?= ""
> >> +
> >> +inherit systemd
> >> +
> >> +def strForBash(s):
> >> +    return s.replace('\\', '\\\\')
> >> +
> >> +def unitFileList(d):
> >> +    fileList = []
> >> +    overlayMountPoints = d.getVarFlags("OVERLAYFS_MOUNT_POINT")
> >> +    for mountPoint in overlayMountPoints:
> >> +        for path in d.getVarFlag('OVERLAYFS_WRITABLE_PATHS', mountPoint).split():
> >> +            fileList.append(mountUnitName(path))
> >> +            fileList.append(helperUnitName(path))
> >> +
> >> +    return fileList
> >> +
> >> +# this function is based on https://github.com/systemd/systemd/blob/main/src/basic/unit-name.c
> >> +def escapeSystemdUnitName(path):
> >> +    escapeMap = {
> >> +        '/': '-',
> >> +        '-': "\\x2d",
> >> +        '\\': "\\x5d"
> >> +    }
> >> +    return "".join([escapeMap.get(c, c) for c in path.strip('/')])
> >> +
> >> +def mountUnitName(unit):
> >> +    return escapeSystemdUnitName(unit) + '.mount'
> >> +
> >> +def helperUnitName(unit):
> >> +    return escapeSystemdUnitName(unit) + '-create-upper-dir.service'
> >> +
> >> +python do_create_overlayfs_units() {
> >> +    CreateDirsUnitTemplate = """[Unit]
> >> +Description=Overlayfs directories setup
> >> +Requires={DATA_MOUNT_UNIT}
> >> +After={DATA_MOUNT_UNIT}
> >> +DefaultDependencies=no
> >> +
> >> +[Service]
> >> +Type=oneshot
> >> +ExecStart=mkdir -p {DATA_MOUNT_POINT}/workdir{LOWERDIR} && mkdir -p {DATA_MOUNT_POINT}/upper{LOWERDIR}
> >> +RemainAfterExit=true
> >> +StandardOutput=journal
> >> +
> >> +[Install]
> >> +WantedBy=multi-user.target
> >> +"""
> >> +    MountUnitTemplate = """[Unit]
> >> +Description=Overlayfs mount unit
> >> +Requires={CREATE_DIRS_SERVICE}
> >> +After={CREATE_DIRS_SERVICE}
> >> +
> >> +[Mount]
> >> +What=overlay
> >> +Where={LOWERDIR}
> >> +Type=overlay
> >> +Options=lowerdir={LOWERDIR},upperdir={DATA_MOUNT_POINT}/upper{LOWERDIR},workdir={DATA_MOUNT_POINT}/workdir{LOWERDIR}
> >> +
> >> +[Install]
> >> +WantedBy=multi-user.target
> >> +"""
> >> +
> >> +    def prepareUnits(data, lower):
> >> +        args = {
> >> +            'DATA_MOUNT_POINT': data,
> >> +            'DATA_MOUNT_UNIT': mountUnitName(data),
> >> +            'CREATE_DIRS_SERVICE': helperUnitName(lower),
> >> +            'LOWERDIR': lower,
> >> +        }
> >> +
> >> +        with open(os.path.join(d.getVar('WORKDIR'), mountUnitName(lower)), 'w') as f:
> >> +            f.write(MountUnitTemplate.format(**args))
> >> +
> >> +        with open(os.path.join(d.getVar('WORKDIR'), helperUnitName(lower)), 'w') as f:
> >> +            f.write(CreateDirsUnitTemplate.format(**args))
> >> +
> >> +    overlayMountPoints = d.getVarFlags("OVERLAYFS_MOUNT_POINT")
> >> +    for mountPoint in overlayMountPoints:
> >> +        for lower in d.getVarFlag('OVERLAYFS_WRITABLE_PATHS', mountPoint).split():
> >> +            prepareUnits(d.getVarFlag('OVERLAYFS_MOUNT_POINT', mountPoint), lower)
> >> +}
> >> +
> >> +# we need to generate file names early during parsing stage
> >> +python () {
> >> +    unitList = unitFileList(d)
> >> +    for unit in unitList:
> >> +        d.appendVar('SYSTEMD_SERVICE_' + d.getVar('PN'), ' ' + unit);
> >> +        d.appendVar('FILES_' + d.getVar('PN'), strForBash(unit))
> >> +
> >> +    d.setVar('OVERLAYFS_UNIT_LIST', ' '.join([strForBash(s) for s in unitList]))
> >> +}
> >> +
> >> +do_install_append() {
> >> +    install -d ${D}${systemd_system_unitdir}
> >> +    for unit in ${OVERLAYFS_UNIT_LIST}; do
> >> +        install -m 0444 ${WORKDIR}/${unit} ${D}${systemd_system_unitdir}
> >> +    done
> >> +}
> >> +
> >> +addtask create_overlayfs_units before do_install
> >> --
> >> 2.28.0
> >>
> >>
> >> 
> >>
>


--
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II