All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH] overlayfs.bbclass: generate overlayfs mount units
@ 2021-06-03 14:21 Vyacheslav Yurkov
  2021-06-03 14:35 ` [OE-core] " Bruce Ashfield
                   ` (3 more replies)
  0 siblings, 4 replies; 10+ messages in thread
From: Vyacheslav Yurkov @ 2021-06-03 14:21 UTC (permalink / raw)
  To: Openembedded-core; +Cc: Vyacheslav Yurkov

It's often desired in Embedded System design to have a read-only rootfs.
But a lot of different applications might want to have a read-write access
to some parts of a filesystem. It can be especially useful when your update
mechanism overwrites the whole rootfs, but you want your application data
to be preserved between updates. This class provides a way to achieve that
by means of overlayfs and at the same time keeping the base rootfs read-only.

Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com>
---
 meta/classes/overlayfs.bbclass | 132 +++++++++++++++++++++++++++++++++
 1 file changed, 132 insertions(+)
 create mode 100644 meta/classes/overlayfs.bbclass

diff --git a/meta/classes/overlayfs.bbclass b/meta/classes/overlayfs.bbclass
new file mode 100644
index 0000000000..7fac3e696e
--- /dev/null
+++ b/meta/classes/overlayfs.bbclass
@@ -0,0 +1,132 @@
+# Class for generation of overlayfs mount units
+#
+# It's often desired in Embedded System design to have a read-only rootfs.
+# But a lot of different applications might want to have a read-write access to
+# some parts of a filesystem. It can be especially useful when your update mechanism
+# overwrites the whole rootfs, but you want your application data to be preserved
+# between updates. This class provides a way to achieve that by means
+# of overlayfs and at the same time keeping the base rootfs read-only.
+#
+# Usage example.
+#
+# Set a mount point for a partition overlayfs is going to use as upper layer
+# in your machine configuration. Underlying file system can be anything that
+# is supported by overlayfs
+#
+#   OVERLAYFS_MOUNT_POINT[data] ?= "/data"
+#
+# The class assumes you have a data.mount systemd unit defined elsewhere in your
+# BSP and installed to the image.
+#
+# Then you can specify writable directories on a recipe base
+#
+#   OVERLAYFS_WRITABLE_PATHS[data] = "/usr/share/my-custom-application"
+#
+# To support several mount points you can use a different variable flag. Assume we
+# what to have a writable location on the file system, but not interested where the data
+# survive a reboot. The we could have a mnt-overlay.mount unit for a tmpfs file system:
+#
+#   OVERLAYFS_MOUNT_POINT[mnt-overlay] = "/mnt/overlay"
+#   OVERLAYFS_WRITABLE_PATHS[mnt-overlay] = "/usr/share/another-application"
+
+OVERLAYFS_WRITABLE_PATHS[data] ?= ""
+
+inherit systemd
+
+def strForBash(s):
+    return s.replace('\\', '\\\\')
+
+def unitFileList(d):
+    fileList = []
+    overlayMountPoints = d.getVarFlags("OVERLAYFS_MOUNT_POINT")
+    for mountPoint in overlayMountPoints:
+        for path in d.getVarFlag('OVERLAYFS_WRITABLE_PATHS', mountPoint).split():
+            fileList.append(mountUnitName(path))
+            fileList.append(helperUnitName(path))
+
+    return fileList
+
+# this function is based on https://github.com/systemd/systemd/blob/main/src/basic/unit-name.c
+def escapeSystemdUnitName(path):
+    escapeMap = {
+        '/': '-',
+        '-': "\\x2d",
+        '\\': "\\x5d"
+    }
+    return "".join([escapeMap.get(c, c) for c in path.strip('/')])
+
+def mountUnitName(unit):
+    return escapeSystemdUnitName(unit) + '.mount'
+
+def helperUnitName(unit):
+    return escapeSystemdUnitName(unit) + '-create-upper-dir.service'
+
+python do_create_overlayfs_units() {
+    CreateDirsUnitTemplate = """[Unit]
+Description=Overlayfs directories setup
+Requires={DATA_MOUNT_UNIT}
+After={DATA_MOUNT_UNIT}
+DefaultDependencies=no
+
+[Service]
+Type=oneshot
+ExecStart=mkdir -p {DATA_MOUNT_POINT}/workdir{LOWERDIR} && mkdir -p {DATA_MOUNT_POINT}/upper{LOWERDIR}
+RemainAfterExit=true
+StandardOutput=journal
+
+[Install]
+WantedBy=multi-user.target
+"""
+    MountUnitTemplate = """[Unit]
+Description=Overlayfs mount unit
+Requires={CREATE_DIRS_SERVICE}
+After={CREATE_DIRS_SERVICE}
+
+[Mount]
+What=overlay
+Where={LOWERDIR}
+Type=overlay
+Options=lowerdir={LOWERDIR},upperdir={DATA_MOUNT_POINT}/upper{LOWERDIR},workdir={DATA_MOUNT_POINT}/workdir{LOWERDIR}
+
+[Install]
+WantedBy=multi-user.target
+"""
+
+    def prepareUnits(data, lower):
+        args = {
+            'DATA_MOUNT_POINT': data,
+            'DATA_MOUNT_UNIT': mountUnitName(data),
+            'CREATE_DIRS_SERVICE': helperUnitName(lower),
+            'LOWERDIR': lower,
+        }
+
+        with open(os.path.join(d.getVar('WORKDIR'), mountUnitName(lower)), 'w') as f:
+            f.write(MountUnitTemplate.format(**args))
+
+        with open(os.path.join(d.getVar('WORKDIR'), helperUnitName(lower)), 'w') as f:
+            f.write(CreateDirsUnitTemplate.format(**args))
+
+    overlayMountPoints = d.getVarFlags("OVERLAYFS_MOUNT_POINT")
+    for mountPoint in overlayMountPoints:
+        for lower in d.getVarFlag('OVERLAYFS_WRITABLE_PATHS', mountPoint).split():
+            prepareUnits(d.getVarFlag('OVERLAYFS_MOUNT_POINT', mountPoint), lower)
+}
+
+# we need to generate file names early during parsing stage
+python () {
+    unitList = unitFileList(d)
+    for unit in unitList:
+        d.appendVar('SYSTEMD_SERVICE_' + d.getVar('PN'), ' ' + unit);
+        d.appendVar('FILES_' + d.getVar('PN'), strForBash(unit))
+
+    d.setVar('OVERLAYFS_UNIT_LIST', ' '.join([strForBash(s) for s in unitList]))
+}
+
+do_install_append() {
+    install -d ${D}${systemd_system_unitdir}
+    for unit in ${OVERLAYFS_UNIT_LIST}; do
+        install -m 0444 ${WORKDIR}/${unit} ${D}${systemd_system_unitdir}
+    done
+}
+
+addtask create_overlayfs_units before do_install
-- 
2.28.0


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [OE-core] [PATCH] overlayfs.bbclass: generate overlayfs mount units
  2021-06-03 14:21 [PATCH] overlayfs.bbclass: generate overlayfs mount units Vyacheslav Yurkov
@ 2021-06-03 14:35 ` Bruce Ashfield
  2021-06-04 17:22   ` Vyacheslav Yurkov
  2021-06-04 20:20 ` Ayoub Zaki
                   ` (2 subsequent siblings)
  3 siblings, 1 reply; 10+ messages in thread
From: Bruce Ashfield @ 2021-06-03 14:35 UTC (permalink / raw)
  To: Vyacheslav Yurkov; +Cc: Patches and discussions about the oe-core layer

On Thu, Jun 3, 2021 at 10:21 AM Vyacheslav Yurkov <uvv.mail@gmail.com> wrote:
>
> It's often desired in Embedded System design to have a read-only rootfs.
> But a lot of different applications might want to have a read-write access
> to some parts of a filesystem. It can be especially useful when your update
> mechanism overwrites the whole rootfs, but you want your application data
> to be preserved between updates. This class provides a way to achieve that
> by means of overlayfs and at the same time keeping the base rootfs read-only.

Hi Vyacheslav,

This looks like it could be quite useful, but a few comments came to
mind on a really quick scan.

Since this is systemd based, it should probably have
REQUIRED_DISTRO_FEATURES = "systemd", to ensure that the rest of the
system and packages are consistently enabled.

As part of getting this into core, it needs some sort of test cases /
test suite (and one that covers the different modes you describe in
the class) .. so something like a test image (a user) and the tests
themselves.

It really also needs a maintainer entry, since otherwise the
maintenance work falls to the same already overloaded folks.

Since this is intended to be used on a per-recipe basis (at least that
is my understanding from reading it), what happens if two recipes
create conflicting overlay mounts ? or if the mentioned data.mount is
not present in the image ?

Bruce

>
> Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com>
> ---
>  meta/classes/overlayfs.bbclass | 132 +++++++++++++++++++++++++++++++++
>  1 file changed, 132 insertions(+)
>  create mode 100644 meta/classes/overlayfs.bbclass
>
> diff --git a/meta/classes/overlayfs.bbclass b/meta/classes/overlayfs.bbclass
> new file mode 100644
> index 0000000000..7fac3e696e
> --- /dev/null
> +++ b/meta/classes/overlayfs.bbclass
> @@ -0,0 +1,132 @@
> +# Class for generation of overlayfs mount units
> +#
> +# It's often desired in Embedded System design to have a read-only rootfs.
> +# But a lot of different applications might want to have a read-write access to
> +# some parts of a filesystem. It can be especially useful when your update mechanism
> +# overwrites the whole rootfs, but you want your application data to be preserved
> +# between updates. This class provides a way to achieve that by means
> +# of overlayfs and at the same time keeping the base rootfs read-only.
> +#
> +# Usage example.
> +#
> +# Set a mount point for a partition overlayfs is going to use as upper layer
> +# in your machine configuration. Underlying file system can be anything that
> +# is supported by overlayfs
> +#
> +#   OVERLAYFS_MOUNT_POINT[data] ?= "/data"
> +#
> +# The class assumes you have a data.mount systemd unit defined elsewhere in your
> +# BSP and installed to the image.
> +#
> +# Then you can specify writable directories on a recipe base
> +#
> +#   OVERLAYFS_WRITABLE_PATHS[data] = "/usr/share/my-custom-application"
> +#
> +# To support several mount points you can use a different variable flag. Assume we
> +# what to have a writable location on the file system, but not interested where the data
> +# survive a reboot. The we could have a mnt-overlay.mount unit for a tmpfs file system:
> +#
> +#   OVERLAYFS_MOUNT_POINT[mnt-overlay] = "/mnt/overlay"
> +#   OVERLAYFS_WRITABLE_PATHS[mnt-overlay] = "/usr/share/another-application"
> +
> +OVERLAYFS_WRITABLE_PATHS[data] ?= ""
> +
> +inherit systemd
> +
> +def strForBash(s):
> +    return s.replace('\\', '\\\\')
> +
> +def unitFileList(d):
> +    fileList = []
> +    overlayMountPoints = d.getVarFlags("OVERLAYFS_MOUNT_POINT")
> +    for mountPoint in overlayMountPoints:
> +        for path in d.getVarFlag('OVERLAYFS_WRITABLE_PATHS', mountPoint).split():
> +            fileList.append(mountUnitName(path))
> +            fileList.append(helperUnitName(path))
> +
> +    return fileList
> +
> +# this function is based on https://github.com/systemd/systemd/blob/main/src/basic/unit-name.c
> +def escapeSystemdUnitName(path):
> +    escapeMap = {
> +        '/': '-',
> +        '-': "\\x2d",
> +        '\\': "\\x5d"
> +    }
> +    return "".join([escapeMap.get(c, c) for c in path.strip('/')])
> +
> +def mountUnitName(unit):
> +    return escapeSystemdUnitName(unit) + '.mount'
> +
> +def helperUnitName(unit):
> +    return escapeSystemdUnitName(unit) + '-create-upper-dir.service'
> +
> +python do_create_overlayfs_units() {
> +    CreateDirsUnitTemplate = """[Unit]
> +Description=Overlayfs directories setup
> +Requires={DATA_MOUNT_UNIT}
> +After={DATA_MOUNT_UNIT}
> +DefaultDependencies=no
> +
> +[Service]
> +Type=oneshot
> +ExecStart=mkdir -p {DATA_MOUNT_POINT}/workdir{LOWERDIR} && mkdir -p {DATA_MOUNT_POINT}/upper{LOWERDIR}
> +RemainAfterExit=true
> +StandardOutput=journal
> +
> +[Install]
> +WantedBy=multi-user.target
> +"""
> +    MountUnitTemplate = """[Unit]
> +Description=Overlayfs mount unit
> +Requires={CREATE_DIRS_SERVICE}
> +After={CREATE_DIRS_SERVICE}
> +
> +[Mount]
> +What=overlay
> +Where={LOWERDIR}
> +Type=overlay
> +Options=lowerdir={LOWERDIR},upperdir={DATA_MOUNT_POINT}/upper{LOWERDIR},workdir={DATA_MOUNT_POINT}/workdir{LOWERDIR}
> +
> +[Install]
> +WantedBy=multi-user.target
> +"""
> +
> +    def prepareUnits(data, lower):
> +        args = {
> +            'DATA_MOUNT_POINT': data,
> +            'DATA_MOUNT_UNIT': mountUnitName(data),
> +            'CREATE_DIRS_SERVICE': helperUnitName(lower),
> +            'LOWERDIR': lower,
> +        }
> +
> +        with open(os.path.join(d.getVar('WORKDIR'), mountUnitName(lower)), 'w') as f:
> +            f.write(MountUnitTemplate.format(**args))
> +
> +        with open(os.path.join(d.getVar('WORKDIR'), helperUnitName(lower)), 'w') as f:
> +            f.write(CreateDirsUnitTemplate.format(**args))
> +
> +    overlayMountPoints = d.getVarFlags("OVERLAYFS_MOUNT_POINT")
> +    for mountPoint in overlayMountPoints:
> +        for lower in d.getVarFlag('OVERLAYFS_WRITABLE_PATHS', mountPoint).split():
> +            prepareUnits(d.getVarFlag('OVERLAYFS_MOUNT_POINT', mountPoint), lower)
> +}
> +
> +# we need to generate file names early during parsing stage
> +python () {
> +    unitList = unitFileList(d)
> +    for unit in unitList:
> +        d.appendVar('SYSTEMD_SERVICE_' + d.getVar('PN'), ' ' + unit);
> +        d.appendVar('FILES_' + d.getVar('PN'), strForBash(unit))
> +
> +    d.setVar('OVERLAYFS_UNIT_LIST', ' '.join([strForBash(s) for s in unitList]))
> +}
> +
> +do_install_append() {
> +    install -d ${D}${systemd_system_unitdir}
> +    for unit in ${OVERLAYFS_UNIT_LIST}; do
> +        install -m 0444 ${WORKDIR}/${unit} ${D}${systemd_system_unitdir}
> +    done
> +}
> +
> +addtask create_overlayfs_units before do_install
> --
> 2.28.0
>
>
> 
>


-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [OE-core] [PATCH] overlayfs.bbclass: generate overlayfs mount units
  2021-06-03 14:35 ` [OE-core] " Bruce Ashfield
@ 2021-06-04 17:22   ` Vyacheslav Yurkov
  2021-06-04 18:24     ` Bruce Ashfield
  0 siblings, 1 reply; 10+ messages in thread
From: Vyacheslav Yurkov @ 2021-06-04 17:22 UTC (permalink / raw)
  To: Bruce Ashfield; +Cc: Patches and discussions about the oe-core layer

Hi Bruce,
Thanks for a quick feedback.

> Hi Vyacheslav,
>
> This looks like it could be quite useful, but a few comments came to
> mind on a really quick scan.
>
> Since this is systemd based, it should probably have
> REQUIRED_DISTRO_FEATURES = "systemd", to ensure that the rest of the
> system and packages are consistently enabled.

Will include it in my v2.

> As part of getting this into core, it needs some sort of test cases /
> test suite (and one that covers the different modes you describe in
> the class) .. so something like a test image (a user) and the tests
> themselves.

Could you please give me a few hints as to where current tests are 
stored and how to run them? Is it meta-selftest layer?

> It really also needs a maintainer entry, since otherwise the
> maintenance work falls to the same already overloaded folks.

You mean just an entry like:
MAINTAINER = "Vyacheslav Yurkov <uvv.mail@gmail.com>

in the class itself?
Because meta/conf/distro/include/maintainers.inc mentions only recipe 
maintainers.

> Since this is intended to be used on a per-recipe basis (at least that
> is my understanding from reading it), what happens if two recipes
> create conflicting overlay mounts ? or if the mentioned data.mount is
> not present in the image ?

These are good questions. I guess if two recipes would try to use the 
same mounts, then do_rootfs should simply fail the same way when two 
recipes install the same file.
As for handling a mount.unit, that has to be done on an image level, I 
guess. The other question is how to do that.. Perhaps a QA-hook in 
ROOTFS_POSTPROCESS_COMMAND, which checks the file existence (not 
validity though!)?

Vyacheslav

> Bruce
>
>> Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com>
>> ---
>>   meta/classes/overlayfs.bbclass | 132 +++++++++++++++++++++++++++++++++
>>   1 file changed, 132 insertions(+)
>>   create mode 100644 meta/classes/overlayfs.bbclass
>>
>> diff --git a/meta/classes/overlayfs.bbclass b/meta/classes/overlayfs.bbclass
>> new file mode 100644
>> index 0000000000..7fac3e696e
>> --- /dev/null
>> +++ b/meta/classes/overlayfs.bbclass
>> @@ -0,0 +1,132 @@
>> +# Class for generation of overlayfs mount units
>> +#
>> +# It's often desired in Embedded System design to have a read-only rootfs.
>> +# But a lot of different applications might want to have a read-write access to
>> +# some parts of a filesystem. It can be especially useful when your update mechanism
>> +# overwrites the whole rootfs, but you want your application data to be preserved
>> +# between updates. This class provides a way to achieve that by means
>> +# of overlayfs and at the same time keeping the base rootfs read-only.
>> +#
>> +# Usage example.
>> +#
>> +# Set a mount point for a partition overlayfs is going to use as upper layer
>> +# in your machine configuration. Underlying file system can be anything that
>> +# is supported by overlayfs
>> +#
>> +#   OVERLAYFS_MOUNT_POINT[data] ?= "/data"
>> +#
>> +# The class assumes you have a data.mount systemd unit defined elsewhere in your
>> +# BSP and installed to the image.
>> +#
>> +# Then you can specify writable directories on a recipe base
>> +#
>> +#   OVERLAYFS_WRITABLE_PATHS[data] = "/usr/share/my-custom-application"
>> +#
>> +# To support several mount points you can use a different variable flag. Assume we
>> +# what to have a writable location on the file system, but not interested where the data
>> +# survive a reboot. The we could have a mnt-overlay.mount unit for a tmpfs file system:
>> +#
>> +#   OVERLAYFS_MOUNT_POINT[mnt-overlay] = "/mnt/overlay"
>> +#   OVERLAYFS_WRITABLE_PATHS[mnt-overlay] = "/usr/share/another-application"
>> +
>> +OVERLAYFS_WRITABLE_PATHS[data] ?= ""
>> +
>> +inherit systemd
>> +
>> +def strForBash(s):
>> +    return s.replace('\\', '\\\\')
>> +
>> +def unitFileList(d):
>> +    fileList = []
>> +    overlayMountPoints = d.getVarFlags("OVERLAYFS_MOUNT_POINT")
>> +    for mountPoint in overlayMountPoints:
>> +        for path in d.getVarFlag('OVERLAYFS_WRITABLE_PATHS', mountPoint).split():
>> +            fileList.append(mountUnitName(path))
>> +            fileList.append(helperUnitName(path))
>> +
>> +    return fileList
>> +
>> +# this function is based on https://github.com/systemd/systemd/blob/main/src/basic/unit-name.c
>> +def escapeSystemdUnitName(path):
>> +    escapeMap = {
>> +        '/': '-',
>> +        '-': "\\x2d",
>> +        '\\': "\\x5d"
>> +    }
>> +    return "".join([escapeMap.get(c, c) for c in path.strip('/')])
>> +
>> +def mountUnitName(unit):
>> +    return escapeSystemdUnitName(unit) + '.mount'
>> +
>> +def helperUnitName(unit):
>> +    return escapeSystemdUnitName(unit) + '-create-upper-dir.service'
>> +
>> +python do_create_overlayfs_units() {
>> +    CreateDirsUnitTemplate = """[Unit]
>> +Description=Overlayfs directories setup
>> +Requires={DATA_MOUNT_UNIT}
>> +After={DATA_MOUNT_UNIT}
>> +DefaultDependencies=no
>> +
>> +[Service]
>> +Type=oneshot
>> +ExecStart=mkdir -p {DATA_MOUNT_POINT}/workdir{LOWERDIR} && mkdir -p {DATA_MOUNT_POINT}/upper{LOWERDIR}
>> +RemainAfterExit=true
>> +StandardOutput=journal
>> +
>> +[Install]
>> +WantedBy=multi-user.target
>> +"""
>> +    MountUnitTemplate = """[Unit]
>> +Description=Overlayfs mount unit
>> +Requires={CREATE_DIRS_SERVICE}
>> +After={CREATE_DIRS_SERVICE}
>> +
>> +[Mount]
>> +What=overlay
>> +Where={LOWERDIR}
>> +Type=overlay
>> +Options=lowerdir={LOWERDIR},upperdir={DATA_MOUNT_POINT}/upper{LOWERDIR},workdir={DATA_MOUNT_POINT}/workdir{LOWERDIR}
>> +
>> +[Install]
>> +WantedBy=multi-user.target
>> +"""
>> +
>> +    def prepareUnits(data, lower):
>> +        args = {
>> +            'DATA_MOUNT_POINT': data,
>> +            'DATA_MOUNT_UNIT': mountUnitName(data),
>> +            'CREATE_DIRS_SERVICE': helperUnitName(lower),
>> +            'LOWERDIR': lower,
>> +        }
>> +
>> +        with open(os.path.join(d.getVar('WORKDIR'), mountUnitName(lower)), 'w') as f:
>> +            f.write(MountUnitTemplate.format(**args))
>> +
>> +        with open(os.path.join(d.getVar('WORKDIR'), helperUnitName(lower)), 'w') as f:
>> +            f.write(CreateDirsUnitTemplate.format(**args))
>> +
>> +    overlayMountPoints = d.getVarFlags("OVERLAYFS_MOUNT_POINT")
>> +    for mountPoint in overlayMountPoints:
>> +        for lower in d.getVarFlag('OVERLAYFS_WRITABLE_PATHS', mountPoint).split():
>> +            prepareUnits(d.getVarFlag('OVERLAYFS_MOUNT_POINT', mountPoint), lower)
>> +}
>> +
>> +# we need to generate file names early during parsing stage
>> +python () {
>> +    unitList = unitFileList(d)
>> +    for unit in unitList:
>> +        d.appendVar('SYSTEMD_SERVICE_' + d.getVar('PN'), ' ' + unit);
>> +        d.appendVar('FILES_' + d.getVar('PN'), strForBash(unit))
>> +
>> +    d.setVar('OVERLAYFS_UNIT_LIST', ' '.join([strForBash(s) for s in unitList]))
>> +}
>> +
>> +do_install_append() {
>> +    install -d ${D}${systemd_system_unitdir}
>> +    for unit in ${OVERLAYFS_UNIT_LIST}; do
>> +        install -m 0444 ${WORKDIR}/${unit} ${D}${systemd_system_unitdir}
>> +    done
>> +}
>> +
>> +addtask create_overlayfs_units before do_install
>> --
>> 2.28.0
>>
>>
>> 
>>


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [OE-core] [PATCH] overlayfs.bbclass: generate overlayfs mount units
  2021-06-04 17:22   ` Vyacheslav Yurkov
@ 2021-06-04 18:24     ` Bruce Ashfield
  0 siblings, 0 replies; 10+ messages in thread
From: Bruce Ashfield @ 2021-06-04 18:24 UTC (permalink / raw)
  To: Vyacheslav Yurkov, Richard Purdie
  Cc: Patches and discussions about the oe-core layer

On Fri, Jun 4, 2021 at 1:22 PM Vyacheslav Yurkov <uvv.mail@gmail.com> wrote:
>
> Hi Bruce,
> Thanks for a quick feedback.
>
> > Hi Vyacheslav,
> >
> > This looks like it could be quite useful, but a few comments came to
> > mind on a really quick scan.
> >
> > Since this is systemd based, it should probably have
> > REQUIRED_DISTRO_FEATURES = "systemd", to ensure that the rest of the
> > system and packages are consistently enabled.
>
> Will include it in my v2.
>
> > As part of getting this into core, it needs some sort of test cases /
> > test suite (and one that covers the different modes you describe in
> > the class) .. so something like a test image (a user) and the tests
> > themselves.
>
> Could you please give me a few hints as to where current tests are
> stored and how to run them? Is it meta-selftest layer?

Richard can guide you better on this front than I can, but the tests
can be in a number of forms. ptests (i.e. if you create the reference
image that uses it, that recipe could have a ptest), and there are also
the lib/oeqa/selftest/ tests.

How they are fully hooked in, so they can be tested with core, I haven't
looked in a while, and might steer you wrong, So my recommendations
are to checkout those components, and follow what is on the wiki and
in the docs (https://wiki.yoctoproject.org/wiki/Oe-selftest),
https://docs.yoctoproject.org/test-manual/intro.html, etc

>
> > It really also needs a maintainer entry, since otherwise the
> > maintenance work falls to the same already overloaded folks.
>
> You mean just an entry like:
> MAINTAINER = "Vyacheslav Yurkov <uvv.mail@gmail.com>

Yep.

>
> in the class itself?
> Because meta/conf/distro/include/maintainers.inc mentions only recipe
> maintainers.

Richard could advise better on this, but yes, even if this isn't a recipe
we'd want someone to look after it if it merges to core. So if we are
breaking into a new category here, that is probably ok.

>
> > Since this is intended to be used on a per-recipe basis (at least that
> > is my understanding from reading it), what happens if two recipes
> > create conflicting overlay mounts ? or if the mentioned data.mount is
> > not present in the image ?
>
> These are good questions. I guess if two recipes would try to use the
> same mounts, then do_rootfs should simply fail the same way when two
> recipes install the same file.

Yes, the image would definitely conflict during package install in that
case. It wasn't entirely clear to me that they'd have to have exactly the
same file.

> As for handling a mount.unit, that has to be done on an image level, I
> guess. The other question is how to do that.. Perhaps a QA-hook in
> ROOTFS_POSTPROCESS_COMMAND, which checks the file existence (not
> validity though!)?

That might be feasible, assuming that it doesn't always run, or if it
always run, that it doesn't take much time.

Cheers,

Bruce

>
> Vyacheslav
>
> > Bruce
> >
> >> Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com>
> >> ---
> >>   meta/classes/overlayfs.bbclass | 132 +++++++++++++++++++++++++++++++++
> >>   1 file changed, 132 insertions(+)
> >>   create mode 100644 meta/classes/overlayfs.bbclass
> >>
> >> diff --git a/meta/classes/overlayfs.bbclass b/meta/classes/overlayfs.bbclass
> >> new file mode 100644
> >> index 0000000000..7fac3e696e
> >> --- /dev/null
> >> +++ b/meta/classes/overlayfs.bbclass
> >> @@ -0,0 +1,132 @@
> >> +# Class for generation of overlayfs mount units
> >> +#
> >> +# It's often desired in Embedded System design to have a read-only rootfs.
> >> +# But a lot of different applications might want to have a read-write access to
> >> +# some parts of a filesystem. It can be especially useful when your update mechanism
> >> +# overwrites the whole rootfs, but you want your application data to be preserved
> >> +# between updates. This class provides a way to achieve that by means
> >> +# of overlayfs and at the same time keeping the base rootfs read-only.
> >> +#
> >> +# Usage example.
> >> +#
> >> +# Set a mount point for a partition overlayfs is going to use as upper layer
> >> +# in your machine configuration. Underlying file system can be anything that
> >> +# is supported by overlayfs
> >> +#
> >> +#   OVERLAYFS_MOUNT_POINT[data] ?= "/data"
> >> +#
> >> +# The class assumes you have a data.mount systemd unit defined elsewhere in your
> >> +# BSP and installed to the image.
> >> +#
> >> +# Then you can specify writable directories on a recipe base
> >> +#
> >> +#   OVERLAYFS_WRITABLE_PATHS[data] = "/usr/share/my-custom-application"
> >> +#
> >> +# To support several mount points you can use a different variable flag. Assume we
> >> +# what to have a writable location on the file system, but not interested where the data
> >> +# survive a reboot. The we could have a mnt-overlay.mount unit for a tmpfs file system:
> >> +#
> >> +#   OVERLAYFS_MOUNT_POINT[mnt-overlay] = "/mnt/overlay"
> >> +#   OVERLAYFS_WRITABLE_PATHS[mnt-overlay] = "/usr/share/another-application"
> >> +
> >> +OVERLAYFS_WRITABLE_PATHS[data] ?= ""
> >> +
> >> +inherit systemd
> >> +
> >> +def strForBash(s):
> >> +    return s.replace('\\', '\\\\')
> >> +
> >> +def unitFileList(d):
> >> +    fileList = []
> >> +    overlayMountPoints = d.getVarFlags("OVERLAYFS_MOUNT_POINT")
> >> +    for mountPoint in overlayMountPoints:
> >> +        for path in d.getVarFlag('OVERLAYFS_WRITABLE_PATHS', mountPoint).split():
> >> +            fileList.append(mountUnitName(path))
> >> +            fileList.append(helperUnitName(path))
> >> +
> >> +    return fileList
> >> +
> >> +# this function is based on https://github.com/systemd/systemd/blob/main/src/basic/unit-name.c
> >> +def escapeSystemdUnitName(path):
> >> +    escapeMap = {
> >> +        '/': '-',
> >> +        '-': "\\x2d",
> >> +        '\\': "\\x5d"
> >> +    }
> >> +    return "".join([escapeMap.get(c, c) for c in path.strip('/')])
> >> +
> >> +def mountUnitName(unit):
> >> +    return escapeSystemdUnitName(unit) + '.mount'
> >> +
> >> +def helperUnitName(unit):
> >> +    return escapeSystemdUnitName(unit) + '-create-upper-dir.service'
> >> +
> >> +python do_create_overlayfs_units() {
> >> +    CreateDirsUnitTemplate = """[Unit]
> >> +Description=Overlayfs directories setup
> >> +Requires={DATA_MOUNT_UNIT}
> >> +After={DATA_MOUNT_UNIT}
> >> +DefaultDependencies=no
> >> +
> >> +[Service]
> >> +Type=oneshot
> >> +ExecStart=mkdir -p {DATA_MOUNT_POINT}/workdir{LOWERDIR} && mkdir -p {DATA_MOUNT_POINT}/upper{LOWERDIR}
> >> +RemainAfterExit=true
> >> +StandardOutput=journal
> >> +
> >> +[Install]
> >> +WantedBy=multi-user.target
> >> +"""
> >> +    MountUnitTemplate = """[Unit]
> >> +Description=Overlayfs mount unit
> >> +Requires={CREATE_DIRS_SERVICE}
> >> +After={CREATE_DIRS_SERVICE}
> >> +
> >> +[Mount]
> >> +What=overlay
> >> +Where={LOWERDIR}
> >> +Type=overlay
> >> +Options=lowerdir={LOWERDIR},upperdir={DATA_MOUNT_POINT}/upper{LOWERDIR},workdir={DATA_MOUNT_POINT}/workdir{LOWERDIR}
> >> +
> >> +[Install]
> >> +WantedBy=multi-user.target
> >> +"""
> >> +
> >> +    def prepareUnits(data, lower):
> >> +        args = {
> >> +            'DATA_MOUNT_POINT': data,
> >> +            'DATA_MOUNT_UNIT': mountUnitName(data),
> >> +            'CREATE_DIRS_SERVICE': helperUnitName(lower),
> >> +            'LOWERDIR': lower,
> >> +        }
> >> +
> >> +        with open(os.path.join(d.getVar('WORKDIR'), mountUnitName(lower)), 'w') as f:
> >> +            f.write(MountUnitTemplate.format(**args))
> >> +
> >> +        with open(os.path.join(d.getVar('WORKDIR'), helperUnitName(lower)), 'w') as f:
> >> +            f.write(CreateDirsUnitTemplate.format(**args))
> >> +
> >> +    overlayMountPoints = d.getVarFlags("OVERLAYFS_MOUNT_POINT")
> >> +    for mountPoint in overlayMountPoints:
> >> +        for lower in d.getVarFlag('OVERLAYFS_WRITABLE_PATHS', mountPoint).split():
> >> +            prepareUnits(d.getVarFlag('OVERLAYFS_MOUNT_POINT', mountPoint), lower)
> >> +}
> >> +
> >> +# we need to generate file names early during parsing stage
> >> +python () {
> >> +    unitList = unitFileList(d)
> >> +    for unit in unitList:
> >> +        d.appendVar('SYSTEMD_SERVICE_' + d.getVar('PN'), ' ' + unit);
> >> +        d.appendVar('FILES_' + d.getVar('PN'), strForBash(unit))
> >> +
> >> +    d.setVar('OVERLAYFS_UNIT_LIST', ' '.join([strForBash(s) for s in unitList]))
> >> +}
> >> +
> >> +do_install_append() {
> >> +    install -d ${D}${systemd_system_unitdir}
> >> +    for unit in ${OVERLAYFS_UNIT_LIST}; do
> >> +        install -m 0444 ${WORKDIR}/${unit} ${D}${systemd_system_unitdir}
> >> +    done
> >> +}
> >> +
> >> +addtask create_overlayfs_units before do_install
> >> --
> >> 2.28.0
> >>
> >>
> >> 
> >>
>


--
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [OE-core] [PATCH] overlayfs.bbclass: generate overlayfs mount units
  2021-06-03 14:21 [PATCH] overlayfs.bbclass: generate overlayfs mount units Vyacheslav Yurkov
  2021-06-03 14:35 ` [OE-core] " Bruce Ashfield
@ 2021-06-04 20:20 ` Ayoub Zaki
  2021-06-05  6:31   ` Vyacheslav Yurkov
  2021-06-05  7:37 ` Adrian Freihofer
  2021-06-09 13:56 ` Andrei Gherzan
  3 siblings, 1 reply; 10+ messages in thread
From: Ayoub Zaki @ 2021-06-04 20:20 UTC (permalink / raw)
  To: openembedded-core

[-- Attachment #1: Type: text/plain, Size: 6542 bytes --]

Hi,


volatile-binds (maybe the *volatile* naming is misleading) recipe in 
yocto is already doing that, just to be appended .ie:

VOLATILE_BINDS += "\
/data/lib/systemd/network /lib/systemd/network\n\
/data/home/root /home/root\n\
"

It will generate for each overlay a mount point service.

BR


On 6/3/21 4:21 PM, Vyacheslav Yurkov wrote:
> It's often desired in Embedded System design to have a read-only rootfs.
> But a lot of different applications might want to have a read-write access
> to some parts of a filesystem. It can be especially useful when your update
> mechanism overwrites the whole rootfs, but you want your application data
> to be preserved between updates. This class provides a way to achieve that
> by means of overlayfs and at the same time keeping the base rootfs read-only.
>
> Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com>
> ---
>   meta/classes/overlayfs.bbclass | 132 +++++++++++++++++++++++++++++++++
>   1 file changed, 132 insertions(+)
>   create mode 100644 meta/classes/overlayfs.bbclass
>
> diff --git a/meta/classes/overlayfs.bbclass b/meta/classes/overlayfs.bbclass
> new file mode 100644
> index 0000000000..7fac3e696e
> --- /dev/null
> +++ b/meta/classes/overlayfs.bbclass
> @@ -0,0 +1,132 @@
> +# Class for generation of overlayfs mount units
> +#
> +# It's often desired in Embedded System design to have a read-only rootfs.
> +# But a lot of different applications might want to have a read-write access to
> +# some parts of a filesystem. It can be especially useful when your update mechanism
> +# overwrites the whole rootfs, but you want your application data to be preserved
> +# between updates. This class provides a way to achieve that by means
> +# of overlayfs and at the same time keeping the base rootfs read-only.
> +#
> +# Usage example.
> +#
> +# Set a mount point for a partition overlayfs is going to use as upper layer
> +# in your machine configuration. Underlying file system can be anything that
> +# is supported by overlayfs
> +#
> +#   OVERLAYFS_MOUNT_POINT[data] ?= "/data"
> +#
> +# The class assumes you have a data.mount systemd unit defined elsewhere in your
> +# BSP and installed to the image.
> +#
> +# Then you can specify writable directories on a recipe base
> +#
> +#   OVERLAYFS_WRITABLE_PATHS[data] = "/usr/share/my-custom-application"
> +#
> +# To support several mount points you can use a different variable flag. Assume we
> +# what to have a writable location on the file system, but not interested where the data
> +# survive a reboot. The we could have a mnt-overlay.mount unit for a tmpfs file system:
> +#
> +#   OVERLAYFS_MOUNT_POINT[mnt-overlay] = "/mnt/overlay"
> +#   OVERLAYFS_WRITABLE_PATHS[mnt-overlay] = "/usr/share/another-application"
> +
> +OVERLAYFS_WRITABLE_PATHS[data] ?= ""
> +
> +inherit systemd
> +
> +def strForBash(s):
> +    return s.replace('\\', '\\\\')
> +
> +def unitFileList(d):
> +    fileList = []
> +    overlayMountPoints = d.getVarFlags("OVERLAYFS_MOUNT_POINT")
> +    for mountPoint in overlayMountPoints:
> +        for path in d.getVarFlag('OVERLAYFS_WRITABLE_PATHS', mountPoint).split():
> +            fileList.append(mountUnitName(path))
> +            fileList.append(helperUnitName(path))
> +
> +    return fileList
> +
> +# this function is based on https://github.com/systemd/systemd/blob/main/src/basic/unit-name.c
> +def escapeSystemdUnitName(path):
> +    escapeMap = {
> +        '/': '-',
> +        '-': "\\x2d",
> +        '\\': "\\x5d"
> +    }
> +    return "".join([escapeMap.get(c, c) for c in path.strip('/')])
> +
> +def mountUnitName(unit):
> +    return escapeSystemdUnitName(unit) + '.mount'
> +
> +def helperUnitName(unit):
> +    return escapeSystemdUnitName(unit) + '-create-upper-dir.service'
> +
> +python do_create_overlayfs_units() {
> +    CreateDirsUnitTemplate = """[Unit]
> +Description=Overlayfs directories setup
> +Requires={DATA_MOUNT_UNIT}
> +After={DATA_MOUNT_UNIT}
> +DefaultDependencies=no
> +
> +[Service]
> +Type=oneshot
> +ExecStart=mkdir -p {DATA_MOUNT_POINT}/workdir{LOWERDIR} && mkdir -p {DATA_MOUNT_POINT}/upper{LOWERDIR}
> +RemainAfterExit=true
> +StandardOutput=journal
> +
> +[Install]
> +WantedBy=multi-user.target
> +"""
> +    MountUnitTemplate = """[Unit]
> +Description=Overlayfs mount unit
> +Requires={CREATE_DIRS_SERVICE}
> +After={CREATE_DIRS_SERVICE}
> +
> +[Mount]
> +What=overlay
> +Where={LOWERDIR}
> +Type=overlay
> +Options=lowerdir={LOWERDIR},upperdir={DATA_MOUNT_POINT}/upper{LOWERDIR},workdir={DATA_MOUNT_POINT}/workdir{LOWERDIR}
> +
> +[Install]
> +WantedBy=multi-user.target
> +"""
> +
> +    def prepareUnits(data, lower):
> +        args = {
> +            'DATA_MOUNT_POINT': data,
> +            'DATA_MOUNT_UNIT': mountUnitName(data),
> +            'CREATE_DIRS_SERVICE': helperUnitName(lower),
> +            'LOWERDIR': lower,
> +        }
> +
> +        with open(os.path.join(d.getVar('WORKDIR'), mountUnitName(lower)), 'w') as f:
> +            f.write(MountUnitTemplate.format(**args))
> +
> +        with open(os.path.join(d.getVar('WORKDIR'), helperUnitName(lower)), 'w') as f:
> +            f.write(CreateDirsUnitTemplate.format(**args))
> +
> +    overlayMountPoints = d.getVarFlags("OVERLAYFS_MOUNT_POINT")
> +    for mountPoint in overlayMountPoints:
> +        for lower in d.getVarFlag('OVERLAYFS_WRITABLE_PATHS', mountPoint).split():
> +            prepareUnits(d.getVarFlag('OVERLAYFS_MOUNT_POINT', mountPoint), lower)
> +}
> +
> +# we need to generate file names early during parsing stage
> +python () {
> +    unitList = unitFileList(d)
> +    for unit in unitList:
> +        d.appendVar('SYSTEMD_SERVICE_' + d.getVar('PN'), ' ' + unit);
> +        d.appendVar('FILES_' + d.getVar('PN'), strForBash(unit))
> +
> +    d.setVar('OVERLAYFS_UNIT_LIST', ' '.join([strForBash(s) for s in unitList]))
> +}
> +
> +do_install_append() {
> +    install -d ${D}${systemd_system_unitdir}
> +    for unit in ${OVERLAYFS_UNIT_LIST}; do
> +        install -m 0444 ${WORKDIR}/${unit} ${D}${systemd_system_unitdir}
> +    done
> +}
> +
> +addtask create_overlayfs_units before do_install
>
> 
>
Mit freundlichen Grüßen / Kind regards

-- 
Ayoub Zaki
Embedded Systems Consultant

Vaihinger Straße 2/1
D-71634 Ludwigsburg

Mobile   : +4917662901545
Email    : ayoub.zaki@embexus.com
Homepage : https://embexus.com
VAT No.  : DE313902634


[-- Attachment #2: Type: text/html, Size: 7491 bytes --]

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [OE-core] [PATCH] overlayfs.bbclass: generate overlayfs mount units
  2021-06-04 20:20 ` Ayoub Zaki
@ 2021-06-05  6:31   ` Vyacheslav Yurkov
  0 siblings, 0 replies; 10+ messages in thread
From: Vyacheslav Yurkov @ 2021-06-05  6:31 UTC (permalink / raw)
  To: Ayoub Zaki, openembedded-core

[-- Attachment #1: Type: text/plain, Size: 7357 bytes --]

Hi Ayoub,
Thanks for the hint. I knew the recipe exists, but I didn't know it uses 
overlay.

I did have a closer look at the mount-copybind script, and it uses 
overlay indeed, but it seems it's used for the other purpose. Originally 
the recipe provided only bind-mount functionality, they added overlayfs 
support later to save on space.

I tried to use the recipe with my use case, but it turns out that 
reverses the options of mount command in mount-copybind. I'm also not 
sure how the dependencies should be injected into the mount unit.

Vyacheslav

On 04.06.2021 22:20, Ayoub Zaki wrote:
>
> Hi,
>
>
> volatile-binds (maybe the *volatile* naming is misleading) recipe in 
> yocto is already doing that, just to be appended .ie:
>
> VOLATILE_BINDS += "\
> /data/lib/systemd/network /lib/systemd/network\n\
> /data/home/root /home/root\n\
> "
>
> It will generate for each overlay a mount point service.
>
> BR
>
> On 6/3/21 4:21 PM, Vyacheslav Yurkov wrote:
>> It's often desired in Embedded System design to have a read-only rootfs.
>> But a lot of different applications might want to have a read-write access
>> to some parts of a filesystem. It can be especially useful when your update
>> mechanism overwrites the whole rootfs, but you want your application data
>> to be preserved between updates. This class provides a way to achieve that
>> by means of overlayfs and at the same time keeping the base rootfs read-only.
>>
>> Signed-off-by: Vyacheslav Yurkov<uvv.mail@gmail.com>
>> ---
>>   meta/classes/overlayfs.bbclass | 132 +++++++++++++++++++++++++++++++++
>>   1 file changed, 132 insertions(+)
>>   create mode 100644 meta/classes/overlayfs.bbclass
>>
>> diff --git a/meta/classes/overlayfs.bbclass b/meta/classes/overlayfs.bbclass
>> new file mode 100644
>> index 0000000000..7fac3e696e
>> --- /dev/null
>> +++ b/meta/classes/overlayfs.bbclass
>> @@ -0,0 +1,132 @@
>> +# Class for generation of overlayfs mount units
>> +#
>> +# It's often desired in Embedded System design to have a read-only rootfs.
>> +# But a lot of different applications might want to have a read-write access to
>> +# some parts of a filesystem. It can be especially useful when your update mechanism
>> +# overwrites the whole rootfs, but you want your application data to be preserved
>> +# between updates. This class provides a way to achieve that by means
>> +# of overlayfs and at the same time keeping the base rootfs read-only.
>> +#
>> +# Usage example.
>> +#
>> +# Set a mount point for a partition overlayfs is going to use as upper layer
>> +# in your machine configuration. Underlying file system can be anything that
>> +# is supported by overlayfs
>> +#
>> +#   OVERLAYFS_MOUNT_POINT[data] ?= "/data"
>> +#
>> +# The class assumes you have a data.mount systemd unit defined elsewhere in your
>> +# BSP and installed to the image.
>> +#
>> +# Then you can specify writable directories on a recipe base
>> +#
>> +#   OVERLAYFS_WRITABLE_PATHS[data] = "/usr/share/my-custom-application"
>> +#
>> +# To support several mount points you can use a different variable flag. Assume we
>> +# what to have a writable location on the file system, but not interested where the data
>> +# survive a reboot. The we could have a mnt-overlay.mount unit for a tmpfs file system:
>> +#
>> +#   OVERLAYFS_MOUNT_POINT[mnt-overlay] = "/mnt/overlay"
>> +#   OVERLAYFS_WRITABLE_PATHS[mnt-overlay] = "/usr/share/another-application"
>> +
>> +OVERLAYFS_WRITABLE_PATHS[data] ?= ""
>> +
>> +inherit systemd
>> +
>> +def strForBash(s):
>> +    return s.replace('\\', '\\\\')
>> +
>> +def unitFileList(d):
>> +    fileList = []
>> +    overlayMountPoints = d.getVarFlags("OVERLAYFS_MOUNT_POINT")
>> +    for mountPoint in overlayMountPoints:
>> +        for path in d.getVarFlag('OVERLAYFS_WRITABLE_PATHS', mountPoint).split():
>> +            fileList.append(mountUnitName(path))
>> +            fileList.append(helperUnitName(path))
>> +
>> +    return fileList
>> +
>> +# this function is based onhttps://github.com/systemd/systemd/blob/main/src/basic/unit-name.c
>> +def escapeSystemdUnitName(path):
>> +    escapeMap = {
>> +        '/': '-',
>> +        '-': "\\x2d",
>> +        '\\': "\\x5d"
>> +    }
>> +    return "".join([escapeMap.get(c, c) for c in path.strip('/')])
>> +
>> +def mountUnitName(unit):
>> +    return escapeSystemdUnitName(unit) + '.mount'
>> +
>> +def helperUnitName(unit):
>> +    return escapeSystemdUnitName(unit) + '-create-upper-dir.service'
>> +
>> +python do_create_overlayfs_units() {
>> +    CreateDirsUnitTemplate = """[Unit]
>> +Description=Overlayfs directories setup
>> +Requires={DATA_MOUNT_UNIT}
>> +After={DATA_MOUNT_UNIT}
>> +DefaultDependencies=no
>> +
>> +[Service]
>> +Type=oneshot
>> +ExecStart=mkdir -p {DATA_MOUNT_POINT}/workdir{LOWERDIR} && mkdir -p {DATA_MOUNT_POINT}/upper{LOWERDIR}
>> +RemainAfterExit=true
>> +StandardOutput=journal
>> +
>> +[Install]
>> +WantedBy=multi-user.target
>> +"""
>> +    MountUnitTemplate = """[Unit]
>> +Description=Overlayfs mount unit
>> +Requires={CREATE_DIRS_SERVICE}
>> +After={CREATE_DIRS_SERVICE}
>> +
>> +[Mount]
>> +What=overlay
>> +Where={LOWERDIR}
>> +Type=overlay
>> +Options=lowerdir={LOWERDIR},upperdir={DATA_MOUNT_POINT}/upper{LOWERDIR},workdir={DATA_MOUNT_POINT}/workdir{LOWERDIR}
>> +
>> +[Install]
>> +WantedBy=multi-user.target
>> +"""
>> +
>> +    def prepareUnits(data, lower):
>> +        args = {
>> +            'DATA_MOUNT_POINT': data,
>> +            'DATA_MOUNT_UNIT': mountUnitName(data),
>> +            'CREATE_DIRS_SERVICE': helperUnitName(lower),
>> +            'LOWERDIR': lower,
>> +        }
>> +
>> +        with open(os.path.join(d.getVar('WORKDIR'), mountUnitName(lower)), 'w') as f:
>> +            f.write(MountUnitTemplate.format(**args))
>> +
>> +        with open(os.path.join(d.getVar('WORKDIR'), helperUnitName(lower)), 'w') as f:
>> +            f.write(CreateDirsUnitTemplate.format(**args))
>> +
>> +    overlayMountPoints = d.getVarFlags("OVERLAYFS_MOUNT_POINT")
>> +    for mountPoint in overlayMountPoints:
>> +        for lower in d.getVarFlag('OVERLAYFS_WRITABLE_PATHS', mountPoint).split():
>> +            prepareUnits(d.getVarFlag('OVERLAYFS_MOUNT_POINT', mountPoint), lower)
>> +}
>> +
>> +# we need to generate file names early during parsing stage
>> +python () {
>> +    unitList = unitFileList(d)
>> +    for unit in unitList:
>> +        d.appendVar('SYSTEMD_SERVICE_' + d.getVar('PN'), ' ' + unit);
>> +        d.appendVar('FILES_' + d.getVar('PN'), strForBash(unit))
>> +
>> +    d.setVar('OVERLAYFS_UNIT_LIST', ' '.join([strForBash(s) for s in unitList]))
>> +}
>> +
>> +do_install_append() {
>> +    install -d ${D}${systemd_system_unitdir}
>> +    for unit in ${OVERLAYFS_UNIT_LIST}; do
>> +        install -m 0444 ${WORKDIR}/${unit} ${D}${systemd_system_unitdir}
>> +    done
>> +}
>> +
>> +addtask create_overlayfs_units before do_install
>>
> Mit freundlichen Grüßen / Kind regards
>
> -- 
> Ayoub Zaki
> Embedded Systems Consultant
>
> Vaihinger Straße 2/1
> D-71634 Ludwigsburg
>
> Mobile   : +4917662901545
> Email    :ayoub.zaki@embexus.com
> Homepage :https://embexus.com
> VAT No.  : DE313902634
>
> 
>


[-- Attachment #2: Type: text/html, Size: 8686 bytes --]

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [OE-core] [PATCH] overlayfs.bbclass: generate overlayfs mount units
  2021-06-03 14:21 [PATCH] overlayfs.bbclass: generate overlayfs mount units Vyacheslav Yurkov
  2021-06-03 14:35 ` [OE-core] " Bruce Ashfield
  2021-06-04 20:20 ` Ayoub Zaki
@ 2021-06-05  7:37 ` Adrian Freihofer
  2021-06-05  9:28   ` Vyacheslav Yurkov
  2021-06-09 13:56 ` Andrei Gherzan
  3 siblings, 1 reply; 10+ messages in thread
From: Adrian Freihofer @ 2021-06-05  7:37 UTC (permalink / raw)
  To: Vyacheslav Yurkov, Openembedded-core

Hi Vyacheslav

It would be very nice to have such a feature in Yocto.

If I remember correctly I tried to do that once with an overlay on
/etc. However, that did not work with systemd. Did you ever try to add
/etc as a mount point?

On Thu, 2021-06-03 at 16:21 +0200, Vyacheslav Yurkov wrote:
> It's often desired in Embedded System design to have a read-only rootfs.
> But a lot of different applications might want to have a read-write access
> to some parts of a filesystem. It can be especially useful when your update
> mechanism overwrites the whole rootfs, but you want your application data
> to be preserved between updates. This class provides a way to achieve that
> by means of overlayfs and at the same time keeping the base rootfs read-only.
> 
> Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com>
> ---
>  meta/classes/overlayfs.bbclass | 132 +++++++++++++++++++++++++++++++++
>  1 file changed, 132 insertions(+)
>  create mode 100644 meta/classes/overlayfs.bbclass
> 
> diff --git a/meta/classes/overlayfs.bbclass b/meta/classes/overlayfs.bbclass
> new file mode 100644
> index 0000000000..7fac3e696e
> --- /dev/null
> +++ b/meta/classes/overlayfs.bbclass
> @@ -0,0 +1,132 @@
> +# Class for generation of overlayfs mount units
> +#
> +# It's often desired in Embedded System design to have a read-only rootfs.
> +# But a lot of different applications might want to have a read-write access to
> +# some parts of a filesystem. It can be especially useful when your update mechanism
> +# overwrites the whole rootfs, but you want your application data to be preserved
> +# between updates. This class provides a way to achieve that by means
> +# of overlayfs and at the same time keeping the base rootfs read-only.
> +#
> +# Usage example.
> +#
> +# Set a mount point for a partition overlayfs is going to use as upper layer
> +# in your machine configuration. Underlying file system can be anything that
> +# is supported by overlayfs
> +#
> +#   OVERLAYFS_MOUNT_POINT[data] ?= "/data"
> +#
> +# The class assumes you have a data.mount systemd unit defined elsewhere in your
> +# BSP and installed to the image.
> +#
> +# Then you can specify writable directories on a recipe base
> +#
> +#   OVERLAYFS_WRITABLE_PATHS[data] = "/usr/share/my-custom-application"
> +#
> +# To support several mount points you can use a different variable flag. Assume we
> +# what to have a writable location on the file system, but not interested where the data
> +# survive a reboot. The we could have a mnt-overlay.mount unit for a tmpfs file system:
> +#
> +#   OVERLAYFS_MOUNT_POINT[mnt-overlay] = "/mnt/overlay"
> +#   OVERLAYFS_WRITABLE_PATHS[mnt-overlay] = "/usr/share/another-application"
> +
> +OVERLAYFS_WRITABLE_PATHS[data] ?= ""
> +
> +inherit systemd
> +
> +def strForBash(s):
> +    return s.replace('\\', '\\\\')
> +
> +def unitFileList(d):
> +    fileList = []
> +    overlayMountPoints = d.getVarFlags("OVERLAYFS_MOUNT_POINT")
> +    for mountPoint in overlayMountPoints:
> +        for path in d.getVarFlag('OVERLAYFS_WRITABLE_PATHS', mountPoint).split():
> +            fileList.append(mountUnitName(path))
> +            fileList.append(helperUnitName(path))
> +
> +    return fileList
> +
> +# this function is based on https://github.com/systemd/systemd/blob/main/src/basic/unit-name.c
> +def escapeSystemdUnitName(path):
> +    escapeMap = {
> +        '/': '-',
> +        '-': "\\x2d",
> +        '\\': "\\x5d"
> +    }
> +    return "".join([escapeMap.get(c, c) for c in path.strip('/')])
> +
> +def mountUnitName(unit):
> +    return escapeSystemdUnitName(unit) + '.mount'
> +
> +def helperUnitName(unit):
> +    return escapeSystemdUnitName(unit) + '-create-upper-dir.service'
> +
> +python do_create_overlayfs_units() {
> +    CreateDirsUnitTemplate = """[Unit]
> +Description=Overlayfs directories setup
> +Requires={DATA_MOUNT_UNIT}
> +After={DATA_MOUNT_UNIT}
> +DefaultDependencies=no
> +
> +[Service]
> +Type=oneshot
> +ExecStart=mkdir -p {DATA_MOUNT_POINT}/workdir{LOWERDIR} && mkdir -p {DATA_MOUNT_POINT}/upper{LOWERDIR}
> +RemainAfterExit=true
> +StandardOutput=journal
> +
> +[Install]
> +WantedBy=multi-user.target
The mounting of overlays should probably happen very early at boot,
before services start using the desired folders.

Regards,
Adrian

> +"""
> +    MountUnitTemplate = """[Unit]
> +Description=Overlayfs mount unit
> +Requires={CREATE_DIRS_SERVICE}
> +After={CREATE_DIRS_SERVICE}
> +
> +[Mount]
> +What=overlay
> +Where={LOWERDIR}
> +Type=overlay
> +Options=lowerdir={LOWERDIR},upperdir={DATA_MOUNT_POINT}/upper{LOWERDIR},workdir={DATA_MOUNT_POINT}/workdir{LOWERDIR}
> +
> +[Install]
> +WantedBy=multi-user.target
> +"""
> +
> +    def prepareUnits(data, lower):
> +        args = {
> +            'DATA_MOUNT_POINT': data,
> +            'DATA_MOUNT_UNIT': mountUnitName(data),
> +            'CREATE_DIRS_SERVICE': helperUnitName(lower),
> +            'LOWERDIR': lower,
> +        }
> +
> +        with open(os.path.join(d.getVar('WORKDIR'), mountUnitName(lower)), 'w') as f:
> +            f.write(MountUnitTemplate.format(**args))
> +
> +        with open(os.path.join(d.getVar('WORKDIR'), helperUnitName(lower)), 'w') as f:
> +            f.write(CreateDirsUnitTemplate.format(**args))
> +
> +    overlayMountPoints = d.getVarFlags("OVERLAYFS_MOUNT_POINT")
> +    for mountPoint in overlayMountPoints:
> +        for lower in d.getVarFlag('OVERLAYFS_WRITABLE_PATHS', mountPoint).split():
> +            prepareUnits(d.getVarFlag('OVERLAYFS_MOUNT_POINT', mountPoint), lower)
> +}
> +
> +# we need to generate file names early during parsing stage
> +python () {
> +    unitList = unitFileList(d)
> +    for unit in unitList:
> +        d.appendVar('SYSTEMD_SERVICE_' + d.getVar('PN'), ' ' + unit);
> +        d.appendVar('FILES_' + d.getVar('PN'), strForBash(unit))
> +
> +    d.setVar('OVERLAYFS_UNIT_LIST', ' '.join([strForBash(s) for s in unitList]))
> +}
> +
> +do_install_append() {
> +    install -d ${D}${systemd_system_unitdir}
> +    for unit in ${OVERLAYFS_UNIT_LIST}; do
> +        install -m 0444 ${WORKDIR}/${unit} ${D}${systemd_system_unitdir}
> +    done
> +}
> +
> +addtask create_overlayfs_units before do_install
> 
> 



^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [OE-core] [PATCH] overlayfs.bbclass: generate overlayfs mount units
  2021-06-05  7:37 ` Adrian Freihofer
@ 2021-06-05  9:28   ` Vyacheslav Yurkov
  0 siblings, 0 replies; 10+ messages in thread
From: Vyacheslav Yurkov @ 2021-06-05  9:28 UTC (permalink / raw)
  To: Adrian Freihofer, Openembedded-core

Hi Adrian,
I don't think using /etc with overlay is a wise idea. You'd like to keep 
system configuration updatable, which wouldn't be simple if the whole 
/etc is an overlay (TBH, I'm not even sure it's possible that way. You'd 
probably need to write custom scripts and add initramfs here). But you 
can export to overlay individual user-specific configuration, for 
example /etc/wpa_supplicant.

Vyacheslav

On 05.06.2021 09:37, Adrian Freihofer wrote:
> Hi Vyacheslav
>
> It would be very nice to have such a feature in Yocto.
>
> If I remember correctly I tried to do that once with an overlay on
> /etc. However, that did not work with systemd. Did you ever try to add
> /etc as a mount point?
>
> On Thu, 2021-06-03 at 16:21 +0200, Vyacheslav Yurkov wrote:
>> It's often desired in Embedded System design to have a read-only rootfs.
>> But a lot of different applications might want to have a read-write access
>> to some parts of a filesystem. It can be especially useful when your update
>> mechanism overwrites the whole rootfs, but you want your application data
>> to be preserved between updates. This class provides a way to achieve that
>> by means of overlayfs and at the same time keeping the base rootfs read-only.
>>
>> Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com>
>> ---
>>   meta/classes/overlayfs.bbclass | 132 +++++++++++++++++++++++++++++++++
>>   1 file changed, 132 insertions(+)
>>   create mode 100644 meta/classes/overlayfs.bbclass
>>
>> diff --git a/meta/classes/overlayfs.bbclass b/meta/classes/overlayfs.bbclass
>> new file mode 100644
>> index 0000000000..7fac3e696e
>> --- /dev/null
>> +++ b/meta/classes/overlayfs.bbclass
>> @@ -0,0 +1,132 @@
>> +# Class for generation of overlayfs mount units
>> +#
>> +# It's often desired in Embedded System design to have a read-only rootfs.
>> +# But a lot of different applications might want to have a read-write access to
>> +# some parts of a filesystem. It can be especially useful when your update mechanism
>> +# overwrites the whole rootfs, but you want your application data to be preserved
>> +# between updates. This class provides a way to achieve that by means
>> +# of overlayfs and at the same time keeping the base rootfs read-only.
>> +#
>> +# Usage example.
>> +#
>> +# Set a mount point for a partition overlayfs is going to use as upper layer
>> +# in your machine configuration. Underlying file system can be anything that
>> +# is supported by overlayfs
>> +#
>> +#   OVERLAYFS_MOUNT_POINT[data] ?= "/data"
>> +#
>> +# The class assumes you have a data.mount systemd unit defined elsewhere in your
>> +# BSP and installed to the image.
>> +#
>> +# Then you can specify writable directories on a recipe base
>> +#
>> +#   OVERLAYFS_WRITABLE_PATHS[data] = "/usr/share/my-custom-application"
>> +#
>> +# To support several mount points you can use a different variable flag. Assume we
>> +# what to have a writable location on the file system, but not interested where the data
>> +# survive a reboot. The we could have a mnt-overlay.mount unit for a tmpfs file system:
>> +#
>> +#   OVERLAYFS_MOUNT_POINT[mnt-overlay] = "/mnt/overlay"
>> +#   OVERLAYFS_WRITABLE_PATHS[mnt-overlay] = "/usr/share/another-application"
>> +
>> +OVERLAYFS_WRITABLE_PATHS[data] ?= ""
>> +
>> +inherit systemd
>> +
>> +def strForBash(s):
>> +    return s.replace('\\', '\\\\')
>> +
>> +def unitFileList(d):
>> +    fileList = []
>> +    overlayMountPoints = d.getVarFlags("OVERLAYFS_MOUNT_POINT")
>> +    for mountPoint in overlayMountPoints:
>> +        for path in d.getVarFlag('OVERLAYFS_WRITABLE_PATHS', mountPoint).split():
>> +            fileList.append(mountUnitName(path))
>> +            fileList.append(helperUnitName(path))
>> +
>> +    return fileList
>> +
>> +# this function is based on https://github.com/systemd/systemd/blob/main/src/basic/unit-name.c
>> +def escapeSystemdUnitName(path):
>> +    escapeMap = {
>> +        '/': '-',
>> +        '-': "\\x2d",
>> +        '\\': "\\x5d"
>> +    }
>> +    return "".join([escapeMap.get(c, c) for c in path.strip('/')])
>> +
>> +def mountUnitName(unit):
>> +    return escapeSystemdUnitName(unit) + '.mount'
>> +
>> +def helperUnitName(unit):
>> +    return escapeSystemdUnitName(unit) + '-create-upper-dir.service'
>> +
>> +python do_create_overlayfs_units() {
>> +    CreateDirsUnitTemplate = """[Unit]
>> +Description=Overlayfs directories setup
>> +Requires={DATA_MOUNT_UNIT}
>> +After={DATA_MOUNT_UNIT}
>> +DefaultDependencies=no
>> +
>> +[Service]
>> +Type=oneshot
>> +ExecStart=mkdir -p {DATA_MOUNT_POINT}/workdir{LOWERDIR} && mkdir -p {DATA_MOUNT_POINT}/upper{LOWERDIR}
>> +RemainAfterExit=true
>> +StandardOutput=journal
>> +
>> +[Install]
>> +WantedBy=multi-user.target
> The mounting of overlays should probably happen very early at boot,
> before services start using the desired folders.
>
> Regards,
> Adrian
>
>> +"""
>> +    MountUnitTemplate = """[Unit]
>> +Description=Overlayfs mount unit
>> +Requires={CREATE_DIRS_SERVICE}
>> +After={CREATE_DIRS_SERVICE}
>> +
>> +[Mount]
>> +What=overlay
>> +Where={LOWERDIR}
>> +Type=overlay
>> +Options=lowerdir={LOWERDIR},upperdir={DATA_MOUNT_POINT}/upper{LOWERDIR},workdir={DATA_MOUNT_POINT}/workdir{LOWERDIR}
>> +
>> +[Install]
>> +WantedBy=multi-user.target
>> +"""
>> +
>> +    def prepareUnits(data, lower):
>> +        args = {
>> +            'DATA_MOUNT_POINT': data,
>> +            'DATA_MOUNT_UNIT': mountUnitName(data),
>> +            'CREATE_DIRS_SERVICE': helperUnitName(lower),
>> +            'LOWERDIR': lower,
>> +        }
>> +
>> +        with open(os.path.join(d.getVar('WORKDIR'), mountUnitName(lower)), 'w') as f:
>> +            f.write(MountUnitTemplate.format(**args))
>> +
>> +        with open(os.path.join(d.getVar('WORKDIR'), helperUnitName(lower)), 'w') as f:
>> +            f.write(CreateDirsUnitTemplate.format(**args))
>> +
>> +    overlayMountPoints = d.getVarFlags("OVERLAYFS_MOUNT_POINT")
>> +    for mountPoint in overlayMountPoints:
>> +        for lower in d.getVarFlag('OVERLAYFS_WRITABLE_PATHS', mountPoint).split():
>> +            prepareUnits(d.getVarFlag('OVERLAYFS_MOUNT_POINT', mountPoint), lower)
>> +}
>> +
>> +# we need to generate file names early during parsing stage
>> +python () {
>> +    unitList = unitFileList(d)
>> +    for unit in unitList:
>> +        d.appendVar('SYSTEMD_SERVICE_' + d.getVar('PN'), ' ' + unit);
>> +        d.appendVar('FILES_' + d.getVar('PN'), strForBash(unit))
>> +
>> +    d.setVar('OVERLAYFS_UNIT_LIST', ' '.join([strForBash(s) for s in unitList]))
>> +}
>> +
>> +do_install_append() {
>> +    install -d ${D}${systemd_system_unitdir}
>> +    for unit in ${OVERLAYFS_UNIT_LIST}; do
>> +        install -m 0444 ${WORKDIR}/${unit} ${D}${systemd_system_unitdir}
>> +    done
>> +}
>> +
>> +addtask create_overlayfs_units before do_install
>> 
>>
>

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [OE-core] [PATCH] overlayfs.bbclass: generate overlayfs mount units
  2021-06-03 14:21 [PATCH] overlayfs.bbclass: generate overlayfs mount units Vyacheslav Yurkov
                   ` (2 preceding siblings ...)
  2021-06-05  7:37 ` Adrian Freihofer
@ 2021-06-09 13:56 ` Andrei Gherzan
  2021-06-09 19:04   ` Vyacheslav Yurkov
  3 siblings, 1 reply; 10+ messages in thread
From: Andrei Gherzan @ 2021-06-09 13:56 UTC (permalink / raw)
  To: openembedded

Hi,

On Thu, 3 Jun 2021, at 15:21, Vyacheslav Yurkov wrote:
> It's often desired in Embedded System design to have a read-only rootfs.
> But a lot of different applications might want to have a read-write access
> to some parts of a filesystem. It can be especially useful when your update
> mechanism overwrites the whole rootfs, but you want your application data
> to be preserved between updates. This class provides a way to achieve that
> by means of overlayfs and at the same time keeping the base rootfs read-only.

I was looking into something like this lately. There are actually two kinds of write access - volatile (tmpfs) and persistent (to disk). So my approach would have been to use bind mounts from a tmpfs/ext4 based on configuration per recipe. Is there any advantage to working with overlayfs in this case? The main idea is to have state that doesn't come from the ro part of the OS because otherwise, updates will be shadowed by the overlayfs. So I fail to see the advantage of using a set of overlayfs mounts instead of bind mounts. At least with bind mounts, it will always shadow the rootfs bits so you can catch early this case.

Depending on your use case, most of the times, you'd want this "state" setup happening pre init (in initramfs, for example), especially if you want to handle persistent systemd logging or machine id, etc.

-- 
Andrei Gherzan 
gpg: rsa4096/D4D94F67AD0E9640

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [OE-core] [PATCH] overlayfs.bbclass: generate overlayfs mount units
  2021-06-09 13:56 ` Andrei Gherzan
@ 2021-06-09 19:04   ` Vyacheslav Yurkov
  0 siblings, 0 replies; 10+ messages in thread
From: Vyacheslav Yurkov @ 2021-06-09 19:04 UTC (permalink / raw)
  To: Andrei Gherzan, openembedded

Hi Andrei,
Like you mentioned, it's application specific. The advantage of 
overlayfs here is that you preserve the data that was changed by user 
(in case of persistent storage), which of course would hide the data 
from rootfs, or your just see the defaults from rootfs if the data was 
not changed. With bind mounts you would need to copy default data first 
into your tmpfs/ext4 bind mount if you wanna have defaults.

Vyacheslav
> I was looking into something like this lately. There are actually two kinds of write access - volatile (tmpfs) and persistent (to disk). So my approach would have been to use bind mounts from a tmpfs/ext4 based on configuration per recipe. Is there any advantage to working with overlayfs in this case? The main idea is to have state that doesn't come from the ro part of the OS because otherwise, updates will be shadowed by the overlayfs. So I fail to see the advantage of using a set of overlayfs mounts instead of bind mounts. At least with bind mounts, it will always shadow the rootfs bits so you can catch early this case.
>
> Depending on your use case, most of the times, you'd want this "state" setup happening pre init (in initramfs, for example), especially if you want to handle persistent systemd logging or machine id, etc.
>


^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2021-06-09 19:04 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-06-03 14:21 [PATCH] overlayfs.bbclass: generate overlayfs mount units Vyacheslav Yurkov
2021-06-03 14:35 ` [OE-core] " Bruce Ashfield
2021-06-04 17:22   ` Vyacheslav Yurkov
2021-06-04 18:24     ` Bruce Ashfield
2021-06-04 20:20 ` Ayoub Zaki
2021-06-05  6:31   ` Vyacheslav Yurkov
2021-06-05  7:37 ` Adrian Freihofer
2021-06-05  9:28   ` Vyacheslav Yurkov
2021-06-09 13:56 ` Andrei Gherzan
2021-06-09 19:04   ` Vyacheslav Yurkov

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.