[PATCH] ref-manual: add overlayfs class

[PATCH] ref-manual: add overlayfs class

[Vyacheslav Yurkov]

Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com>
---
 documentation/ref-manual/classes.rst | 46 ++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)

diff --git a/documentation/ref-manual/classes.rst b/documentation/ref-manual/classes.rst
index a98a64c43..5d33e81a1 100644
--- a/documentation/ref-manual/classes.rst
+++ b/documentation/ref-manual/classes.rst
@@ -1710,6 +1710,52 @@ one such example. However, being aware of this class can reduce the
 proliferation of different versions of similar classes across multiple
 layers.
 
+.. _ref-classes-overlayfs:
+
+``overlayfs.bbclass``
+=======================
+
+
+It's often desired in Embedded System design to have a read-only rootfs.
+But a lot of different applications might want to have a read-write access to
+some parts of a filesystem. It can be especially useful when your update mechanism
+overwrites the whole rootfs, but you want your application data to be preserved
+between updates. The ``overlayfs`` class provides a way to achieve that by means
+of overlayfs and at the same time keeping the base rootfs read-only.
+
+To use this class, set a mount point for a partition overlayfs is going to use as upper
+layer in your machine configuration. Underlying file system can be anything that
+is supported by overlayfs. This has to be done in your machine configuration::
+
+  OVERLAYFS_MOUNT_POINT[data] = "/data"
+
+.. note::
+
+  QA check fails to catch file existence if you redefine this variable in your recipe!
+
+The class assumes you have a data.mount systemd unit defined elsewhere in your BSP
+(e.g. in systemd-machine-units recipe) and it's installed to the image.
+
+Then you can specify writable directories on a recipe basis::
+
+  OVERLAYFS_WRITABLE_PATHS[data] = "/usr/share/my-custom-application"
+
+To support several mount points you can use a different variable flag. Assume we
+want to have a writable location on the file system, but not interested where the data
+survive a reboot. Then we could have a mnt-overlay.mount unit for a tmpfs file system.
+
+In your machine configuration::
+
+  OVERLAYFS_MOUNT_POINT[mnt-overlay] = "/mnt/overlay"
+
+and then in your recipe::
+
+  OVERLAYFS_WRITABLE_PATHS[mnt-overlay] = "/usr/share/another-application"
+
+.. note::
+
+   The class does not support /etc directory itself, because systemd depends on it
+
 .. _ref-classes-own-mirrors:
 
 ``own-mirrors.bbclass``
-- 
2.28.0

Re: [docs] [PATCH] ref-manual: add overlayfs class

[Quentin Schulz]

Hi Vyacheslav,

Thanks for the patch!

On Sat, Aug 14, 2021 at 06:28:25AM +0200, Vyacheslav Yurkov wrote:
> Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com>
> ---
>  documentation/ref-manual/classes.rst | 46 ++++++++++++++++++++++++++++
>  1 file changed, 46 insertions(+)
> 
> diff --git a/documentation/ref-manual/classes.rst b/documentation/ref-manual/classes.rst
> index a98a64c43..5d33e81a1 100644
> --- a/documentation/ref-manual/classes.rst
> +++ b/documentation/ref-manual/classes.rst
> @@ -1710,6 +1710,52 @@ one such example. However, being aware of this class can reduce the
>  proliferation of different versions of similar classes across multiple
>  layers.
>  
> +.. _ref-classes-overlayfs:
> +
> +``overlayfs.bbclass``
> +=======================
> +
> +

One newline is enough :)

> +It's often desired in Embedded System design to have a read-only rootfs.
> +But a lot of different applications might want to have a read-write access to
> +some parts of a filesystem. It can be especially useful when your update mechanism
> +overwrites the whole rootfs, but you want your application data to be preserved
> +between updates. The ``overlayfs`` class provides a way to achieve that by means

s/``overlayfs``/:ref:`overlayfs <ref-classes-overlayfs>`/

> +of overlayfs and at the same time keeping the base rootfs read-only.
> +

I would highlight overlayfs with tick quotes (``overlayfs``).

> +To use this class, set a mount point for a partition overlayfs is going to use as upper
> +layer in your machine configuration. Underlying file system can be anything that
> +is supported by overlayfs. This has to be done in your machine configuration::
> +

How do I know what's supported by overlayfs? Where can I find this
important piece of information?

> +  OVERLAYFS_MOUNT_POINT[data] = "/data"
> +
> +.. note::
> +
> +  QA check fails to catch file existence if you redefine this variable in your recipe!
> +

What about:
This variable must only be set from a configuration file as QA check will fail
to catch file existence if this variable is also set in a recipe.

Also, I'm not sure to understand what "catch file existence" means?
Could you explain to me please :)

> +The class assumes you have a data.mount systemd unit defined elsewhere in your BSP

s/data.mount/``data.mount``/

> +(e.g. in systemd-machine-units recipe) and it's installed to the image.
> +

s/systemd-machine-units/``systemd-machine-units``/

> +Then you can specify writable directories on a recipe basis::
> +

If package recipes are supported, does it actually make sense to have
such variable in there instead of in image recipes?

Just trying to know if we should highlight whether it's a bad idea for
users to do so.

> +  OVERLAYFS_WRITABLE_PATHS[data] = "/usr/share/my-custom-application"
> +
> +To support several mount points you can use a different variable flag. Assume we

s/Assume/Assuming/ ?

> +want to have a writable location on the file system, but not interested where the data
> +survive a reboot. Then we could have a mnt-overlay.mount unit for a tmpfs file system.
> +

s/mnt-overlay.mount/``mnt-overlay.mount``/

s/tmpfs/``tmpfs``/ ?

> +In your machine configuration::
> +
> +  OVERLAYFS_MOUNT_POINT[mnt-overlay] = "/mnt/overlay"
> +
> +and then in your recipe::
> +
> +  OVERLAYFS_WRITABLE_PATHS[mnt-overlay] = "/usr/share/another-application"
> +
> +.. note::
> +
> +   The class does not support /etc directory itself, because systemd depends on it
> +

s/\/etc/``\/etc``/

s/systemd/``systemd``/

My nitpicks excluded, looks good :) ~

Cheers,
Quentin

Re: [docs] [PATCH] ref-manual: add overlayfs class

[Vyacheslav Yurkov]

[-- Attachment #1: Type: text/plain, Size: 4936 bytes --]

Hi Quentin,
Thanks for the review! I'll address the notes in v2. A few answers to your
questions are inlined below.

On Mon, Aug 16, 2021, 13:14 Quentin Schulz <
quentin.schulz@theobroma-systems.com> wrote:

> Hi Vyacheslav,
>
> Thanks for the patch!
>
> On Sat, Aug 14, 2021 at 06:28:25AM +0200, Vyacheslav Yurkov wrote:
> > Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com>
> > ---
> >  documentation/ref-manual/classes.rst | 46 ++++++++++++++++++++++++++++
> >  1 file changed, 46 insertions(+)
> >
> > diff --git a/documentation/ref-manual/classes.rst
> b/documentation/ref-manual/classes.rst
> > index a98a64c43..5d33e81a1 100644
> > --- a/documentation/ref-manual/classes.rst
> > +++ b/documentation/ref-manual/classes.rst
> > @@ -1710,6 +1710,52 @@ one such example. However, being aware of this
> class can reduce the
> >  proliferation of different versions of similar classes across multiple
> >  layers.
> >
> > +.. _ref-classes-overlayfs:
> > +
> > +``overlayfs.bbclass``
> > +=======================
> > +
> > +
>
> One newline is enough :)
>
> > +It's often desired in Embedded System design to have a read-only rootfs.
> > +But a lot of different applications might want to have a read-write
> access to
> > +some parts of a filesystem. It can be especially useful when your
> update mechanism
> > +overwrites the whole rootfs, but you want your application data to be
> preserved
> > +between updates. The ``overlayfs`` class provides a way to achieve that
> by means
>
> s/``overlayfs``/:ref:`overlayfs <ref-classes-overlayfs>`/
>
> > +of overlayfs and at the same time keeping the base rootfs read-only.
> > +
>
> I would highlight overlayfs with tick quotes (``overlayfs``).
>
> > +To use this class, set a mount point for a partition overlayfs is going
> to use as upper
> > +layer in your machine configuration. Underlying file system can be
> anything that
> > +is supported by overlayfs. This has to be done in your machine
> configuration::
> > +
>
> How do I know what's supported by overlayfs? Where can I find this
> important piece of information?
>
> > +  OVERLAYFS_MOUNT_POINT[data] = "/data"
> > +
> > +.. note::
> > +
> > +  QA check fails to catch file existence if you redefine this variable
> in your recipe!
> > +
>
> What about:
> This variable must only be set from a configuration file as QA check will
> fail
> to catch file existence if this variable is also set in a recipe.
>
> Also, I'm not sure to understand what "catch file existence" means?
> Could you explain to me please :)
>

That means I check only that file exists in QA check, I do not check
whether it's a valid systemd mount unit (to avoid additional dependency on
systemd-native recipe).


> > +The class assumes you have a data.mount systemd unit defined elsewhere
> in your BSP
>
> s/data.mount/``data.mount``/
>
> > +(e.g. in systemd-machine-units recipe) and it's installed to the image.
> > +
>
> s/systemd-machine-units/``systemd-machine-units``/
>
> > +Then you can specify writable directories on a recipe basis::
> > +
>
> If package recipes are supported, does it actually make sense to have
> such variable in there instead of in image recipes?
>
> Just trying to know if we should highlight whether it's a bad idea for
> users to do so.
>

There are actually two pitfalls here.

Semantically directory belongs to a recipe it's used in. If you redefine
OVERLAYFS_WRITABLE_PATHS in another recipe, do_rootfs would fail when two
recipes install the same file.

Another trade-off I had to deal with is if you redefine
OVERLAYFS_MOUNT_POINT in a recipe, then I can't catch it in my QA check.
This is due to a bitbake limitation, in particular I can't forbid further
variable overrides. That's why I mentioned above that this variable has to
be set in machine configuration.

I hope that makes sense :)

Regards,
Vyacheslav


> > +  OVERLAYFS_WRITABLE_PATHS[data] = "/usr/share/my-custom-application"
> > +
> > +To support several mount points you can use a different variable flag.
> Assume we
>
> s/Assume/Assuming/ ?
>
> > +want to have a writable location on the file system, but not interested
> where the data
> > +survive a reboot. Then we could have a mnt-overlay.mount unit for a
> tmpfs file system.
> > +
>
> s/mnt-overlay.mount/``mnt-overlay.mount``/
>
> s/tmpfs/``tmpfs``/ ?
>
> > +In your machine configuration::
> > +
> > +  OVERLAYFS_MOUNT_POINT[mnt-overlay] = "/mnt/overlay"
> > +
> > +and then in your recipe::
> > +
> > +  OVERLAYFS_WRITABLE_PATHS[mnt-overlay] =
> "/usr/share/another-application"
> > +
> > +.. note::
> > +
> > +   The class does not support /etc directory itself, because systemd
> depends on it
> > +
>
> s/\/etc/``\/etc``/
>
> s/systemd/``systemd``/
>
> My nitpicks excluded, looks good :) ~
>
> Cheers,
> Quentin

[-- Attachment #2: Type: text/html, Size: 6616 bytes --]

Re: [docs] [PATCH] ref-manual: add overlayfs class

[Quentin Schulz]

Hi Vyacheslav,

On Tue, Aug 17, 2021 at 07:29:35AM +0400, Vyacheslav Yurkov wrote:
> Hi Quentin,
> Thanks for the review! I'll address the notes in v2. A few answers to your
> questions are inlined below.
> 
> On Mon, Aug 16, 2021, 13:14 Quentin Schulz <
> quentin.schulz@theobroma-systems.com> wrote:
> 
> > Hi Vyacheslav,
> >
> > Thanks for the patch!
> >
> > On Sat, Aug 14, 2021 at 06:28:25AM +0200, Vyacheslav Yurkov wrote:
> > > Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com>
> > > ---
> > >  documentation/ref-manual/classes.rst | 46 ++++++++++++++++++++++++++++
> > >  1 file changed, 46 insertions(+)
> > >
> > > diff --git a/documentation/ref-manual/classes.rst
> > b/documentation/ref-manual/classes.rst
> > > index a98a64c43..5d33e81a1 100644
> > > --- a/documentation/ref-manual/classes.rst
> > > +++ b/documentation/ref-manual/classes.rst
> > > @@ -1710,6 +1710,52 @@ one such example. However, being aware of this
> > class can reduce the
> > >  proliferation of different versions of similar classes across multiple
> > >  layers.
> > >
> > > +.. _ref-classes-overlayfs:
> > > +
> > > +``overlayfs.bbclass``
> > > +=======================
> > > +
> > > +
> >
> > One newline is enough :)
> >
> > > +It's often desired in Embedded System design to have a read-only rootfs.
> > > +But a lot of different applications might want to have a read-write
> > access to
> > > +some parts of a filesystem. It can be especially useful when your
> > update mechanism
> > > +overwrites the whole rootfs, but you want your application data to be
> > preserved
> > > +between updates. The ``overlayfs`` class provides a way to achieve that
> > by means
> >
> > s/``overlayfs``/:ref:`overlayfs <ref-classes-overlayfs>`/
> >
> > > +of overlayfs and at the same time keeping the base rootfs read-only.
> > > +
> >
> > I would highlight overlayfs with tick quotes (``overlayfs``).
> >
> > > +To use this class, set a mount point for a partition overlayfs is going
> > to use as upper
> > > +layer in your machine configuration. Underlying file system can be
> > anything that
> > > +is supported by overlayfs. This has to be done in your machine
> > configuration::
> > > +
> >
> > How do I know what's supported by overlayfs? Where can I find this
> > important piece of information?
> >
> > > +  OVERLAYFS_MOUNT_POINT[data] = "/data"
> > > +
> > > +.. note::
> > > +
> > > +  QA check fails to catch file existence if you redefine this variable
> > in your recipe!
> > > +
> >
> > What about:
> > This variable must only be set from a configuration file as QA check will
> > fail
> > to catch file existence if this variable is also set in a recipe.
> >
> > Also, I'm not sure to understand what "catch file existence" means?
> > Could you explain to me please :)
> >
> 
> That means I check only that file exists in QA check, I do not check
> whether it's a valid systemd mount unit (to avoid additional dependency on
> systemd-native recipe).
> 

I think what I was missing is that the QA check is actually part of the
ROOTFS_POSTPROCESS_COMMAND variable. Therefore,  only the image recipe
will do the QA check.

I think we should highlight that this variable should only be set in
configuration files OR image recipes. (or basically, say that it
shouldn't be set in package recipes).

So I'd actually add two notes:

1) Variables to be defined in conf files or image recipes (or say the
opposite: not in package recipes)
2) Only existence of the systemd mount unit file is checked, not its
content.

> 
> > > +The class assumes you have a data.mount systemd unit defined elsewhere
> > in your BSP
> >
> > s/data.mount/``data.mount``/
> >
> > > +(e.g. in systemd-machine-units recipe) and it's installed to the image.
> > > +
> >
> > s/systemd-machine-units/``systemd-machine-units``/
> >
> > > +Then you can specify writable directories on a recipe basis::
> > > +
> >
> > If package recipes are supported, does it actually make sense to have
> > such variable in there instead of in image recipes?
> >
> > Just trying to know if we should highlight whether it's a bad idea for
> > users to do so.
> >
> 
> There are actually two pitfalls here.
> 
> Semantically directory belongs to a recipe it's used in. If you redefine
> OVERLAYFS_WRITABLE_PATHS in another recipe, do_rootfs would fail when two
> recipes install the same file.
> 
> Another trade-off I had to deal with is if you redefine
> OVERLAYFS_MOUNT_POINT in a recipe, then I can't catch it in my QA check.
> This is due to a bitbake limitation, in particular I can't forbid further
> variable overrides. That's why I mentioned above that this variable has to
> be set in machine configuration.
> 

I feel like OVERLAYFS configuration makes sense only when defined
globally (conf files) or in image recipes.

Just thinking out loud right now but couldn't we add the following in
overlayfs.bbclass anonymous python code?

if not bb.data.inherits_class('image', d):
	bb.fatal('Only image recipes can inherit overlayfs.bbclass. %s
		 is a package recipe.' % d.getVar('${PN}'))

or something like that?

What are your thoughts on my remarks?

Lemme know!

Regards,
Quentin

Re: [docs] [PATCH] ref-manual: add overlayfs class

[Vyacheslav Yurkov]

Hi Quentin

>> That means I check only that file exists in QA check, I do not check
>> whether it's a valid systemd mount unit (to avoid additional dependency on
>> systemd-native recipe).
>>
> I think what I was missing is that the QA check is actually part of the
> ROOTFS_POSTPROCESS_COMMAND variable. Therefore,  only the image recipe
> will do the QA check.
>
> I think we should highlight that this variable should only be set in
> configuration files OR image recipes. (or basically, say that it
> shouldn't be set in package recipes).
>
> So I'd actually add two notes:
>
> 1) Variables to be defined in conf files or image recipes (or say the
> opposite: not in package recipes)

Not entirely correct, see my other answer below.

> 2) Only existence of the systemd mount unit file is checked, not its
> content.

Agreed.

> I feel like OVERLAYFS configuration makes sense only when defined
> globally (conf files) or in image recipes.
>
> Just thinking out loud right now but couldn't we add the following in
> overlayfs.bbclass anonymous python code?
>
> if not bb.data.inherits_class('image', d):
> 	bb.fatal('Only image recipes can inherit overlayfs.bbclass. %s
> 		 is a package recipe.' % d.getVar('${PN}'))
>
> or something like that?
>
> What are your thoughts on my remarks?

To sum it up:
- OVERLAYFS_MOUNT_POINT should be defined globally and _not_ redefined 
afterwards.
- OVERLAYFS_WRITABLE_PATHS can be used in any recipe (I guess not a 
package group though). It doesn't have to be image recipe, because a 
directory specified there can be application / recipe specific. For 
example, if we have a recipe my-app.bb and an application there wants to 
have a writable path in /usr/share/my-app, then it makes sense to set it 
in the my-app recipe, not in the image recipe, right?
- QA check is part of ROOTFS_POSTPROCESS_COMMAND as you noted before and 
is done as one of the last step during image creation.

> Lemme know!
>
> Regards,
> Quentin

Cheers,
Vyacheslav