Alternative to the filesystem overlay

Alternative to the filesystem overlay

[Adriana Kobylak]

In OpenBMC we've been using an overlay to make the read-only filesystem 
(or part of it) writable. It hasn't come without its issues, like 
missing directories (openbmc/openbmc#3564) and various flavors of 
corruption (openbmc/openbmc#3578 and issues when adding files to /etc in 
the build causing files that had been previously modified on the bmc to 
become corrupted).

An alternative would be to use a bind mount and manage the file merging 
by an OpenBMC app. Doing some experimentation, seems this solves the 
issues seen above. The way that it'd work would be:
- The init script or initramfs would do an rsync from /etc to the bind 
mount destination directory in the read-write filesystem, skipping the 
contents of a list which would contain the files that have been modified 
in the BMC and should be preserved.
- A new OpenBMC app would monitor /etc for changes via inotify for 
example, and if a file is modified, it'd add it to the "list".

Considerations:
- The /etc dir mainly being a configuration destination does not get 
modified often so the app that is watching the directory would not be 
triggered often.
- The bind mount would duplicate the contents of etc although it's not 
much. If space is a concern, some exploration could be done to have the 
bind mount be a tmpfs and have the monitor app copy the modified files 
to the read-write filesystem instead of adding them to a list.

Thoughts?

Re: Alternative to the filesystem overlay

[Andrew Geissler]

On Thu, Aug 15, 2019 at 12:38 PM Adriana Kobylak <anoo@linux.ibm.com> wrote:
>
> In OpenBMC we've been using an overlay to make the read-only filesystem
> (or part of it) writable. It hasn't come without its issues, like
> missing directories (openbmc/openbmc#3564) and various flavors of
> corruption (openbmc/openbmc#3578 and issues when adding files to /etc in
> the build causing files that had been previously modified on the bmc to
> become corrupted).

The overlay sounds ideal (when it works) because we don't need all of the
special code and logic you mention below. Any chance we can just fix
whatever is causing the corruption?

>
> An alternative would be to use a bind mount and manage the file merging
> by an OpenBMC app. Doing some experimentation, seems this solves the
> issues seen above. The way that it'd work would be:
> - The init script or initramfs would do an rsync from /etc to the bind
> mount destination directory in the read-write filesystem, skipping the
> contents of a list which would contain the files that have been modified
> in the BMC and should be preserved.
> - A new OpenBMC app would monitor /etc for changes via inotify for
> example, and if a file is modified, it'd add it to the "list".

We always have race condition here though? Someone updates the
BMC network info and before the app can detect it they reboot the
BMC and we lose the settings.

Any idea what other flash based distros do out there? Seems like
this would be a common problem for a lot of projects.

>
> Considerations:
> - The /etc dir mainly being a configuration destination does not get
> modified often so the app that is watching the directory would not be
> triggered often.
> - The bind mount would duplicate the contents of etc although it's not
> much. If space is a concern, some exploration could be done to have the
> bind mount be a tmpfs and have the monitor app copy the modified files
> to the read-write filesystem instead of adding them to a list.
>
> Thoughts?
>

Re: Alternative to the filesystem overlay

[Alexander A. Filippov]

On Thu, Aug 15, 2019 at 12:38:16PM -0500, Adriana Kobylak wrote:
> In OpenBMC we've been using an overlay to make the read-only filesystem (or
> part of it) writable. It hasn't come without its issues, like missing
> directories (openbmc/openbmc#3564) and various flavors of corruption
> (openbmc/openbmc#3578 and issues when adding files to /etc in the build
> causing files that had been previously modified on the bmc to become
> corrupted).

There is one more issue for the same type - openbmc/openbmc#3468.
And it can't be solved by the way you propose.

> 
> An alternative would be to use a bind mount and manage the file merging by
> an OpenBMC app. Doing some experimentation, seems this solves the issues
> seen above. The way that it'd work would be:
> - The init script or initramfs would do an rsync from /etc to the bind mount
> destination directory in the read-write filesystem, skipping the contents of
> a list which would contain the files that have been modified in the BMC and
> should be preserved.
> - A new OpenBMC app would monitor /etc for changes via inotify for example,
> and if a file is modified, it'd add it to the "list".
> 
> Considerations:
> - The /etc dir mainly being a configuration destination does not get
> modified often so the app that is watching the directory would not be
> triggered often.
> - The bind mount would duplicate the contents of etc although it's not much.
> If space is a concern, some exploration could be done to have the bind mount
> be a tmpfs and have the monitor app copy the modified files to the
> read-write filesystem instead of adding them to a list.
> 
> Thoughts?
> 

Most of these issues can be solved by the simple shell scripts. We could modify
the initramfs/init scenario to exec these scripts at the moment when ROFS and
RWFS have mounted but are not combined to OVERLAY yet.

Of course, this method doesn't prevent an occurrence of any new issues of this
type but brings an easy way to solve them.

--
Regards,
Alexander

Re: Alternative to the filesystem overlay

[William Kennington]

Sounds like this is just a data schema update type of issue, where you
just need a service to run at first update boot and upgrade the files
as expected by the change.

We could probably do this with systemd services that are
ConditionNeedsUpdate= and
https://www.freedesktop.org/software/systemd/man/systemd-update-done.service.html

I'd prefer to see us move to a mostly immutable filesystem with
symlinks that map into a rwfs for specific files / directories we
expect to mutate at runtime. I don't think we should do any smart
types of writeback caching at the filesystem level as that will just
lead to confusion about persistence that is hard to debug. Not to
mention tools like rsync don't guarantee atomic snapshotting during
the copy for file consistency. Our applications should be designed in
such a way that they are always consistent on disk. Ex. writing text
files should be done to a temporary, then moved over the old one. That
way the update is atomic and we don't have partially written files.

On Fri, Aug 16, 2019 at 2:10 AM Alexander A. Filippov
<a.filippov@yadro.com> wrote:
>
> On Thu, Aug 15, 2019 at 12:38:16PM -0500, Adriana Kobylak wrote:
> > In OpenBMC we've been using an overlay to make the read-only filesystem (or
> > part of it) writable. It hasn't come without its issues, like missing
> > directories (openbmc/openbmc#3564) and various flavors of corruption
> > (openbmc/openbmc#3578 and issues when adding files to /etc in the build
> > causing files that had been previously modified on the bmc to become
> > corrupted).
>
> There is one more issue for the same type - openbmc/openbmc#3468.
> And it can't be solved by the way you propose.
>
> >
> > An alternative would be to use a bind mount and manage the file merging by
> > an OpenBMC app. Doing some experimentation, seems this solves the issues
> > seen above. The way that it'd work would be:
> > - The init script or initramfs would do an rsync from /etc to the bind mount
> > destination directory in the read-write filesystem, skipping the contents of
> > a list which would contain the files that have been modified in the BMC and
> > should be preserved.
> > - A new OpenBMC app would monitor /etc for changes via inotify for example,
> > and if a file is modified, it'd add it to the "list".
> >
> > Considerations:
> > - The /etc dir mainly being a configuration destination does not get
> > modified often so the app that is watching the directory would not be
> > triggered often.
> > - The bind mount would duplicate the contents of etc although it's not much.
> > If space is a concern, some exploration could be done to have the bind mount
> > be a tmpfs and have the monitor app copy the modified files to the
> > read-write filesystem instead of adding them to a list.
> >
> > Thoughts?
> >
>
> Most of these issues can be solved by the simple shell scripts. We could modify
> the initramfs/init scenario to exec these scripts at the moment when ROFS and
> RWFS have mounted but are not combined to OVERLAY yet.
>
> Of course, this method doesn't prevent an occurrence of any new issues of this
> type but brings an easy way to solve them.
>
> --
> Regards,
> Alexander

Re: Alternative to the filesystem overlay

[Terry Duncan]

On 8/16/19 2:09 AM, Alexander A. Filippov wrote:

> Most of these issues can be solved by the simple shell scripts. We could modify
> the initramfs/init scenario to exec these scripts at the moment when ROFS and
> RWFS have mounted but are not combined to OVERLAY yet.
> 
> Of course, this method doesn't prevent an occurrence of any new issues of this
> type but brings an easy way to solve them.
> 
> --
> Regards,
> Alexander
> 

Possibly the overlay FS is not expecting and does not support the ROFS 
to change as it does when a new build adds a file to an existing 
directory. When this occurs an overlayed file in the same directory 
could cause corruption in the directory. Are you suggesting scripts that 
detect a change to the underlying FS and attempt to patch the overlay?

Thanks,
Terry

Re: Alternative to the filesystem overlay

[Alexander A. Filippov]

On Tue, Aug 20, 2019 at 05:56:32PM -0700, Terry Duncan wrote:
> On 8/16/19 2:09 AM, Alexander A. Filippov wrote:
> 
> > Most of these issues can be solved by the simple shell scripts. We could modify
> > the initramfs/init scenario to exec these scripts at the moment when ROFS and
> > RWFS have mounted but are not combined to OVERLAY yet.
> > 
> > Of course, this method doesn't prevent an occurrence of any new issues of this
> > type but brings an easy way to solve them.
> > 
> > --
> > Regards,
> > Alexander
> > 
> 
> Possibly the overlay FS is not expecting and does not support the ROFS to
> change as it does when a new build adds a file to an existing directory.
> When this occurs an overlayed file in the same directory could cause
> corruption in the directory. Are you suggesting scripts that detect a change
> to the underlying FS and attempt to patch the overlay?
> 
> Thanks,
> Terry

That's not entirely true. I'm suggesting to keep the scripts to solve each
particular issue. But if we can detect and resolve such issues automatically
that would be nice.

Regards,
Alexander

Re: Alternative to the filesystem overlay

[Alexander A. Filippov]

On Tue, Aug 20, 2019 at 11:00:25AM -0700, William Kennington wrote:
> Sounds like this is just a data schema update type of issue, where you
> just need a service to run at first update boot and upgrade the files
> as expected by the change.
> 
> We could probably do this with systemd services that are
> ConditionNeedsUpdate= and
> https://www.freedesktop.org/software/systemd/man/systemd-update-done.service.html
> 

Yes, it looks like a good solution. I will try it.
Thanks for the tip.

> I'd prefer to see us move to a mostly immutable filesystem with
> symlinks that map into a rwfs for specific files / directories we
> expect to mutate at runtime. I don't think we should do any smart
> types of writeback caching at the filesystem level as that will just
> lead to confusion about persistence that is hard to debug. Not to
> mention tools like rsync don't guarantee atomic snapshotting during
> the copy for file consistency. Our applications should be designed in
> such a way that they are always consistent on disk. Ex. writing text
> files should be done to a temporary, then moved over the old one. That
> way the update is atomic and we don't have partially written files.
> 

Regards,
Alexander