userns mount and metacopy redirects (Was: Re: [PATCH v3 0/6] Composefs: an opportunistically sharing verified image filesystem)

userns mount and metacopy redirects (Was: Re: [PATCH v3 0/6] Composefs: an opportunistically sharing verified image filesystem)

[Amir Goldstein]

[spawning overlayfs sub-topic]

On Wed, Jan 25, 2023 at 10:29 PM Amir Goldstein <amir73il@gmail.com> wrote:
>
> On Wed, Jan 25, 2023 at 10:23 PM Amir Goldstein <amir73il@gmail.com> wrote:
> >
> > On Wed, Jan 25, 2023 at 9:45 PM Giuseppe Scrivano <gscrivan@redhat.com> wrote:
> > >
> > > Amir Goldstein <amir73il@gmail.com> writes:
> > >
> > > >> >> I previously mentioned my wish of using it from a user namespace, the
> > > >> >> goal seems more challenging with EROFS or any other block devices.  I

For those who are starting to read here, the context is userns mounting
of overlayfs with a lower EROFS layer containing metacopy references to
lower data blobs in another fs (a.k.a the composefs model).

IMO, mounting a readonly image of whatever on-disk format
is a very high risk for userns mount.
A privileged mount helper that verifies and mounts the EROFS
layer sounds like a more feasible solution.

> > > >> >> don't know about the difficulty of getting overlay metacopy working in a
> > > >> >> user namespace, even though it would be helpful for other use cases as
> > > >> >> well.
> > > >> >>
> > > >> >
> > > >> > There is no restriction of metacopy in user namespace.
> > > >> > overlayfs needs to be mounted with -o userxattr and the overlay
> > > >> > xattrs needs to use user.overlay. prefix.
> > > >>
> > > >> if I specify both userxattr and metacopy=on then the mount ends up in
> > > >> the following check:
> > > >>
> > > >> if (config->userxattr) {
> > > >>         [...]
> > > >>         if (config->metacopy && metacopy_opt) {
> > > >>                 pr_err("conflicting options: userxattr,metacopy=on\n");
> > > >>                 return -EINVAL;
> > > >>         }
> > > >> }
> > > >>
> > > >
> > > > Right, my bad.
> > > >
> > > >> to me it looks like it was done on purpose to prevent metacopy from a
> > > >> user namespace, but I don't know the reason for sure.
> > > >>
> > > >
> > > > With hand crafted metacopy, an unpriv user can chmod
> > > > any files to anything by layering another file with different
> > > > mode on top of it....
> > >
> > > I might be missing something obvious about metacopy, so please correct
> > > me if I am wrong, but I don't see how it is any different than just
> > > copying the file and chowning it.  Of course, as long as overlay uses
> > > the same security model so that a file that wasn't originally possible
> > > to access must be still blocked, even if referenced through metacopy.
> > >
> >
> > You're right.
> > The reason for mutual exclusion maybe related to the
> > comment in ovl_check_metacopy_xattr() about EACCES.
> > Need to check with Vivek or Miklos.
> >
> > But get this - you do not need metacopy=on to follow lower inode.
> > It should work without metacopy=on.
> > metacopy=on only instructs overlayfs whether to copy up data
> > or only metadata when changing metadata of lower object, so it is
> > not relevant for readonly mount.
> >
>
> However, you do need redirect=follow and that one is only mutually
> exclusive with userxattr.
> Again, need to ask Miklos whether that could be relaxed under
> some conditions.
>

I can see some possible problems with userns mount and redirect:
- referencing same dir inode from different paths
- referencing same inode from different paths with
  wrong nlink and inconsistent metadata

However, I think a mode that only follows a redirect from a
lower metacopy file to its data should be safe for userns mount.

In this special case (lower metacopy file) we may also be able to
implement the lazy lookup of the data file on open to optimize
'find' performance, but need to figure out what to do with st_blocks
of stat() in that case.

Thanks,
Amir.

Re: userns mount and metacopy redirects (Was: Re: [PATCH v3 0/6] Composefs: an opportunistically sharing verified image filesystem)

[Christian Brauner]

On Thu, Jan 26, 2023 at 07:26:49AM +0200, Amir Goldstein wrote:
> [spawning overlayfs sub-topic]
> 
> On Wed, Jan 25, 2023 at 10:29 PM Amir Goldstein <amir73il@gmail.com> wrote:
> >
> > On Wed, Jan 25, 2023 at 10:23 PM Amir Goldstein <amir73il@gmail.com> wrote:
> > >
> > > On Wed, Jan 25, 2023 at 9:45 PM Giuseppe Scrivano <gscrivan@redhat.com> wrote:
> > > >
> > > > Amir Goldstein <amir73il@gmail.com> writes:
> > > >
> > > > >> >> I previously mentioned my wish of using it from a user namespace, the
> > > > >> >> goal seems more challenging with EROFS or any other block devices.  I
> 
> For those who are starting to read here, the context is userns mounting
> of overlayfs with a lower EROFS layer containing metacopy references to
> lower data blobs in another fs (a.k.a the composefs model).
> 
> IMO, mounting a readonly image of whatever on-disk format
> is a very high risk for userns mount.
> A privileged mount helper that verifies and mounts the EROFS
> layer sounds like a more feasible solution.

Very much agreed. This filesystem specific userns mountable stuff where
filesystems with any kind of on-disk format guarantees the safety is not
something we should support.

I'm starting to think about how to make it possible for a privileged
process to delegate/allow a filesystem mount to an unprivileged one. The
policy belongs in userspace. Something which I've talked about before a
few years ago but now I actually have time to work on this.

Re: [PATCH v3 0/6] Composefs: an opportunistically sharing verified image filesystem

[Amir Goldstein]

[+overlayfs list]

On Mon, Feb 6, 2023 at 7:16 PM Amir Goldstein <amir73il@gmail.com> wrote:
>
> On Mon, Feb 6, 2023 at 6:34 PM Miklos Szeredi <miklos@szeredi.hu> wrote:
> >
> > On Mon, 6 Feb 2023 at 14:31, Amir Goldstein <amir73il@gmail.com> wrote:
> > >
> > > > > > My little request again, could you help benchmark on your real workload
> > > > > > rather than "ls -lR" stuff?  If your hard KPI is really what as you
> > > > > > said, why not just benchmark the real workload now and write a detailed
> > > > > > analysis to everyone to explain it's a _must_ that we should upstream
> > > > > > a new stacked fs for this?
> > > > > >
> > > > >
> > > > > I agree that benchmarking the actual KPI (boot time) will have
> > > > > a much stronger impact and help to build a much stronger case
> > > > > for composefs if you can prove that the boot time difference really matters.
> > > > >
> > > > > In order to test boot time on fair grounds, I prepared for you a POC
> > > > > branch with overlayfs lazy lookup:
> > > > > https://github.com/amir73il/linux/commits/ovl-lazy-lowerdata
> > > >
> > > > Sorry about being late to the party...
> > > >
> > > > Can you give a little detail about what exactly this does?
> > > >
> > >
> > > Consider a container image distribution system, with base images
> > > and derived images and instruction on how to compose these images
> > > using overlayfs or other methods.
> > >
> > > Consider a derived image L3 that depends on images L2, L1.
> > >
> > > With the composefs methodology, the image distribution server splits
> > > each image is split into metadata only (metacopy) images M3, M2, M1
> > > and their underlying data images containing content addressable blobs
> > > D3, D2, D1.
> > >
> > > The image distribution server goes on to merge the metadata layers
> > > on the server, so U3 = M3 + M2 + M1.
> > >
> > > In order to start image L3, the container client will unpack the data layers
> > > D3, D2, D1 to local fs normally, but the server merged U3 metadata image
> > > will be distributed as a read-only fsverity signed image that can be mounted
> > > by mount -t composefs U3.img (much like mount -t erofs -o loop U3.img).
> > >
> > > The composefs image format contains "redirect" instruction to the data blob
> > > path and an fsverity signature that can be used to verify the redirected data
> > > content.
> > >
> > > When composefs authors proposed to merge composefs, Gao and me
> > > pointed out that the same functionality can be achieved with minimal changes
> > > using erofs+overlayfs.
> > >
> > > Composefs authors have presented ls -lR time and memory usage benchmarks
> > > that demonstrate how composefs performs better that erofs+overlayfs in
> > > this workload and explained that the lookup of the data blobs is what takes
> > > the extra time and memory in the erofs+overlayfs ls -lR test.
> > >
> > > The lazyfollow POC optimizes-out the lowerdata lookup for the ls -lR
> > > benchmark, so that composefs could be compared to erofs+overlayfs.
> >
> > Got it, thanks.
> >
> > >
> > > To answer Alexander's question:
> > >
> > > > Cool. I'll play around with this. Does this need to be an opt-in
> > > > option in the final version? It feels like this could be useful to
> > > > improve performance in general for overlayfs, for example when
> > > > metacopy is used in container layers.
> > >
> > > I think lazyfollow could be enabled by default after we hashed out
> > > all the bugs and corner cases and most importantly remove the
> > > POC limitation of lower-only overlay.
> > >
> > > The feedback that composefs authors are asking from you
> > > is whether you will agree to consider adding the "lazyfollow
> > > lower data" optimization and "fsverity signature for metacopy"
> > > feature to overlayfs?
> > >
> > > If you do agree, the I think they should invest their resources
> > > in making those improvements to overlayfs and perhaps
> > > other improvements to erofs, rather than proposing a new
> > > specialized filesystem.
> >
> > Lazy follow seems to make sense.  Why does it need to be optional?
>
> It doesn't.
>
> > Does it have any advantage to *not* do lazy follow?
> >
>
> Not that I can think of.
>
> > Not sure I follow the fsverity requirement.  For overlay+erofs case
> > itsn't it enough to verify the erofs image?
> >
>
> it's not overlay{erofs+erofs}
> it's overlay{erofs+ext4} (or another fs-verity [1] supporting fs)
> the lower layer is a mutable fs with /objects/ dir containing
> the blobs.
>
> The way to ensure the integrity of erofs is to setup dm-verity at
> erofs mount time.
>
> The way to ensure the integrity of the blobs is to store an fs-verity
> signature of each blob file in trusted.overlay.verify xattr on the
> metacopy and for overlayfs to enable fsverity on the blob file before
> allowing access to the lowerdata.
>

Perhaps I should have mentioned that the lower /objects dir, despite
being mutable (mostly append-only) is shared among several overlays.

This technically breaks the law of no modification to lower layer,
but the /objects dir itself is a whiteout in the metadata layer, so
the blobs are only accessible via absolute path redirect and there
is no /objects overlay dir, so there is no readdir cache to invalidate.
Naturally, the content addressable blobs are not expected to be
renamed/unlinked while an overlayfs that references them is mounted.

Thanks,
Amir.

Lazy lowerdata lookup and data-only layers (Was: Re: Composefs:)

[Amir Goldstein]

> > >
> > > I think lazyfollow could be enabled by default after we hashed out
> > > all the bugs and corner cases and most importantly remove the
> > > POC limitation of lower-only overlay.
> > >
[...]
> > >
> >
> > Lazy follow seems to make sense.  Why does it need to be optional?
>
> It doesn't.
>
> > Does it have any advantage to *not* do lazy follow?
> >
>
> Not that I can think of.

Miklos,

I completed writing the lazy lookup patches [1].

It wasn't trivial and the first versions had many traps that took time to
trip on, so I've made some design choices to make it safer and easier to
land an initial improvement that will cater the composefs use case.

The main design choice has to do with making lazy lowerdata lookup
completely opt-in by defining a new type of data-only layers, such as
the content addressable lower layer of composefs.
The request for the data-only layers came from Alexander.

The current patches only do lazy lookup in data-only layers and the lookup
in data-only layers is always lazy.

Data-only layers have some other advantages, for example, multiple
data-only uuid-less layers are allowed.
Please see the text below taken from the patches.

What do you think about this direction?

Alexander has started to test these patches.
If he finds no issues and if you have no objections to the concept,
then I will post the patches for wider review.

Thanks,
Amir.

[1] https://github.com/amir73il/linux/commits/ovl-lazy-lowerdata-rc2

Data-only lower layers
----------------------

With "metacopy" feature enabled, an overlayfs regular file may be a
composition of information from up to three different layers:

 1) metadata from a file in the upper layer

 2) st_ino and st_dev object identifier from a file in a lower layer

 3) data from a file in another lower layer (further below)

The "lower data" file can be on any lower layer, except from the top most
lower layer.

Below the top most lower layer, any number of lower most layers may be
defined as "data-only" lower layers, using the double collon ("::") separator.

For example:

  mount -t overlay overlay -olowerdir=/lower1::/lower2:/lower3 /merged

The paths of files in the "data-only" lower layers are not visible in the
merged overlayfs directories and the metadata and st_ino/st_dev of files
in the "data-only" lower layers are not visible in overlayfs inodes.

Only the data of the files in the "data-only" lower layers may be visible
when a "metacopy" file in one of the lower layers above it, has a "redirect"
to the absolute path of the "lower data" file in the "data-only" lower layer.

Re: Lazy lowerdata lookup and data-only layers (Was: Re: Composefs:)

[Miklos Szeredi]

On Mon, 3 Apr 2023 at 21:00, Amir Goldstein <amir73il@gmail.com> wrote:
>
> > > >
> > > > I think lazyfollow could be enabled by default after we hashed out
> > > > all the bugs and corner cases and most importantly remove the
> > > > POC limitation of lower-only overlay.
> > > >
> [...]
> > > >
> > >
> > > Lazy follow seems to make sense.  Why does it need to be optional?
> >
> > It doesn't.
> >
> > > Does it have any advantage to *not* do lazy follow?
> > >
> >
> > Not that I can think of.
>
> Miklos,
>
> I completed writing the lazy lookup patches [1].
>
> It wasn't trivial and the first versions had many traps that took time to
> trip on, so I've made some design choices to make it safer and easier to
> land an initial improvement that will cater the composefs use case.
>
> The main design choice has to do with making lazy lowerdata lookup
> completely opt-in by defining a new type of data-only layers, such as
> the content addressable lower layer of composefs.
> The request for the data-only layers came from Alexander.
>
> The current patches only do lazy lookup in data-only layers and the lookup
> in data-only layers is always lazy.
>
> Data-only layers have some other advantages, for example, multiple
> data-only uuid-less layers are allowed.
> Please see the text below taken from the patches.
>
> What do you think about this direction?
>
> Alexander has started to test these patches.
> If he finds no issues and if you have no objections to the concept,
> then I will post the patches for wider review.
>
>
> Thanks,
> Amir.
>
> [1] https://github.com/amir73il/linux/commits/ovl-lazy-lowerdata-rc2
>
> Data-only lower layers
> ----------------------
>
> With "metacopy" feature enabled, an overlayfs regular file may be a
> composition of information from up to three different layers:
>
>  1) metadata from a file in the upper layer
>
>  2) st_ino and st_dev object identifier from a file in a lower layer
>
>  3) data from a file in another lower layer (further below)
>
> The "lower data" file can be on any lower layer, except from the top most
> lower layer.
>
> Below the top most lower layer, any number of lower most layers may be
> defined as "data-only" lower layers, using the double collon ("::") separator.
>
> For example:
>
>   mount -t overlay overlay -olowerdir=/lower1::/lower2:/lower3 /merged

What are the rules?

Is "do1::do2::lower" allowed?
Is "do1::lower1:do2::lower2 allowed?

>
> The paths of files in the "data-only" lower layers are not visible in the
> merged overlayfs directories and the metadata and st_ino/st_dev of files
> in the "data-only" lower layers are not visible in overlayfs inodes.
>
> Only the data of the files in the "data-only" lower layers may be visible
> when a "metacopy" file in one of the lower layers above it, has a "redirect"
> to the absolute path of the "lower data" file in the "data-only" lower layer.

Okay.

Thanks,
Miklos

Re: Lazy lowerdata lookup and data-only layers (Was: Re: Composefs:)

[Amir Goldstein]

On Tue, Apr 11, 2023 at 6:50 PM Miklos Szeredi <miklos@szeredi.hu> wrote:
>
> On Mon, 3 Apr 2023 at 21:00, Amir Goldstein <amir73il@gmail.com> wrote:
> >
> > > > >
> > > > > I think lazyfollow could be enabled by default after we hashed out
> > > > > all the bugs and corner cases and most importantly remove the
> > > > > POC limitation of lower-only overlay.
> > > > >
> > [...]
> > > > >
> > > >
> > > > Lazy follow seems to make sense.  Why does it need to be optional?
> > >
> > > It doesn't.
> > >
> > > > Does it have any advantage to *not* do lazy follow?
> > > >
> > >
> > > Not that I can think of.
> >
> > Miklos,
> >
> > I completed writing the lazy lookup patches [1].
> >
> > It wasn't trivial and the first versions had many traps that took time to
> > trip on, so I've made some design choices to make it safer and easier to
> > land an initial improvement that will cater the composefs use case.
> >
> > The main design choice has to do with making lazy lowerdata lookup
> > completely opt-in by defining a new type of data-only layers, such as
> > the content addressable lower layer of composefs.
> > The request for the data-only layers came from Alexander.
> >
> > The current patches only do lazy lookup in data-only layers and the lookup
> > in data-only layers is always lazy.
> >
> > Data-only layers have some other advantages, for example, multiple
> > data-only uuid-less layers are allowed.
> > Please see the text below taken from the patches.
> >
> > What do you think about this direction?
> >
> > Alexander has started to test these patches.
> > If he finds no issues and if you have no objections to the concept,
> > then I will post the patches for wider review.
> >
> >
> > Thanks,
> > Amir.
> >
> > [1] https://github.com/amir73il/linux/commits/ovl-lazy-lowerdata-rc2
> >
> > Data-only lower layers
> > ----------------------
> >
> > With "metacopy" feature enabled, an overlayfs regular file may be a
> > composition of information from up to three different layers:
> >
> >  1) metadata from a file in the upper layer
> >
> >  2) st_ino and st_dev object identifier from a file in a lower layer
> >
> >  3) data from a file in another lower layer (further below)
> >
> > The "lower data" file can be on any lower layer, except from the top most
> > lower layer.
> >
> > Below the top most lower layer, any number of lower most layers may be
> > defined as "data-only" lower layers, using the double collon ("::") separator.
> >
> > For example:
> >
> >   mount -t overlay overlay -olowerdir=/lower1::/lower2:/lower3 /merged
>
> What are the rules?
>
> Is "do1::do2::lower" allowed?
> Is "do1::lower1:do2::lower2 allowed?
>

To elaborate:

lowerdir="lo1:lo2:lo3::do1:do2:do3" is allowed

:: must have non-zero lower layers on the left side
and non-zero data-only layers on the right side.

Actually, this feature originates from a request from Alexander to
respect opaque root dir in lower layers, but I preferred to make this
change of behavior opt-in so it can be tested by userspace.

I took it one step further than the opaque root dir request -
the lookup in data-only is a generic vfs_path_lookup() of an
absolute path redirect from one of the lowerdirs, with no
checking of redirect/metacopy/opque xattrs.

And then I only implemented lazy lookup for the lookup
in those new data-only layers, which made things simpler.
Please see the patches I just posted for details [1].

Thanks,
Amir.

[1] https://lore.kernel.org/linux-unionfs/20230412135412.1684197-1-amir73il@gmail.com/

Re: Lazy lowerdata lookup and data-only layers (Was: Re: Composefs:)

[Miklos Szeredi]

On Wed, 12 Apr 2023 at 16:07, Amir Goldstein <amir73il@gmail.com> wrote:

> To elaborate:
>
> lowerdir="lo1:lo2:lo3::do1:do2:do3" is allowed
>
> :: must have non-zero lower layers on the left side
> and non-zero data-only layers on the right side.

Okay.   Can you please add this to the documentation?

>
> Actually, this feature originates from a request from Alexander to
> respect opaque root dir in lower layers, but I preferred to make this
> change of behavior opt-in so it can be tested by userspace.

Not sure I get that.  Does "opaque root dir" mean that only absolute
redirects can access layers below such a layer?

I guess that's not something that works today.  Or am I mistaken?

I also don't get what you mean by testing in userspace.  Can you ellaborate?

>
> I took it one step further than the opaque root dir request -
> the lookup in data-only is a generic vfs_path_lookup() of an
> absolute path redirect from one of the lowerdirs, with no
> checking of redirect/metacopy/opque xattrs.
>
> And then I only implemented lazy lookup for the lookup
> in those new data-only layers, which made things simpler.

Okay, makes sense.  If someone hits this limitation, then we can
always start thinking about generalizing this feature.

> Please see the patches I just posted for details [1].

Will do.

Thanks,
Miklos

Re: Lazy lowerdata lookup and data-only layers (Was: Re: Composefs:)

[Amir Goldstein]

On Wed, Apr 12, 2023 at 5:21 PM Miklos Szeredi <miklos@szeredi.hu> wrote:
>
> On Wed, 12 Apr 2023 at 16:07, Amir Goldstein <amir73il@gmail.com> wrote:
>
> > To elaborate:
> >
> > lowerdir="lo1:lo2:lo3::do1:do2:do3" is allowed
> >
> > :: must have non-zero lower layers on the left side
> > and non-zero data-only layers on the right side.
>
> Okay.   Can you please add this to the documentation?
>

OK.

> >
> > Actually, this feature originates from a request from Alexander to
> > respect opaque root dir in lower layers, but I preferred to make this
> > change of behavior opt-in so it can be tested by userspace.
>
> Not sure I get that.  Does "opaque root dir" mean that only absolute
> redirects can access layers below such a layer?

Yes, that's what he wanted. to hide the subdirs being redirected to
from the namespace.

>
> I guess that's not something that works today.  Or am I mistaken?

You are not mistaken, it is not working today.

>
> I also don't get what you mean by testing in userspace.  Can you ellaborate?
>
> >
> > I took it one step further than the opaque root dir request -
> > the lookup in data-only is a generic vfs_path_lookup() of an
> > absolute path redirect from one of the lowerdirs, with no
> > checking of redirect/metacopy/opque xattrs.
> >
> > And then I only implemented lazy lookup for the lookup
> > in those new data-only layers, which made things simpler.
>
> Okay, makes sense.  If someone hits this limitation, then we can
> always start thinking about generalizing this feature.

Yap, that's what I thought.

Thanks,
Amir.