Re: Lazy lowerdata lookup and data-only layers (Was: Re: Composefs:)

[Miklos Szeredi <miklos@szeredi.hu>]

On Mon, 3 Apr 2023 at 21:00, Amir Goldstein <amir73il@gmail.com> wrote:
>
> > > >
> > > > I think lazyfollow could be enabled by default after we hashed out
> > > > all the bugs and corner cases and most importantly remove the
> > > > POC limitation of lower-only overlay.
> > > >
> [...]
> > > >
> > >
> > > Lazy follow seems to make sense.  Why does it need to be optional?
> >
> > It doesn't.
> >
> > > Does it have any advantage to *not* do lazy follow?
> > >
> >
> > Not that I can think of.
>
> Miklos,
>
> I completed writing the lazy lookup patches [1].
>
> It wasn't trivial and the first versions had many traps that took time to
> trip on, so I've made some design choices to make it safer and easier to
> land an initial improvement that will cater the composefs use case.
>
> The main design choice has to do with making lazy lowerdata lookup
> completely opt-in by defining a new type of data-only layers, such as
> the content addressable lower layer of composefs.
> The request for the data-only layers came from Alexander.
>
> The current patches only do lazy lookup in data-only layers and the lookup
> in data-only layers is always lazy.
>
> Data-only layers have some other advantages, for example, multiple
> data-only uuid-less layers are allowed.
> Please see the text below taken from the patches.
>
> What do you think about this direction?
>
> Alexander has started to test these patches.
> If he finds no issues and if you have no objections to the concept,
> then I will post the patches for wider review.
>
>
> Thanks,
> Amir.
>
> [1] https://github.com/amir73il/linux/commits/ovl-lazy-lowerdata-rc2
>
> Data-only lower layers
> ----------------------
>
> With "metacopy" feature enabled, an overlayfs regular file may be a
> composition of information from up to three different layers:
>
>  1) metadata from a file in the upper layer
>
>  2) st_ino and st_dev object identifier from a file in a lower layer
>
>  3) data from a file in another lower layer (further below)
>
> The "lower data" file can be on any lower layer, except from the top most
> lower layer.
>
> Below the top most lower layer, any number of lower most layers may be
> defined as "data-only" lower layers, using the double collon ("::") separator.
>
> For example:
>
>   mount -t overlay overlay -olowerdir=/lower1::/lower2:/lower3 /merged

What are the rules?

Is "do1::do2::lower" allowed?
Is "do1::lower1:do2::lower2 allowed?

>
> The paths of files in the "data-only" lower layers are not visible in the
> merged overlayfs directories and the metadata and st_ino/st_dev of files
> in the "data-only" lower layers are not visible in overlayfs inodes.
>
> Only the data of the files in the "data-only" lower layers may be visible
> when a "metacopy" file in one of the lower layers above it, has a "redirect"
> to the absolute path of the "lower data" file in the "data-only" lower layer.

Okay.

Thanks,
Miklos