Re: [PATCH v3 0/6] Composefs: an opportunistically sharing verified image filesystem

[Miklos Szeredi <miklos@szeredi.hu>]

On Mon, 6 Feb 2023 at 14:31, Amir Goldstein <amir73il@gmail.com> wrote:
>
> > > > My little request again, could you help benchmark on your real workload
> > > > rather than "ls -lR" stuff?  If your hard KPI is really what as you
> > > > said, why not just benchmark the real workload now and write a detailed
> > > > analysis to everyone to explain it's a _must_ that we should upstream
> > > > a new stacked fs for this?
> > > >
> > >
> > > I agree that benchmarking the actual KPI (boot time) will have
> > > a much stronger impact and help to build a much stronger case
> > > for composefs if you can prove that the boot time difference really matters.
> > >
> > > In order to test boot time on fair grounds, I prepared for you a POC
> > > branch with overlayfs lazy lookup:
> > > https://github.com/amir73il/linux/commits/ovl-lazy-lowerdata
> >
> > Sorry about being late to the party...
> >
> > Can you give a little detail about what exactly this does?
> >
>
> Consider a container image distribution system, with base images
> and derived images and instruction on how to compose these images
> using overlayfs or other methods.
>
> Consider a derived image L3 that depends on images L2, L1.
>
> With the composefs methodology, the image distribution server splits
> each image is split into metadata only (metacopy) images M3, M2, M1
> and their underlying data images containing content addressable blobs
> D3, D2, D1.
>
> The image distribution server goes on to merge the metadata layers
> on the server, so U3 = M3 + M2 + M1.
>
> In order to start image L3, the container client will unpack the data layers
> D3, D2, D1 to local fs normally, but the server merged U3 metadata image
> will be distributed as a read-only fsverity signed image that can be mounted
> by mount -t composefs U3.img (much like mount -t erofs -o loop U3.img).
>
> The composefs image format contains "redirect" instruction to the data blob
> path and an fsverity signature that can be used to verify the redirected data
> content.
>
> When composefs authors proposed to merge composefs, Gao and me
> pointed out that the same functionality can be achieved with minimal changes
> using erofs+overlayfs.
>
> Composefs authors have presented ls -lR time and memory usage benchmarks
> that demonstrate how composefs performs better that erofs+overlayfs in
> this workload and explained that the lookup of the data blobs is what takes
> the extra time and memory in the erofs+overlayfs ls -lR test.
>
> The lazyfollow POC optimizes-out the lowerdata lookup for the ls -lR
> benchmark, so that composefs could be compared to erofs+overlayfs.

Got it, thanks.

>
> To answer Alexander's question:
>
> > Cool. I'll play around with this. Does this need to be an opt-in
> > option in the final version? It feels like this could be useful to
> > improve performance in general for overlayfs, for example when
> > metacopy is used in container layers.
>
> I think lazyfollow could be enabled by default after we hashed out
> all the bugs and corner cases and most importantly remove the
> POC limitation of lower-only overlay.
>
> The feedback that composefs authors are asking from you
> is whether you will agree to consider adding the "lazyfollow
> lower data" optimization and "fsverity signature for metacopy"
> feature to overlayfs?
>
> If you do agree, the I think they should invest their resources
> in making those improvements to overlayfs and perhaps
> other improvements to erofs, rather than proposing a new
> specialized filesystem.

Lazy follow seems to make sense.  Why does it need to be optional?
Does it have any advantage to *not* do lazy follow?

Not sure I follow the fsverity requirement.  For overlay+erofs case
itsn't it enough to verify the erofs image?

Thanks,
Miklos






>
> Thanks,
> Amir.