Re: Overlayfs with writable lower layer

From: Amir Goldstein <amir73il@gmail.com>
To: Hao Luo <haoluo@google.com>
Cc: Miklos Szeredi <miklos@szeredi.hu>,
	"linux-fsdevel@vger.kernel.org" <linux-fsdevel@vger.kernel.org>
Subject: Re: Overlayfs with writable lower layer
Date: Mon, 12 Sep 2022 21:28:43 +0300	[thread overview]
Message-ID: <CAOQ4uxjUbwKmLAO-jTE3y6EnH2PNw0+V=oXNqNyD+H9U+nX49g@mail.gmail.com> (raw)
In-Reply-To: <CA+khW7hviAT6DbNORYKcatOV1cigGyrd_1mH-oMwehafobVXVg@mail.gmail.com>

[Changing subject, reduce CC list and add move to overlayfs list]

On Mon, Sep 12, 2022 at 8:43 PM Hao Luo <haoluo@google.com> wrote:
>
> Sorry, resend, my response was bounced back by mail system due to not
> using plain text.
>
> On Mon, Sep 12, 2022 at 8:40 AM Amir Goldstein <amir73il@gmail.com> wrote:
> >
> > On Mon, Sep 12, 2022 at 5:22 PM Miklos Szeredi <miklos@szeredi.hu> wrote:
> > >
> > > I imagine the "switch" layer for a HSM would be simple enough:
> > >
> > > a) if file exists on fastest layer (upper) then take that
> > > b) if not then fall back to fuse layer (lower) .
> > >
> > > It's almost like a read-only overlayfs (no copy up) except it would be
> > > read-write and copy-up/down would be performed by the server as
> > > needed. No page cache duplication for upper, and AFAICS no corner
> > > cases that overlayfs has, since all layers are consistent (the fuse
> > > layer would reference the upper if that is currently the up-to-date
> > > one).
> >
> > On recent LSF/MM/BPF, BPF developers asked me about using overlayfs
> > for something that looks like the above - merging of non overlapping layers
> > without any copy-up/down, but with write to lower.
> >
> > I gave them the same solution (overlayfs without copy-up)
> > but I said I didn't know what you would think about this overlayfs mode
> > and I also pointed them to the eBPF-FUSE developers as another
> > possible solution to their use case.
> >
>
> Thanks Amir for adding me in the thread. This idea is very useful for
> BPF use cases.

Hi Hao,

Thanks for chiming in.
This thread has long diverged from the FUSE_PASSTHROUGH
patch review so I started a new thread to discuss the overlayfs
option.

Am I to understand that the eBPF-FUSE option currently
does not fit your needs (maybe because it is not ready yet)?

>
> A bit more context here: we were thinking of overlaying two
> filesystems together to create a view that extends the filesystem at
> the lower layer. In our design, the lower layer is a pseudo
> filesystem, which one can _not_ create/delete files, but make
> directories _indirectly_, via creating a kernel object; the upper is
> bpf filesystem, from which, one can create files. The file's purpose
> is to describe the directory in the lower layer, that is, to describe
> the kernel object that directory corresponds to.
>
> With the flexibility brought by BPF, it can be a quite flexible
> solution to query kernel objects' states.
>

Can't say I was able to understand the description, but let me
try to write the requirement in overlayfs terminology.
Please correct me if I am wrong.

1. The "lower" fs (cgroupfs?) is a "remote" fs where directories
    may appear or disappear due to "remote" events

I think there were similar requirements to support changes
to lower fs which in a network fs in the past.
As long as those directories are strictly lower that should be
possible.

Does upper fs have directories of the same name that need to
be merged with those lower dirs?

2. You did not mention this but IIRC, the lower fs has (pseudo)
    files that you want to be able to write to, and those files never
    exist in the upper fs

That translates to allowing write to lower fs without triggering copy-up

3. The upper fs files are all "pure" - that means those file paths
    do not exist in lower fs

Creating files in the upper fs is normal in overlayfs, so not a problem.

4. Merging upper and lower directory content is usually not needed??

It must be needed for the root dir at least but where else?
In which directories are the upper files created?
Which directories are "pure" (containing either lower or upper files)
and which directories are "merge" of upper and lower children?

If the answer is that some directories exist both in upper and in lower
and that those directories should be merged for the unified view then
this is standard overlayfs behavior.

Once I get the requirement in order I will try to write coherent
semantics for this mode and see if it can meet the needs of other
projects, such as HSM with FUSE as lower fs.

Thanks,
Amir.