Re: Overlayfs with writable lower layer

From: Hao Luo <haoluo@google.com>
To: Amir Goldstein <amir73il@gmail.com>
Cc: Miklos Szeredi <miklos@szeredi.hu>,
	"linux-fsdevel@vger.kernel.org" <linux-fsdevel@vger.kernel.org>
Subject: Re: Overlayfs with writable lower layer
Date: Tue, 13 Sep 2022 11:26:07 -0700	[thread overview]
Message-ID: <CA+khW7jQ6fZbEgzxCafsaaTyv7ze58bd9hQ0HBH4R+dQyRaqog@mail.gmail.com> (raw)
In-Reply-To: <CAOQ4uxjUbwKmLAO-jTE3y6EnH2PNw0+V=oXNqNyD+H9U+nX49g@mail.gmail.com>

On Mon, Sep 12, 2022 at 11:28 AM Amir Goldstein <amir73il@gmail.com> wrote:
>
> On Mon, Sep 12, 2022 at 8:43 PM Hao Luo <haoluo@google.com> wrote:
> >
> > On Mon, Sep 12, 2022 at 8:40 AM Amir Goldstein <amir73il@gmail.com> wrote:
> > >
> > > On Mon, Sep 12, 2022 at 5:22 PM Miklos Szeredi <miklos@szeredi.hu> wrote:
> > > >
> > > > I imagine the "switch" layer for a HSM would be simple enough:
> > > >
> > > > a) if file exists on fastest layer (upper) then take that
> > > > b) if not then fall back to fuse layer (lower) .
> > > >
> > > > It's almost like a read-only overlayfs (no copy up) except it would be
> > > > read-write and copy-up/down would be performed by the server as
> > > > needed. No page cache duplication for upper, and AFAICS no corner
> > > > cases that overlayfs has, since all layers are consistent (the fuse
> > > > layer would reference the upper if that is currently the up-to-date
> > > > one).
> > >
> > > On recent LSF/MM/BPF, BPF developers asked me about using overlayfs
> > > for something that looks like the above - merging of non overlapping layers
> > > without any copy-up/down, but with write to lower.
> > >
> > > I gave them the same solution (overlayfs without copy-up)
> > > but I said I didn't know what you would think about this overlayfs mode
> > > and I also pointed them to the eBPF-FUSE developers as another
> > > possible solution to their use case.
> > >
> >
> > Thanks Amir for adding me in the thread. This idea is very useful for
> > BPF use cases.
>
[...]
>
> Am I to understand that the eBPF-FUSE option currently
> does not fit your needs (maybe because it is not ready yet)?
>

Yeah, mostly because eBPF-FUSE is not ready. I talked to Alessio and
his colleague after LSF/MM/BPF. They were distracted from eBPF-FUSE
development at that time and I didn't follow up, working on other BPF
stuff.

> >
[...]
> Can't say I was able to understand the description, but let me
> try to write the requirement in overlayfs terminology.
> Please correct me if I am wrong.
>
> 1. The "lower" fs (cgroupfs?) is a "remote" fs where directories
>     may appear or disappear due to "remote" events
>

Right. Seems we are aligned on this.

> I think there were similar requirements to support changes
> to lower fs which in a network fs in the past.
> As long as those directories are strictly lower that should be
> possible.
>
> Does upper fs have directories of the same name that need to
> be merged with those lower dirs?
>

No, I don't think so. Upper fs should only have files in my use case.

> 2. You did not mention this but IIRC, the lower fs has (pseudo)
>     files that you want to be able to write to, and those files never
>     exist in the upper fs
>
> That translates to allowing write to lower fs without triggering copy-up
>

Writing to lower is not needed right now.

> 3. The upper fs files are all "pure" - that means those file paths
>     do not exist in lower fs
>
> Creating files in the upper fs is normal in overlayfs, so not a problem.
>

Yes. If used in an expected way, the lower fs won't have file paths
that also exist in the upper.

> 4. Merging upper and lower directory content is usually not needed??
>
> It must be needed for the root dir at least but where else?
> In which directories are the upper files created?
> Which directories are "pure" (containing either lower or upper files)
> and which directories are "merge" of upper and lower children?
>

In my use case, if that's doable, all the directories are "pure",
except the root dir, and they are in the lower. The files are either
from upper or from lower, so no merge. This should be sufficient for
the BPF use case.

> If the answer is that some directories exist both in upper and in lower
> and that those directories should be merged for the unified view then
> this is standard overlayfs behavior.
>
> Once I get the requirement in order I will try to write coherent
> semantics for this mode and see if it can meet the needs of other
> projects, such as HSM with FUSE as lower fs.

Thanks Amir. I want to clarify that with my very very limited
knowledge on FS, I can't say what I am thinking right now is
absolutely correct. Please take my request for features with a grain
of salt and feel free to pick the semantics that you see make most
sense.

Hao