archive mirror
 help / color / mirror / Atom feed
From: Amir Goldstein <>
To: Sargun Dhillon <>
Cc: overlayfs <>,
	Miklos Szeredi <>
Subject: Re: Detaching lower layers (Was: Lazy Loading Layers)
Date: Mon, 29 May 2023 18:15:48 +0300	[thread overview]
Message-ID: <> (raw)
In-Reply-To: <20210125194848.GA12389@ircssh-2.c.rugged-nimbus-611.internal>

On Mon, Jan 25, 2021 at 9:54 PM Sargun Dhillon <> wrote:
> One of the projects I'm playing with for containers is lazy-loading of layers.
> We've found that less than 10% of the files on a layer actually get used, which
> is an unfortunate waste. It also means in some cases downloading ~100s of MB, or
> ~1s of GB of files before starting a container workload. This is unfortunate.
> It would be nice if there was a way to start a container workload, and have
> it so that if it tries to access and unpopulated (not yet downloaded) part
> of the filesystem block while trying to be accessed. This is trivial to do
> if the "lowest" layer is FUSE, where one can just stall in userspace on
> loads. Unfortunately, AFAIK, there's not a good way to swap out the FUSE
> filesystem with the "real" filesystem once it's done fully populating,
> and you have to pay for the full FUSE cost on each read / write.
> I've tossed around:
> 1. Mutable lowerdirs and having something like this:
> layer0 --> Writeable space
> layer1 --> Real XFS filesystem
> layer2 --> FUSE FS
> and if there is a "miss" on layer 1, it will then look it up on
> layer 2 while layer 1 is being populated. Then the FUSE FS can block.
> This is neat, but it requires the FUSE FS to always be up, and incurs
> a userspace bounce on every miss.
> It also means things like metadata only copies don't work.
> Does anyone have a suggestion of a mechanism to handle this? I've looked into
> swapping out layers on the fly, and what it would take to add a mechanism like
> userfaultfd to overlayfs, but I was wondering if anything like this was already
> built, or if someone has thought it through more than me.

Hi Sargun,

I believe that this is the use case that you asked me about in LSFMM,
at least the lower part of layer1+layer2. Is that correct?

You did not mention three layers in the use case that you described
Is that because you decided that layer0 and layer1 can be combined?

Technically, you can also setup a nested overlay with the lower overlay
layer1+layer2 only doing the lazy loading of the remote read-only layer
and the upper overlay is composed of layer0+ovl(layer1+layer2), but this
nested overlay configuration has some limitations.

Anyway, I have talked with Miklos about the use case that requires
detaching the lowermost FUSE layer eventually and the solution that
we discussed was to gradually "opaquify" directories whose entire
descendant hierarchy is fully copied up at readdir time.

I have prepared POC patches for this design:

This was tested using the following patch to unionmount-testsuite:

commit 026e73c37f3993f56e76128a267e54faedf2322c
Author: Amir Goldstein <>
Date:   Mon May 29 17:01:55 2023 +0300

    Test detaching lower fs

    Test that with xino=nofollow, after copying up all files and listing
    all the directories in DFS order, the lower fs can be detached.

    Signed-off-by: Amir Goldstein <>

diff --git a/ b/
index e905b83..4fad5dd 100644
--- a/
+++ b/
@@ -54,3 +54,13 @@ def mount_union(ctx):
         ctx.note_upper_fs(upper_mntroot, testdir, union_mntroot + "/f")
+        if cfg.is_xino():
+            # Copy up everything, set all dirs opaque and then detach lower fs.
+            # Instead of iterating in DFS order we iterate 4 times as the depth
+            # of the dataset tree - on every iteration, level 4-i
becomes opaque.
+            system("chown -R 0.0 " + union_mntroot)
+            system("find " + union_mntroot + " -inum 0")
+            system("find " + union_mntroot + " -inum 0")
+            system("find " + union_mntroot + " -inum 0")
+            system("find " + union_mntroot + " -inum 0")
+            system("xfs_io -x -c shutdown " + lower_mntroot)
diff --git a/run b/run
index 3a6efc3..f8116c1 100755
--- a/run
+++ b/run
@@ -219,7 +219,7 @@ if redirect_dir is False:

 # Auto-upgrade xino=auto to xino=on for kernel < v5.7
 if xino:
-    cfg.add_mntopt("xino=on")
+    cfg.add_mntopt("xino=nofollow")


It should be pretty self-explanatory - after mounting the overlay, all
lower files
are copied up using chown -R (no metacopy) and then the overlay is iterated
several times, until all the merge directories iterations notice that
there is nothing
interesting in the lower dirs, so they all become opaque.
At this point, the lowest xfs layer is being shutdown and the tests are run.
With the 4*find iterations, none of the tests get EIO.

This does not mean that the lower xfs can be cleanly unmounted - there may
still be references to dentries/inodes from the lower fs, but
overlayfs never calls
any filesystem methods on the lower dentry/inodes - specifically lookup misses
in the upper dir do not end up looking in the lower dir.

The reason that I used an opt-in mount option (xino=nofollow) to enable this
functionality is because even after all files have been copied up,
overlayfs does
currently access one bit of information from the lower fs - it calls
getattr() to get
st_ino from the lower file/directory in order to preserve st_ino across copy up.

I used an opt-in mount option to allow st_ino to change across copy up.
I hope this change of behavior is acceptable for your use case.
Note that after the completion of the migration process (e.g. chown -R + 4*find)
all inode numbers are stabilized.

Are you interested in testing these patches?
If you indicate that they are useful to you, I can post them for review
and in that case, I would appreciate if you can write the xfstests
for the feature.


  parent reply	other threads:[~2023-05-29 15:16 UTC|newest]

Thread overview: 5+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-01-25 19:48 Lazy Loading Layers (Userfaultfd for filesystems?) Sargun Dhillon
2021-01-26  5:18 ` Amir Goldstein
2021-01-26 13:12   ` Alessio Balsini
2023-05-29 15:15 ` Amir Goldstein [this message]
2023-05-29 17:50   ` Detaching lower layers (Was: Lazy Loading Layers) Rodrigo Campos

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='' \ \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).