[PATCH 0/3] Support negative dentry cache for FUSE and virtiofs

[PATCH 0/3] Support negative dentry cache for FUSE and virtiofs

[Keiichi Watanabe]

This patch series adds a new mount option called negative_dentry_timeout
for FUSE and virtio-fs filesystems. This option allows the kernel to cache
negative dentries, which are dentries that represent a non-existent file.
When this option is enabled, the kernel will skip FUSE_LOOKUP requests for
second and subsequent lookups to a non-existent file.

Unlike negative dentry caches on a normal filesystem such as ext4, the
kernel may need to refresh the cache for FUSE and virtio-fs filesystems.
This is because the kernel does not know when a FUSE server or a virtio-fs
device creates or deletes files. To address this, the new
negative_dentry_timeout option takes an expiration time for cache entries.
The appropriate timeout duration should be determined by considering how
often a FUSE server updates file paths and the amount of memory the kernel
can use for the cache.

As we evaluated the virtio-fs's performance on a guest Linux on crosvm
[1]'s virtiofs device[2], the `negative_dentry_timeout` option saved ~1
second per 10000 `stat` call against a non-existent path. The experiment
settings and results are as follows:

* Command to start VM with crosvm:
sudo crosvm run \
  --disable-sandbox \
  --cpus 1 \
  --mem 2048 \
  --rwroot debian.img \
  --shared-dir \
  /path/:my_virtiofs:type=fs:cache=always:timeout=3600 \
  -p "console=hvc0 init=/bin/bash" \
  vmlinux

* Mount command in the guest
Default:
$ mount -t virtiofs my_virtiofs ./workspace/
With negative dentry cache:
$ mount -t virtiofs -o negative_dentry_timeout=10 my_virtiofs ./workspace/

* Test commands
$ cd workspace
$ echo 3 > /proc/sys/vm/drop_caches
$ time for i in `seq 1 10000`; \
  do stat non-existent.txt 2> /dev/null; \
  done

* Results:
Default:
real 0m12.606s
user 0m3.624s
sys 0m7.756s

With `-o negative_dentry_timeout=10`:
real 0m11.276s
user 0m3.514s
sys 0m7.544s

[1]: https://crosvm.dev/book/
[2]: https://crosvm.dev/book/devices/fs.html


Keiichi Watanabe (3):
  docs: virtiofs: Fix descriptions about virtiofs mount option
  fuse: Add negative_dentry_timeout option
  virtiofs: Add negative_dentry_timeout option

 Documentation/filesystems/dax.rst      |  1 +
 Documentation/filesystems/fuse.rst     |  6 ++++++
 Documentation/filesystems/virtiofs.rst |  9 ++++++++-
 fs/fuse/dir.c                          |  3 ++-
 fs/fuse/fuse_i.h                       |  4 ++++
 fs/fuse/inode.c                        | 12 +++++++++++-
 fs/fuse/virtio_fs.c                    |  8 ++++++++
 7 files changed, 40 insertions(+), 3 deletions(-)

-- 
2.41.0.185.g7c58973941-goog

[PATCH 1/3] docs: virtiofs: Fix descriptions about virtiofs mount option

[Keiichi Watanabe]

Since virtiofs and FUSE don't share mount options, fixes the
description. Also, explains DAX option as the only virtiofs-specific
option so far.

Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
---

 Documentation/filesystems/dax.rst      | 1 +
 Documentation/filesystems/virtiofs.rst | 5 ++++-
 2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/Documentation/filesystems/dax.rst b/Documentation/filesystems/dax.rst
index c04609d8ee24..77c5d4550ecd 100644
--- a/Documentation/filesystems/dax.rst
+++ b/Documentation/filesystems/dax.rst
@@ -167,6 +167,7 @@ Setting the `FS_XFLAG_DAX` flag (specifically or through inheritance) occurs eve
 if the underlying media does not support dax and/or the filesystem is
 overridden with a mount option.
 
+.. _virtiofs-dax:
 
 Enabling DAX on virtiofs
 ----------------------------
diff --git a/Documentation/filesystems/virtiofs.rst b/Documentation/filesystems/virtiofs.rst
index fd4d2484e949..fdec5a7840f7 100644
--- a/Documentation/filesystems/virtiofs.rst
+++ b/Documentation/filesystems/virtiofs.rst
@@ -43,7 +43,10 @@ Mount options
 -------------
 
 virtiofs supports general VFS mount options, for example, remount,
-ro, rw, context, etc. It also supports FUSE mount options.
+ro, rw, context, etc. Also, virtiofs has its own options.
+
+dax[=always,never,inode]
+  Enable direct access for files. See :ref:`virtiofs-dax`.
 
 atime behavior
 ^^^^^^^^^^^^^^
-- 
2.41.0.185.g7c58973941-goog

[PATCH 2/3] fuse: Add negative_dentry_timeout option

[Keiichi Watanabe]

Add `negative_dentry_timeout` mount option for FUSE to cache negative
dentries for the specified duration.
With this option enabled, second and subsequent lookups to non-existent
files can be omitted.

A user needs to specify how often each negative_dentry cache should be
refreshed. This is because the kernel has no idea when a FUSE server
creates or deletes files, unlike normal filesystems such as ext4 where
all of file operations are managed by the kernel.
The appropriate timeout duration should be determined by considering how
often a FUSE server update file paths and the amount of memory the
kernel can use the cache.

Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
---

 Documentation/filesystems/fuse.rst |  6 ++++++
 fs/fuse/dir.c                      |  3 ++-
 fs/fuse/fuse_i.h                   |  4 ++++
 fs/fuse/inode.c                    | 12 +++++++++++-
 4 files changed, 23 insertions(+), 2 deletions(-)

diff --git a/Documentation/filesystems/fuse.rst b/Documentation/filesystems/fuse.rst
index 1e31e87aee68..6d3b55476504 100644
--- a/Documentation/filesystems/fuse.rst
+++ b/Documentation/filesystems/fuse.rst
@@ -103,6 +103,12 @@ blksize=N
   Set the block size for the filesystem.  The default is 512.  This
   option is only valid for 'fuseblk' type mounts.
 
+negative_dentry_timeout=N
+  Set the time in seconds to keep negative dentry cache. If a lookup for
+  a path fails, the kernel won't do another lookup for this period of
+  time. The default value is 0, which means that negative dentries are not
+  cached.
+
 Control filesystem
 ==================
 
diff --git a/fs/fuse/dir.c b/fs/fuse/dir.c
index 35bc174f9ba2..21ff395467ab 100644
--- a/fs/fuse/dir.c
+++ b/fs/fuse/dir.c
@@ -453,7 +453,8 @@ static struct dentry *fuse_lookup(struct inode *dir, struct dentry *entry,
 	if (outarg_valid)
 		fuse_change_entry_timeout(entry, &outarg);
 	else
-		fuse_invalidate_entry_cache(entry);
+		fuse_dentry_settime(entry,
+		time_to_jiffies(get_fuse_mount(dir)->negative_dentry_timeout, 0));
 
 	if (inode)
 		fuse_advise_use_readdirplus(dir);
diff --git a/fs/fuse/fuse_i.h b/fs/fuse/fuse_i.h
index 9b7fc7d3c7f1..bbfe53635329 100644
--- a/fs/fuse/fuse_i.h
+++ b/fs/fuse/fuse_i.h
@@ -516,6 +516,7 @@ struct fuse_fs_context {
 	unsigned int max_read;
 	unsigned int blksize;
 	const char *subtype;
+	unsigned int negative_dentry_timeout;
 
 	/* DAX device, may be NULL */
 	struct dax_device *dax_dev;
@@ -860,6 +861,9 @@ struct fuse_mount {
 	 */
 	struct super_block *sb;
 
+	/* Timeout on negative denty caches in seconds */
+	u32 negative_dentry_timeout;
+
 	/* Entry on fc->mounts */
 	struct list_head fc_entry;
 };
diff --git a/fs/fuse/inode.c b/fs/fuse/inode.c
index d66070af145d..45ed0c52f8a6 100644
--- a/fs/fuse/inode.c
+++ b/fs/fuse/inode.c
@@ -654,7 +654,8 @@ enum {
 	OPT_ALLOW_OTHER,
 	OPT_MAX_READ,
 	OPT_BLKSIZE,
-	OPT_ERR
+	OPT_ERR,
+	OPT_NEGATIVE_DENTRY_TIMEOUT
 };
 
 static const struct fs_parameter_spec fuse_fs_parameters[] = {
@@ -668,6 +669,7 @@ static const struct fs_parameter_spec fuse_fs_parameters[] = {
 	fsparam_u32	("max_read",		OPT_MAX_READ),
 	fsparam_u32	("blksize",		OPT_BLKSIZE),
 	fsparam_string	("subtype",		OPT_SUBTYPE),
+	fsparam_u32	("negative_dentry_timeout", OPT_NEGATIVE_DENTRY_TIMEOUT),
 	{}
 };
 
@@ -751,6 +753,10 @@ static int fuse_parse_param(struct fs_context *fsc, struct fs_parameter *param)
 		ctx->blksize = result.uint_32;
 		break;
 
+	case OPT_NEGATIVE_DENTRY_TIMEOUT:
+		ctx->negative_dentry_timeout = result.uint_32;
+		break;
+
 	default:
 		return -EINVAL;
 	}
@@ -1482,6 +1488,10 @@ static int fuse_get_tree_submount(struct fs_context *fsc)
 		return -ENOMEM;
 
 	fm->fc = fuse_conn_get(fc);
+	fm->negative_dentry_timeout =
+		((struct fuse_fs_context *)fsc->fs_private)
+			->negative_dentry_timeout;
+
 	fsc->s_fs_info = fm;
 	sb = sget_fc(fsc, NULL, set_anon_super_fc);
 	if (fsc->s_fs_info)
-- 
2.41.0.185.g7c58973941-goog

Re: [PATCH 2/3] fuse: Add negative_dentry_timeout option

[Miklos Szeredi]

On Tue, 20 Jun 2023 at 17:14, Keiichi Watanabe <keiichiw@chromium.org> wrote:
>
> Add `negative_dentry_timeout` mount option for FUSE to cache negative
> dentries for the specified duration.

This is already possible, no kernel changes needed.  See e.g.
xmp_init() in libfuse/example/passthrough.c.

Thanks,
Miklos

Re: [PATCH 2/3] fuse: Add negative_dentry_timeout option

[Miklos Szeredi]

On Wed, 21 Jun 2023 at 00:53, Junichi Uekawa (上川純一) <uekawa@google.com> wrote:
>
> Hi
>
>
>
> 2023年6月21日(水) 4:28 Miklos Szeredi <miklos@szeredi.hu>:
>>
>> On Tue, 20 Jun 2023 at 17:14, Keiichi Watanabe <keiichiw@chromium.org> wrote:
>> >
>> > Add `negative_dentry_timeout` mount option for FUSE to cache negative
>> > dentries for the specified duration.
>>
>> This is already possible, no kernel changes needed.  See e.g.
>> xmp_init() in libfuse/example/passthrough.c.
>>
>
> Thank you for the pointer!
>
> So reading libfuse/fuse.c, fuse_lib_lookup does a reply with e.ino=0 err=0 (instead of ENOENT) with e.entry_timeout=negative_timeout,
> for each lookup (and there's no global configuration but that's okay) ?

Yes.

Thanks,
Miklos

Re: [PATCH 2/3] fuse: Add negative_dentry_timeout option

[Keiichi Watanabe]

On Wed, Jun 21, 2023 at 1:07 PM Miklos Szeredi <miklos@szeredi.hu> wrote:
>
> On Wed, 21 Jun 2023 at 00:53, Junichi Uekawa (上川純一) <uekawa@google.com> wrote:
> >
> > Hi
> >
> >
> >
> > 2023年6月21日(水) 4:28 Miklos Szeredi <miklos@szeredi.hu>:
> >>
> >> On Tue, 20 Jun 2023 at 17:14, Keiichi Watanabe <keiichiw@chromium.org> wrote:
> >> >
> >> > Add `negative_dentry_timeout` mount option for FUSE to cache negative
> >> > dentries for the specified duration.
> >>
> >> This is already possible, no kernel changes needed.  See e.g.
> >> xmp_init() in libfuse/example/passthrough.c.
> >>
> >
> > Thank you for the pointer!
> >
> > So reading libfuse/fuse.c, fuse_lib_lookup does a reply with e.ino=0 err=0 (instead of ENOENT) with e.entry_timeout=negative_timeout,
> > for each lookup (and there's no global configuration but that's okay) ?
>
> Yes.


Oh, good to know!
I could make it work in our VMM (crosvm) without any kernel changes.
https://crrev.com/c/4630879
Thanks a lot!

Keiichi


>
>
> Thanks,
> Miklos

[PATCH 3/3] virtiofs: Add negative_dentry_timeout option

[Keiichi Watanabe]

Add negative_dentry_timeout mount option to virtiofs to caching negative
dentry on the guest side. When the host virito-fs device has an exclusive
access to the file system and the machine has enough memory, one can
specify a long time as the timeout.

This option saves ~1 second per 10,000 stat request for non-existing paths.

Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
---

 Documentation/filesystems/virtiofs.rst | 4 ++++
 fs/fuse/virtio_fs.c                    | 8 ++++++++
 2 files changed, 12 insertions(+)

diff --git a/Documentation/filesystems/virtiofs.rst b/Documentation/filesystems/virtiofs.rst
index fdec5a7840f7..b045ef2223de 100644
--- a/Documentation/filesystems/virtiofs.rst
+++ b/Documentation/filesystems/virtiofs.rst
@@ -48,6 +48,10 @@ ro, rw, context, etc. Also, virtiofs has its own options.
 dax[=always,never,inode]
   Enable direct access for files. See :ref:`virtiofs-dax`.
 
+negative_dentry_timeout=N
+  Set the time in seconds to keep negative dentry cache. Same as the FUSE's
+  mount option.
+
 atime behavior
 ^^^^^^^^^^^^^^
 
diff --git a/fs/fuse/virtio_fs.c b/fs/fuse/virtio_fs.c
index 4d8d4f16c727..bbbd840510f9 100644
--- a/fs/fuse/virtio_fs.c
+++ b/fs/fuse/virtio_fs.c
@@ -99,11 +99,13 @@ static const struct constant_table dax_param_enums[] = {
 enum {
 	OPT_DAX,
 	OPT_DAX_ENUM,
+	OPT_NEGATIVE_DENTRY_TIMEOUT,
 };
 
 static const struct fs_parameter_spec virtio_fs_parameters[] = {
 	fsparam_flag("dax", OPT_DAX),
 	fsparam_enum("dax", OPT_DAX_ENUM, dax_param_enums),
+	fsparam_u32 ("negative_dentry_timeout", OPT_NEGATIVE_DENTRY_TIMEOUT),
 	{}
 };
 
@@ -125,6 +127,9 @@ static int virtio_fs_parse_param(struct fs_context *fsc,
 	case OPT_DAX_ENUM:
 		ctx->dax_mode = result.uint_32;
 		break;
+	case OPT_NEGATIVE_DENTRY_TIMEOUT:
+		ctx->negative_dentry_timeout = result.uint_32;
+		break;
 	default:
 		return -EINVAL;
 	}
@@ -1416,6 +1421,7 @@ static int virtio_fs_get_tree(struct fs_context *fsc)
 	struct super_block *sb;
 	struct fuse_conn *fc = NULL;
 	struct fuse_mount *fm;
+	struct fuse_fs_context *ffc;
 	unsigned int virtqueue_size;
 	int err = -EIO;
 
@@ -1468,6 +1474,8 @@ static int virtio_fs_get_tree(struct fs_context *fsc)
 
 		sb->s_flags |= SB_ACTIVE;
 	}
+	ffc = fsc->fs_private;
+	fm->negative_dentry_timeout = ffc->negative_dentry_timeout;
 
 	WARN_ON(fsc->root);
 	fsc->root = dget(sb->s_root);
-- 
2.41.0.185.g7c58973941-goog