RE: [fuse-devel] Cross-host entry caching and file open/create

[Ken Schalk <kschalk@nvidia.com>]

On Tue, Nov 24, 2020 Miklos Szeredi <miklos@szeredi.hu> wrote:
> ESTALE is the correct error value here, not ENOENT.  I agree this is
> subtle and not well documented, but it's quite logical: the cache
> contains stale lookup information and hence open finds the file
> non-existent.

Thank you for explaining the use of ESTALE here.  It definitely does
resolve my O_CREAT without O_EXCL issue in a better way.  I would
agree that this is not well documented, as other than the kernel
source itself the only reference I can find to this is in
Documentation/filesystems/path-lookup.rst (which I admit I had not
read prior to this).

Do you have any recommendation on the portion of my patch relating to
negative directory entries?  This is another area where currently we
don't seem to be able to get precisely the same behavior under FUSE as
under NFS.  Specifically, a file open under NFS when there is a
negative directory entry in the kernel's cache always seems to result
in a re-validating GETATTR RPC to the NFS server (which, if the file
has been recently created, allows the client to become aware of its
existence).  Is there a better way we can achieve this with FUSE than
the change in the other portion of my patch (forcing a lookup call on
any open when there is a negative directory entry for the target of
the open)?

To again be specific, the sequence of events that I'm trying to get to
work differently is:

1. A lookup occurs on host X for name which does not exist, and our
   FUSE daemon's response causes the kernel to cache a negative
   directory entry.  (I know that we can disable kernel-level caching
   of negative lookup results, but we would rather not do so for
   performance.)

2. The name is created as a file through our through our FUSE
   distributed filesystem on host Y.

3. An open(2) call for the recently created file occurs on host X.
   Because the kernel has a cached negative dentry for the file (which
   now exists), the open(2) call fails with ENOENT without any FUSE
   requests.

This comes up in the context of a distributed build system where the
work of building an intermediate file may be dispatched to a remote
host.  Under NFS, the open(2) consults the NFS server and discovers
that the name now exists.

--Ken Schalk