[Virtio-fs] Deleting files when using NFS as a shared folder

[Virtio-fs] Deleting files when using NFS as a shared folder

[Gal Hammer]

[-- Attachment #1: Type: text/plain, Size: 1154 bytes --]

Hello,

When using NFS as a shared folder (mount type nfs4) with a Linux guest I
have the following issue:

Guest:
$ ls -la /mnt/shared
total 8
drwxr-xrwx.  2  135  135 4096 Aug  2 13:08 .
dr-xr-xr-x. 17 root   root    224 May 23 10:58 ..
-rw-r--rw-.  1  135  135   27 Aug  2 13:07 readme.txt

Host:
$ rm readme.txt

Guest:
$ ls -la /mnt/shared
total 8
drwxr-xrwx.  2  135  135 4096 Aug  2 13:10 .
dr-xr-xr-x. 17 root   root    224 May 23 10:58 ..
-rw-r--rw-.  1  135  135   27 Aug  2 13:07 .nfs0000000001b600d000000005

Guest:
$ cat /mnt/shared/readme.txt
This is a readme.txt file.

So it seems that the virtiofsd has a reference to the file which the guest
is not aware of and is unable to send a FUSE_FORGET message. This results
in a file not actually deleted (renamed to .nfsXXX) and is still accessible
by the guest.

I have a similar problem when deleting a file from a Windows guest side.
The FUSE_READDIR(PLUS) commands add a reference count to files which the OS
doesn't have a file context for. However I was able to solve it (for now?)
by keeping track of returned files' inodes.

Is this behaviour current and by design?

Thanks,

    Gal.

[-- Attachment #2: Type: text/html, Size: 3559 bytes --]

Re: [Virtio-fs] Deleting files when using NFS as a shared folder

[Dr. David Alan Gilbert]

* Gal Hammer (ghammer@redhat.com) wrote:
> Hello,
> 
> When using NFS as a shared folder (mount type nfs4) with a Linux guest I
> have the following issue:
> 
> Guest:
> $ ls -la /mnt/shared
> total 8
> drwxr-xrwx.  2  135  135 4096 Aug  2 13:08 .
> dr-xr-xr-x. 17 root   root    224 May 23 10:58 ..
> -rw-r--rw-.  1  135  135   27 Aug  2 13:07 readme.txt
> 
> Host:
> $ rm readme.txt
> 
> Guest:
> $ ls -la /mnt/shared
> total 8
> drwxr-xrwx.  2  135  135 4096 Aug  2 13:10 .
> dr-xr-xr-x. 17 root   root    224 May 23 10:58 ..
> -rw-r--rw-.  1  135  135   27 Aug  2 13:07 .nfs0000000001b600d000000005
> 
> Guest:
> $ cat /mnt/shared/readme.txt
> This is a readme.txt file.
> 
> So it seems that the virtiofsd has a reference to the file which the guest
> is not aware of and is unable to send a FUSE_FORGET message. This results
> in a file not actually deleted (renamed to .nfsXXX) and is still accessible
> by the guest.
> 
> I have a similar problem when deleting a file from a Windows guest side.
> The FUSE_READDIR(PLUS) commands add a reference count to files which the OS
> doesn't have a file context for. However I was able to solve it (for now?)
> by keeping track of returned files' inodes.
> 
> Is this behaviour current and by design?

Current problem, not really by design; the problem is the O_PATH files
that we have open for the inodes.  I thought if the guest sent the
forget for the file then it got closed.

Dave


> Thanks,
> 
>     Gal.

> _______________________________________________
> Virtio-fs mailing list
> Virtio-fs@redhat.com
> https://listman.redhat.com/mailman/listinfo/virtio-fs

-- 
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Virtio-fs] Deleting files when using NFS as a shared folder

[Gal Hammer]

[-- Attachment #1: Type: text/plain, Size: 2049 bytes --]

On Mon, 2 Aug 2021 at 13:36, Dr. David Alan Gilbert <dgilbert@redhat.com>
wrote:

> * Gal Hammer (ghammer@redhat.com) wrote:
> > Hello,
> >
> > When using NFS as a shared folder (mount type nfs4) with a Linux guest I
> > have the following issue:
> >
> > Guest:
> > $ ls -la /mnt/shared
> > total 8
> > drwxr-xrwx.  2  135  135 4096 Aug  2 13:08 .
> > dr-xr-xr-x. 17 root   root    224 May 23 10:58 ..
> > -rw-r--rw-.  1  135  135   27 Aug  2 13:07 readme.txt
> >
> > Host:
> > $ rm readme.txt
> >
> > Guest:
> > $ ls -la /mnt/shared
> > total 8
> > drwxr-xrwx.  2  135  135 4096 Aug  2 13:10 .
> > dr-xr-xr-x. 17 root   root    224 May 23 10:58 ..
> > -rw-r--rw-.  1  135  135   27 Aug  2 13:07 .nfs0000000001b600d000000005
> >
> > Guest:
> > $ cat /mnt/shared/readme.txt
> > This is a readme.txt file.
> >
> > So it seems that the virtiofsd has a reference to the file which the
> guest
> > is not aware of and is unable to send a FUSE_FORGET message. This results
> > in a file not actually deleted (renamed to .nfsXXX) and is still
> accessible
> > by the guest.
> >
> > I have a similar problem when deleting a file from a Windows guest side.
> > The FUSE_READDIR(PLUS) commands add a reference count to files which the
> OS
> > doesn't have a file context for. However I was able to solve it (for
> now?)
> > by keeping track of returned files' inodes.
> >
> > Is this behaviour current and by design?
>
> Current problem, not really by design; the problem is the O_PATH files
> that we have open for the inodes.  I thought if the guest sent the
> forget for the file then it got closed.
>

So if I understand then sending forget message for each inode returned by
readdir won't solve the problem because you need the open files for inodes?

Thanks,

    Gal.


>
> Dave
>
>
> > Thanks,
> >
> >     Gal.
>
> > _______________________________________________
> > Virtio-fs mailing list
> > Virtio-fs@redhat.com
> > https://listman.redhat.com/mailman/listinfo/virtio-fs
>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
>

[-- Attachment #2: Type: text/html, Size: 3468 bytes --]

Re: [Virtio-fs] Deleting files when using NFS as a shared folder

[Max Reitz]

On 02.08.21 12:44, Gal Hammer wrote:
>
>
> On Mon, 2 Aug 2021 at 13:36, Dr. David Alan Gilbert 
> <dgilbert@redhat.com <mailto:dgilbert@redhat.com>> wrote:
>
>     * Gal Hammer (ghammer@redhat.com <mailto:ghammer@redhat.com>) wrote:
>     > Hello,
>     >
>     > When using NFS as a shared folder (mount type nfs4) with a Linux
>     guest I
>     > have the following issue:
>     >
>     > Guest:
>     > $ ls -la /mnt/shared
>     > total 8
>     > drwxr-xrwx.  2  135  135 4096 Aug  2 13:08 .
>     > dr-xr-xr-x. 17 root   root    224 May 23 10:58 ..
>     > -rw-r--rw-.  1  135  135   27 Aug  2 13:07 readme.txt
>     >
>     > Host:
>     > $ rm readme.txt
>     >
>     > Guest:
>     > $ ls -la /mnt/shared
>     > total 8
>     > drwxr-xrwx.  2  135  135 4096 Aug  2 13:10 .
>     > dr-xr-xr-x. 17 root   root    224 May 23 10:58 ..
>     > -rw-r--rw-.  1  135  135   27 Aug  2 13:07
>     .nfs0000000001b600d000000005
>     >
>     > Guest:
>     > $ cat /mnt/shared/readme.txt
>     > This is a readme.txt file.
>     >
>     > So it seems that the virtiofsd has a reference to the file which
>     the guest
>     > is not aware of and is unable to send a FUSE_FORGET message.
>     This results
>     > in a file not actually deleted (renamed to .nfsXXX) and is still
>     accessible
>     > by the guest.
>     >
>     > I have a similar problem when deleting a file from a Windows
>     guest side.
>     > The FUSE_READDIR(PLUS) commands add a reference count to files
>     which the OS
>     > doesn't have a file context for. However I was able to solve it
>     (for now?)
>     > by keeping track of returned files' inodes.
>     >
>     > Is this behaviour current and by design?
>
>     Current problem, not really by design; the problem is the O_PATH files
>     that we have open for the inodes.  I thought if the guest sent the
>     forget for the file then it got closed.
>
>
> So if I understand then sending forget message for each inode returned 
> by readdir won't solve the problem because you need the open files for 
> inodes?

virtiofsd internally keeps an lo_inode object for every inode that has 
been looked up at some point, and every such lo_inode contains an O_PATH 
fd referencing that inode.  I don’t know by heart what the conditions 
for dropping those lo_inode objects are.

However, once it’s possible to use file handles to reference inodes 
instead of O_PATH fds (already in virtiofsd-rs, for virtiofsd there’s 
this series: 
https://listman.redhat.com/archives/virtio-fs/2021-July/msg00050.html), 
then giving the appropriate options (-o inode_file_handles -o 
modcaps=+dac_read_search) should result in no O_PATH fds being kept 
around anymore, so that deleting an inode on the host will result in the 
inode being truly deleted (unless the guest still has it open).

But with O_PATH fds, it’s kind of by design, I would say.

Max

Re: [Virtio-fs] Deleting files when using NFS as a shared folder

[Gal Hammer]

[-- Attachment #1: Type: text/plain, Size: 3389 bytes --]

On Mon, 2 Aug 2021 at 13:49, Max Reitz <mreitz@redhat.com> wrote:

> On 02.08.21 12:44, Gal Hammer wrote:
> >
> >
> > On Mon, 2 Aug 2021 at 13:36, Dr. David Alan Gilbert
> > <dgilbert@redhat.com <mailto:dgilbert@redhat.com>> wrote:
> >
> >     * Gal Hammer (ghammer@redhat.com <mailto:ghammer@redhat.com>) wrote:
> >     > Hello,
> >     >
> >     > When using NFS as a shared folder (mount type nfs4) with a Linux
> >     guest I
> >     > have the following issue:
> >     >
> >     > Guest:
> >     > $ ls -la /mnt/shared
> >     > total 8
> >     > drwxr-xrwx.  2  135  135 4096 Aug  2 13:08 .
> >     > dr-xr-xr-x. 17 root   root    224 May 23 10:58 ..
> >     > -rw-r--rw-.  1  135  135   27 Aug  2 13:07 readme.txt
> >     >
> >     > Host:
> >     > $ rm readme.txt
> >     >
> >     > Guest:
> >     > $ ls -la /mnt/shared
> >     > total 8
> >     > drwxr-xrwx.  2  135  135 4096 Aug  2 13:10 .
> >     > dr-xr-xr-x. 17 root   root    224 May 23 10:58 ..
> >     > -rw-r--rw-.  1  135  135   27 Aug  2 13:07
> >     .nfs0000000001b600d000000005
> >     >
> >     > Guest:
> >     > $ cat /mnt/shared/readme.txt
> >     > This is a readme.txt file.
> >     >
> >     > So it seems that the virtiofsd has a reference to the file which
> >     the guest
> >     > is not aware of and is unable to send a FUSE_FORGET message.
> >     This results
> >     > in a file not actually deleted (renamed to .nfsXXX) and is still
> >     accessible
> >     > by the guest.
> >     >
> >     > I have a similar problem when deleting a file from a Windows
> >     guest side.
> >     > The FUSE_READDIR(PLUS) commands add a reference count to files
> >     which the OS
> >     > doesn't have a file context for. However I was able to solve it
> >     (for now?)
> >     > by keeping track of returned files' inodes.
> >     >
> >     > Is this behaviour current and by design?
> >
> >     Current problem, not really by design; the problem is the O_PATH
> files
> >     that we have open for the inodes.  I thought if the guest sent the
> >     forget for the file then it got closed.
> >
> >
> > So if I understand then sending forget message for each inode returned
> > by readdir won't solve the problem because you need the open files for
> > inodes?
>
> virtiofsd internally keeps an lo_inode object for every inode that has
> been looked up at some point, and every such lo_inode contains an O_PATH
> fd referencing that inode.  I don’t know by heart what the conditions
> for dropping those lo_inode objects are.
>

I think it depends on the guest's forget message.


> However, once it’s possible to use file handles to reference inodes
> instead of O_PATH fds (already in virtiofsd-rs, for virtiofsd there’s
> this series:
> https://listman.redhat.com/archives/virtio-fs/2021-July/msg00050.html),
> then giving the appropriate options (-o inode_file_handles -o
> modcaps=+dac_read_search) should result in no O_PATH fds being kept
> around anymore, so that deleting an inode on the host will result in the
> inode being truly deleted (unless the guest still has it open).
>

Will the guest will still need to send forget messages with this new
feature?


>
> But with O_PATH fds, it’s kind of by design, I would say.
>

Thanks for the clarification.

    Gal.


>
> Max
>
>

[-- Attachment #2: Type: text/html, Size: 5629 bytes --]

Re: [Virtio-fs] Deleting files when using NFS as a shared folder

[Max Reitz]

On 02.08.21 13:30, Gal Hammer wrote:
>
>
> On Mon, 2 Aug 2021 at 13:49, Max Reitz <mreitz@redhat.com 
> <mailto:mreitz@redhat.com>> wrote:
>
>     On 02.08.21 12:44, Gal Hammer wrote:
>     >
>     >
>     > On Mon, 2 Aug 2021 at 13:36, Dr. David Alan Gilbert
>     > <dgilbert@redhat.com <mailto:dgilbert@redhat.com>
>     <mailto:dgilbert@redhat.com <mailto:dgilbert@redhat.com>>> wrote:
>     >
>     >     * Gal Hammer (ghammer@redhat.com <mailto:ghammer@redhat.com>
>     <mailto:ghammer@redhat.com <mailto:ghammer@redhat.com>>) wrote:
>     >     > Hello,
>     >     >
>     >     > When using NFS as a shared folder (mount type nfs4) with a
>     Linux
>     >     guest I
>     >     > have the following issue:
>     >     >
>     >     > Guest:
>     >     > $ ls -la /mnt/shared
>     >     > total 8
>     >     > drwxr-xrwx.  2  135  135 4096 Aug  2 13:08 .
>     >     > dr-xr-xr-x. 17 root   root    224 May 23 10:58 ..
>     >     > -rw-r--rw-.  1  135  135   27 Aug  2 13:07 readme.txt
>     >     >
>     >     > Host:
>     >     > $ rm readme.txt
>     >     >
>     >     > Guest:
>     >     > $ ls -la /mnt/shared
>     >     > total 8
>     >     > drwxr-xrwx.  2  135  135 4096 Aug  2 13:10 .
>     >     > dr-xr-xr-x. 17 root   root    224 May 23 10:58 ..
>     >     > -rw-r--rw-.  1  135  135   27 Aug  2 13:07
>     >     .nfs0000000001b600d000000005
>     >     >
>     >     > Guest:
>     >     > $ cat /mnt/shared/readme.txt
>     >     > This is a readme.txt file.
>     >     >
>     >     > So it seems that the virtiofsd has a reference to the file
>     which
>     >     the guest
>     >     > is not aware of and is unable to send a FUSE_FORGET message.
>     >     This results
>     >     > in a file not actually deleted (renamed to .nfsXXX) and is
>     still
>     >     accessible
>     >     > by the guest.
>     >     >
>     >     > I have a similar problem when deleting a file from a Windows
>     >     guest side.
>     >     > The FUSE_READDIR(PLUS) commands add a reference count to files
>     >     which the OS
>     >     > doesn't have a file context for. However I was able to
>     solve it
>     >     (for now?)
>     >     > by keeping track of returned files' inodes.
>     >     >
>     >     > Is this behaviour current and by design?
>     >
>     >     Current problem, not really by design; the problem is the
>     O_PATH files
>     >     that we have open for the inodes.  I thought if the guest
>     sent the
>     >     forget for the file then it got closed.
>     >
>     >
>     > So if I understand then sending forget message for each inode
>     returned
>     > by readdir won't solve the problem because you need the open
>     files for
>     > inodes?
>
>     virtiofsd internally keeps an lo_inode object for every inode that
>     has
>     been looked up at some point, and every such lo_inode contains an
>     O_PATH
>     fd referencing that inode.  I don’t know by heart what the conditions
>     for dropping those lo_inode objects are.
>
>
> I think it depends on the guest's forget message.

Yes, it looks like it.

>     However, once it’s possible to use file handles to reference inodes
>     instead of O_PATH fds (already in virtiofsd-rs, for virtiofsd there’s
>     this series:
>     https://listman.redhat.com/archives/virtio-fs/2021-July/msg00050.html
>     <https://listman.redhat.com/archives/virtio-fs/2021-July/msg00050.html>),
>
>     then giving the appropriate options (-o inode_file_handles -o
>     modcaps=+dac_read_search) should result in no O_PATH fds being kept
>     around anymore, so that deleting an inode on the host will result
>     in the
>     inode being truly deleted (unless the guest still has it open).
>
>
> Will the guest will still need to send forget messages with this new 
> feature?

I don’t think so.  With file handles, FDs should only be opened (and 
kept open) when the guest actually opens some file.  (Aside from 
temporary O_PATH FDs e.g. during a lookup.)

Max

>
>     But with O_PATH fds, it’s kind of by design, I would say.
>
>
> Thanks for the clarification.
>
>     Gal.
>
>
>     Max
>

Re: [Virtio-fs] Deleting files when using NFS as a shared folder

[Vivek Goyal]

On Mon, Aug 02, 2021 at 06:34:17PM +0200, Max Reitz wrote:
> On 02.08.21 13:30, Gal Hammer wrote:
> > 
> > 
> > On Mon, 2 Aug 2021 at 13:49, Max Reitz <mreitz@redhat.com
> > <mailto:mreitz@redhat.com>> wrote:
> > 
> >     On 02.08.21 12:44, Gal Hammer wrote:
> >     >
> >     >
> >     > On Mon, 2 Aug 2021 at 13:36, Dr. David Alan Gilbert
> >     > <dgilbert@redhat.com <mailto:dgilbert@redhat.com>
> >     <mailto:dgilbert@redhat.com <mailto:dgilbert@redhat.com>>> wrote:
> >     >
> >     >     * Gal Hammer (ghammer@redhat.com <mailto:ghammer@redhat.com>
> >     <mailto:ghammer@redhat.com <mailto:ghammer@redhat.com>>) wrote:
> >     >     > Hello,
> >     >     >
> >     >     > When using NFS as a shared folder (mount type nfs4) with a
> >     Linux
> >     >     guest I
> >     >     > have the following issue:
> >     >     >
> >     >     > Guest:
> >     >     > $ ls -la /mnt/shared
> >     >     > total 8
> >     >     > drwxr-xrwx.  2  135  135 4096 Aug  2 13:08 .
> >     >     > dr-xr-xr-x. 17 root   root    224 May 23 10:58 ..
> >     >     > -rw-r--rw-.  1  135  135   27 Aug  2 13:07 readme.txt
> >     >     >
> >     >     > Host:
> >     >     > $ rm readme.txt
> >     >     >
> >     >     > Guest:
> >     >     > $ ls -la /mnt/shared
> >     >     > total 8
> >     >     > drwxr-xrwx.  2  135  135 4096 Aug  2 13:10 .
> >     >     > dr-xr-xr-x. 17 root   root    224 May 23 10:58 ..
> >     >     > -rw-r--rw-.  1  135  135   27 Aug  2 13:07
> >     >     .nfs0000000001b600d000000005
> >     >     >
> >     >     > Guest:
> >     >     > $ cat /mnt/shared/readme.txt
> >     >     > This is a readme.txt file.
> >     >     >
> >     >     > So it seems that the virtiofsd has a reference to the file
> >     which
> >     >     the guest
> >     >     > is not aware of and is unable to send a FUSE_FORGET message.
> >     >     This results
> >     >     > in a file not actually deleted (renamed to .nfsXXX) and is
> >     still
> >     >     accessible
> >     >     > by the guest.
> >     >     >
> >     >     > I have a similar problem when deleting a file from a Windows
> >     >     guest side.
> >     >     > The FUSE_READDIR(PLUS) commands add a reference count to files
> >     >     which the OS
> >     >     > doesn't have a file context for. However I was able to
> >     solve it
> >     >     (for now?)
> >     >     > by keeping track of returned files' inodes.
> >     >     >
> >     >     > Is this behaviour current and by design?
> >     >
> >     >     Current problem, not really by design; the problem is the
> >     O_PATH files
> >     >     that we have open for the inodes.  I thought if the guest
> >     sent the
> >     >     forget for the file then it got closed.
> >     >
> >     >
> >     > So if I understand then sending forget message for each inode
> >     returned
> >     > by readdir won't solve the problem because you need the open
> >     files for
> >     > inodes?
> > 
> >     virtiofsd internally keeps an lo_inode object for every inode that
> >     has
> >     been looked up at some point, and every such lo_inode contains an
> >     O_PATH
> >     fd referencing that inode.  I don’t know by heart what the conditions
> >     for dropping those lo_inode objects are.
> > 
> > 
> > I think it depends on the guest's forget message.
> 
> Yes, it looks like it.
> 
> >     However, once it’s possible to use file handles to reference inodes
> >     instead of O_PATH fds (already in virtiofsd-rs, for virtiofsd there’s
> >     this series:
> >     https://listman.redhat.com/archives/virtio-fs/2021-July/msg00050.html
> >     <https://listman.redhat.com/archives/virtio-fs/2021-July/msg00050.html>),
> > 
> >     then giving the appropriate options (-o inode_file_handles -o
> >     modcaps=+dac_read_search) should result in no O_PATH fds being kept
> >     around anymore, so that deleting an inode on the host will result
> >     in the
> >     inode being truly deleted (unless the guest still has it open).
> > 
> > 
> > Will the guest will still need to send forget messages with this new
> > feature?
> 
> I don’t think so.  With file handles, FDs should only be opened (and kept
> open) when the guest actually opens some file.  (Aside from temporary O_PATH
> FDs e.g. during a lookup.)

I guess FORGET messages will still have to be sent so that virtiofsd can
free lo_inode() and associated data structrues when reference count
reaches zero. So FORGET message is more like a dropping guest's reference
count on lo_inode.

Gal, I think we had discussed this nfs issue in the past. And problem
probably is that dentry/inode is cached in guest. And that's why
lo_inode is around hence O_PATH fd is around. If you do drop caches
in guest, that might lead to removal of this temp file (sync; echo 3 >
/proc/sys/vm/drop_caches).

Max, interesting point that using file handles should help with this
situation.

Vivek

Re: [Virtio-fs] Deleting files when using NFS as a shared folder

[Gal Hammer]

[-- Attachment #1: Type: text/plain, Size: 5941 bytes --]

Hi Vivek,

On Tue, 3 Aug 2021 at 00:15, Vivek Goyal <vgoyal@redhat.com> wrote:

> On Mon, Aug 02, 2021 at 06:34:17PM +0200, Max Reitz wrote:
> > On 02.08.21 13:30, Gal Hammer wrote:
> > >
> > >
> > > On Mon, 2 Aug 2021 at 13:49, Max Reitz <mreitz@redhat.com
> > > <mailto:mreitz@redhat.com>> wrote:
> > >
> > >     On 02.08.21 12:44, Gal Hammer wrote:
> > >     >
> > >     >
> > >     > On Mon, 2 Aug 2021 at 13:36, Dr. David Alan Gilbert
> > >     > <dgilbert@redhat.com <mailto:dgilbert@redhat.com>
> > >     <mailto:dgilbert@redhat.com <mailto:dgilbert@redhat.com>>> wrote:
> > >     >
> > >     >     * Gal Hammer (ghammer@redhat.com <mailto:ghammer@redhat.com>
> > >     <mailto:ghammer@redhat.com <mailto:ghammer@redhat.com>>) wrote:
> > >     >     > Hello,
> > >     >     >
> > >     >     > When using NFS as a shared folder (mount type nfs4) with a
> > >     Linux
> > >     >     guest I
> > >     >     > have the following issue:
> > >     >     >
> > >     >     > Guest:
> > >     >     > $ ls -la /mnt/shared
> > >     >     > total 8
> > >     >     > drwxr-xrwx.  2  135  135 4096 Aug  2 13:08 .
> > >     >     > dr-xr-xr-x. 17 root   root    224 May 23 10:58 ..
> > >     >     > -rw-r--rw-.  1  135  135   27 Aug  2 13:07 readme.txt
> > >     >     >
> > >     >     > Host:
> > >     >     > $ rm readme.txt
> > >     >     >
> > >     >     > Guest:
> > >     >     > $ ls -la /mnt/shared
> > >     >     > total 8
> > >     >     > drwxr-xrwx.  2  135  135 4096 Aug  2 13:10 .
> > >     >     > dr-xr-xr-x. 17 root   root    224 May 23 10:58 ..
> > >     >     > -rw-r--rw-.  1  135  135   27 Aug  2 13:07
> > >     >     .nfs0000000001b600d000000005
> > >     >     >
> > >     >     > Guest:
> > >     >     > $ cat /mnt/shared/readme.txt
> > >     >     > This is a readme.txt file.
> > >     >     >
> > >     >     > So it seems that the virtiofsd has a reference to the file
> > >     which
> > >     >     the guest
> > >     >     > is not aware of and is unable to send a FUSE_FORGET
> message.
> > >     >     This results
> > >     >     > in a file not actually deleted (renamed to .nfsXXX) and is
> > >     still
> > >     >     accessible
> > >     >     > by the guest.
> > >     >     >
> > >     >     > I have a similar problem when deleting a file from a
> Windows
> > >     >     guest side.
> > >     >     > The FUSE_READDIR(PLUS) commands add a reference count to
> files
> > >     >     which the OS
> > >     >     > doesn't have a file context for. However I was able to
> > >     solve it
> > >     >     (for now?)
> > >     >     > by keeping track of returned files' inodes.
> > >     >     >
> > >     >     > Is this behaviour current and by design?
> > >     >
> > >     >     Current problem, not really by design; the problem is the
> > >     O_PATH files
> > >     >     that we have open for the inodes.  I thought if the guest
> > >     sent the
> > >     >     forget for the file then it got closed.
> > >     >
> > >     >
> > >     > So if I understand then sending forget message for each inode
> > >     returned
> > >     > by readdir won't solve the problem because you need the open
> > >     files for
> > >     > inodes?
> > >
> > >     virtiofsd internally keeps an lo_inode object for every inode that
> > >     has
> > >     been looked up at some point, and every such lo_inode contains an
> > >     O_PATH
> > >     fd referencing that inode.  I don’t know by heart what the
> conditions
> > >     for dropping those lo_inode objects are.
> > >
> > >
> > > I think it depends on the guest's forget message.
> >
> > Yes, it looks like it.
> >
> > >     However, once it’s possible to use file handles to reference inodes
> > >     instead of O_PATH fds (already in virtiofsd-rs, for virtiofsd
> there’s
> > >     this series:
> > >
> https://listman.redhat.com/archives/virtio-fs/2021-July/msg00050.html
> > >     <
> https://listman.redhat.com/archives/virtio-fs/2021-July/msg00050.html>),
> > >
> > >     then giving the appropriate options (-o inode_file_handles -o
> > >     modcaps=+dac_read_search) should result in no O_PATH fds being kept
> > >     around anymore, so that deleting an inode on the host will result
> > >     in the
> > >     inode being truly deleted (unless the guest still has it open).
> > >
> > >
> > > Will the guest will still need to send forget messages with this new
> > > feature?
> >
> > I don’t think so.  With file handles, FDs should only be opened (and kept
> > open) when the guest actually opens some file.  (Aside from temporary
> O_PATH
> > FDs e.g. during a lookup.)
>
> I guess FORGET messages will still have to be sent so that virtiofsd can
> free lo_inode() and associated data structrues when reference count
> reaches zero. So FORGET message is more like a dropping guest's reference
> count on lo_inode.
>

I can live with that as long as I won't need to send FORGET messages for
files that the guest didn't open or create.


>
> Gal, I think we had discussed this nfs issue in the past. And problem
> probably is that dentry/inode is cached in guest. And that's why
> lo_inode is around hence O_PATH fd is around. If you do drop caches
> in guest, that might lead to removal of this temp file (sync; echo 3 >
> /proc/sys/vm/drop_caches).
>

Even though I wrote the driver and implemented an inodes' cache, I can't be
sure that Windows or WinFSP (the user-space file system library that I use)
is having some kind of internal cache. Saying that, the problem that I have
is that virtiofsd is increasing its own reference count on READDIR when the
guest doesn't hold a files' context for.

Oh... and I don't have /proc on Windows ;-).


>
> Max, interesting point that using file handles should help with this
> situation.
>
> Vivek
>
>
    Gal.

[-- Attachment #2: Type: text/html, Size: 9607 bytes --]

Re: [Virtio-fs] Deleting files when using NFS as a shared folder

[Vivek Goyal]

On Wed, Aug 04, 2021 at 12:08:38PM +0300, Gal Hammer wrote:

[..]
> > > >     virtiofsd internally keeps an lo_inode object for every inode that
> > > >     has
> > > >     been looked up at some point, and every such lo_inode contains an
> > > >     O_PATH
> > > >     fd referencing that inode.  I don’t know by heart what the
> > conditions
> > > >     for dropping those lo_inode objects are.
> > > >
> > > >
> > > > I think it depends on the guest's forget message.
> > >
> > > Yes, it looks like it.
> > >
> > > >     However, once it’s possible to use file handles to reference inodes
> > > >     instead of O_PATH fds (already in virtiofsd-rs, for virtiofsd
> > there’s
> > > >     this series:
> > > >
> > https://listman.redhat.com/archives/virtio-fs/2021-July/msg00050.html
> > > >     <
> > https://listman.redhat.com/archives/virtio-fs/2021-July/msg00050.html>),
> > > >
> > > >     then giving the appropriate options (-o inode_file_handles -o
> > > >     modcaps=+dac_read_search) should result in no O_PATH fds being kept
> > > >     around anymore, so that deleting an inode on the host will result
> > > >     in the
> > > >     inode being truly deleted (unless the guest still has it open).
> > > >
> > > >
> > > > Will the guest will still need to send forget messages with this new
> > > > feature?
> > >
> > > I don’t think so.  With file handles, FDs should only be opened (and kept
> > > open) when the guest actually opens some file.  (Aside from temporary
> > O_PATH
> > > FDs e.g. during a lookup.)
> >
> > I guess FORGET messages will still have to be sent so that virtiofsd can
> > free lo_inode() and associated data structrues when reference count
> > reaches zero. So FORGET message is more like a dropping guest's reference
> > count on lo_inode.
> >
> 
> I can live with that as long as I won't need to send FORGET messages for
> files that the guest didn't open or create.

FORGET message will need to be sent for every inode looked up by guest. So
you don't have to necessarily open/create the file. Any action (say
readdir), which triggers lookup from guest, will return inode info
and will require a FORGET message to be sent to virtiofsd so that
virtiofsd knows that inode is not being used anymore and it can
free data structures associated with that.

IOW, FORGET message seems to be essential part of FUSE protocol. I 
am not sure what's the problem in sending FORGET message.

Also I am not sure what FORGET message has to do with nfs problem
you are facing.

> 
> 
> >
> > Gal, I think we had discussed this nfs issue in the past. And problem
> > probably is that dentry/inode is cached in guest. And that's why
> > lo_inode is around hence O_PATH fd is around. If you do drop caches
> > in guest, that might lead to removal of this temp file (sync; echo 3 >
> > /proc/sys/vm/drop_caches).
> >
> 
> Even though I wrote the driver and implemented an inodes' cache, I can't be
> sure that Windows or WinFSP (the user-space file system library that I use)
> is having some kind of internal cache. Saying that, the problem that I have
> is that virtiofsd is increasing its own reference count on READDIR when the
> guest doesn't hold a files' context for.

IIUC, it is FUSE_READDIRPLUS (and not FUSE_READDIR) which takes the
reference on inodes of the directory being read, hence requires a
FORGET message. I see that linux guest is adding those inodes to the
inode cache. So when inode is dropped from inode cache, we will end
up sending a FORGET message for that inode.

fuse_lowleve.h also mentions this.

========================
     * In contrast to readdir() (which does not affect the lookup counts),
     * the lookup count of every entry returned by readdirplus(), except "."
     * and "..", is incremented by one.
==================

So if windows guest is using READDIRPLUS, then you will have to send
FORGET messages for every inode returned.

W.r.t caching in windows guest, I have no idea how it works. Linux
does cache inodes in inode cache and these are freed later and FORGET
message sent.

You can try using "cache=none" option in virtiofsd. And that should
tell client to not cache data and metadata and that means client
should send FORGET right away as soon as it is done using inode and
you should not see this nfs temporary file issue.

But if there are issues w.r.t FORGET and proper FORGET messages are
not being sent as needed by protocol, then that's a different issue
in windows client and will need to be fixed.

Thanks
Vivek