[LSF/MM/BPF TOPIC] Enabling change notification for network and cluster fs

[LSF/MM/BPF TOPIC] Enabling change notification for network and cluster fs

[Steve French]

Currently only local events can be waited on with the current notify
kernel API since the requests to wait on these events is not passed to
the filesystem.   Especially for network and cluster filesystems it is
important that they be told that applications want to be notified of
these file or directory change events.

A few years ago, discussions began on the changes needed to enable
support for this.   Would be timely to finish those discussions, as
waiting on file and directory change events to network mounts is very
common for other OS, and would be valuable for Linux to fix.

-- 
Thanks,

Steve

Re: [LSF/MM/BPF TOPIC] Enabling change notification for network and cluster fs

[Vivek Goyal]

On Wed, Feb 23, 2022 at 11:16:33PM -0600, Steve French wrote:
> Currently only local events can be waited on with the current notify
> kernel API since the requests to wait on these events is not passed to
> the filesystem.   Especially for network and cluster filesystems it is
> important that they be told that applications want to be notified of
> these file or directory change events.
> 
> A few years ago, discussions began on the changes needed to enable
> support for this.   Would be timely to finish those discussions, as
> waiting on file and directory change events to network mounts is very
> common for other OS, and would be valuable for Linux to fix.
> 

This sounds like which might have some overlap with what we are trying
to do.

Currently inotify/fanotify only work for local filesystems. We were
thinking is it possible to extend it for remote filesystems as well. My
interest primarily was to make notifications work on virtiofs. So I
asked Ioannis (an intern with us) to try to prototype it and see what are
the challenges and roadblocks.

He posted one version of patches just as proof of concept and only tried
to make remote inotify work. One primary feedback from Amir was that
this is too specific to inotify and if you are extending fsnotify, then
it should have some support for fanotify as well. There is bunch of
other feedback too. So Ioannis is trying to rework his patches now.

https://lore.kernel.org/linux-fsdevel/20211025204634.2517-1-iangelak@redhat.com/

Anyway, you had pointed to following commit.

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/fs/cifs/ioctl.c?id=d26c2ddd33569667e3eeb577c4c1d966ca9192e2

So looks like application calls this cifs specific ioctl and blocks and
unblocks when notifications comes, IIUC.

I don't know about SMB and what kind of other notifications does it
support. With this proposal, you are trying to move away from cifs
specific ioctl? What will user use to either block or poll for the
said notification.

Sorry, I might be just completely off the mark. Just trying to find out
if there is any overlap in what you are looking for and what we are
trying to do. 

Thanks
Vivek

> -- 
> Thanks,
> 
> Steve
> 

Re: [LSF/MM/BPF TOPIC] Enabling change notification for network and cluster fs

[Steve French]

>  With this proposal, you are trying to move away from cifs specific ioctl?

Yes - since there are some applications that already use inotify
(although presumably less common than the similar API on Macs and
Windows), just fixing inotify to call into the fs to register events
is a good first step.  See inotify(7) - Linux manual page e.g.

For the case of SMB3.1.1 the wire protocol should be fine for this
(although perhaps could be a few flags added to inotify to better
match what servers support) See
https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb2/598f395a-e7a2-4cc8-afb3-ccb30dd2df7c

But there may be other APIs other than inotify that could be mapped to
what is already supported on the wire (with minor changes to the vfs)

On Thu, Feb 24, 2022 at 3:52 PM Vivek Goyal <vgoyal@redhat.com> wrote:
>
> On Wed, Feb 23, 2022 at 11:16:33PM -0600, Steve French wrote:
> > Currently only local events can be waited on with the current notify
> > kernel API since the requests to wait on these events is not passed to
> > the filesystem.   Especially for network and cluster filesystems it is
> > important that they be told that applications want to be notified of
> > these file or directory change events.
> >
> > A few years ago, discussions began on the changes needed to enable
> > support for this.   Would be timely to finish those discussions, as
> > waiting on file and directory change events to network mounts is very
> > common for other OS, and would be valuable for Linux to fix.
> >
>
> This sounds like which might have some overlap with what we are trying
> to do.
>
> Currently inotify/fanotify only work for local filesystems. We were
> thinking is it possible to extend it for remote filesystems as well. My
> interest primarily was to make notifications work on virtiofs. So I
> asked Ioannis (an intern with us) to try to prototype it and see what are
> the challenges and roadblocks.
>
> He posted one version of patches just as proof of concept and only tried
> to make remote inotify work. One primary feedback from Amir was that
> this is too specific to inotify and if you are extending fsnotify, then
> it should have some support for fanotify as well. There is bunch of
> other feedback too. So Ioannis is trying to rework his patches now.
>
> https://lore.kernel.org/linux-fsdevel/20211025204634.2517-1-iangelak@redhat.com/
>
> Anyway, you had pointed to following commit.
>
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/fs/cifs/ioctl.c?id=d26c2ddd33569667e3eeb577c4c1d966ca9192e2
>
> So looks like application calls this cifs specific ioctl and blocks and
> unblocks when notifications comes, IIUC.
>
> I don't know about SMB and what kind of other notifications does it
> support. With this proposal, you are trying to move away from cifs
> specific ioctl? What will user use to either block or poll for the
> said notification.
>
> Sorry, I might be just completely off the mark. Just trying to find out
> if there is any overlap in what you are looking for and what we are
> trying to do.
>
> Thanks
> Vivek
>
> > --
> > Thanks,
> >
> > Steve
> >
>


-- 
Thanks,

Steve

Re: [LSF/MM/BPF TOPIC] Enabling change notification for network and cluster fs

[Vivek Goyal]

On Thu, Feb 24, 2022 at 04:55:42PM -0600, Steve French wrote:
> >  With this proposal, you are trying to move away from cifs specific ioctl?
> 
> Yes - since there are some applications that already use inotify
> (although presumably less common than the similar API on Macs and
> Windows), just fixing inotify to call into the fs to register events
> is a good first step.  See inotify(7) - Linux manual page e.g.

Ok. Fixing one first makes it little simpler. But ultimately we probably
would like to get to fixing boty inotify and fanotify.

> 
> For the case of SMB3.1.1 the wire protocol should be fine for this
> (although perhaps could be a few flags added to inotify to better
> match what servers support) See
> https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb2/598f395a-e7a2-4cc8-afb3-ccb30dd2df7c
> 

I see that "completion_filter" allows one to specify what kind of event
client is interested in.

Some of the events can probably mapped to existing inotify/fanotify events
but others don't seem to have 1:1 mapping. For example,
FILE_NOTIFY_CHANGE_SECURITY or FILE_NOTIFY_CHANGE_EA.

I guess this will be a generic problem. Remote filesystem protocol might
support a set of notifications which don't have any 1:1 mapping with
what inotify/fanotiy support. For the case of virtiofs we are initially
thinking of running linux host and server can use inoitfy/fanotify on
host and then lot of events can have 1:1 mapping. But if we were to
run virtiofsd on some other platform macOS or windows, then set of
events supported might be very different. So mapping events will
be a challege there as well.

I guess client will have to keep a list of events it can support and
if application tries to monitor for an event which remote fs can't
support, just deny putting that watch. For example, I did not see in
SMB, what's the equivalent of IN_CLOSE_WRITE or IN_OPEN.

This will be a challege for applications though because while a set of event
works on one filesystem it will not work on other fs and life will be hard
for application writers.

What about local events. I am assuming you want to supress local events
and only deliver remote events. Because having both local and remote
events delivered at the same time will be just confusing at best.

Thanks
Vivek

> But there may be other APIs other than inotify that could be mapped to
> what is already supported on the wire (with minor changes to the vfs)

> 
> On Thu, Feb 24, 2022 at 3:52 PM Vivek Goyal <vgoyal@redhat.com> wrote:
> >
> > On Wed, Feb 23, 2022 at 11:16:33PM -0600, Steve French wrote:
> > > Currently only local events can be waited on with the current notify
> > > kernel API since the requests to wait on these events is not passed to
> > > the filesystem.   Especially for network and cluster filesystems it is
> > > important that they be told that applications want to be notified of
> > > these file or directory change events.
> > >
> > > A few years ago, discussions began on the changes needed to enable
> > > support for this.   Would be timely to finish those discussions, as
> > > waiting on file and directory change events to network mounts is very
> > > common for other OS, and would be valuable for Linux to fix.
> > >
> >
> > This sounds like which might have some overlap with what we are trying
> > to do.
> >
> > Currently inotify/fanotify only work for local filesystems. We were
> > thinking is it possible to extend it for remote filesystems as well. My
> > interest primarily was to make notifications work on virtiofs. So I
> > asked Ioannis (an intern with us) to try to prototype it and see what are
> > the challenges and roadblocks.
> >
> > He posted one version of patches just as proof of concept and only tried
> > to make remote inotify work. One primary feedback from Amir was that
> > this is too specific to inotify and if you are extending fsnotify, then
> > it should have some support for fanotify as well. There is bunch of
> > other feedback too. So Ioannis is trying to rework his patches now.
> >
> > https://lore.kernel.org/linux-fsdevel/20211025204634.2517-1-iangelak@redhat.com/
> >
> > Anyway, you had pointed to following commit.
> >
> > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/fs/cifs/ioctl.c?id=d26c2ddd33569667e3eeb577c4c1d966ca9192e2
> >
> > So looks like application calls this cifs specific ioctl and blocks and
> > unblocks when notifications comes, IIUC.
> >
> > I don't know about SMB and what kind of other notifications does it
> > support. With this proposal, you are trying to move away from cifs
> > specific ioctl? What will user use to either block or poll for the
> > said notification.
> >
> > Sorry, I might be just completely off the mark. Just trying to find out
> > if there is any overlap in what you are looking for and what we are
> > trying to do.
> >
> > Thanks
> > Vivek
> >
> > > --
> > > Thanks,
> > >
> > > Steve
> > >
> >
> 
> 
> -- 
> Thanks,
> 
> Steve
> 

Re: [LSF/MM/BPF TOPIC] Enabling change notification for network and cluster fs

[Matthew Wilcox]

On Fri, Feb 25, 2022 at 08:23:20AM -0500, Vivek Goyal wrote:
> What about local events. I am assuming you want to supress local events
> and only deliver remote events. Because having both local and remote
> events delivered at the same time will be just confusing at best.

This paragraph confuses me.  If I'm writing, for example, a file manager
and I want it to update its display automatically when another task alters
the contents of a directory, I don't care whether the modification was
done locally or remotely.

If I understand the SMB protocol correctly, it allows the client to take
out a lease on a directory and not send its modifications back to the
server until the client chooses to (or the server breaks the lease).
So you wouldn't get any remote notifications because the client hasn't
told the server.

Re: [LSF/MM/BPF TOPIC] Enabling change notification for network and cluster fs

[Vivek Goyal]

On Fri, Feb 25, 2022 at 01:49:19PM +0000, Matthew Wilcox wrote:
> On Fri, Feb 25, 2022 at 08:23:20AM -0500, Vivek Goyal wrote:
> > What about local events. I am assuming you want to supress local events
> > and only deliver remote events. Because having both local and remote
> > events delivered at the same time will be just confusing at best.
> 
> This paragraph confuses me.  If I'm writing, for example, a file manager
> and I want it to update its display automatically when another task alters
> the contents of a directory, I don't care whether the modification was
> done locally or remotely.
> 
> If I understand the SMB protocol correctly, it allows the client to take
> out a lease on a directory and not send its modifications back to the
> server until the client chooses to (or the server breaks the lease).
> So you wouldn't get any remote notifications because the client hasn't
> told the server.

So we will get remote notifications when client flushes changes to server,
IIUC. But in this case, given changes are happening on same client, local
events will make sense because we will come to know about changes much
sooner.

But if another client was watching for changes too, it will not come to
know about these events till first client flushes these changes to
server.

Anyway, it is a good point. This is a good example of where we might want
local events too. 

This raises question how applications will handle the situation, if we allow
both local and remote events, then there will be too many duplicate events.
One event for local change and another will be sent by server when server
notices the change.

May be there needs to be a way to supress remote event if we already
generated local event. But not sure how would one figure that out. If
server can somehow not send remote events to the client which triggered
the event (and send remote events to all other clients), may be that will
help.

Havid said that, it might not be easy for server to figure out which
client triggered the event and not send remote event back to that client.

May be we should allow both local and remote events. And probably event
should carry additional property which says whether event was local or
remote. And then let application deal with it?  I am not sure, just
thinking loud. 

Thanks
Vivek

Re: [LSF/MM/BPF TOPIC] Enabling change notification for network and cluster fs

[Steve French]

On Fri, Feb 25, 2022 at 7:49 AM Matthew Wilcox <willy@infradead.org> wrote:
>
> On Fri, Feb 25, 2022 at 08:23:20AM -0500, Vivek Goyal wrote:
> > What about local events. I am assuming you want to supress local events
> > and only deliver remote events. Because having both local and remote
> > events delivered at the same time will be just confusing at best.
>
> This paragraph confuses me.  If I'm writing, for example, a file manager
> and I want it to update its display automatically when another task alters
> the contents of a directory, I don't care whether the modification was
> done locally or remotely.
>
> If I understand the SMB protocol correctly, it allows the client to take
> out a lease on a directory and not send its modifications back to the
> server until the client chooses to (or the server breaks the lease).
> So you wouldn't get any remote notifications because the client hasn't
> told the server.

Directory leases would be broken by file create so the more important
question is what happens when client 1 has a change notification on writes
to files in a directory then client 2 opens a file in the same directory and is
granted a file lease and starts writing to the file (which means the
writes could get cached).   This is probably a minor point because when
writes get flushed from client 2, client 1 (and any others with notifications
requested) will get notified of the event (changes to files in a directory
that they are watching).

Local applications watching a file on a network or cluster mount in Linux
(just as is the case with Windows, Macs etc.) should be able to be notified of
local (cached) writes to a remote file or remote writes to the file from another
client.  I don't think the change is large, and there was an earlier version of
a patch circulated for this

-- 
Thanks,

Steve

Re: [LSF/MM/BPF TOPIC] Enabling change notification for network and cluster fs

[Vivek Goyal]

On Fri, Feb 25, 2022 at 09:27:55AM -0600, Steve French wrote:
> On Fri, Feb 25, 2022 at 7:49 AM Matthew Wilcox <willy@infradead.org> wrote:
> >
> > On Fri, Feb 25, 2022 at 08:23:20AM -0500, Vivek Goyal wrote:
> > > What about local events. I am assuming you want to supress local events
> > > and only deliver remote events. Because having both local and remote
> > > events delivered at the same time will be just confusing at best.
> >
> > This paragraph confuses me.  If I'm writing, for example, a file manager
> > and I want it to update its display automatically when another task alters
> > the contents of a directory, I don't care whether the modification was
> > done locally or remotely.
> >
> > If I understand the SMB protocol correctly, it allows the client to take
> > out a lease on a directory and not send its modifications back to the
> > server until the client chooses to (or the server breaks the lease).
> > So you wouldn't get any remote notifications because the client hasn't
> > told the server.
> 
> Directory leases would be broken by file create so the more important
> question is what happens when client 1 has a change notification on writes
> to files in a directory then client 2 opens a file in the same directory and is
> granted a file lease and starts writing to the file (which means the
> writes could get cached).   This is probably a minor point because when
> writes get flushed from client 2, client 1 (and any others with notifications
> requested) will get notified of the event (changes to files in a directory
> that they are watching).
> 
> Local applications watching a file on a network or cluster mount in Linux
> (just as is the case with Windows, Macs etc.) should be able to be notified of
> local (cached) writes to a remote file or remote writes to the file from another
> client.  I don't think the change is large, and there was an earlier version of
> a patch circulated for this

So local notifications are generated by filesystem code as needed?

IOW, in general disable all local events and let filesystems decide which
local events to generate? And locally cached write is one such example?

Thanks
Vivek

Re: [Lsf-pc] [LSF/MM/BPF TOPIC] Enabling change notification for network and cluster fs

[Amir Goldstein]

On Fri, Feb 25, 2022 at 8:11 PM Steve French <smfrench@gmail.com> wrote:
>
> > IOW, in general disable all local events and let filesystems decide which
> local events to generate? And locally cached write is one such example?
>
> The fs doesn't see cached writes so probably best to still use the common
> existing code for notification on local writes
>

I guess SMB protocol does not allow client B to request a NOTIFY on change
when client A has a directory lease, because requesting NOTIFY requires
getting a read file handle on the dir?

Effectively, smb client needs to open the remote directory for read in order
to prove that the client has read access to the directory, which is the
prerequisite for getting directory change notifications.

The local check for permissions is not enough for remote notifications:
        /* you can only watch an inode if you have read permissions on it */
        error = path_permission(path, MAY_READ);

Thanks,
Amir.

Re: [LSF/MM/BPF TOPIC] Enabling change notification for network and cluster fs

[Vivek Goyal]

On Fri, Feb 25, 2022 at 01:11:10PM -0500, Steve French wrote:
> > IOW, in general disable all local events and let filesystems decide which
> local events to generate? And locally cached write is one such example?
> 
> The fs doesn't see cached writes so probably best to still use the common
> existing code for notification on local writes

Filesystems could provide a hook/function which local notifications could
call and then filesystem could take a decision whether to allow that
particular local event. 

For example, Ioannis implemented inode operation ->fsnotify_event()

https://lore.kernel.org/linux-fsdevel/20211025204634.2517-6-iangelak@redhat.com/

This gives local event to fuse. And now fuse can decide whether to
dispatch the event to user space by calling __fsnotify() or drop it
on the floor.

As of now we drop everything on the floor if remote notifications are
enabled. But I realize that cache=writeback mode can cache writes, so
fs could selectively allow some local events.

I guess for virtiofs, we could allow all local events. Supress remote
events if event is triggered due to the action of same client. If we
use fanotify on server, then it will report PID of process that caused
the event. And if PID belongs to this virtiofsd instance, it probably
could supress remote event. 

That way an application running inside the guest will see all local
events + remote events triggered by other clients. (But should not
see duplicate events).

Thanks
Vivek

> 
> On Fri, Feb 25, 2022, 11:35 Vivek Goyal <vgoyal@redhat.com> wrote:
> 
> > On Fri, Feb 25, 2022 at 09:27:55AM -0600, Steve French wrote:
> > > On Fri, Feb 25, 2022 at 7:49 AM Matthew Wilcox <willy@infradead.org>
> > wrote:
> > > >
> > > > On Fri, Feb 25, 2022 at 08:23:20AM -0500, Vivek Goyal wrote:
> > > > > What about local events. I am assuming you want to supress local
> > events
> > > > > and only deliver remote events. Because having both local and remote
> > > > > events delivered at the same time will be just confusing at best.
> > > >
> > > > This paragraph confuses me.  If I'm writing, for example, a file
> > manager
> > > > and I want it to update its display automatically when another task
> > alters
> > > > the contents of a directory, I don't care whether the modification was
> > > > done locally or remotely.
> > > >
> > > > If I understand the SMB protocol correctly, it allows the client to
> > take
> > > > out a lease on a directory and not send its modifications back to the
> > > > server until the client chooses to (or the server breaks the lease).
> > > > So you wouldn't get any remote notifications because the client hasn't
> > > > told the server.
> > >
> > > Directory leases would be broken by file create so the more important
> > > question is what happens when client 1 has a change notification on
> > writes
> > > to files in a directory then client 2 opens a file in the same directory
> > and is
> > > granted a file lease and starts writing to the file (which means the
> > > writes could get cached).   This is probably a minor point because when
> > > writes get flushed from client 2, client 1 (and any others with
> > notifications
> > > requested) will get notified of the event (changes to files in a
> > directory
> > > that they are watching).
> > >
> > > Local applications watching a file on a network or cluster mount in Linux
> > > (just as is the case with Windows, Macs etc.) should be able to be
> > notified of
> > > local (cached) writes to a remote file or remote writes to the file from
> > another
> > > client.  I don't think the change is large, and there was an earlier
> > version of
> > > a patch circulated for this
> >
> > So local notifications are generated by filesystem code as needed?
> >
> > IOW, in general disable all local events and let filesystems decide which
> > local events to generate? And locally cached write is one such example?
> >
> > Thanks
> > Vivek
> >
> >

Re: [Lsf-pc] [LSF/MM/BPF TOPIC] Enabling change notification for network and cluster fs

[Ralph Boehme]

[-- Attachment #1.1: Type: text/plain, Size: 332 bytes --]

Hi Amir,

On 2/26/22 11:22, Amir Goldstein wrote:
> I guess SMB protocol does not allow client B to request a NOTIFY on change
> when client A has a directory lease, because requesting NOTIFY requires
> getting a read file handle on the dir?
fwiw, you don't get a "W" SMB3 directory lease, so this is not a problem.

-slow

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]

Re: [LSF/MM/BPF TOPIC] Enabling change notification for network and cluster fs

[Vivek Goyal]

On Wed, Feb 23, 2022 at 11:16:33PM -0600, Steve French wrote:
> Currently only local events can be waited on with the current notify
> kernel API since the requests to wait on these events is not passed to
> the filesystem.   Especially for network and cluster filesystems it is
> important that they be told that applications want to be notified of
> these file or directory change events.
> 
> A few years ago, discussions began on the changes needed to enable
> support for this.   Would be timely to finish those discussions, as
> waiting on file and directory change events to network mounts is very
> common for other OS, and would be valuable for Linux to fix.

If this topic gets selected for discussion, I will be interested in
joining the conversation.

Vivek
> 
> -- 
> Thanks,
> 
> Steve
>