* [LFS/MM TOPIC] Enabling file and directory change notification for network and cluster file systems
@ 2020-01-21 3:55 Steve French
2020-01-21 7:47 ` Amir Goldstein
0 siblings, 1 reply; 6+ messages in thread
From: Steve French @ 2020-01-21 3:55 UTC (permalink / raw)
To: linux-fsdevel; +Cc: CIFS, samba-technical
Currently the inotify interface in the kernel can only be used for
local file systems (unlike the previous change notify API used years
ago, and the change notify interface in Windows and other OS which is
primarily of interest for network file systems).
I wanted to discuss the VFS changes needed to allow inotify requests
to be passed into file systems so network and cluster file systems (as
an example in the SMB3 case this simply means sending a
SMB3_CHANGE_NOTIFY request to the server, whether Samba or Cloud
(Azure) or Mac or Windows or Network Appliance - all support the API
on the server side, the problem is that the network or cluster fs
client isn't told about the request to wait on the inotify event).
Although user space tools can use file system specific ioctls to wait
on events, it is obviously preferable to allow network and cluster
file systems to wait on events using the calls which current Linux
GUIs use.
This would allow gnome file manager GUI for example to be
automatically updated when a file is added to an open directory window
from another remote client.
It would also fix the embarrassing problem noted in the inotify man page:
"Inotify reports only events that a user-space program triggers
through the filesystem
API. As a result, it does not catch remote events that occur
on network filesystems."
but that is precisely the types of notifications that are most useful
... users often are aware of updates to local directories from the
same system, but ... automatic notifications that allow GUIs to be
updated on changes from **other** clients is of more value (and this
is exactly what the equivalent API allows on other OS).
The changes to the Linux VFS are small.
--
Thanks,
Steve
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [LFS/MM TOPIC] Enabling file and directory change notification for network and cluster file systems 2020-01-21 3:55 [LFS/MM TOPIC] Enabling file and directory change notification for network and cluster file systems Steve French @ 2020-01-21 7:47 ` Amir Goldstein 2020-01-21 7:49 ` Amir Goldstein 2020-01-21 8:30 ` ronnie sahlberg 0 siblings, 2 replies; 6+ messages in thread From: Amir Goldstein @ 2020-01-21 7:47 UTC (permalink / raw) To: Steve French Cc: linux-fsdevel, CIFS, samba-technical, Jan Kara, Miklos Szeredi On Tue, Jan 21, 2020 at 5:55 AM Steve French <smfrench@gmail.com> wrote: > > Currently the inotify interface in the kernel can only be used for > local file systems (unlike the previous change notify API used years > ago, and the change notify interface in Windows and other OS which is > primarily of interest for network file systems). > > I wanted to discuss the VFS changes needed to allow inotify requests > to be passed into file systems so network and cluster file systems (as > an example in the SMB3 case this simply means sending a > SMB3_CHANGE_NOTIFY request to the server, whether Samba or Cloud > (Azure) or Mac or Windows or Network Appliance - all support the API > on the server side, the problem is that the network or cluster fs > client isn't told about the request to wait on the inotify event). > Although user space tools can use file system specific ioctls to wait > on events, it is obviously preferable to allow network and cluster > file systems to wait on events using the calls which current Linux > GUIs use. > > This would allow gnome file manager GUI for example to be > automatically updated when a file is added to an open directory window > from another remote client. > > It would also fix the embarrassing problem noted in the inotify man page: > > "Inotify reports only events that a user-space program triggers > through the filesystem > API. As a result, it does not catch remote events that occur > on network filesystems." > > but that is precisely the types of notifications that are most useful > ... users often are aware of updates to local directories from the > same system, but ... automatic notifications that allow GUIs to be > updated on changes from **other** clients is of more value (and this > is exactly what the equivalent API allows on other OS). > > The changes to the Linux VFS are small. > > Miklos has already posted an RFC patch: https://lore.kernel.org/linux-fsdevel/20190507085707.GD30899@veci.piliscsaba.redhat.com/ Did you try it? You also did not answer Miklos' question: does the smb protocol support whole filesystem (or subtree) notifications? (or just per-directory notifications)? Thanks, Amir. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [LFS/MM TOPIC] Enabling file and directory change notification for network and cluster file systems 2020-01-21 7:47 ` Amir Goldstein @ 2020-01-21 7:49 ` Amir Goldstein 2020-01-21 8:30 ` ronnie sahlberg 1 sibling, 0 replies; 6+ messages in thread From: Amir Goldstein @ 2020-01-21 7:49 UTC (permalink / raw) To: Steve French Cc: linux-fsdevel, CIFS, samba-technical, Jan Kara, Miklos Szeredi, lsf-pc CC: lsf-pc On Tue, Jan 21, 2020 at 9:47 AM Amir Goldstein <amir73il@gmail.com> wrote: > > On Tue, Jan 21, 2020 at 5:55 AM Steve French <smfrench@gmail.com> wrote: > > > > Currently the inotify interface in the kernel can only be used for > > local file systems (unlike the previous change notify API used years > > ago, and the change notify interface in Windows and other OS which is > > primarily of interest for network file systems). > > > > I wanted to discuss the VFS changes needed to allow inotify requests > > to be passed into file systems so network and cluster file systems (as > > an example in the SMB3 case this simply means sending a > > SMB3_CHANGE_NOTIFY request to the server, whether Samba or Cloud > > (Azure) or Mac or Windows or Network Appliance - all support the API > > on the server side, the problem is that the network or cluster fs > > client isn't told about the request to wait on the inotify event). > > Although user space tools can use file system specific ioctls to wait > > on events, it is obviously preferable to allow network and cluster > > file systems to wait on events using the calls which current Linux > > GUIs use. > > > > This would allow gnome file manager GUI for example to be > > automatically updated when a file is added to an open directory window > > from another remote client. > > > > It would also fix the embarrassing problem noted in the inotify man page: > > > > "Inotify reports only events that a user-space program triggers > > through the filesystem > > API. As a result, it does not catch remote events that occur > > on network filesystems." > > > > but that is precisely the types of notifications that are most useful > > ... users often are aware of updates to local directories from the > > same system, but ... automatic notifications that allow GUIs to be > > updated on changes from **other** clients is of more value (and this > > is exactly what the equivalent API allows on other OS). > > > > The changes to the Linux VFS are small. > > > > > > Miklos has already posted an RFC patch: > https://lore.kernel.org/linux-fsdevel/20190507085707.GD30899@veci.piliscsaba.redhat.com/ > > Did you try it? > > You also did not answer Miklos' question: > does the smb protocol support whole filesystem (or subtree) notifications? > (or just per-directory notifications)? > > Thanks, > Amir. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [LFS/MM TOPIC] Enabling file and directory change notification for network and cluster file systems 2020-01-21 7:47 ` Amir Goldstein 2020-01-21 7:49 ` Amir Goldstein @ 2020-01-21 8:30 ` ronnie sahlberg 2020-01-21 9:43 ` Amir Goldstein 1 sibling, 1 reply; 6+ messages in thread From: ronnie sahlberg @ 2020-01-21 8:30 UTC (permalink / raw) To: Amir Goldstein Cc: Steve French, linux-fsdevel, CIFS, samba-technical, Jan Kara, Miklos Szeredi On Tue, Jan 21, 2020 at 5:48 PM Amir Goldstein <amir73il@gmail.com> wrote: > > On Tue, Jan 21, 2020 at 5:55 AM Steve French <smfrench@gmail.com> wrote: > > > > Currently the inotify interface in the kernel can only be used for > > local file systems (unlike the previous change notify API used years > > ago, and the change notify interface in Windows and other OS which is > > primarily of interest for network file systems). > > > > I wanted to discuss the VFS changes needed to allow inotify requests > > to be passed into file systems so network and cluster file systems (as > > an example in the SMB3 case this simply means sending a > > SMB3_CHANGE_NOTIFY request to the server, whether Samba or Cloud > > (Azure) or Mac or Windows or Network Appliance - all support the API > > on the server side, the problem is that the network or cluster fs > > client isn't told about the request to wait on the inotify event). > > Although user space tools can use file system specific ioctls to wait > > on events, it is obviously preferable to allow network and cluster > > file systems to wait on events using the calls which current Linux > > GUIs use. > > > > This would allow gnome file manager GUI for example to be > > automatically updated when a file is added to an open directory window > > from another remote client. > > > > It would also fix the embarrassing problem noted in the inotify man page: > > > > "Inotify reports only events that a user-space program triggers > > through the filesystem > > API. As a result, it does not catch remote events that occur > > on network filesystems." > > > > but that is precisely the types of notifications that are most useful > > ... users often are aware of updates to local directories from the > > same system, but ... automatic notifications that allow GUIs to be > > updated on changes from **other** clients is of more value (and this > > is exactly what the equivalent API allows on other OS). > > > > The changes to the Linux VFS are small. > > > > > > Miklos has already posted an RFC patch: > https://lore.kernel.org/linux-fsdevel/20190507085707.GD30899@veci.piliscsaba.redhat.com/ > > Did you try it? > > You also did not answer Miklos' question: > does the smb protocol support whole filesystem (or subtree) notifications? > (or just per-directory notifications)? SMB can do both. There is a flag that specifies if you want to just get notified about the directory itself or whether you want notifications from the whole subtree. > > Thanks, > Amir. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [LFS/MM TOPIC] Enabling file and directory change notification for network and cluster file systems 2020-01-21 8:30 ` ronnie sahlberg @ 2020-01-21 9:43 ` Amir Goldstein 2020-01-21 10:02 ` ronnie sahlberg 0 siblings, 1 reply; 6+ messages in thread From: Amir Goldstein @ 2020-01-21 9:43 UTC (permalink / raw) To: ronnie sahlberg Cc: Steve French, linux-fsdevel, CIFS, samba-technical, Jan Kara, Miklos Szeredi On Tue, Jan 21, 2020 at 10:30 AM ronnie sahlberg <ronniesahlberg@gmail.com> wrote: > > On Tue, Jan 21, 2020 at 5:48 PM Amir Goldstein <amir73il@gmail.com> wrote: > > > > On Tue, Jan 21, 2020 at 5:55 AM Steve French <smfrench@gmail.com> wrote: > > > > > > Currently the inotify interface in the kernel can only be used for > > > local file systems (unlike the previous change notify API used years > > > ago, and the change notify interface in Windows and other OS which is > > > primarily of interest for network file systems). > > > > > > I wanted to discuss the VFS changes needed to allow inotify requests > > > to be passed into file systems so network and cluster file systems (as > > > an example in the SMB3 case this simply means sending a > > > SMB3_CHANGE_NOTIFY request to the server, whether Samba or Cloud > > > (Azure) or Mac or Windows or Network Appliance - all support the API > > > on the server side, the problem is that the network or cluster fs > > > client isn't told about the request to wait on the inotify event). > > > Although user space tools can use file system specific ioctls to wait > > > on events, it is obviously preferable to allow network and cluster > > > file systems to wait on events using the calls which current Linux > > > GUIs use. > > > > > > This would allow gnome file manager GUI for example to be > > > automatically updated when a file is added to an open directory window > > > from another remote client. > > > > > > It would also fix the embarrassing problem noted in the inotify man page: > > > > > > "Inotify reports only events that a user-space program triggers > > > through the filesystem > > > API. As a result, it does not catch remote events that occur > > > on network filesystems." > > > > > > but that is precisely the types of notifications that are most useful > > > ... users often are aware of updates to local directories from the > > > same system, but ... automatic notifications that allow GUIs to be > > > updated on changes from **other** clients is of more value (and this > > > is exactly what the equivalent API allows on other OS). > > > > > > The changes to the Linux VFS are small. > > > > > > > > > > Miklos has already posted an RFC patch: > > https://lore.kernel.org/linux-fsdevel/20190507085707.GD30899@veci.piliscsaba.redhat.com/ > > > > Did you try it? > > > > You also did not answer Miklos' question: > > does the smb protocol support whole filesystem (or subtree) notifications? > > (or just per-directory notifications)? > > SMB can do both. There is a flag that specifies if you want to just > get notified about the directory itself > or whether you want notifications from the whole subtree. > I see. There is no user API in Linux to request a "subtree" watch. For the private case that the user requests a FAN_MARK_FILESYSTEM, cifs may translate that into a SMB2_WATCH_TREE for the share root dir. For that, Miklos' RFC of vfs interface inode->i_op->notify_update(inode) should be enriched with sb->s_op->notify_update(sb). Thanks, Amir. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [LFS/MM TOPIC] Enabling file and directory change notification for network and cluster file systems 2020-01-21 9:43 ` Amir Goldstein @ 2020-01-21 10:02 ` ronnie sahlberg 0 siblings, 0 replies; 6+ messages in thread From: ronnie sahlberg @ 2020-01-21 10:02 UTC (permalink / raw) To: Amir Goldstein Cc: Steve French, linux-fsdevel, CIFS, samba-technical, Jan Kara, Miklos Szeredi On Tue, Jan 21, 2020 at 7:43 PM Amir Goldstein <amir73il@gmail.com> wrote: > > On Tue, Jan 21, 2020 at 10:30 AM ronnie sahlberg > <ronniesahlberg@gmail.com> wrote: > > > > On Tue, Jan 21, 2020 at 5:48 PM Amir Goldstein <amir73il@gmail.com> wrote: > > > > > > On Tue, Jan 21, 2020 at 5:55 AM Steve French <smfrench@gmail.com> wrote: > > > > > > > > Currently the inotify interface in the kernel can only be used for > > > > local file systems (unlike the previous change notify API used years > > > > ago, and the change notify interface in Windows and other OS which is > > > > primarily of interest for network file systems). > > > > > > > > I wanted to discuss the VFS changes needed to allow inotify requests > > > > to be passed into file systems so network and cluster file systems (as > > > > an example in the SMB3 case this simply means sending a > > > > SMB3_CHANGE_NOTIFY request to the server, whether Samba or Cloud > > > > (Azure) or Mac or Windows or Network Appliance - all support the API > > > > on the server side, the problem is that the network or cluster fs > > > > client isn't told about the request to wait on the inotify event). > > > > Although user space tools can use file system specific ioctls to wait > > > > on events, it is obviously preferable to allow network and cluster > > > > file systems to wait on events using the calls which current Linux > > > > GUIs use. > > > > > > > > This would allow gnome file manager GUI for example to be > > > > automatically updated when a file is added to an open directory window > > > > from another remote client. > > > > > > > > It would also fix the embarrassing problem noted in the inotify man page: > > > > > > > > "Inotify reports only events that a user-space program triggers > > > > through the filesystem > > > > API. As a result, it does not catch remote events that occur > > > > on network filesystems." > > > > > > > > but that is precisely the types of notifications that are most useful > > > > ... users often are aware of updates to local directories from the > > > > same system, but ... automatic notifications that allow GUIs to be > > > > updated on changes from **other** clients is of more value (and this > > > > is exactly what the equivalent API allows on other OS). > > > > > > > > The changes to the Linux VFS are small. > > > > > > > > > > > > > > Miklos has already posted an RFC patch: > > > https://lore.kernel.org/linux-fsdevel/20190507085707.GD30899@veci.piliscsaba.redhat.com/ > > > > > > Did you try it? > > > > > > You also did not answer Miklos' question: > > > does the smb protocol support whole filesystem (or subtree) notifications? > > > (or just per-directory notifications)? > > > > SMB can do both. There is a flag that specifies if you want to just > > get notified about the directory itself > > or whether you want notifications from the whole subtree. > > > > I see. There is no user API in Linux to request a "subtree" watch. > For the private case that the user requests a FAN_MARK_FILESYSTEM, > cifs may translate that into a SMB2_WATCH_TREE for the share root dir. > > For that, Miklos' RFC of vfs interface inode->i_op->notify_update(inode) > should be enriched with sb->s_op->notify_update(sb). It all depends on what actual linux applications want/need. On windows, things like the windows explorer file manager only uses "this directory only" watches. I imagine that linux gui filemanagers would do the same and be fine with this directory only watches. On windows you also have a large class of applications that does file caching in the userspace application itself. Like IIS or other applications such as indexing services or things that keep track of changes to know what to backup at a later stage. They always use "watch the whole subtree" version of notifications. I don't think there are any linux native applications that would depend on this as the functionality was never present. Maybe an exception would be windows applications running under wine, but surely no currently existing linux native applications. If we do get an API that would allow to watch an entire subtree then that would make it possible for future backup applications to become much more effective for incremental backups. Things like "git status" would likely also benefit greatly from such an api if it became available. > > Thanks, > Amir. ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2020-01-21 10:03 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-01-21 3:55 [LFS/MM TOPIC] Enabling file and directory change notification for network and cluster file systems Steve French 2020-01-21 7:47 ` Amir Goldstein 2020-01-21 7:49 ` Amir Goldstein 2020-01-21 8:30 ` ronnie sahlberg 2020-01-21 9:43 ` Amir Goldstein 2020-01-21 10:02 ` ronnie sahlberg
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).