[Virtio-fs] Lack of inotify for virtio-fs backed files/directories

[Virtio-fs] Lack of inotify for virtio-fs backed files/directories

[Eric Ernst]

[-- Attachment #1: Type: text/plain, Size: 1154 bytes --]

Hey ya'll,

As highlighted nicely at [1], a common pattern in Kubernetes is to watch
for changes to files/directories passed in as Configmaps or secrets.
Sidecars typically use inotify to watch for changes and then signal the
primary container or perform other changes based on the reconfiguration.

In the context of Kata, these files/dirs are passed in utilizing virtio-fs.
Since FUSE itself doesn't support inotify, this pattern will fail. I'm
looking at ways to work-around this gap, including:

1) documenting this as a compatibility limitation ("don't do this!").
2) suggesting end-users modify their sidecars and utilize poll
3) doing an inotify on the host, and when changes are identified, notify
the guest agent who can then modify the files on disk to trigger an inotify
4) Working on contributing to libfuse in order to add inotify support.

(4) is the right thing, but will take quite a bit of time before it can be
used. I'm curious if other folks have suggestions for workarounds (I don't
like any of the ones I suggested), and if anyone has taken a look at (4)?

Thanks,
Eric

[1] - https://github.com/kata-containers/runtime/issues/1505

[-- Attachment #2: Type: text/html, Size: 1486 bytes --]

Re: [Virtio-fs] Lack of inotify for virtio-fs backed files/directories

[Miklos Szeredi]

On Fri, Jan 22, 2021 at 5:58 PM Eric Ernst <eric.g.ernst@gmail.com> wrote:
>
> Hey ya'll,
>
> As highlighted nicely at [1], a common pattern in Kubernetes is to watch for changes to files/directories passed in as Configmaps or secrets. Sidecars typically use inotify to watch for changes and then signal the primary container or perform other changes based on the reconfiguration.
>
> In the context of Kata, these files/dirs are passed in utilizing virtio-fs. Since FUSE itself doesn't support inotify, this pattern will fail. I'm looking at ways to work-around this gap, including:
>
> 1) documenting this as a compatibility limitation ("don't do this!").
> 2) suggesting end-users modify their sidecars and utilize poll
> 3) doing an inotify on the host, and when changes are identified, notify the guest agent who can then modify the files on disk to trigger an inotify
> 4) Working on contributing to libfuse in order to add inotify support.
>
> (4) is the right thing, but will take quite a bit of time before it can be used. I'm curious if other folks have suggestions for workarounds (I don't like any of the ones I suggested), and if anyone has taken a look at (4)?

Here's a thread on enabling inotify in remote fs:

https://lore.kernel.org/linux-fsdevel/20190507085707.GD30899@veci.piliscsaba.redhat.com/

The discussion didn't get too far, it seems there's not a very big
interest in this, but conceptually it doesn't seem too difficult.

At the moment I'm busy with other things, but it's an interesting
issue so I'd be happy to look into it sometime if nobody is willing to
pick it up.

Thanks,
Miklos

Re: [Virtio-fs] Lack of inotify for virtio-fs backed files/directories

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1992 bytes --]

On Mon, Jan 25, 2021 at 08:22:32PM +0100, Miklos Szeredi wrote:
> On Fri, Jan 22, 2021 at 5:58 PM Eric Ernst <eric.g.ernst@gmail.com> wrote:
> >
> > Hey ya'll,
> >
> > As highlighted nicely at [1], a common pattern in Kubernetes is to watch for changes to files/directories passed in as Configmaps or secrets. Sidecars typically use inotify to watch for changes and then signal the primary container or perform other changes based on the reconfiguration.
> >
> > In the context of Kata, these files/dirs are passed in utilizing virtio-fs. Since FUSE itself doesn't support inotify, this pattern will fail. I'm looking at ways to work-around this gap, including:
> >
> > 1) documenting this as a compatibility limitation ("don't do this!").
> > 2) suggesting end-users modify their sidecars and utilize poll
> > 3) doing an inotify on the host, and when changes are identified, notify the guest agent who can then modify the files on disk to trigger an inotify
> > 4) Working on contributing to libfuse in order to add inotify support.
> >
> > (4) is the right thing, but will take quite a bit of time before it can be used. I'm curious if other folks have suggestions for workarounds (I don't like any of the ones I suggested), and if anyone has taken a look at (4)?
> 
> Here's a thread on enabling inotify in remote fs:
> 
> https://lore.kernel.org/linux-fsdevel/20190507085707.GD30899@veci.piliscsaba.redhat.com/
> 
> The discussion didn't get too far, it seems there's not a very big
> interest in this, but conceptually it doesn't seem too difficult.

Vivek Goyal previously experimented with adding a notification virtqueue
to the virtio-fs device so that FUSE_NOTIFY_* messages can be supported.
That code can be brought back and merged.

However, due to the number of flags in inotify(7) I think this is a
non-trivial project. Whoever wants to tackle this feature should be
aware that it may take a few months (3?) to fully implement and test.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]