Re: robinhood, fanotify name info events and lustre changelog

[Amir Goldstein <amir73il@gmail.com>]

> > I would be happy to work with you on a POC for adapting fanotify
> > test code with robinhood v4, but before I invest time on that, I would
> > need to know there is a good chance that people are going to test and
> > use robinhood with Linux vfs.
> >
> > Do you have actual users requesting to use robinhood with non-Lustre
> > fs?
>
> I would run it at home, but that isn't much :D
> As I wrote previously we have users for large nfs shares out of lustre,
> but I honestly don't think there will be much use for local filesystems
> at least in the short term.
>
> Filesystem indexers like tracker[1] or similar would definitely get much
> more use for that; from an objective point of view I wouldn't suggest
> you spend time on robinhood for this: local filesytems are rarely large
> enough to warrant using something like robinhood, and without something
> like fanotify we wouldn't be efficient for a local disk with hundreds of
> millions of files anyway because of the prohibitive rescan cost - so
> it's a bit like chicken and egg maybe,

I very much agree with that statement.
I have written the kernel side to facilitate a file server system with many
millions of files. We use in-house software for the user side, not very
different in concept from robinhood.
So I am looking for a similar use case out there using open source
software, and they are not easy to find.

> I don't know, but if you want
> many users to test different configurations I wouldn't recommend
> robinhood (OTOH, we run CI tests so would be happy to add that to the
> tests once it's available on vanilla kernels; but that's still not real
> users)
>
> [1] https://wiki.gnome.org/Projects/Tracker
>

The problem with Track/Watchman is that they are running as
as unprivileged services per user and fanotify requires
CAP_SYS_ADMIN (for good reasons).
Also, if they are not used for watching very large scale of directories,
there is no strong incentive to switch from inotify to fanotify.

My plan was to create a privileged system watchman service that
feeds off of fanotify and serves unprivileged watchman services.
This is not unlike MacOS fseventsd.
I never got around to asses the size of that task.

Looking at robinhood (especially v4), I seems like it could fit
very well into the vacuum in Linux and act as "fsnotifyd".
unprivileged applications and services could register to event streams
and get fed from db, so applications not running will not loose events.
Events delivered to unprivileged applications need to be filtered by
subtree those applications, something that fanotify does not do and
will not likely do and filtered by access permissions of application
to the path of the reported object.

This is not going to be an easy task, but without it, fanotify can serve
some niche use cases and not be as helpful to the wider community.

>
> > May I ask, what is the reason for embarking on the project to decouple
> > robinhood v4 API from Lustre changelog API?
> > Is it because you had other fsevent producers in mind?
>
> I've been planning to at least add some recursive-inotifywatch a
> subfolder at least (like watchman does) before these new fanotify events
> came up, so I might be partly to blame for that.
>
> There also are advantages for lustre though; the point is to be able to
> ingest changelogs directly with some daemon (it's only at proof of
> concept level for v4 at this point), but also to split the load by
> involving multiple lustre clients.
> So you would get a pool of lustre clients to read changelogs, a pool of
> lustre clients to stat files as required to enrich the fsevents (file
> size etc), and a pool of servers to read fsevents and commit changes to
> the database (this part is still at the design level afaik)
>

Sounds interesting. I hope I was able to plant enough seeds
in your mind to steer robinhood in the direction of a future "fsnotifyd" ;-)

Anyway, I will CC you with new posting of my work, so if you want
you can take it for a test drive.

Thanks,
Amir.