Better support for (desktop) file search / indexing applications

Better support for (desktop) file search / indexing applications

[Martin Steigerwald]

Hi!

Some time ago I stumpled over a blog entry that kernel user inotify watch
limit is often not enough for Nepomuk File Watcher to be notified of file
renames, new files and file deletes reliably[1].

There has been a discussion about that on various places[2,3,4] and likely
others.


I am writing to help the Nepomuk team to get in contact with Kernel
developers who could advise or help on how to solve the issues they
have with the current filesystem notification APIs in the kernel.

I thus added to CC any DNotify, INotify and FANotify maintainers as well
as Jan Kara who analyzed the advantages and disadvantages of each approach
and also developed some patches about recursive mtimes. I can dig out the
links to that as well, just ask if you want that. I also cc LKML,
linux-fsdevel and Nepomuk mailinglist. Feel free to drop CCs that you
deem inapprobiate or to add some for other Linux desktop or server
file indexing projects. Please tell me if I missed other kernel developers
who worked on file notification stuff.


The following two main issues led to the discussion about adding
notification about user inotify watch limit or even having it raised
automatically via some policy kit mechanism:

1) Watches are not working recursively. Thus one has to add a watch to
each sub directory.

2) There are inotify file move events. But one has to watch source and
destination directory to get notified of a file move between these. Thus
one has to watch each directory again. File moves outside the watched
home directory will go unnotified unless every other accessible directory
is watched as well.


What would be nice to have for file indexers would be:

1) Recursive notifications. I.e. one watch for /home/martin can notify 
about everything what happens in sub directories of that directory.

2) File move events that work from the source directory. I.e. if
watching a directory like /home/martin recursively it would be nice to
be notified about:

a) A file is moved from one sub directory inside /home/martin to another
one inside it.

b) A file is moved outside /home/martin

While these enhancement would likely fix the issues desktop file search
applications have with the kernel notification APIs, there might be other
approaches I did not yet thought off... so feel free to comment with your
thoughts on it.


Furthermore there is an issue with updating the file index on login or
service start. In order to catch all other file renames a indexer would
have to run over every directory whose modification time stamp has changed
again in order to see whether a (checksummed) file has moved.

An approach like recursive mtime as proposed by Jan Kara can help to
improve initial scan times a lot.

As to what I know this scan has been enabled in Nepomuk recently, with the
hope that files are moved mainly during the user session is active. I
think thats an assumption that may be accurate for many cases.

Still something like recursive mtime or BTRFS generation numbers with
btrfs subvolume find-new PATH LASTGENERATION would help that case a lot.
The issue with the BTRFS approach is that it only works as root. A
solution to this would be to integrate it in some daemon that works as
root and have applications communicate via socket or DBUS with it.


Some of this issues may apply to server side services like constellio or
Apache SolR (Lucene) as well. For example when there has been a service
downtime and after service restart the service wants to pick up last
changes. Or for near realtime indexing.


I hope to help to unstick the current state. I think its important for
kernel and userspace developers to talk to each other about good ways
to move forward.

So maybe some time in the future:

martin@merkaba:~> cat /etc/sysctl.d/nepomuk.conf 
# Für Nepomuk File Indexer
# martin@merkaba:~> find -type d | wc -l
# 34515
#
# merkaba:/proc/sys/fs/inotify> cat max_user_watches 
# 8192

fs.inotify.max_user_watches = 200000

Wont be necessary anymore.

I found that SLES 11 SP 2, maybe earlier versions as well, raise the
user watch limit to 65536 by default. So this seems to have been an
issue in a server-oriented enterprise distribution as well.



[1] Alvaro Soliverez: Nepomuk not indexing a large home:
http://soliverez.com.ar/home/2012/10/nepomuk-not-indexing-a-large-home/

[2] [Nepomuk] User limit reached. Please raise the inotify user watch limit:
http://lists.kde.org/?l=nepomuk&m=134954456529570&w=2

[3] Vishesh Handa, Nepomuk Without Files: 
http://vhanda.in/blog/2012/08/nepomuk-without-files/

[4] Martin Sandsmark, KFileMon,: 
http://martinsandsmark.wordpress.com/2012/08/07/kfilemon/

Thanks,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: [Nepomuk] Better support for (desktop) file search / indexing applications

[Martin Steigerwald]

Am Donnerstag, 1. November 2012 schrieb Martin Steigerwald:
> Furthermore there is an issue with updating the file index on login or
> service start. In order to catch all other file renames a indexer would
> have to run over every directory whose modification time stamp has
> changed again in order to see whether a (checksummed) file has moved.
> 
> An approach like recursive mtime as proposed by Jan Kara can help to
> improve initial scan times a lot.
> 
> As to what I know this scan has been enabled in Nepomuk recently, with
> the hope that files are moved mainly during the user session is
> active. I think thats an assumption that may be accurate for many
> cases.

disabled, not enabled.

I read over this before but did not see the typo.

Sorry,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: Better support for (desktop) file search / indexing applications

[Tvrtko Ursulin]

On Thursday 01 November 2012 13:52:42 Martin Steigerwald wrote:
...
> The following two main issues led to the discussion about adding
> notification about user inotify watch limit or even having it raised
> automatically via some policy kit mechanism:
> 
> 1) Watches are not working recursively. Thus one has to add a watch to
> each sub directory.
> 
> 2) There are inotify file move events. But one has to watch source and
> destination directory to get notified of a file move between these. Thus
> one has to watch each directory again. File moves outside the watched
> home directory will go unnotified unless every other accessible directory
> is watched as well.
> 
> 
> What would be nice to have for file indexers would be:
> 
> 1) Recursive notifications. I.e. one watch for /home/martin can notify
> about everything what happens in sub directories of that directory.
> 
> 2) File move events that work from the source directory. I.e. if
> watching a directory like /home/martin recursively it would be nice to
> be notified about:
> 
> a) A file is moved from one sub directory inside /home/martin to another
> one inside it.
> 
> b) A file is moved outside /home/martin
> 
> While these enhancement would likely fix the issues desktop file search
> applications have with the kernel notification APIs, there might be other
> approaches I did not yet thought off... so feel free to comment with your
> thoughts on it.

I will only comment on the real time indexing part since I had some part in 
the inception of fanotify and still remember a thing or two about it.

Perhaps you should look into how hard would it be to add directory or rename, 
and unlink events to fanotify. It may not be too hard.

In that case, even though it does not support recursive directory watches (I 
tried to implement this some time around 2009. but found it impossible to 
wedge into the fanotify locking model), it does support mount point watches. 
Which for the desktop use might be sufficient, assuming /home is typically a 
separate filesystem.

Downside with  this approach is that you have to filter out the events you do 
not care about like /home/some-other-user, or even more if /home is not a 
separate filesystem. Which with the current fanotify state can be done using 
paths, but that includes resolving a link in procfs which may be a too 
expensive thing to do.

Or perhaps it is acceptable, if you for example only cared about CLOSE_WRITE 
events (closure of file which were open for writing).

So I think for this part you have two options, have a go of extending 
directory watches to be recursive, or live with the mount watches giving you 
too much traffic.

Regards,

Tvrtko

Re: Better support for (desktop) file search / indexing applications

[Martin Steigerwald]

Am Donnerstag, 1. November 2012 schrieb Tvrtko Ursulin:
> On Thursday 01 November 2012 13:52:42 Martin Steigerwald wrote:
> ...
> 
> > The following two main issues led to the discussion about adding
> > notification about user inotify watch limit or even having it raised
> > automatically via some policy kit mechanism:
> > 
> > 1) Watches are not working recursively. Thus one has to add a watch
> > to each sub directory.
> > 
> > 2) There are inotify file move events. But one has to watch source
> > and destination directory to get notified of a file move between
> > these. Thus one has to watch each directory again. File moves
> > outside the watched home directory will go unnotified unless every
> > other accessible directory is watched as well.
> > 
> > 
> > What would be nice to have for file indexers would be:
> > 
> > 1) Recursive notifications. I.e. one watch for /home/martin can
> > notify about everything what happens in sub directories of that
> > directory.
> > 
> > 2) File move events that work from the source directory. I.e. if
> > watching a directory like /home/martin recursively it would be nice
> > to be notified about:
> > 
> > a) A file is moved from one sub directory inside /home/martin to
> > another one inside it.
> > 
> > b) A file is moved outside /home/martin
> > 
> > While these enhancement would likely fix the issues desktop file
> > search applications have with the kernel notification APIs, there
> > might be other approaches I did not yet thought off... so feel free
> > to comment with your thoughts on it.
> 
> I will only comment on the real time indexing part since I had some
> part in the inception of fanotify and still remember a thing or two
> about it.
> 
> Perhaps you should look into how hard would it be to add directory or
> rename, and unlink events to fanotify. It may not be too hard.
> 
> In that case, even though it does not support recursive directory
> watches (I tried to implement this some time around 2009. but found it
> impossible to wedge into the fanotify locking model), it does support
> mount point watches. Which for the desktop use might be sufficient,
> assuming /home is typically a separate filesystem.
> 
> Downside with  this approach is that you have to filter out the events
> you do not care about like /home/some-other-user, or even more if
> /home is not a separate filesystem. Which with the current fanotify
> state can be done using paths, but that includes resolving a link in
> procfs which may be a too expensive thing to do.
> 
> Or perhaps it is acceptable, if you for example only cared about
> CLOSE_WRITE events (closure of file which were open for writing).
> 
> So I think for this part you have two options, have a go of extending
> directory watches to be recursive, or live with the mount watches
> giving you too much traffic.

Thanks for your suggestions.

Still fanotify needs root access and thus this would need a daemon running 
as root and some policy kit stuff to access it and in case of mount point 
watches robust and secure code so that each user may only see his/her own 
results.

Any other ideas from anyone?

Thanks.
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: Better support for (desktop) file search / indexing applications

[Tvrtko Ursulin]

On Saturday 10 November 2012 17:53:45 Martin Steigerwald wrote:
> Still fanotify needs root access and thus this would need a daemon running
> as root and some policy kit stuff to access it and in case of mount point
> watches robust and secure code so that each user may only see his/her own
> results.

Perhaps then also extend fanotify to support user watches, from the top of my 
head I can't think of a reason it would be very difficult to implement. But it 
has been a few years since I actively worked with that code.

Since you are not the only group having issues with fanotify feature set I can 
see this mini-project (together with extensions from me previous reply) being 
useful. It is also better to evolve it than neglect due a few shortcomings and 
then in a few years someone will come up with something completely new and 
then we will have yet another notification system.

Tvrtko

Fwd: [Nepomuk] Better support for (desktop) file search / indexing applications

[Simeon Bird]

Sent again in plain-text - apologies.

---------- Forwarded message ----------
From: Simeon Bird <bladud@gmail.com>
Date: 9 March 2013 23:49
Subject: Re: [Nepomuk] Better support for (desktop) file search /
indexing applications
To: Tvrtko Ursulin <tvrtko.ursulin@onelan.co.uk>
Cc: Martin Steigerwald <Martin@lichtvoll.de>, Jan Kara <jack@suse.cz>,
Robert Love <rlove@rlove.org>, linux-kernel@vger.kernel.org, Eric
Paris <eparis@redhat.com>, Nepomuk Mailing List <nepomuk@kde.org>,
Linux Filesystem Development Mailinglist
<linux-fsdevel@vger.kernel.org>, Eric Paris <eparis@parisplace.org>,
John McCutchan <john@johnmccutchan.com>


On 12 November 2012 04:10, Tvrtko Ursulin <tvrtko.ursulin@onelan.co.uk> wrote:
>
> On Saturday 10 November 2012 17:53:45 Martin Steigerwald wrote:
> > Still fanotify needs root access and thus this would need a daemon running
> > as root and some policy kit stuff to access it and in case of mount point
> > watches robust and secure code so that each user may only see his/her own
> > results.
>
> Perhaps then also extend fanotify to support user watches, from the top of my
> head I can't think of a reason it would be very difficult to implement. But it
> has been a few years since I actively worked with that code.
>
> Since you are not the only group having issues with fanotify feature set I can
> see this mini-project (together with extensions from me previous reply) being
> useful. It is also better to evolve it than neglect due a few shortcomings and
> then in a few years someone will come up with something completely new and
> then we will have yet another notification system.
>
> Tvrtko


Hi,

We (nepomuk) recently looked at using fanotify, and indeed we would
need user watches, support for moves and recursive directory watches
(we need to support the case where /home is not a separate filesystem)
before it would be useful to us. If you are interested in adding
these, we would be delighted to use nepomuk as a test-case for them.

We were wondering also if it would be possible to extend inotify a
little? Our wishlist is:

1) Recursive folder watches
2) When a file moves, some way to get the destination without watching
the directory it moved to, so moves can be tracked without watching
every file on the system.

I understand that there are reasons of security and performance why
you cannot implement 1), but is 2) possible? Maybe by extending
IN_MOVED_TO, or adding a new event type?

2) is actually in some ways the more severe problem for us. As well as
being an indexer, nepomuk is a system that allows you to store file
metadata such as ratings. When users move the files, they want the
metadata to move too, so we need to track where the file moved, and
thus at the moment we recursively watch everything. This is
particularly problematic with removable media; because a lot of people
will plug in an external drive and then move files onto it, we have to
watch every drive as soon as it is plugged in. If we were able to get
the destination of move events without watching the destination
directory, we could watch only those directories with interesting
metadata in, which would make things a lot easier.

inotify move tracking would also be useful for other things - eg, a
text editor could use inotify to see if a file it has open has moved
and offer to re-open the file in its new location, which is impossible
at the moment.

Since the lack of recursive watches is really a problem because we
have a tendency to run out of watches, it would also really help if
the default limit was a bit higher -  most people seem to have > 8000
folders, but I suspect far fewer have > 32000 (probably excepting
those who are indexing kernel source trees: I have 21000, and half of
that is KDE source).

Would any of this be possible? If you happen to know of a better way
to track moves using existing tools, that would be even better.

Thanks,
Simeon

Re: Fwd: [Nepomuk] Better support for (desktop) file search / indexing applications

[Lijo Antony]

On 03/10/2013 08:51 AM, Simeon Bird wrote:
>
> Hi,
>
> We (nepomuk) recently looked at using fanotify, and indeed we would
> need user watches, support for moves and recursive directory watches
> (we need to support the case where /home is not a separate filesystem)
> before it would be useful to us. If you are interested in adding
> these, we would be delighted to use nepomuk as a test-case for them.
>
> We were wondering also if it would be possible to extend inotify a
> little? Our wishlist is:
>
> 1) Recursive folder watches
> 2) When a file moves, some way to get the destination without watching
> the directory it moved to, so moves can be tracked without watching
> every file on the system.

I am also interested in these features. As of now, my solutions are,

1) When the limit is reached, ask the user to increase the limit and 
restart the application.

2) When top level directory move is detected, do a file system search 
based on inode to find out the new location. Very slow and not fool proof.

I would also like to any other solutions for these problems. I am yet to 
look into fanotify.

-lijo

[leaving the rest for reference]

>
> I understand that there are reasons of security and performance why
> you cannot implement 1), but is 2) possible? Maybe by extending
> IN_MOVED_TO, or adding a new event type?
>
> 2) is actually in some ways the more severe problem for us. As well as
> being an indexer, nepomuk is a system that allows you to store file
> metadata such as ratings. When users move the files, they want the
> metadata to move too, so we need to track where the file moved, and
> thus at the moment we recursively watch everything. This is
> particularly problematic with removable media; because a lot of people
> will plug in an external drive and then move files onto it, we have to
> watch every drive as soon as it is plugged in. If we were able to get
> the destination of move events without watching the destination
> directory, we could watch only those directories with interesting
> metadata in, which would make things a lot easier.
>
> inotify move tracking would also be useful for other things - eg, a
> text editor could use inotify to see if a file it has open has moved
> and offer to re-open the file in its new location, which is impossible
> at the moment.
>
> Since the lack of recursive watches is really a problem because we
> have a tendency to run out of watches, it would also really help if
> the default limit was a bit higher -  most people seem to have > 8000
> folders, but I suspect far fewer have > 32000 (probably excepting
> those who are indexing kernel source trees: I have 21000, and half of
> that is KDE source).
>
> Would any of this be possible? If you happen to know of a better way
> to track moves using existing tools, that would be even better.
>
> Thanks,
> Simeon
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: [Nepomuk] Better support for (desktop) file search / indexing applications

[Andreas Dilger]

On 2013-03-10, at 6:06, Lijo Antony <lijo.kernel@gmail.com> wrote:
> On 03/10/2013 08:51 AM, Simeon Bird wrote:
>> 
>> We (nepomuk) recently looked at using fanotify, and indeed we would
>> need user watches, support for moves and recursive directory watches
>> (we need to support the case where /home is not a separate filesystem)
>> before it would be useful to us. If you are interested in adding
>> these, we would be delighted to use nepomuk as a test-case for them.
>> 
>> We were wondering also if it would be possible to extend inotify a
>> little? Our wishlist is:
>> 
>> 1) Recursive folder watches
>> 2) When a file moves, some way to get the destination without watching
>> the directory it moved to, so moves can be tracked without watching
>> every file on the system.
> 
> I am also interested in these features. As of now, my solutions are,
> 
> 1) When the limit is reached, ask the user to increase the limit and restart the application.
> 
> 2) When top level directory move is detected, do a file system search based on inode to find out the new location. Very slow and not fool proof.

For Lustre, we implemented something similar to inotify with some
improvements that are possible because we limit the backend
filesystems that it runs on (ext4 and ZFS currently, but would also be
possible on Btrfs as well). 

For #1 (event recording) we have a persistent transactional ChangeLog
that is updated atomically with the metadata operation (create, rename,
unlink, etc).  This allows external applications to be notified of changes
in the whole filesystem, even if there are modifications while the watcher was
not running (to some limited extent). It is possible to limit the types of events
that are recorded in the ChangeLog, but not necessarily by pathname yet.
This is used for HSM and remote filesystem replication today.

For #2, we have a function "fid2path" that will generate in O(1) each
pathname of a file given the FID (essentially the inode number). This is
possible because each inode keeps an xattr ("link") that is updated for each link 
or rename of the inode with the parent directory FID and directory entry name. 

The "link" xattr is relatively low cost, since the inode needs to be updated for 
each link/rename/unlink anyway (nlinks and ctime), and in the overwhelmingly
common case if a single link on a file there is only a single entry in the xattr,
so it can fit inside the inode.

>From the list of links, we can walk the namespace in reverse order with
fid2path() to generate all if the pathnames of an inode. Something
similar would also allow you to find the target directory of a renamed file
without having to watch all of the directories. 

Cheers, Andreas

> I would also like to any other solutions for these problems. I am yet to look into fanotify.
> 
> -lijo
> 
> [leaving the rest for reference]
> 
>> 
>> I understand that there are reasons of security and performance why
>> you cannot implement 1), but is 2) possible? Maybe by extending
>> IN_MOVED_TO, or adding a new event type?
>> 
>> 2) is actually in some ways the more severe problem for us. As well as
>> being an indexer, nepomuk is a system that allows you to store file
>> metadata such as ratings. When users move the files, they want the
>> metadata to move too, so we need to track where the file moved, and
>> thus at the moment we recursively watch everything. This is
>> particularly problematic with removable media; because a lot of people
>> will plug in an external drive and then move files onto it, we have to
>> watch every drive as soon as it is plugged in. If we were able to get
>> the destination of move events without watching the destination
>> directory, we could watch only those directories with interesting
>> metadata in, which would make things a lot easier.
>> 
>> inotify move tracking would also be useful for other things - eg, a
>> text editor could use inotify to see if a file it has open has moved
>> and offer to re-open the file in its new location, which is impossible
>> at the moment.
>> 
>> Since the lack of recursive watches is really a problem because we
>> have a tendency to run out of watches, it would also really help if
>> the default limit was a bit higher -  most people seem to have > 8000
>> folders, but I suspect far fewer have > 32000 (probably excepting
>> those who are indexing kernel source trees: I have 21000, and half of
>> that is KDE source).
>> 
>> Would any of this be possible? If you happen to know of a better way
>> to track moves using existing tools, that would be even better.
>> 
>> Thanks,
>> Simeon
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> Please read the FAQ at  http://www.tux.org/lkml/
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html