inotify, new idea?

inotify, new idea?

[Jos Huisken]

I was trying to maintain a local and remote directory in sync with
lsync, using inotify.
I happen to have >4M files and >400k directories... running over
/proc/sys/fs/inotify/max_user_watches

What if inotify could just provide all events (limited for instance by
euid), which can be filtered by the application requesting
inotification?
Would something like that be possible?

BTW, I'm not a kernel expert, and not following this list... just
wanted to throw in this idea. Pls Cc: personally.

Thanks,
Jos

Re: inotify, new idea?

[Lennart Sorensen]

On Thu, Apr 17, 2014 at 11:00:37PM +0200, Jos Huisken wrote:
> I was trying to maintain a local and remote directory in sync with
> lsync, using inotify.
> I happen to have >4M files and >400k directories... running over
> /proc/sys/fs/inotify/max_user_watches

Would fanotify perhaps be a better interface to use?

-- 
Len Sorensen

Re: inotify, new idea?

[Michael Kerrisk]

On Thu, Apr 17, 2014 at 11:28 PM, Lennart Sorensen
<lsorense@csclub.uwaterloo.ca> wrote:
> On Thu, Apr 17, 2014 at 11:00:37PM +0200, Jos Huisken wrote:
>> I was trying to maintain a local and remote directory in sync with
>> lsync, using inotify.
>> I happen to have >4M files and >400k directories... running over
>> /proc/sys/fs/inotify/max_user_watches
>
> Would fanotify perhaps be a better interface to use?

(One of us is misunderstanding fanotify; it might be me.)

Did you look at fanotify closely? I don't think it could be used for
this task -- does not notify linka dn unlink events, difficult to set
up recursive monitoring, etc.

Cheers,

Michael

Re: inotify, new idea?

[Lennart Sorensen]

On Fri, Apr 18, 2014 at 06:15:40AM +0200, Michael Kerrisk wrote:
> (One of us is misunderstanding fanotify; it might be me.)

No I think you are right, and that the misunderstanding is on my part.
I had not realized that fanotify did not provide create/delete events,
only access events.

> Did you look at fanotify closely? I don't think it could be used for
> this task -- does not notify linka dn unlink events, difficult to set
> up recursive monitoring, etc.

It would appear you are correct.

-- 
Len Sorensen

Re: inotify, new idea?

[Richard Weinberger]

On Fri, Apr 18, 2014 at 6:15 AM, Michael Kerrisk <mtk.manpages@gmail.com> wrote:
> On Thu, Apr 17, 2014 at 11:28 PM, Lennart Sorensen
> <lsorense@csclub.uwaterloo.ca> wrote:
>> On Thu, Apr 17, 2014 at 11:00:37PM +0200, Jos Huisken wrote:
>>> I was trying to maintain a local and remote directory in sync with
>>> lsync, using inotify.
>>> I happen to have >4M files and >400k directories... running over
>>> /proc/sys/fs/inotify/max_user_watches
>>
>> Would fanotify perhaps be a better interface to use?
>
> (One of us is misunderstanding fanotify; it might be me.)
>
> Did you look at fanotify closely? I don't think it could be used for
> this task -- does not notify linka dn unlink events, difficult to set
> up recursive monitoring, etc.

Does recursive monitoring even work with inotify?
Last time I've tried it did failed as soon I did a mkdir -p a/b/c/d because
mkdir() raced against the thread which installes the new watches.

-- 
Thanks,
//richard

Re: inotify, new idea?

[Michael Kerrisk]

On Mon, Apr 21, 2014 at 12:22 AM, Richard Weinberger
<richard.weinberger@gmail.com> wrote:
> On Fri, Apr 18, 2014 at 6:15 AM, Michael Kerrisk <mtk.manpages@gmail.com> wrote:
>> On Thu, Apr 17, 2014 at 11:28 PM, Lennart Sorensen
>> <lsorense@csclub.uwaterloo.ca> wrote:
>>> On Thu, Apr 17, 2014 at 11:00:37PM +0200, Jos Huisken wrote:
>>>> I was trying to maintain a local and remote directory in sync with
>>>> lsync, using inotify.
>>>> I happen to have >4M files and >400k directories... running over
>>>> /proc/sys/fs/inotify/max_user_watches
>>>
>>> Would fanotify perhaps be a better interface to use?
>>
>> (One of us is misunderstanding fanotify; it might be me.)
>>
>> Did you look at fanotify closely? I don't think it could be used for
>> this task -- does not notify linka dn unlink events, difficult to set
>> up recursive monitoring, etc.
>
> Does recursive monitoring even work with inotify?
> Last time I've tried it did failed as soon I did a mkdir -p a/b/c/d because
> mkdir() raced against the thread which installes the new watches.

As I understand it, you have to program to deal with the races (rescan
directories after adding watches). I recently did a lot of work
updating the inotify(7) man page to discuss all the issues that I know
of, and their remedies. If I missed anything, I'd appreciate a note on
it, so that it can be added. See
http://man7.org/linux/man-pages/man7/inotify.7.html#NOTES

Cheers,

Michael


-- 
Michael Kerrisk Linux man-pages maintainer;
http://www.kernel.org/doc/man-pages/
Author of "The Linux Programming Interface", http://blog.man7.org/

Re: inotify, new idea?

[Richard Weinberger]

Am 21.04.2014 09:24, schrieb Michael Kerrisk:
>> Does recursive monitoring even work with inotify?
>> Last time I've tried it did failed as soon I did a mkdir -p a/b/c/d because
>> mkdir() raced against the thread which installes the new watches.
> 
> As I understand it, you have to program to deal with the races (rescan
> directories after adding watches). I recently did a lot of work
> updating the inotify(7) man page to discuss all the issues that I know
> of, and their remedies. If I missed anything, I'd appreciate a note on
> it, so that it can be added. See
> http://man7.org/linux/man-pages/man7/inotify.7.html#NOTES

I'm aware of the rescan hack, but in my case it does not help
because my program must not miss any event.
Currently I'm using a fuse overlay filesystem to log everything.
Not perfect but works... :-)

Thanks,
//richard

Re: inotify, new idea?

[Michael Kerrisk (man-pages)]

On 04/21/2014 10:42 AM, Richard Weinberger wrote:
> Am 21.04.2014 09:24, schrieb Michael Kerrisk:
>>> Does recursive monitoring even work with inotify?
>>> Last time I've tried it did failed as soon I did a mkdir -p a/b/c/d because
>>> mkdir() raced against the thread which installes the new watches.
>>
>> As I understand it, you have to program to deal with the races (rescan
>> directories after adding watches). I recently did a lot of work
>> updating the inotify(7) man page to discuss all the issues that I know
>> of, and their remedies. If I missed anything, I'd appreciate a note on
>> it, so that it can be added. See
>> http://man7.org/linux/man-pages/man7/inotify.7.html#NOTES
> 
> I'm aware of the rescan hack, but in my case it does not help
> because my program must not miss any event.

Then, we're understand the same thing: you're out of luck :-}.

> Currently I'm using a fuse overlay filesystem to log everything.
> Not perfect but works... :-)

Interesting notion. I need to get to grips with FUSE...


-- 
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/

Re: inotify, new idea?

[Richard Weinberger]

Am 21.04.2014 15:31, schrieb Michael Kerrisk (man-pages):
> On 04/21/2014 10:42 AM, Richard Weinberger wrote:
>> Am 21.04.2014 09:24, schrieb Michael Kerrisk:
>>>> Does recursive monitoring even work with inotify?
>>>> Last time I've tried it did failed as soon I did a mkdir -p a/b/c/d because
>>>> mkdir() raced against the thread which installes the new watches.
>>>
>>> As I understand it, you have to program to deal with the races (rescan
>>> directories after adding watches). I recently did a lot of work
>>> updating the inotify(7) man page to discuss all the issues that I know
>>> of, and their remedies. If I missed anything, I'd appreciate a note on
>>> it, so that it can be added. See
>>> http://man7.org/linux/man-pages/man7/inotify.7.html#NOTES
>>
>> I'm aware of the rescan hack, but in my case it does not help
>> because my program must not miss any event.
> 
> Then, we're understand the same thing: you're out of luck :-}.
> 
>> Currently I'm using a fuse overlay filesystem to log everything.
>> Not perfect but works... :-)
> 
> Interesting notion. I need to get to grips with FUSE...

e.g. http://loggedfs.sourceforge.net/

Thanks,
//richard

Re: inotify, new idea?

[Jos Huisken]

Seems like also 'mv' events are also not supported by fanotify, like
creation/deletion.
Therefore not an option (for lsyncd) probably. Maybe loggedfs is (if
time permits...).
Thanks for the discussion.

Jos

On Mon, Apr 21, 2014 at 5:01 PM, Richard Weinberger <richard@nod.at> wrote:
>
>
> Am 21.04.2014 15:31, schrieb Michael Kerrisk (man-pages):
>> On 04/21/2014 10:42 AM, Richard Weinberger wrote:
>>> Am 21.04.2014 09:24, schrieb Michael Kerrisk:
>>>>> Does recursive monitoring even work with inotify?
>>>>> Last time I've tried it did failed as soon I did a mkdir -p a/b/c/d because
>>>>> mkdir() raced against the thread which installes the new watches.
>>>>
>>>> As I understand it, you have to program to deal with the races (rescan
>>>> directories after adding watches). I recently did a lot of work
>>>> updating the inotify(7) man page to discuss all the issues that I know
>>>> of, and their remedies. If I missed anything, I'd appreciate a note on
>>>> it, so that it can be added. See
>>>> http://man7.org/linux/man-pages/man7/inotify.7.html#NOTES
>>>
>>> I'm aware of the rescan hack, but in my case it does not help
>>> because my program must not miss any event.
>>
>> Then, we're understand the same thing: you're out of luck :-}.
>>
>>> Currently I'm using a fuse overlay filesystem to log everything.
>>> Not perfect but works... :-)
>>
>> Interesting notion. I need to get to grips with FUSE...
>
> e.g. http://loggedfs.sourceforge.net/
>
> Thanks,
> //richard

Re: inotify, new idea?

[Michael Kerrisk (man-pages)]

On 04/21/2014 10:42 AM, Richard Weinberger wrote:
> Am 21.04.2014 09:24, schrieb Michael Kerrisk:
>>> Does recursive monitoring even work with inotify?
>>> Last time I've tried it did failed as soon I did a mkdir -p a/b/c/d because
>>> mkdir() raced against the thread which installes the new watches.
>>
>> As I understand it, you have to program to deal with the races (rescan
>> directories after adding watches). I recently did a lot of work
>> updating the inotify(7) man page to discuss all the issues that I know
>> of, and their remedies. If I missed anything, I'd appreciate a note on
>> it, so that it can be added. See
>> http://man7.org/linux/man-pages/man7/inotify.7.html#NOTES
> 
> I'm aware of the rescan hack, but in my case it does not help
> because my program must not miss any event.
> Currently I'm using a fuse overlay filesystem to log everything.
> Not perfect but works... :-)

Richard,

A late follow up question. How does your application deal with the
event overflow problem (i.e., when you get a large number of events 
much faster than your application can deal with them?

Cheers,

Michael


-- 
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/

Re: inotify, new idea?

[Richard Weinberger]

Am 24.05.2014 09:52, schrieb Michael Kerrisk (man-pages):
> On 04/21/2014 10:42 AM, Richard Weinberger wrote:
>> Am 21.04.2014 09:24, schrieb Michael Kerrisk:
>>>> Does recursive monitoring even work with inotify?
>>>> Last time I've tried it did failed as soon I did a mkdir -p a/b/c/d because
>>>> mkdir() raced against the thread which installes the new watches.
>>>
>>> As I understand it, you have to program to deal with the races (rescan
>>> directories after adding watches). I recently did a lot of work
>>> updating the inotify(7) man page to discuss all the issues that I know
>>> of, and their remedies. If I missed anything, I'd appreciate a note on
>>> it, so that it can be added. See
>>> http://man7.org/linux/man-pages/man7/inotify.7.html#NOTES
>>
>> I'm aware of the rescan hack, but in my case it does not help
>> because my program must not miss any event.
>> Currently I'm using a fuse overlay filesystem to log everything.
>> Not perfect but works... :-)
> 
> Richard,
> 
> A late follow up question. How does your application deal with the
> event overflow problem (i.e., when you get a large number of events 
> much faster than your application can deal with them?

The downside of the FUSE approach is that you have to intercept
every filesystem function.
This can be a performance issue.
But due to this design the overflow problem cannot happen as the
FUSE filesystem blocks until the event has been proceed.

Thanks,
//richard

Re: inotify, new idea?

[Marian Marinov]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 05/24/2014 03:34 PM, Richard Weinberger wrote:
> Am 24.05.2014 09:52, schrieb Michael Kerrisk (man-pages):
>> On 04/21/2014 10:42 AM, Richard Weinberger wrote:
>>> Am 21.04.2014 09:24, schrieb Michael Kerrisk:
>>>>> Does recursive monitoring even work with inotify? Last time I've tried it did failed as soon I did a mkdir
>>>>> -p a/b/c/d because mkdir() raced against the thread which installes the new watches.
>>>> 
>>>> As I understand it, you have to program to deal with the races (rescan directories after adding watches). I
>>>> recently did a lot of work updating the inotify(7) man page to discuss all the issues that I know of, and
>>>> their remedies. If I missed anything, I'd appreciate a note on it, so that it can be added. See 
>>>> http://man7.org/linux/man-pages/man7/inotify.7.html#NOTES
>>> 
>>> I'm aware of the rescan hack, but in my case it does not help because my program must not miss any event. 
>>> Currently I'm using a fuse overlay filesystem to log everything. Not perfect but works... :-)
>> 
>> Richard,
>> 
>> A late follow up question. How does your application deal with the event overflow problem (i.e., when you get a
>> large number of events much faster than your application can deal with them?
> 
> The downside of the FUSE approach is that you have to intercept every filesystem function. This can be a
> performance issue. But due to this design the overflow problem cannot happen as the FUSE filesystem blocks until
> the event has been proceed.
> 

Have anyone of you looked at this: https://github.com/1and1/linux-filemon/

I haven't stress tested it, but in the past I ported it to more recent kernels:
  https://github.com/hackman/filemon-patches

It is not polished, but it works.

I'm considering fixing some of the issues at has for use in my home setup.

Marian

> Thanks, //richard -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a
> message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html Please read
> the FAQ at  http://www.tux.org/lkml/
> 


- -- 
Marian Marinov
Founder & CEO of 1H Ltd.
Jabber/GTalk: hackman@jabber.org
ICQ: 7556201
Mobile: +359 886 660 270
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)

iEYEARECAAYFAlOCgOwACgkQ4mt9JeIbjJSXrgCdHCFADxvKN3Hxb4qinc/5Fj6V
WYMAnArk3Zb8YqTUNoyNCh1zYplUUx9B
=y/Sm
-----END PGP SIGNATURE-----

Re: inotify, new idea?

[Michael Kerrisk (man-pages)]

On 05/24/2014 02:34 PM, Richard Weinberger wrote:
> Am 24.05.2014 09:52, schrieb Michael Kerrisk (man-pages):
>> On 04/21/2014 10:42 AM, Richard Weinberger wrote:
>>> Am 21.04.2014 09:24, schrieb Michael Kerrisk:
>>>>> Does recursive monitoring even work with inotify?
>>>>> Last time I've tried it did failed as soon I did a mkdir -p a/b/c/d because
>>>>> mkdir() raced against the thread which installes the new watches.
>>>>
>>>> As I understand it, you have to program to deal with the races (rescan
>>>> directories after adding watches). I recently did a lot of work
>>>> updating the inotify(7) man page to discuss all the issues that I know
>>>> of, and their remedies. If I missed anything, I'd appreciate a note on
>>>> it, so that it can be added. See
>>>> http://man7.org/linux/man-pages/man7/inotify.7.html#NOTES
>>>
>>> I'm aware of the rescan hack, but in my case it does not help
>>> because my program must not miss any event.
>>> Currently I'm using a fuse overlay filesystem to log everything.
>>> Not perfect but works... :-)
>>
>> Richard,
>>
>> A late follow up question. How does your application deal with the
>> event overflow problem (i.e., when you get a large number of events 
>> much faster than your application can deal with them?
> 
> The downside of the FUSE approach is that you have to intercept
> every filesystem function.
> This can be a performance issue.
> But due to this design the overflow problem cannot happen as the
> FUSE filesystem blocks until the event has been proceed.

Ahh -- that clears things up for me. Thanks, Richard.

Cheers,

Michael

-- 
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/

Re: inotify, new idea?

[Richard Weinberger]

[-- Attachment #1: Type: text/plain, Size: 634 bytes --]

Am 26.05.2014 01:46, schrieb Marian Marinov:
> Have anyone of you looked at this: https://github.com/1and1/linux-filemon/
> 
> I haven't stress tested it, but in the past I ported it to more recent kernels:
>   https://github.com/hackman/filemon-patches
> 
> It is not polished, but it works.
> 
> I'm considering fixing some of the issues at has for use in my home setup.

Well, the patch itself has documented its major flaw:
+ * WARNING! This patch has been designed as EXPERIMENTAL. Its usage
+ * is DANGEROUS, because some filesystems could get exhausted by the
+ * masses of ORPHAN INODES!

Thanks,
//richard


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 901 bytes --]