All of lore.kernel.org
 help / color / mirror / Atom feed
* Reproducible, long-standing fanotify+autofs problem
@ 2017-01-25 11:48 Marko Rauhamaa
  2017-01-25 12:03 ` Amir Goldstein
  0 siblings, 1 reply; 11+ messages in thread
From: Marko Rauhamaa @ 2017-01-25 11:48 UTC (permalink / raw)
  To: linux-fsdevel; +Cc: marko.rauhamaa, Amir Goldstein


We have run into an easily reproducible fanotify hang that affects
several distros as well as the latest development kernel. (It isn't
fixed by Amir Goldstein's recent fanotify patch, BTW.)

Here is a simple reproduction:

 1. Compile and build the tiny test program below, which has been
    adapted from the example in "man fanotify":

    cc -o fantest3 fantest.c

 2. Install autofs.

 3. Run these commands as root:

      mkdir -p foo/home/test foo/user/test
      echo /sshome /etc/auto.sshome >/etc/auto.master
      echo "test / :/root/foo/home/test /user :/root/foo/user/test" \
          >/etc/auto.sshome
      systemctl start autofs
      ls -l /sshome/test
      ./fantest3 / /sshome/test /sshome/test/user/

      # NB! The final slash in the fantest3 command can be significant!

  4. Observe the last command hang. While it is hung, other file system
     operations from other processes are blocked.

  5. Press Ctrl-C to force fantest3 to exit. The file system becomes
     alive. Rerunning the fantest3 command no longer hangs.

  6. To get back into the problem state, execute (as root):

      systemctl restart autofs
      ls -l /sshome/test
      ./fantest3 / /sshome/test /sshome/test/user/


===Begin fantest.c======================================================
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/fanotify.h>

int main(int argc, char *argv[])
{
    int fd, i;
    if (argc < 2) {
        fprintf(stderr, "Usage: %s MOUNT ...\n", argv[0]);
        exit(EXIT_FAILURE);
    }
    fd = fanotify_init(FAN_CLOEXEC | FAN_CLASS_CONTENT | FAN_NONBLOCK,
                       O_RDONLY);
    if (fd == -1) {
        perror("fanotify_init");
        exit(EXIT_FAILURE);
    }
    for (i = 1; i < argc; i++)
        if (fanotify_mark(fd, FAN_MARK_ADD | FAN_MARK_MOUNT,
                          FAN_OPEN_PERM | FAN_CLOSE_WRITE, AT_FDCWD,
                          argv[i]) == -1) {
            perror("fanotify_mark");
            exit(EXIT_FAILURE);
        }
    exit(EXIT_SUCCESS);
}
===End fantest.c========================================================


Marko

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: Reproducible, long-standing fanotify+autofs problem
  2017-01-25 11:48 Reproducible, long-standing fanotify+autofs problem Marko Rauhamaa
@ 2017-01-25 12:03 ` Amir Goldstein
  2017-01-25 14:06   ` Marko Rauhamaa
  0 siblings, 1 reply; 11+ messages in thread
From: Amir Goldstein @ 2017-01-25 12:03 UTC (permalink / raw)
  To: Marko Rauhamaa; +Cc: linux-fsdevel, Jan Kara

On Wed, Jan 25, 2017 at 1:48 PM, Marko Rauhamaa
<marko.rauhamaa@f-secure.com> wrote:
>
> We have run into an easily reproducible fanotify hang that affects
> several distros as well as the latest development kernel. (It isn't
> fixed by Amir Goldstein's recent fanotify patch, BTW.)
>

Not sure what is the role that autofs is playing in this hang, but it is worth
checking if Jan's latest work fixes the problem. I estimate that it should,
because it isolated the user space handling of permission events from
affecting processes generating fsnotify events on other fs objects.

git://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-fs.git #for_testing

> Here is a simple reproduction:
>
>  1. Compile and build the tiny test program below, which has been
>     adapted from the example in "man fanotify":
>
>     cc -o fantest3 fantest.c
>
>  2. Install autofs.
>
>  3. Run these commands as root:
>
>       mkdir -p foo/home/test foo/user/test
>       echo /sshome /etc/auto.sshome >/etc/auto.master
>       echo "test / :/root/foo/home/test /user :/root/foo/user/test" \
>           >/etc/auto.sshome
>       systemctl start autofs
>       ls -l /sshome/test
>       ./fantest3 / /sshome/test /sshome/test/user/
>
>       # NB! The final slash in the fantest3 command can be significant!
>
>   4. Observe the last command hang. While it is hung, other file system
>      operations from other processes are blocked.

You mean other processes not trying to access objects under the
watched mounts. right?

>
>   5. Press Ctrl-C to force fantest3 to exit. The file system becomes
>      alive. Rerunning the fantest3 command no longer hangs.
>
>   6. To get back into the problem state, execute (as root):
>
>       systemctl restart autofs
>       ls -l /sshome/test
>       ./fantest3 / /sshome/test /sshome/test/user/
>
>
> ===Begin fantest.c======================================================
> #include <fcntl.h>
> #include <stdio.h>
> #include <stdlib.h>
> #include <sys/fanotify.h>
>
> int main(int argc, char *argv[])
> {
>     int fd, i;
>     if (argc < 2) {
>         fprintf(stderr, "Usage: %s MOUNT ...\n", argv[0]);
>         exit(EXIT_FAILURE);
>     }
>     fd = fanotify_init(FAN_CLOEXEC | FAN_CLASS_CONTENT | FAN_NONBLOCK,
>                        O_RDONLY);
>     if (fd == -1) {
>         perror("fanotify_init");
>         exit(EXIT_FAILURE);
>     }
>     for (i = 1; i < argc; i++)
>         if (fanotify_mark(fd, FAN_MARK_ADD | FAN_MARK_MOUNT,
>                           FAN_OPEN_PERM | FAN_CLOSE_WRITE, AT_FDCWD,
>                           argv[i]) == -1) {
>             perror("fanotify_mark");
>             exit(EXIT_FAILURE);
>         }
>     exit(EXIT_SUCCESS);
> }
> ===End fantest.c========================================================
>
>
> Marko

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: Reproducible, long-standing fanotify+autofs problem
  2017-01-25 12:03 ` Amir Goldstein
@ 2017-01-25 14:06   ` Marko Rauhamaa
  2017-01-25 14:11     ` Amir Goldstein
  0 siblings, 1 reply; 11+ messages in thread
From: Marko Rauhamaa @ 2017-01-25 14:06 UTC (permalink / raw)
  To: Amir Goldstein; +Cc: linux-fsdevel, Jan Kara

Amir Goldstein <amir73il@gmail.com>:

> On Wed, Jan 25, 2017 at 1:48 PM, Marko Rauhamaa
> <marko.rauhamaa@f-secure.com> wrote:
>> We have run into an easily reproducible fanotify hang that affects
>> several distros as well as the latest development kernel. (It isn't
>> fixed by Amir Goldstein's recent fanotify patch, BTW.)
>
> Not sure what is the role that autofs is playing in this hang, but it
> is worth checking if Jan's latest work fixes the problem. I estimate
> that it should, because it isolated the user space handling of
> permission events from affecting processes generating fsnotify events
> on other fs objects.
>
> git://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-fs.git #for_testing

I could try.

>>   4. Observe the last command hang. While it is hung, other file system
>>      operations from other processes are blocked.
>
> You mean other processes not trying to access objects under the
> watched mounts. right?

No, accessing the watched mounts. Don't know if the hang is limited to
them.


Marko

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: Reproducible, long-standing fanotify+autofs problem
  2017-01-25 14:06   ` Marko Rauhamaa
@ 2017-01-25 14:11     ` Amir Goldstein
  2017-01-25 14:42       ` Marko Rauhamaa
  0 siblings, 1 reply; 11+ messages in thread
From: Amir Goldstein @ 2017-01-25 14:11 UTC (permalink / raw)
  To: Marko Rauhamaa; +Cc: linux-fsdevel, Jan Kara

On Wed, Jan 25, 2017 at 4:06 PM, Marko Rauhamaa
<marko.rauhamaa@f-secure.com> wrote:
> Amir Goldstein <amir73il@gmail.com>:
>
>> On Wed, Jan 25, 2017 at 1:48 PM, Marko Rauhamaa
>> <marko.rauhamaa@f-secure.com> wrote:
>>> We have run into an easily reproducible fanotify hang that affects
>>> several distros as well as the latest development kernel. (It isn't
>>> fixed by Amir Goldstein's recent fanotify patch, BTW.)
>>
>> Not sure what is the role that autofs is playing in this hang, but it
>> is worth checking if Jan's latest work fixes the problem. I estimate
>> that it should, because it isolated the user space handling of
>> permission events from affecting processes generating fsnotify events
>> on other fs objects.
>>
>> git://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-fs.git #for_testing
>
> I could try.
>
>>>   4. Observe the last command hang. While it is hung, other file system
>>>      operations from other processes are blocked.
>>
>> You mean other processes not trying to access objects under the
>> watched mounts. right?
>
> No, accessing the watched mounts. Don't know if the hang is limited to
> them.

So I am confused.
Your test program sets a watch on permission events on the mount
and does not respond to permission events on the mount, so all
file system operations on the mount SHOULD be blocked.
What am I missing?

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: Reproducible, long-standing fanotify+autofs problem
  2017-01-25 14:11     ` Amir Goldstein
@ 2017-01-25 14:42       ` Marko Rauhamaa
  2017-01-25 14:54         ` Amir Goldstein
  0 siblings, 1 reply; 11+ messages in thread
From: Marko Rauhamaa @ 2017-01-25 14:42 UTC (permalink / raw)
  To: Amir Goldstein; +Cc: linux-fsdevel, Jan Kara

Amir Goldstein <amir73il@gmail.com>:

> On Wed, Jan 25, 2017 at 4:06 PM, Marko Rauhamaa
> <marko.rauhamaa@f-secure.com> wrote:
>> Amir Goldstein <amir73il@gmail.com>:
> So I am confused.
> Your test program sets a watch on permission events on the mount
> and does not respond to permission events on the mount, so all
> file system operations on the mount SHOULD be blocked.
> What am I missing?

The problem is that the fanotify_mark() system call is hanging and as a
side effect blocks everything.

The side effect is probably completely understandable, but
fanotify_mark() should never hang, should it?


Marko

-- 
+358 44 990 4795
Skype: marko.rauhamaa_f-secure

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: Reproducible, long-standing fanotify+autofs problem
  2017-01-25 14:42       ` Marko Rauhamaa
@ 2017-01-25 14:54         ` Amir Goldstein
  2017-01-25 15:15           ` Jan Kara
  2017-01-25 15:20           ` Marko Rauhamaa
  0 siblings, 2 replies; 11+ messages in thread
From: Amir Goldstein @ 2017-01-25 14:54 UTC (permalink / raw)
  To: Marko Rauhamaa; +Cc: linux-fsdevel, Jan Kara

On Wed, Jan 25, 2017 at 4:42 PM, Marko Rauhamaa
<marko.rauhamaa@f-secure.com> wrote:
> Amir Goldstein <amir73il@gmail.com>:
>
>> On Wed, Jan 25, 2017 at 4:06 PM, Marko Rauhamaa
>> <marko.rauhamaa@f-secure.com> wrote:
>>> Amir Goldstein <amir73il@gmail.com>:
>> So I am confused.
>> Your test program sets a watch on permission events on the mount
>> and does not respond to permission events on the mount, so all
>> file system operations on the mount SHOULD be blocked.
>> What am I missing?
>
> The problem is that the fanotify_mark() system call is hanging and as a
> side effect blocks everything.
>
> The side effect is probably completely understandable, but
> fanotify_mark() should never hang, should it?
>
>

I see. I guess it shouldn't block.
Also with this understanding, I am not sure Jan's work is relevant to
your issue.

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: Reproducible, long-standing fanotify+autofs problem
  2017-01-25 14:54         ` Amir Goldstein
@ 2017-01-25 15:15           ` Jan Kara
       [not found]             ` <87efzrrwec.fsf@drapion.f-secure.com>
  2017-01-25 15:20           ` Marko Rauhamaa
  1 sibling, 1 reply; 11+ messages in thread
From: Jan Kara @ 2017-01-25 15:15 UTC (permalink / raw)
  To: Amir Goldstein; +Cc: Marko Rauhamaa, linux-fsdevel, Jan Kara

On Wed 25-01-17 16:54:46, Amir Goldstein wrote:
> On Wed, Jan 25, 2017 at 4:42 PM, Marko Rauhamaa
> <marko.rauhamaa@f-secure.com> wrote:
> > Amir Goldstein <amir73il@gmail.com>:
> >
> >> On Wed, Jan 25, 2017 at 4:06 PM, Marko Rauhamaa
> >> <marko.rauhamaa@f-secure.com> wrote:
> >>> Amir Goldstein <amir73il@gmail.com>:
> >> So I am confused.
> >> Your test program sets a watch on permission events on the mount
> >> and does not respond to permission events on the mount, so all
> >> file system operations on the mount SHOULD be blocked.
> >> What am I missing?
> >
> > The problem is that the fanotify_mark() system call is hanging and as a
> > side effect blocks everything.
> >
> > The side effect is probably completely understandable, but
> > fanotify_mark() should never hang, should it?
> >
> >
> 
> I see. I guess it shouldn't block.
> Also with this understanding, I am not sure Jan's work is relevant to
> your issue.

Yeah, it should not, although I expect this is some weird interaction with
autofs which tries to open something, which generates fanotify open event
for already created mark and thus following mark creation gets hung. Can
you reproduce the hang and send contents of dmesg after you do
'echo t >/proc/sysrq-trigger'? Thanks.

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: Reproducible, long-standing fanotify+autofs problem
  2017-01-25 14:54         ` Amir Goldstein
  2017-01-25 15:15           ` Jan Kara
@ 2017-01-25 15:20           ` Marko Rauhamaa
  2017-01-26 17:47             ` Marko Rauhamaa
  1 sibling, 1 reply; 11+ messages in thread
From: Marko Rauhamaa @ 2017-01-25 15:20 UTC (permalink / raw)
  To: Amir Goldstein; +Cc: linux-fsdevel, Jan Kara

Amir Goldstein <amir73il@gmail.com>:

> I see. I guess it shouldn't block. Also with this understanding, I am
> not sure Jan's work is relevant to your issue.

I have narrowed the problem down a bit in the kernel. The hanging
function is "user_path_at":

    user_path_at(filename="/sshome/test/user/", lookup_flags=0x1)
      called from: fanotify_find_path()
      called from: fanotify_mark()

And Ctrl-C wakes it up.


Marko

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: Reproducible, long-standing fanotify+autofs problem
  2017-01-25 15:20           ` Marko Rauhamaa
@ 2017-01-26 17:47             ` Marko Rauhamaa
  2017-01-26 18:05               ` Marko Rauhamaa
  0 siblings, 1 reply; 11+ messages in thread
From: Marko Rauhamaa @ 2017-01-26 17:47 UTC (permalink / raw)
  To: Amir Goldstein; +Cc: linux-fsdevel, Jan Kara

Marko Rauhamaa <marko.rauhamaa@f-secure.com>:

> Amir Goldstein <amir73il@gmail.com>:
> I have narrowed the problem down a bit in the kernel. The hanging
> function is "user_path_at":
>
>     user_path_at(filename="/sshome/test/user/", lookup_flags=0x1)
>       called from: fanotify_find_path()
>       called from: fanotify_mark()
>
> And Ctrl-C wakes it up.

I have dug a bit deeper. What hangs is this statement:

    mnt = path->dentry->d_op->d_automount(path);

in:

    follow_automount() // fs/namei.c
      called from: follow_managed()
      called from: walk_component()
      called from: lookup_last()
      called from: path_lookupat()
      called from: filename_lookup()
      called from: user_path_at_empty()
      called from: user_path_at(filename="/sshome/test/user/", lookup_flags=0x1)
      called from: fanotify_find_path() // fs/notify/fanotify_user.c
      called from: fanotify_mark()

And Ctrl-C wakes it up.

The d_automount() method is bound to autofs4_d_automount() in
fs/autofs4/root.c. The hanging statement is:

     wait_event_interruptible(wq->queue, wq->name.name == NULL);

in:

    autofs4_wait() // fs/autofs4/waitq.c
      called from: autofs4_mount_wait() // fs/autofs4/root.c
      called from: autofs4_d_automount()


Marko

-- 
+358 44 990 4795
Skype: marko.rauhamaa_f-secure

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: Reproducible, long-standing fanotify+autofs problem
  2017-01-26 17:47             ` Marko Rauhamaa
@ 2017-01-26 18:05               ` Marko Rauhamaa
  0 siblings, 0 replies; 11+ messages in thread
From: Marko Rauhamaa @ 2017-01-26 18:05 UTC (permalink / raw)
  To: Amir Goldstein; +Cc: linux-fsdevel, Jan Kara

Marko Rauhamaa <marko.rauhamaa@f-secure.com>:
> I have dug a bit deeper. What hangs is this statement:
>
>     mnt = path->dentry->d_op->d_automount(path);
>
> in:
>
>     follow_automount() // fs/namei.c
>       called from: follow_managed()
        called from: lookup_fast()
>       called from: walk_component()
>       called from: lookup_last()
>       called from: path_lookupat()
>       called from: filename_lookup()
>       called from: user_path_at_empty()
>       called from: user_path_at(filename="/sshome/test/user/", lookup_flags=0x1)
>       called from: fanotify_find_path() // fs/notify/fanotify_user.c
>       called from: fanotify_mark()
>
> And Ctrl-C wakes it up.
>
> The d_automount() method is bound to autofs4_d_automount() in
> fs/autofs4/root.c. The hanging statement is:
>
>      wait_event_interruptible(wq->queue, wq->name.name == NULL);
>
> in:
>
>     autofs4_wait() // fs/autofs4/waitq.c
>       called from: autofs4_mount_wait() // fs/autofs4/root.c
>       called from: autofs4_d_automount()

Added a missing call to the call trace.


Marko

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: Reproducible, long-standing fanotify+autofs problem
       [not found]               ` <20170127134620.GA16757@quack2.suse.cz>
@ 2017-01-27 14:21                 ` Marko Rauhamaa
  0 siblings, 0 replies; 11+ messages in thread
From: Marko Rauhamaa @ 2017-01-27 14:21 UTC (permalink / raw)
  To: Jan Kara; +Cc: Amir Goldstein, linux-fsdevel

Jan Kara <jack@suse.cz>:

> Yes, so from the backtraces it is pretty clear what is going on:
>
> You have placed OPEN_PERM fanotify mark on a / mountpoint. Once you do
> that, any open attempt anywhere in / must be approved by your process.
> Then you want to place an fanotify mark on /sshome/test. As a part of
> directory lookup of that path automounter triggers and wants to mount
> /sshome. That tries to open something in / (likely the config file in
> /etc) and blocks on approval from your process. Deadlock.
>
> So the kernel behaves as it was asked to, you just have to be *very*
> careful when placing permission marks into the system as it is very
> easy to deadlock it. Another similar type of deadlocks users of
> fanotify permission events hit is that the process responding to
> fanotify permission events does something (perhaps indirectly through
> some library) that generates another fanotify event.

That means it is not possible, in the general case, to use fanotify from
a single thread/process. A worthwhile lesson.

Thanks, Jan, for the clarification. Now we all know that fanotify_mark()
*can* block. The fanotify_mark() man page could benefit from the
addition of EINTR in the ERRORS section.


Marko

^ permalink raw reply	[flat|nested] 11+ messages in thread

end of thread, other threads:[~2017-01-27 14:21 UTC | newest]

Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-01-25 11:48 Reproducible, long-standing fanotify+autofs problem Marko Rauhamaa
2017-01-25 12:03 ` Amir Goldstein
2017-01-25 14:06   ` Marko Rauhamaa
2017-01-25 14:11     ` Amir Goldstein
2017-01-25 14:42       ` Marko Rauhamaa
2017-01-25 14:54         ` Amir Goldstein
2017-01-25 15:15           ` Jan Kara
     [not found]             ` <87efzrrwec.fsf@drapion.f-secure.com>
     [not found]               ` <20170127134620.GA16757@quack2.suse.cz>
2017-01-27 14:21                 ` Marko Rauhamaa
2017-01-25 15:20           ` Marko Rauhamaa
2017-01-26 17:47             ` Marko Rauhamaa
2017-01-26 18:05               ` Marko Rauhamaa

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.