Re: stuck in inotify_release

[Olivier Chapelliere <olivier.chapelliere@alcmeon.com>]

[-- Attachment #1: Type: text/plain, Size: 2327 bytes --]

Hi all,

If it can help troubleshoot the issue, I found another host with the
same symptoms.
I ran sysrq commands and attached the kernel log file.

Thanks
Olivier



On Mon, May 6, 2019 at 8:54 PM Olivier Chapelliere
<olivier.chapelliere@alcmeon.com> wrote:
>
> Jan and all,
>
> It finally took a month to happen again : python processes watching a
> directory are stuck in inotify_release.
> I ran the sysrq commands as you requested and attached the result.
>
> Thanks for your help
>
> On Thu, Mar 28, 2019 at 10:52 AM Jan Kara <jack@suse.cz> wrote:
> >
> > Hello,
> >
> > On Thu 28-03-19 09:26:45, Olivier Chapelliere wrote:
> > > According to what I read on internet you seem to be the right person to get
> > > in touch with when one has problems with inotify.
> >
> > Yes, there's also linux-fsdevel@vger.kernel.org mailing list which we use
> > (added to CC).
> >
> > > We are monitoring several directories in python processes through inotify.
> > > But after few days all processes are stuck in a call to inotify_release.
> > > Once I detected the problem, I dumped info to dmesg with sysrq-trigger
> > > (dmesg content attached):
> > > echo w > /proc/sysrq-trigger
> >
> > Looking through the stack traces, all of them wait in fput() ->
> > inotify_release() -> ... -> fsnotify_wait_marks_destroyed() ->
> > flush_delayed_work(&reaper_work). So they wait for worker process to
> > destroy all marks for the group. However that worker (kworker/u8:4) is
> > stuck in:
> >
> > fsnotify_mark_destroy_workfn() -> synchronize_srcu(&fsnotify_mark_srcu)
> >
> > So the question is who is holding fsnotify_mark_srcu so that SRCU cannot
> > declare new grace period. I don't see any such process among the processes
> > you've shown in the dump (but it should be there) so it's a bit of a
> > mystery.
> >
> > > Our production env is ubuntu 18.04 kernel 4.15 fs ext4
> > > This problem appears on a weekly basis so I will be able to run additional
> > > commands to track down the issue if needed.
> >
> > So when this happens again, try grabbing output of sysrq-l and sysrq-t if
> > we can find the task holding fsnotify_mark_srcu.
> >
> >                                                                 Honza
> > --
> > Jan Kara <jack@suse.com>
> > SUSE Labs, CR
>
>
>
> --
> Olivier Chapelliere



-- 
Olivier Chapelliere

[-- Attachment #2: kern.log.tar.gz --]
[-- Type: application/gzip, Size: 57484 bytes --]