From: Amir Goldstein <amir73il@gmail.com>
To: Jan Kara <jack@suse.cz>
Cc: Stephen Brennan <stephen.s.brennan@oracle.com>,
linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org,
Al Viro <viro@zeniv.linux.org.uk>
Subject: Re: [PATCH v4 0/5] fsnotify: fix softlockups iterating over d_subdirs
Date: Tue, 22 Nov 2022 16:03:22 +0200 [thread overview]
Message-ID: <CAOQ4uxhic7+q0Q9SjO_qkp71gCphZy8cAmzHbacY63Wxr35akg@mail.gmail.com> (raw)
In-Reply-To: <20221122115042.qssn25wbtxxhaeys@quack3>
On Tue, Nov 22, 2022 at 1:50 PM Jan Kara <jack@suse.cz> wrote:
>
> Hi Stephen!
>
> On Fri 11-11-22 14:06:09, Stephen Brennan wrote:
> > Here's my v4 patch series that aims to eliminate soft lockups when updating
> > dentry flags in fsnotify. I've incorporated Jan's suggestion of simply
> > allowing the flag to be lazily cleared in the fsnotify_parent() function,
> > via Amir's patch. This allowed me to drop patch #2 from my previous series
> > (fsnotify: Protect i_fsnotify_mask and child flags with inode rwsem). I
> > replaced it with "fsnotify: require inode lock held during child flag
> > update", patch #5 in this series. I also added "dnotify: move
> > fsnotify_recalc_mask() outside spinlock" to address the sleep-during-atomic
> > issues with dnotify.
>
> Yes, the series is now much simpler. Thanks!
>
> > Jan expressed concerns about lock ordering of the inode rwsem with the
> > fsnotify group mutex. I built this with lockdep enabled (see below for the
> > lock debugging .config section -- I'm not too familiar with lockdep so I
> > wanted a sanity check). I ran all the fanotify, inotify, and dnotify tests
> > I could find in LTP, with no lockdep splats to be found. I don't know that
> > this can completely satisfy the concerns about lock ordering: I'm reading
> > through the code to better understand the concern about "the removal of
> > oneshot mark during modify event generation". But I'm encouraged by the
> > LTP+lockdep results.
>
> So I had a look and I think your patches could cause deadlock at least for
> nfsd. The problem is with things like inotify IN_ONESHOT marks. They get
> autodeleted as soon as they trigger. Thus e.g. fsnotify_mkdir() can trigger
> IN_ONESHOT mark and goes on removing it by calling fsnotify_destroy_mark()
> from inotify_handle_inode_event(). And nfsd calls e.g. fsnotify_mkdir()
> while holding dir->i_rwsem held. So we have lock ordering like:
>
> nfsd_mkdir()
> inode_lock(dir);
> ...
> __nfsd_mkdir(dir, ...)
> fsnotify_mkdir(dir, dentry);
> ...
> inotify_handle_inode_event()
> ...
> fsnotify_destroy_mark()
> fsnotify_group_lock(group)
>
> So we have dir->i_rwsem > group->mark_mutex. But we also have callchains
> like:
>
> inotify_add_watch()
> inotify_update_watch()
> fsnotify_group_lock(group)
> inotify_update_existing_watch()
> ...
> fsnotify_recalc_mask()
> inode_lock(dir); -> added by your series
>
> which creates ordering group->mark_mutex > dir->i_rwsem.
>
> It is even worse with dnotify which (even with your patches) ends up
> calling fsnotify_recalc_mask() from dnotify_handle_event() so we have a
> possibility of direct A->A deadlock. But I'd leave dnotify aside, I think
> that can be massaged to not need to call fsnotify_recalc_mask()
> (__fsnotify_recalc_mask() would be enough there).
>
> Still I'm not 100% sure about a proper way out of this. The simplicity of
> alias->d_subdirs iteration with i_rwsem held is compeling.
Agreed.
> We could mandate
> that fsnotify hooks cannot be called with inode->i_rwsem held (and fixup
> nfsd) but IMO that is pushing the complexity from the fsnotify core into
> its users which is undesirable.
I think inode in this context is the parent inode, so all fsnotify hooks
in namei.c are holding inode->i_rwsem by design.
> Maybe we could grab inode->i_rwsem in those
> places adding / removing notification marks before we grab
> group->mark_mutex, just verify (with lockdep) that fsnotify_recalc_mask()
> has the inode->i_rwsem held and be done with it? That pushes a bit of
> complexity into the fsnotify backends but it is not too bad.
> fsnotify_recalc_mask() gets only called by dnotify, inotify, and fanotify.
> Amir?
>
Absolutely agree - I think it makes sense and will simplify things a lot.
Obviously if we need to assert inode_is_locked() in fsnotify_recalc_mask()
only for (conn->type == FSNOTIFY_OBJ_TYPE_INODE).
Thanks,
Amir.
prev parent reply other threads:[~2022-11-22 14:05 UTC|newest]
Thread overview: 67+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-10-13 22:27 [RFC] fsnotify: allow sleepable child dentry flag update Stephen Brennan
2022-10-13 23:51 ` Al Viro
2022-11-01 21:47 ` Stephen Brennan
2022-10-14 8:01 ` Amir Goldstein
2022-10-17 7:59 ` Stephen Brennan
2022-10-17 11:44 ` Amir Goldstein
2022-10-17 16:59 ` Stephen Brennan
2022-10-17 17:42 ` Amir Goldstein
2022-10-17 9:09 ` Jan Kara
2022-10-18 4:12 ` [PATCH 0/2] fsnotify: fix softlockups iterating over d_subdirs Stephen Brennan
2022-10-18 4:12 ` [PATCH 1/2] fsnotify: Protect i_fsnotify_mask and child flags with inode rwsem Stephen Brennan
2022-10-18 7:39 ` Amir Goldstein
2022-10-21 0:33 ` Stephen Brennan
2022-10-21 7:22 ` Amir Goldstein
2022-10-18 4:12 ` [PATCH 2/2] fsnotify: allow sleepable child flag update Stephen Brennan
2022-10-18 5:36 ` Amir Goldstein
2022-10-27 7:50 ` kernel test robot
2022-10-27 8:44 ` Yujie Liu
2022-10-27 22:12 ` Stephen Brennan
2022-10-18 8:07 ` [PATCH 0/2] fsnotify: fix softlockups iterating over d_subdirs Amir Goldstein
2022-10-18 23:52 ` Stephen Brennan
2022-10-19 5:33 ` Amir Goldstein
2022-10-27 22:06 ` Stephen Brennan
2022-10-28 8:58 ` Amir Goldstein
2022-10-21 1:03 ` [PATCH v2 0/3] " Stephen Brennan
2022-10-21 1:03 ` [PATCH v2 1/3] fsnotify: Use d_find_any_alias to get dentry associated with inode Stephen Brennan
2022-10-21 9:25 ` Amir Goldstein
2022-10-21 1:03 ` [PATCH v2 2/3] fsnotify: Protect i_fsnotify_mask and child flags with inode rwsem Stephen Brennan
2022-10-21 4:01 ` kernel test robot
2022-10-21 8:22 ` Amir Goldstein
2022-10-21 9:18 ` Amir Goldstein
2022-10-25 18:02 ` Stephen Brennan
2022-10-26 5:41 ` Amir Goldstein
2022-10-21 9:17 ` Christian Brauner
2022-10-21 9:21 ` Amir Goldstein
2022-10-21 1:03 ` [PATCH v2 3/3] fsnotify: allow sleepable child flag update Stephen Brennan
2022-10-28 0:10 ` [PATCH v3 0/3] fsnotify: fix softlockups iterating over d_subdirs Stephen Brennan
2022-10-28 0:10 ` [PATCH v3 1/3] fsnotify: Use d_find_any_alias to get dentry associated with inode Stephen Brennan
2022-11-10 1:12 ` Stephen Brennan
2022-10-28 0:10 ` [PATCH v3 2/3] fsnotify: Protect i_fsnotify_mask and child flags with inode rwsem Stephen Brennan
2022-10-28 9:11 ` Amir Goldstein
2022-11-10 0:03 ` kernel test robot
2022-11-10 1:06 ` Stephen Brennan
2022-10-28 0:10 ` [PATCH v3 3/3] fsnotify: allow sleepable child flag update Stephen Brennan
2022-10-28 9:32 ` Amir Goldstein
2022-11-01 21:25 ` Stephen Brennan
2022-11-01 17:51 ` [PATCH v3 0/3] fsnotify: fix softlockups iterating over d_subdirs Jan Kara
2022-11-01 20:48 ` Stephen Brennan
2022-11-02 8:55 ` Amir Goldstein
2022-11-10 20:04 ` Stephen Brennan
2022-11-02 17:52 ` Jan Kara
2022-11-04 23:33 ` Stephen Brennan
2022-11-07 11:56 ` Jan Kara
2022-11-11 22:06 ` [PATCH v4 0/5] " Stephen Brennan
2022-11-11 22:06 ` [PATCH v4 1/5] fsnotify: clear PARENT_WATCHED flags lazily Stephen Brennan
2022-11-11 22:06 ` [PATCH v4 2/5] fsnotify: Use d_find_any_alias to get dentry associated with inode Stephen Brennan
2022-11-12 8:53 ` Amir Goldstein
2022-11-11 22:06 ` [PATCH v4 3/5] dnotify: move fsnotify_recalc_mask() outside spinlock Stephen Brennan
2022-11-12 9:06 ` Amir Goldstein
2022-11-11 22:06 ` [PATCH v4 4/5] fsnotify: allow sleepable child flag update Stephen Brennan
2022-11-12 10:00 ` Amir Goldstein
2022-11-15 7:10 ` kernel test robot
2022-11-11 22:06 ` [PATCH v4 5/5] fsnotify: require inode lock held during " Stephen Brennan
2022-11-12 9:42 ` Amir Goldstein
2022-11-11 22:08 ` [PATCH v4 0/5] fsnotify: fix softlockups iterating over d_subdirs Stephen Brennan
2022-11-22 11:50 ` Jan Kara
2022-11-22 14:03 ` Amir Goldstein [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAOQ4uxhic7+q0Q9SjO_qkp71gCphZy8cAmzHbacY63Wxr35akg@mail.gmail.com \
--to=amir73il@gmail.com \
--cc=jack@suse.cz \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=stephen.s.brennan@oracle.com \
--cc=viro@zeniv.linux.org.uk \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).