From: Stephen Brennan <stephen.s.brennan@oracle.com>
To: linux-kernel@vger.kernel.org
Cc: Stephen Brennan <stephen.s.brennan@oracle.com>,
Gautham Ananthakrishna <gautham.ananthakrishna@oracle.com>,
Konstantin Khlebnikov <khlebnikov@yandex-team.ru>,
linux-fsdevel@vger.kernel.org
Subject: [PATCH 0/4] Fix softlockup when adding inotify watch
Date: Mon, 13 Dec 2021 16:53:33 -0800 [thread overview]
Message-ID: <20211214005337.161885-1-stephen.s.brennan@oracle.com> (raw)
When a system with large amounts of memory has several millions of
negative dentries in a single directory, a softlockup can occur while
adding an inotify watch:
watchdog: BUG: soft lockup - CPU#20 stuck for 9s! [inotifywait:9528]
CPU: 20 PID: 9528 Comm: inotifywait Kdump: loaded Not tainted 5.16.0-rc4.20211208.el8uek.rc1.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.4.1 12/03/2020
RIP: 0010:__fsnotify_update_child_dentry_flags+0xad/0x120
Call Trace:
<TASK>
fsnotify_add_mark_locked+0x113/0x160
inotify_new_watch+0x130/0x190
inotify_update_watch+0x11a/0x140
__x64_sys_inotify_add_watch+0xef/0x140
do_syscall_64+0x3b/0x90
entry_SYSCALL_64_after_hwframe+0x44/0xae
This patch series is a modified version of the following:
https://lore.kernel.org/linux-fsdevel/1611235185-1685-1-git-send-email-gautham.ananthakrishna@oracle.com/
The strategy employed by this series is to move negative dentries to the
end of the d_subdirs list, and mark them with a flag as "tail negative".
Then, readers of the d_subdirs list, which are only interested in
positive dentries, can stop reading once they reach the first tail
negative dentry. By applying this patch, I'm able to avoid the above
softlockup caused by 200 million negative dentries on my test system.
Inotify watches are set up nearly instantly.
Previously, Al expressed concern for:
1. Possible memory corruption due to use of lock_parent() in
sweep_negative(), see patch 01 for fix.
2. The previous patch didn't catch all ways a negative dentry could
become positive (d_add, d_instantiate_new), see patch 01.
3. The previous series contained a new negative dentry limit, which
capped the negative dentry count at around 3 per hash bucket. I've
dropped this patch from the series.
Patches 2-4 are unmodified from the previous posting.
Konstantin Khlebnikov (3):
fsnotify: stop walking child dentries if remaining tail is negative
dcache: add action D_WALK_SKIP_SIBLINGS to d_walk()
dcache: stop walking siblings if remaining dentries all negative
Stephen Brennan (1):
dcache: sweep cached negative dentries to the end of list of siblings
fs/dcache.c | 101 +++++++++++++++++++++++++++++++++++++++--
fs/libfs.c | 3 ++
fs/notify/fsnotify.c | 6 ++-
include/linux/dcache.h | 6 +++
4 files changed, 110 insertions(+), 6 deletions(-)
--
2.30.2
next reply other threads:[~2021-12-14 0:53 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-12-14 0:53 Stephen Brennan [this message]
2021-12-14 0:53 ` [PATCH 1/4] dcache: sweep cached negative dentries to the end of list of siblings Stephen Brennan
2021-12-18 8:17 ` [dcache] 5384bd38a8: WARNING:at_fs/nfsd/nfsctl.c:#nfsdfs_remove_files[nfsd] kernel test robot
2021-12-14 0:53 ` [PATCH 2/4] fsnotify: stop walking child dentries if remaining tail is negative Stephen Brennan
2021-12-14 0:53 ` [PATCH 3/4] dcache: add action D_WALK_SKIP_SIBLINGS to d_walk() Stephen Brennan
2021-12-14 0:53 ` [PATCH 4/4] dcache: stop walking siblings if remaining dentries all negative Stephen Brennan
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20211214005337.161885-1-stephen.s.brennan@oracle.com \
--to=stephen.s.brennan@oracle.com \
--cc=gautham.ananthakrishna@oracle.com \
--cc=khlebnikov@yandex-team.ru \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).