[PATCH RFC 0/6] fix the negative dentres bloating system memory usage

[PATCH RFC 0/6] fix the negative dentres bloating system memory usage

[Gautham Ananthakrishna]

For most filesystems result of every negative lookup is cached, content of
directories is usually cached too. Production of negative dentries isn't
limited with disk speed. It's really easy to generate millions of them if
system has enough memory.

Getting this memory back ins't that easy because slab frees pages only when
all related objects are gone. While dcache shrinker works in LRU order.

Typical scenario is an idle system where some process periodically creates
temporary files and removes them. After some time, memory will be filled
with negative dentries for these random file names.

Simple lookup of random names also generates negative dentries very fast.
Constant flow of such negative denries drains all other inactive caches.
Too many negative dentries in the system can cause memory fragmentation
and memory compaction.

Negative dentries are linked into siblings list along with normal positive
dentries. Some operations walks dcache tree but looks only for positive
dentries: most important is fsnotify/inotify. Hordes of negative dentries
slow down these operations significantly.

Time of dentry lookup is usually unaffected because hash table grows along
with size of memory. Unless somebody especially crafts hash collisions.

This patch set solves all of these problems:

Move negative denries to the end of sliblings list, thus walkers could
skip them at first sight (patches 1-4).

Keep in dcache at most three unreferenced negative denties in row in each
hash bucket (patches 5-6).

We tested this patch set recently and found it limiting negative dentry to a
small part of total memory. The following is the test result we ran on two
types of servers, one is 256G memory with 24 CPUS and another is 3T memory
with 384 CPUS. The test case is using a lot of processes to generate negative
dentry in parallel, the following is the test result after 72 hours, the
negative dentry number is stable around that number even after running longer
for much longer time. Without the patch set, in less than half an hour 197G was
taken by negative dentry on 256G system, in 1 day 2.4T was taken on 3T system.

system memory   neg-dentry-number   neg-dentry-mem-usage
256G            55259084            10.6G
3T              202306756           38.8G

For perf test, we ran the following, and no regression found.

1. create 1M negative dentry and then touch them to convert them to positive
   dentry

2. create 10K/100K/1M files

3. remove 10K/100K/1M files

4. kernel compile

To verify the fsnotify fix, we used inotifywait to watch file create/open in
some directory where there is a lot of negative dentry, without the patch set,
the system would run into soft lockup, with it, no soft lockup was found.

We also tried to defeat the limitation by making different processes generate
negative dentry with the same name, that will make one negative dentry being
accessed couple times around same time, DCACHE_REFERENCED will be set on it
and it can't be trimmed easily.

There were a lot of customer cases on this issue. It makes no sense to leave
so many negative dentry, it just causes memory fragmentation and compaction
and does not help a lot.

Konstantin Khlebnikov (6):
  dcache: sweep cached negative dentries to the end of list of siblings
  fsnotify: stop walking child dentries if remaining tail is negative
  dcache: add action D_WALK_SKIP_SIBLINGS to d_walk()
  dcache: stop walking siblings if remaining dentries all negative
  dcache: push releasing dentry lock into sweep_negative
  dcache: prevent flooding with negative dentries

 fs/dcache.c            | 135 +++++++++++++++++++++++++++++++++++++++++++++++--
 fs/libfs.c             |   3 ++
 fs/notify/fsnotify.c   |   6 ++-
 include/linux/dcache.h |   6 +++
 4 files changed, 145 insertions(+), 5 deletions(-)

-- 
1.8.3.1

[PATCH RFC 1/6] dcache: sweep cached negative dentries to the end of list of siblings

[Gautham Ananthakrishna]

From: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>

For disk filesystems result of every negative lookup is cached, content of
directories is usually cached too. Production of negative dentries isn't
limited with disk speed. It's really easy to generate millions of them if
system has enough memory. Negative dentries are linked into siblings list
along with normal positive dentries. Some operations walks dcache tree but
looks only for positive dentries: most important is fsnotify/inotify.

This patch moves negative dentries to the end of list at final dput() and
marks with flag which tells that all following dentries are negative too.
Reverse operation is required before instantiating negative dentry.

Signed-off-by: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>
Signed-off-by: Gautham Ananthakrishna <gautham.ananthakrishna@oracle.com>
---
 fs/dcache.c            | 59 +++++++++++++++++++++++++++++++++++++++++++++++---
 include/linux/dcache.h |  6 +++++
 2 files changed, 62 insertions(+), 3 deletions(-)

diff --git a/fs/dcache.c b/fs/dcache.c
index ea04858..a506169 100644
--- a/fs/dcache.c
+++ b/fs/dcache.c
@@ -632,6 +632,48 @@ static inline struct dentry *lock_parent(struct dentry *dentry)
 	return __lock_parent(dentry);
 }
 
+/*
+ * Move cached negative dentry to the tail of parent->d_subdirs.
+ * This lets walkers skip them all together at first sight.
+ * Must be called at dput of negative dentry.
+ */
+static void sweep_negative(struct dentry *dentry)
+{
+	struct dentry *parent;
+
+	if (!d_is_tail_negative(dentry)) {
+		parent = lock_parent(dentry);
+		if (!parent)
+			return;
+
+		if (!d_count(dentry) && d_is_negative(dentry) &&
+		    !d_is_tail_negative(dentry)) {
+			dentry->d_flags |= DCACHE_TAIL_NEGATIVE;
+			list_move_tail(&dentry->d_child, &parent->d_subdirs);
+		}
+
+		spin_unlock(&parent->d_lock);
+	}
+}
+
+/*
+ * Undo sweep_negative() and move to the head of parent->d_subdirs.
+ * Must be called before converting negative dentry into positive.
+ */
+static void recycle_negative(struct dentry *dentry)
+{
+	struct dentry *parent;
+
+	spin_lock(&dentry->d_lock);
+	parent = lock_parent(dentry);
+	dentry->d_flags &= ~DCACHE_TAIL_NEGATIVE;
+	if (parent) {
+		list_move(&dentry->d_child, &parent->d_subdirs);
+		spin_unlock(&parent->d_lock);
+	}
+	spin_unlock(&dentry->d_lock);
+}
+
 static inline bool retain_dentry(struct dentry *dentry)
 {
 	WARN_ON(d_in_lookup(dentry));
@@ -737,7 +779,7 @@ static struct dentry *dentry_kill(struct dentry *dentry)
 static inline bool fast_dput(struct dentry *dentry)
 {
 	int ret;
-	unsigned int d_flags;
+	unsigned int d_flags, required;
 
 	/*
 	 * If we have a d_op->d_delete() operation, we sould not
@@ -785,6 +827,8 @@ static inline bool fast_dput(struct dentry *dentry)
 	 * a 'delete' op, and it's referenced and already on
 	 * the LRU list.
 	 *
+	 * Cached negative dentry must be swept to the tail.
+	 *
 	 * NOTE! Since we aren't locked, these values are
 	 * not "stable". However, it is sufficient that at
 	 * some point after we dropped the reference the
@@ -796,10 +840,15 @@ static inline bool fast_dput(struct dentry *dentry)
 	 */
 	smp_rmb();
 	d_flags = READ_ONCE(dentry->d_flags);
-	d_flags &= DCACHE_REFERENCED | DCACHE_LRU_LIST | DCACHE_DISCONNECTED;
+
+	required = DCACHE_REFERENCED | DCACHE_LRU_LIST |
+		(d_flags_negative(d_flags) ? DCACHE_TAIL_NEGATIVE : 0);
+
+	d_flags &= DCACHE_REFERENCED | DCACHE_LRU_LIST |
+		DCACHE_DISCONNECTED | DCACHE_TAIL_NEGATIVE;
 
 	/* Nothing to do? Dropping the reference was all we needed? */
-	if (d_flags == (DCACHE_REFERENCED | DCACHE_LRU_LIST) && !d_unhashed(dentry))
+	if (d_flags == required && !d_unhashed(dentry))
 		return true;
 
 	/*
@@ -871,6 +920,8 @@ void dput(struct dentry *dentry)
 		rcu_read_unlock();
 
 		if (likely(retain_dentry(dentry))) {
+			if (d_is_negative(dentry))
+				sweep_negative(dentry);
 			spin_unlock(&dentry->d_lock);
 			return;
 		}
@@ -1970,6 +2021,8 @@ void d_instantiate(struct dentry *entry, struct inode * inode)
 {
 	BUG_ON(!hlist_unhashed(&entry->d_u.d_alias));
 	if (inode) {
+		if (d_is_tail_negative(entry))
+			recycle_negative(entry);
 		security_d_instantiate(entry, inode);
 		spin_lock(&inode->i_lock);
 		__d_instantiate(entry, inode);
diff --git a/include/linux/dcache.h b/include/linux/dcache.h
index 6f95c33..5f4ce3a 100644
--- a/include/linux/dcache.h
+++ b/include/linux/dcache.h
@@ -219,6 +219,7 @@ struct dentry_operations {
 #define DCACHE_PAR_LOOKUP		0x10000000 /* being looked up (with parent locked shared) */
 #define DCACHE_DENTRY_CURSOR		0x20000000
 #define DCACHE_NORCU			0x40000000 /* No RCU delay for freeing */
+#define DCACHE_TAIL_NEGATIVE		0x80000000 /* All following siblings are negative */
 
 extern seqlock_t rename_lock;
 
@@ -495,6 +496,11 @@ static inline int simple_positive(const struct dentry *dentry)
 	return d_really_is_positive(dentry) && !d_unhashed(dentry);
 }
 
+static inline bool d_is_tail_negative(const struct dentry *dentry)
+{
+	return unlikely(dentry->d_flags & DCACHE_TAIL_NEGATIVE);
+}
+
 extern void d_set_fallthru(struct dentry *dentry);
 
 static inline bool d_is_fallthru(const struct dentry *dentry)
-- 
1.8.3.1

Re: [PATCH RFC 1/6] dcache: sweep cached negative dentries to the end of list of siblings

[Al Viro]

On Thu, Jan 21, 2021 at 06:49:40PM +0530, Gautham Ananthakrishna wrote:
> From: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>
> 
> For disk filesystems result of every negative lookup is cached, content of
> directories is usually cached too. Production of negative dentries isn't
> limited with disk speed. It's really easy to generate millions of them if
> system has enough memory. Negative dentries are linked into siblings list
> along with normal positive dentries. Some operations walks dcache tree but
> looks only for positive dentries: most important is fsnotify/inotify.
> 
> This patch moves negative dentries to the end of list at final dput() and
> marks with flag which tells that all following dentries are negative too.
> Reverse operation is required before instantiating negative dentry.

> +static void sweep_negative(struct dentry *dentry)
> +{
> +	struct dentry *parent;
> +
> +	if (!d_is_tail_negative(dentry)) {
> +		parent = lock_parent(dentry);
> +		if (!parent)
> +			return;
> +
> +		if (!d_count(dentry) && d_is_negative(dentry) &&
> +		    !d_is_tail_negative(dentry)) {
> +			dentry->d_flags |= DCACHE_TAIL_NEGATIVE;
> +			list_move_tail(&dentry->d_child, &parent->d_subdirs);
> +		}
> +
> +		spin_unlock(&parent->d_lock);
> +	}
> +}

Ugh...  So when dput() drives the refcount down to 0 you hit lock_parent()
and only then bother to check if the sucker had been negative in the first
place?

> @@ -1970,6 +2021,8 @@ void d_instantiate(struct dentry *entry, struct inode * inode)
>  {
>  	BUG_ON(!hlist_unhashed(&entry->d_u.d_alias));
>  	if (inode) {
> +		if (d_is_tail_negative(entry))
> +			recycle_negative(entry);
>  		security_d_instantiate(entry, inode);
>  		spin_lock(&inode->i_lock);
>  		__d_instantiate(entry, inode);

Wait a bloody minute.  What about d_instantiate_new() right next to it?

Re: [PATCH RFC 1/6] dcache: sweep cached negative dentries to the end of list of siblings

[Al Viro]

On Wed, Apr 14, 2021 at 03:00:48AM +0000, Al Viro wrote:

> Ugh...  So when dput() drives the refcount down to 0 you hit lock_parent()
> and only then bother to check if the sucker had been negative in the first
                                              ^^^^^^^^^^^^^^^^^
                                              had zero refcount, of course.
> place?

> > @@ -1970,6 +2021,8 @@ void d_instantiate(struct dentry *entry, struct inode * inode)
> >  {
> >  	BUG_ON(!hlist_unhashed(&entry->d_u.d_alias));
> >  	if (inode) {
> > +		if (d_is_tail_negative(entry))
> > +			recycle_negative(entry);
> >  		security_d_instantiate(entry, inode);
> >  		spin_lock(&inode->i_lock);
> >  		__d_instantiate(entry, inode);
> 
> Wait a bloody minute.  What about d_instantiate_new() right next to it?

Another fun question: where's the proof that __d_add(dentry, non_NULL_inode)
won't happen to dentry marked tail-negative?  From a quick grep I see at
least one such place - on success cifs_do_create() does
        d_drop(direntry);
	d_add(direntry, newinode);
and it would bloody well evade what you are doing in d_instantiate().
Same seems to be true for nfs_link()...

Re: [PATCH RFC 1/6] dcache: sweep cached negative dentries to the end of list of siblings

[Al Viro]

On Thu, Jan 21, 2021 at 06:49:40PM +0530, Gautham Ananthakrishna wrote:

> +static void sweep_negative(struct dentry *dentry)
> +{
> +	struct dentry *parent;
> +
> +	if (!d_is_tail_negative(dentry)) {
> +		parent = lock_parent(dentry);
> +		if (!parent)
> +			return;

Wait a minute.  It's not a good environment for calling lock_parent().
Who said that dentry won't get freed right under it?

Right now callers of __lock_parent() either hold a reference to dentry
*or* are called for a positive dentry, with inode->i_lock held.
You are introducing something very different - 

>  		if (likely(retain_dentry(dentry))) {
> +			if (d_is_negative(dentry))
> +				sweep_negative(dentry);
>  			spin_unlock(&dentry->d_lock);

Here we can be called for a negative dentry with refcount already *NOT*
held by us.  Look:

static inline struct dentry *lock_parent(struct dentry *dentry)
{
        struct dentry *parent = dentry->d_parent;
	if (IS_ROOT(dentry))
		return NULL;
isn't a root

	if (likely(spin_trylock(&parent->d_lock)))
		return parent;

no such luck - someone's already holding parent's ->d_lock

	return __lock_parent(dentry);
and here we have
static struct dentry *__lock_parent(struct dentry *dentry)
{
	struct dentry *parent;
	rcu_read_lock();  

OK, anything we see in its ->d_parent is guaranteed to stay
allocated until we get to matching rcu_read_unlock()

	spin_unlock(&dentry->d_lock);
dropped the spinlock, now it's fair game for d_move(), d_drop(), etc.

again:
	parent = READ_ONCE(dentry->d_parent);
dentry couldn't have been reused, so it's the last value stored there.
Points to still allocated struct dentry instance, so we can...

	spin_lock(&parent->d_lock);
grab its ->d_lock.

	/*
	 * We can't blindly lock dentry until we are sure
	 * that we won't violate the locking order.
	 * Any changes of dentry->d_parent must have
	 * been done with parent->d_lock held, so
	 * spin_lock() above is enough of a barrier
	 * for checking if it's still our child.
	 */
	if (unlikely(parent != dentry->d_parent)) {
		spin_unlock(&parent->d_lock);
		goto again;
	}
Nevermind, it's still equal to our ->d_parent.  So we have
the last valid parent's ->d_lock held

	rcu_read_unlock();
What's to hold dentry allocated now?  IF we held its refcount - no
problem, it can't go away.  If we held its ->d_inode->i_lock - ditto
(it wouldn't get to __dentry_kill() until we drop that, since all
callers do acquire that lock and it couldn't get scheduled for
freeing until it gets through most of __dentry_kill()).

IOW, we are free to grab dentry->d_lock again.
	if (parent != dentry)
		spin_lock_nested(&dentry->d_lock, DENTRY_D_LOCK_NESTED);
	else
		parent = NULL;
	return parent;
}

With your patch, though, you've got a call site where neither condition
is guaranteed.  Current kernel is fine - we are holding ->d_lock there,
and we don't touch dentry after it gets dropped.  Again, it can't get
scheduled for freeing until after we drop ->d_lock, so we are safe.
With that change, however, you've got a hard-to-hit memory corruptor
there...

Re: [PATCH RFC 1/6] dcache: sweep cached negative dentries to the end of list of siblings

[Al Viro]

On Wed, Apr 14, 2021 at 03:41:10AM +0000, Al Viro wrote:

> > +	if (!d_is_tail_negative(dentry)) {
> > +		parent = lock_parent(dentry);
> > +		if (!parent)
> > +			return;
> 
> Wait a minute.  It's not a good environment for calling lock_parent().
> Who said that dentry won't get freed right under it?

[snip]

FWIW, in that state (dentry->d_lock held) we have
	* stable ->d_flags
	* stable ->d_count
	* stable ->d_inode
IOW, we can bloody well check ->d_count *before* bothering with lock_parent().
It does not get rid of the problems with lifetimes, though.  We could
do something along the lines of

	rcu_read_lock()
	if retain_dentry()
		parent = NULL
		if that dentry might need to be moved in list
			parent = lock_parent()
			// if reached __dentry_kill(), d_count() will be -128,
			// so the check below will exclude those
			if that dentry does need to be moved
				move it to the end of list
		unlock dentry and parent (if any)
		rcu_read_unlock()
		return
here, but your other uses of lock_parent() also need attention.  And
recursive call of dput() in trim_negative() (#6/6) is really asking
for trouble.

[PATCH RFC 2/6] fsnotify: stop walking child dentries if remaining tail is negative

[Gautham Ananthakrishna]

From: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>

When notification starts/stops listening events from inode's children it
have to update dentry->d_flags of all positive child dentries. Scanning
may took a long time if directory has a lot of negative child dentries.

This is main beneficiary of sweeping cached negative dentries to the end.

Before patch:

nr_dentry = 24172597    24.2M
nr_buckets = 8388608    2.9 avg
nr_unused = 24158110    99.9%
nr_negative = 24142810  99.9%

inotify time: 0.507182 seconds

After patch:

nr_dentry = 24562747    24.6M
nr_buckets = 8388608    2.9 avg
nr_unused = 24548714    99.9%
nr_negative = 24543867  99.9%

inotify time: 0.000010 seconds

Negative dentries no longer slow down inotify op at parent directory.

Signed-off-by: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>
Signed-off-by: Gautham Ananthakrishna <gautham.ananthakrishna@oracle.com>
---
 fs/notify/fsnotify.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/fs/notify/fsnotify.c b/fs/notify/fsnotify.c
index 8d3ad5e..4ccb59d 100644
--- a/fs/notify/fsnotify.c
+++ b/fs/notify/fsnotify.c
@@ -127,8 +127,12 @@ void __fsnotify_update_child_dentry_flags(struct inode *inode)
 		 * original inode) */
 		spin_lock(&alias->d_lock);
 		list_for_each_entry(child, &alias->d_subdirs, d_child) {
-			if (!child->d_inode)
+			if (!child->d_inode) {
+				/* all remaining children are negative */
+				if (d_is_tail_negative(child))
+					break;
 				continue;
+			}
 
 			spin_lock_nested(&child->d_lock, DENTRY_D_LOCK_NESTED);
 			if (watched)
-- 
1.8.3.1

[PATCH RFC 3/6] dcache: add action D_WALK_SKIP_SIBLINGS to d_walk()

[Gautham Ananthakrishna]

From: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>

This lets skip remaining siblings at seeing d_is_tail_negative().

Signed-off-by: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>
Signed-off-by: Gautham Ananthakrishna <gautham.ananthakrishna@oracle.com>
---
 fs/dcache.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/fs/dcache.c b/fs/dcache.c
index a506169..894e6da 100644
--- a/fs/dcache.c
+++ b/fs/dcache.c
@@ -1320,12 +1320,14 @@ void shrink_dcache_sb(struct super_block *sb)
  * @D_WALK_QUIT:	quit walk
  * @D_WALK_NORETRY:	quit when retry is needed
  * @D_WALK_SKIP:	skip this dentry and its children
+ * @D_WALK_SKIP_SIBLINGS: skip siblings and their children
  */
 enum d_walk_ret {
 	D_WALK_CONTINUE,
 	D_WALK_QUIT,
 	D_WALK_NORETRY,
 	D_WALK_SKIP,
+	D_WALK_SKIP_SIBLINGS,
 };
 
 /**
@@ -1356,6 +1358,7 @@ static void d_walk(struct dentry *parent, void *data,
 		break;
 	case D_WALK_QUIT:
 	case D_WALK_SKIP:
+	case D_WALK_SKIP_SIBLINGS:
 		goto out_unlock;
 	case D_WALK_NORETRY:
 		retry = false;
@@ -1387,6 +1390,9 @@ static void d_walk(struct dentry *parent, void *data,
 		case D_WALK_SKIP:
 			spin_unlock(&dentry->d_lock);
 			continue;
+		case D_WALK_SKIP_SIBLINGS:
+			spin_unlock(&dentry->d_lock);
+			goto skip_siblings;
 		}
 
 		if (!list_empty(&dentry->d_subdirs)) {
@@ -1398,6 +1404,7 @@ static void d_walk(struct dentry *parent, void *data,
 		}
 		spin_unlock(&dentry->d_lock);
 	}
+skip_siblings:
 	/*
 	 * All done at this level ... ascend and resume the search.
 	 */
-- 
1.8.3.1

[PATCH RFC 4/6] dcache: stop walking siblings if remaining dentries all negative

[Gautham Ananthakrishna]

From: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>

Most walkers are interested only in positive dentries.

Changes in simple_* libfs helpers are mostly cosmetic: it shouldn't cache
negative dentries unless uses d_delete other than always_delete_dentry().

Signed-off-by: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>
Signed-off-by: Gautham Ananthakrishna <gautham.ananthakrishna@oracle.com>
---
 fs/dcache.c | 9 +++++++++
 fs/libfs.c  | 3 +++
 2 files changed, 12 insertions(+)

diff --git a/fs/dcache.c b/fs/dcache.c
index 894e6da..492a42f 100644
--- a/fs/dcache.c
+++ b/fs/dcache.c
@@ -1459,6 +1459,8 @@ static enum d_walk_ret path_check_mount(void *data, struct dentry *dentry)
 	struct check_mount *info = data;
 	struct path path = { .mnt = info->mnt, .dentry = dentry };
 
+	if (d_is_tail_negative(dentry))
+		return D_WALK_SKIP_SIBLINGS;
 	if (likely(!d_mountpoint(dentry)))
 		return D_WALK_CONTINUE;
 	if (__path_is_mountpoint(&path)) {
@@ -1705,6 +1707,10 @@ void shrink_dcache_for_umount(struct super_block *sb)
 static enum d_walk_ret find_submount(void *_data, struct dentry *dentry)
 {
 	struct dentry **victim = _data;
+
+	if (d_is_tail_negative(dentry))
+		return D_WALK_SKIP_SIBLINGS;
+
 	if (d_mountpoint(dentry)) {
 		__dget_dlock(dentry);
 		*victim = dentry;
@@ -3174,6 +3180,9 @@ static enum d_walk_ret d_genocide_kill(void *data, struct dentry *dentry)
 {
 	struct dentry *root = data;
 	if (dentry != root) {
+		if (d_is_tail_negative(dentry))
+			return D_WALK_SKIP_SIBLINGS;
+
 		if (d_unhashed(dentry) || !dentry->d_inode)
 			return D_WALK_SKIP;
 
diff --git a/fs/libfs.c b/fs/libfs.c
index 7124c2e..15d5ecf 100644
--- a/fs/libfs.c
+++ b/fs/libfs.c
@@ -410,6 +410,9 @@ int simple_empty(struct dentry *dentry)
 
 	spin_lock(&dentry->d_lock);
 	list_for_each_entry(child, &dentry->d_subdirs, d_child) {
+		if (d_is_tail_negative(child))
+			break;
+
 		spin_lock_nested(&child->d_lock, DENTRY_D_LOCK_NESTED);
 		if (simple_positive(child)) {
 			spin_unlock(&child->d_lock);
-- 
1.8.3.1

[PATCH RFC 5/6] dcache: push releasing dentry lock into sweep_negative

[Gautham Ananthakrishna]

From: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>

Release the dentry lock inside the sweep_negative() function.  This
is in preparation for a follow up patch and doesn't change runtime
behavior.

Signed-off-by: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>
Signed-off-by: Gautham Ananthakrisha <gautham.ananthakrishna@oracle.com>
---
 fs/dcache.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/fs/dcache.c b/fs/dcache.c
index 492a42f..22c990b 100644
--- a/fs/dcache.c
+++ b/fs/dcache.c
@@ -638,13 +638,14 @@ static inline struct dentry *lock_parent(struct dentry *dentry)
  * Must be called at dput of negative dentry.
  */
 static void sweep_negative(struct dentry *dentry)
+	__releases(dentry->d_lock)
 {
 	struct dentry *parent;
 
 	if (!d_is_tail_negative(dentry)) {
 		parent = lock_parent(dentry);
 		if (!parent)
-			return;
+			goto out;
 
 		if (!d_count(dentry) && d_is_negative(dentry) &&
 		    !d_is_tail_negative(dentry)) {
@@ -654,6 +655,8 @@ static void sweep_negative(struct dentry *dentry)
 
 		spin_unlock(&parent->d_lock);
 	}
+out:
+	spin_unlock(&dentry->d_lock);
 }
 
 /*
@@ -922,7 +925,8 @@ void dput(struct dentry *dentry)
 		if (likely(retain_dentry(dentry))) {
 			if (d_is_negative(dentry))
 				sweep_negative(dentry);
-			spin_unlock(&dentry->d_lock);
+			else
+				spin_unlock(&dentry->d_lock);
 			return;
 		}
 
-- 
1.8.3.1

[PATCH RFC 6/6] dcache: prevent flooding with negative dentries

[Gautham Ananthakrishna]

From: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>

Without memory pressure count of negative dentries isn't bounded.
They could consume all memory and drain all other inactive caches.

Typical scenario is an idle system where some process periodically creates
temporary files and removes them. After some time, memory will be filled
with negative dentries for these random file names. Reclaiming them took
some time because slab frees pages only when all related objects are gone.
Time of dentry lookup is usually unaffected because hash table grows along
with size of memory. Unless somebody especially crafts hash collisions.
Simple lookup of random names also generates negative dentries very fast.

This patch implements heuristic which detects such scenarios and prevents
unbounded growth of completely unneeded negative dentries. It keeps up to
three latest negative dentry in each bucket unless they were referenced.

At first dput of negative dentry when it swept to the tail of siblings
we'll also clear it's reference flag and look at next dentries in chain.
Then kill third in series of negative, unused and unreferenced denries.

This way each hash bucket will preserve three negative dentry to let them
get reference and survive. Adding positive or used dentry into hash chain
also protects few recent negative dentries. In result total size of dcache
asymptotically limited by count of buckets and positive or used dentries.

Before patch: tool 'dcache_stress' could fill entire memory with dentries.

nr_dentry = 104913261   104.9M
nr_buckets = 8388608    12.5 avg
nr_unused = 104898729   100.0%
nr_negative = 104883218 100.0%

After this patch count of dentries saturates at around 3 per bucket:

nr_dentry = 24619259    24.6M
nr_buckets = 8388608    2.9 avg
nr_unused = 24605226    99.9%
nr_negative = 24600351  99.9%

This heuristic isn't bulletproof and solves only most practical case.
It's easy to deceive: just touch same random name twice.

Signed-off-by: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>
Signed-off-by: Gautham Ananthakrishna <gautham.ananthakrishna@oracle.com>
---
 fs/dcache.c | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 54 insertions(+)

diff --git a/fs/dcache.c b/fs/dcache.c
index 22c990b..6281938 100644
--- a/fs/dcache.c
+++ b/fs/dcache.c
@@ -633,6 +633,58 @@ static inline struct dentry *lock_parent(struct dentry *dentry)
 }
 
 /*
+ * Called at first dput of each negative dentry.
+ * Prevents filling cache with never reused negative dentries.
+ *
+ * This clears reference and then looks at following dentries in hash chain.
+ * If they are negative, unused and unreferenced then keep two and kill third.
+ */
+static void trim_negative(struct dentry *dentry)
+	__releases(dentry->d_lock)
+{
+	struct dentry *victim, *parent;
+	struct hlist_bl_node *next;
+	int keep = 2;
+
+	rcu_read_lock();
+
+	dentry->d_flags &= ~DCACHE_REFERENCED;
+	spin_unlock(&dentry->d_lock);
+
+	next = rcu_dereference_raw(dentry->d_hash.next);
+	while (1) {
+		victim = hlist_bl_entry(next, struct dentry, d_hash);
+
+		if (!next || d_count(victim) || !d_is_negative(victim) ||
+		    (victim->d_flags & DCACHE_REFERENCED)) {
+			rcu_read_unlock();
+			return;
+		}
+
+		if (!keep--)
+			break;
+
+		next = rcu_dereference_raw(next->next);
+	}
+
+	spin_lock(&victim->d_lock);
+	parent = lock_parent(victim);
+
+	rcu_read_unlock();
+
+	if (d_count(victim) || !d_is_negative(victim) ||
+	    (victim->d_flags & DCACHE_REFERENCED)) {
+		if (parent)
+			spin_unlock(&parent->d_lock);
+		spin_unlock(&victim->d_lock);
+		return;
+	}
+
+	__dentry_kill(victim);
+	dput(parent);
+}
+
+/*
  * Move cached negative dentry to the tail of parent->d_subdirs.
  * This lets walkers skip them all together at first sight.
  * Must be called at dput of negative dentry.
@@ -654,6 +706,8 @@ static void sweep_negative(struct dentry *dentry)
 		}
 
 		spin_unlock(&parent->d_lock);
+
+		return trim_negative(dentry);
 	}
 out:
 	spin_unlock(&dentry->d_lock);
-- 
1.8.3.1

Re: [PATCH RFC 6/6] dcache: prevent flooding with negative dentries

[Al Viro]

On Thu, Jan 21, 2021 at 06:49:45PM +0530, Gautham Ananthakrishna wrote:

> +	spin_lock(&victim->d_lock);
> +	parent = lock_parent(victim);
> +
> +	rcu_read_unlock();

Similar story.  As soon as you hit that rcu_read_unlock(), the memory
pointed to by victim might be reused.  If you have hit __lock_parent(),
victim->d_lock had been dropped and regained.  Which means that freeing
might've been already scheduled.  Unlike #1/6, here you won't get
memory corruption in lock_parent() itself, but...

> +
> +	if (d_count(victim) || !d_is_negative(victim) ||
> +	    (victim->d_flags & DCACHE_REFERENCED)) {
> +		if (parent)
> +			spin_unlock(&parent->d_lock);
> +		spin_unlock(&victim->d_lock);

... starting from here you just might.

Re: [PATCH RFC 0/6] fix the negative dentres bloating system memory usage

[Matthew Wilcox]

Ping?  These patches are looking pretty good in our internal testing.

On Thu, Jan 21, 2021 at 06:49:39PM +0530, Gautham Ananthakrishna wrote:
> For most filesystems result of every negative lookup is cached, content of
> directories is usually cached too. Production of negative dentries isn't
> limited with disk speed. It's really easy to generate millions of them if
> system has enough memory.
> 
> Getting this memory back ins't that easy because slab frees pages only when
> all related objects are gone. While dcache shrinker works in LRU order.
> 
> Typical scenario is an idle system where some process periodically creates
> temporary files and removes them. After some time, memory will be filled
> with negative dentries for these random file names.
> 
> Simple lookup of random names also generates negative dentries very fast.
> Constant flow of such negative denries drains all other inactive caches.
> Too many negative dentries in the system can cause memory fragmentation
> and memory compaction.
> 
> Negative dentries are linked into siblings list along with normal positive
> dentries. Some operations walks dcache tree but looks only for positive
> dentries: most important is fsnotify/inotify. Hordes of negative dentries
> slow down these operations significantly.
> 
> Time of dentry lookup is usually unaffected because hash table grows along
> with size of memory. Unless somebody especially crafts hash collisions.
> 
> This patch set solves all of these problems:
> 
> Move negative denries to the end of sliblings list, thus walkers could
> skip them at first sight (patches 1-4).
> 
> Keep in dcache at most three unreferenced negative denties in row in each
> hash bucket (patches 5-6).
> 
> We tested this patch set recently and found it limiting negative dentry to a
> small part of total memory. The following is the test result we ran on two
> types of servers, one is 256G memory with 24 CPUS and another is 3T memory
> with 384 CPUS. The test case is using a lot of processes to generate negative
> dentry in parallel, the following is the test result after 72 hours, the
> negative dentry number is stable around that number even after running longer
> for much longer time. Without the patch set, in less than half an hour 197G was
> taken by negative dentry on 256G system, in 1 day 2.4T was taken on 3T system.
> 
> system memory   neg-dentry-number   neg-dentry-mem-usage
> 256G            55259084            10.6G
> 3T              202306756           38.8G
> 
> For perf test, we ran the following, and no regression found.
> 
> 1. create 1M negative dentry and then touch them to convert them to positive
>    dentry
> 
> 2. create 10K/100K/1M files
> 
> 3. remove 10K/100K/1M files
> 
> 4. kernel compile
> 
> To verify the fsnotify fix, we used inotifywait to watch file create/open in
> some directory where there is a lot of negative dentry, without the patch set,
> the system would run into soft lockup, with it, no soft lockup was found.
> 
> We also tried to defeat the limitation by making different processes generate
> negative dentry with the same name, that will make one negative dentry being
> accessed couple times around same time, DCACHE_REFERENCED will be set on it
> and it can't be trimmed easily.
> 
> There were a lot of customer cases on this issue. It makes no sense to leave
> so many negative dentry, it just causes memory fragmentation and compaction
> and does not help a lot.
> 
> Konstantin Khlebnikov (6):
>   dcache: sweep cached negative dentries to the end of list of siblings
>   fsnotify: stop walking child dentries if remaining tail is negative
>   dcache: add action D_WALK_SKIP_SIBLINGS to d_walk()
>   dcache: stop walking siblings if remaining dentries all negative
>   dcache: push releasing dentry lock into sweep_negative
>   dcache: prevent flooding with negative dentries
> 
>  fs/dcache.c            | 135 +++++++++++++++++++++++++++++++++++++++++++++++--
>  fs/libfs.c             |   3 ++
>  fs/notify/fsnotify.c   |   6 ++-
>  include/linux/dcache.h |   6 +++
>  4 files changed, 145 insertions(+), 5 deletions(-)
> 
> -- 
> 1.8.3.1
> 
> 

Re: [PATCH RFC 0/6] fix the negative dentres bloating system memory usage

[Al Viro]

On Thu, Jan 21, 2021 at 06:49:39PM +0530, Gautham Ananthakrishna wrote:

> We tested this patch set recently and found it limiting negative dentry to a
> small part of total memory. The following is the test result we ran on two
> types of servers, one is 256G memory with 24 CPUS and another is 3T memory
> with 384 CPUS. The test case is using a lot of processes to generate negative
> dentry in parallel, the following is the test result after 72 hours, the
> negative dentry number is stable around that number even after running longer
> for much longer time. Without the patch set, in less than half an hour 197G was
> taken by negative dentry on 256G system, in 1 day 2.4T was taken on 3T system.
> 
> system memory   neg-dentry-number   neg-dentry-mem-usage
> 256G            55259084            10.6G
> 3T              202306756           38.8G
> 
> For perf test, we ran the following, and no regression found.
> 
> 1. create 1M negative dentry and then touch them to convert them to positive
>    dentry
> 
> 2. create 10K/100K/1M files
> 
> 3. remove 10K/100K/1M files
> 
> 4. kernel compile

Good for you; how would that work for thinner boxen, though?  I agree that if you
have 8M hash buckets your "no more than 3 unused negatives per bucket" is generous
enough for everything, but that's less obvious for something with e.g 4 or 8 gigs.
And believe it or not, there are real-world boxen like that ;-)

Re: [PATCH RFC 1/6] dcache: sweep cached negative dentries to the end of list of siblings

[kernel test robot]

[-- Attachment #1: Type: text/plain, Size: 13817 bytes --]

CC: kbuild-all(a)lists.01.org
In-Reply-To: <1611235185-1685-2-git-send-email-gautham.ananthakrishna@oracle.com>
References: <1611235185-1685-2-git-send-email-gautham.ananthakrishna@oracle.com>
TO: Gautham Ananthakrishna <gautham.ananthakrishna@oracle.com>

Hi Gautham,

[FYI, it's a private test report for your RFC patch.]
[auto build test WARNING on ext3/fsnotify]
[cannot apply to linux/master linus/master v5.11-rc4 next-20210121]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Gautham-Ananthakrishna/fix-the-negative-dentres-bloating-system-memory-usage/20210121-212603
base:   https://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-fs.git fsnotify
:::::: branch date: 3 hours ago
:::::: commit date: 3 hours ago
config: nios2-randconfig-s032-20210121 (attached as .config)
compiler: nios2-linux-gcc (GCC) 9.3.0
reproduce:
        wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # apt-get install sparse
        # sparse version: v0.6.3-208-g46a52ca4-dirty
        # https://github.com/0day-ci/linux/commit/65ea9583a5de81f48ddc324932035ce2a4f0f8db
        git remote add linux-review https://github.com/0day-ci/linux
        git fetch --no-tags linux-review Gautham-Ananthakrishna/fix-the-negative-dentres-bloating-system-memory-usage/20210121-212603
        git checkout 65ea9583a5de81f48ddc324932035ce2a4f0f8db
        # save the attached .config to linux build tree
        COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross C=1 CF='-fdiagnostic-prefix -D__CHECK_ENDIAN__' ARCH=nios2 

If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>


"sparse warnings: (new ones prefixed by >>)"
   fs/dcache.c:577:28: sparse: sparse: context imbalance in '__dentry_kill' - unexpected unlock
   fs/dcache.c:622:9: sparse: sparse: context imbalance in '__lock_parent' - wrong count at exit
>> fs/dcache.c:640:13: sparse: sparse: context imbalance in 'sweep_negative' - different lock contexts for basic block
   fs/dcache.c:674:20: sparse: sparse: context imbalance in 'recycle_negative' - unexpected unlock
   fs/dcache.c:767:20: sparse: sparse: context imbalance in 'dentry_kill' - different lock contexts for basic block
   fs/dcache.c:929:17: sparse: sparse: context imbalance in 'dput' - unexpected unlock
   fs/dcache.c:958:20: sparse: sparse: context imbalance in 'dput_to_list' - unexpected unlock
   fs/dcache.c:1097:18: sparse: sparse: context imbalance in 'd_prune_aliases' - different lock contexts for basic block
   fs/dcache.c:1127:13: sparse: sparse: context imbalance in 'shrink_lock_dentry' - different lock contexts for basic block
   fs/dcache.c:1170:27: sparse: sparse: context imbalance in 'shrink_dentry_list' - different lock contexts for basic block
   fs/dcache.c:1277:24: sparse: sparse: context imbalance in 'dentry_lru_isolate_shrink' - different lock contexts for basic block
   fs/dcache.c:1339:13: sparse: sparse: context imbalance in 'd_walk' - different lock contexts for basic block
   fs/dcache.c:1573:24: sparse: sparse: context imbalance in 'select_collect2' - different lock contexts for basic block
   fs/dcache.c:1632:44: sparse: sparse: context imbalance in 'shrink_dcache_parent' - unexpected unlock
   fs/dcache.c: note: in included file (through include/linux/dcache.h, include/linux/fs.h, include/linux/huge_mm.h, include/linux/mm.h):
   include/linux/rculist_bl.h:24:33: sparse: sparse: incompatible types in comparison expression (different address spaces):
   include/linux/rculist_bl.h:24:33: sparse:    struct hlist_bl_node [noderef] __rcu *
   include/linux/rculist_bl.h:24:33: sparse:    struct hlist_bl_node *
   include/linux/rculist_bl.h:24:33: sparse: sparse: incompatible types in comparison expression (different address spaces):
   include/linux/rculist_bl.h:24:33: sparse:    struct hlist_bl_node [noderef] __rcu *
   include/linux/rculist_bl.h:24:33: sparse:    struct hlist_bl_node *
   include/linux/rculist_bl.h:17:9: sparse: sparse: incompatible types in comparison expression (different address spaces):
   include/linux/rculist_bl.h:17:9: sparse:    struct hlist_bl_node [noderef] __rcu *
   include/linux/rculist_bl.h:17:9: sparse:    struct hlist_bl_node *
   include/linux/rculist_bl.h:17:9: sparse: sparse: incompatible types in comparison expression (different address spaces):
   include/linux/rculist_bl.h:17:9: sparse:    struct hlist_bl_node [noderef] __rcu *
   include/linux/rculist_bl.h:17:9: sparse:    struct hlist_bl_node *

vim +/sweep_negative +640 fs/dcache.c

ba65dc5ef16f82f Al Viro               2016-06-10  549  
e55fd011549eae0 Al Viro               2014-05-28  550  static void __dentry_kill(struct dentry *dentry)
77812a1ef139d84 Nick Piggin           2011-01-07  551  {
41edf278fc2f042 Al Viro               2014-05-01  552  	struct dentry *parent = NULL;
41edf278fc2f042 Al Viro               2014-05-01  553  	bool can_free = true;
41edf278fc2f042 Al Viro               2014-05-01  554  	if (!IS_ROOT(dentry))
77812a1ef139d84 Nick Piggin           2011-01-07  555  		parent = dentry->d_parent;
31e6b01f4183ff4 Nick Piggin           2011-01-07  556  
0d98439ea3c6ffb Linus Torvalds        2013-09-08  557  	/*
0d98439ea3c6ffb Linus Torvalds        2013-09-08  558  	 * The dentry is now unrecoverably dead to the world.
0d98439ea3c6ffb Linus Torvalds        2013-09-08  559  	 */
0d98439ea3c6ffb Linus Torvalds        2013-09-08  560  	lockref_mark_dead(&dentry->d_lockref);
0d98439ea3c6ffb Linus Torvalds        2013-09-08  561  
f0023bc617ba600 Sage Weil             2011-10-28  562  	/*
f0023bc617ba600 Sage Weil             2011-10-28  563  	 * inform the fs via d_prune that this dentry is about to be
f0023bc617ba600 Sage Weil             2011-10-28  564  	 * unhashed and destroyed.
f0023bc617ba600 Sage Weil             2011-10-28  565  	 */
2926620145095ff Al Viro               2014-05-30  566  	if (dentry->d_flags & DCACHE_OP_PRUNE)
61572bb1f40b9be Yan, Zheng            2013-04-15  567  		dentry->d_op->d_prune(dentry);
61572bb1f40b9be Yan, Zheng            2013-04-15  568  
01b6035190b0242 Al Viro               2014-04-29  569  	if (dentry->d_flags & DCACHE_LRU_LIST) {
01b6035190b0242 Al Viro               2014-04-29  570  		if (!(dentry->d_flags & DCACHE_SHRINK_LIST))
01b6035190b0242 Al Viro               2014-04-29  571  			d_lru_del(dentry);
01b6035190b0242 Al Viro               2014-04-29  572  	}
77812a1ef139d84 Nick Piggin           2011-01-07  573  	/* if it was on the hash then remove it */
77812a1ef139d84 Nick Piggin           2011-01-07  574  	__d_drop(dentry);
ba65dc5ef16f82f Al Viro               2016-06-10  575  	dentry_unlist(dentry, parent);
03b3b889e79cdb6 Al Viro               2014-04-29  576  	if (parent)
03b3b889e79cdb6 Al Viro               2014-04-29 @577  		spin_unlock(&parent->d_lock);
550dce01dd606c8 Al Viro               2016-05-29  578  	if (dentry->d_inode)
550dce01dd606c8 Al Viro               2016-05-29  579  		dentry_unlink_inode(dentry);
550dce01dd606c8 Al Viro               2016-05-29  580  	else
550dce01dd606c8 Al Viro               2016-05-29  581  		spin_unlock(&dentry->d_lock);
03b3b889e79cdb6 Al Viro               2014-04-29  582  	this_cpu_dec(nr_dentry);
03b3b889e79cdb6 Al Viro               2014-04-29  583  	if (dentry->d_op && dentry->d_op->d_release)
03b3b889e79cdb6 Al Viro               2014-04-29  584  		dentry->d_op->d_release(dentry);
03b3b889e79cdb6 Al Viro               2014-04-29  585  
41edf278fc2f042 Al Viro               2014-05-01  586  	spin_lock(&dentry->d_lock);
41edf278fc2f042 Al Viro               2014-05-01  587  	if (dentry->d_flags & DCACHE_SHRINK_LIST) {
41edf278fc2f042 Al Viro               2014-05-01  588  		dentry->d_flags |= DCACHE_MAY_FREE;
41edf278fc2f042 Al Viro               2014-05-01  589  		can_free = false;
41edf278fc2f042 Al Viro               2014-05-01  590  	}
41edf278fc2f042 Al Viro               2014-05-01  591  	spin_unlock(&dentry->d_lock);
41edf278fc2f042 Al Viro               2014-05-01  592  	if (likely(can_free))
b4f0354e968f5fa Al Viro               2014-04-29  593  		dentry_free(dentry);
9c5f1d30199d09f Al Viro               2018-04-15  594  	cond_resched();
e55fd011549eae0 Al Viro               2014-05-28  595  }
e55fd011549eae0 Al Viro               2014-05-28  596  
8b987a46a1e0e93 Al Viro               2018-02-23  597  static struct dentry *__lock_parent(struct dentry *dentry)
046b961b45f93a9 Al Viro               2014-05-29  598  {
8b987a46a1e0e93 Al Viro               2018-02-23  599  	struct dentry *parent;
046b961b45f93a9 Al Viro               2014-05-29  600  	rcu_read_lock();
c2338f2dc7c1e9f Al Viro               2014-06-12  601  	spin_unlock(&dentry->d_lock);
046b961b45f93a9 Al Viro               2014-05-29  602  again:
66702eb59064f10 Mark Rutland          2017-10-23  603  	parent = READ_ONCE(dentry->d_parent);
046b961b45f93a9 Al Viro               2014-05-29  604  	spin_lock(&parent->d_lock);
046b961b45f93a9 Al Viro               2014-05-29  605  	/*
046b961b45f93a9 Al Viro               2014-05-29  606  	 * We can't blindly lock dentry until we are sure
046b961b45f93a9 Al Viro               2014-05-29  607  	 * that we won't violate the locking order.
046b961b45f93a9 Al Viro               2014-05-29  608  	 * Any changes of dentry->d_parent must have
046b961b45f93a9 Al Viro               2014-05-29  609  	 * been done with parent->d_lock held, so
046b961b45f93a9 Al Viro               2014-05-29  610  	 * spin_lock() above is enough of a barrier
046b961b45f93a9 Al Viro               2014-05-29  611  	 * for checking if it's still our child.
046b961b45f93a9 Al Viro               2014-05-29  612  	 */
046b961b45f93a9 Al Viro               2014-05-29  613  	if (unlikely(parent != dentry->d_parent)) {
046b961b45f93a9 Al Viro               2014-05-29  614  		spin_unlock(&parent->d_lock);
046b961b45f93a9 Al Viro               2014-05-29  615  		goto again;
046b961b45f93a9 Al Viro               2014-05-29  616  	}
65d8eb5a8f54807 Al Viro               2018-02-23  617  	rcu_read_unlock();
65d8eb5a8f54807 Al Viro               2018-02-23  618  	if (parent != dentry)
9f12600fe425bc2 Linus Torvalds        2014-05-31  619  		spin_lock_nested(&dentry->d_lock, DENTRY_D_LOCK_NESTED);
65d8eb5a8f54807 Al Viro               2018-02-23  620  	else
046b961b45f93a9 Al Viro               2014-05-29  621  		parent = NULL;
046b961b45f93a9 Al Viro               2014-05-29  622  	return parent;
046b961b45f93a9 Al Viro               2014-05-29  623  }
046b961b45f93a9 Al Viro               2014-05-29  624  
8b987a46a1e0e93 Al Viro               2018-02-23  625  static inline struct dentry *lock_parent(struct dentry *dentry)
8b987a46a1e0e93 Al Viro               2018-02-23  626  {
8b987a46a1e0e93 Al Viro               2018-02-23  627  	struct dentry *parent = dentry->d_parent;
8b987a46a1e0e93 Al Viro               2018-02-23  628  	if (IS_ROOT(dentry))
8b987a46a1e0e93 Al Viro               2018-02-23  629  		return NULL;
8b987a46a1e0e93 Al Viro               2018-02-23  630  	if (likely(spin_trylock(&parent->d_lock)))
8b987a46a1e0e93 Al Viro               2018-02-23  631  		return parent;
8b987a46a1e0e93 Al Viro               2018-02-23  632  	return __lock_parent(dentry);
8b987a46a1e0e93 Al Viro               2018-02-23  633  }
8b987a46a1e0e93 Al Viro               2018-02-23  634  
65ea9583a5de81f Konstantin Khlebnikov 2021-01-21  635  /*
65ea9583a5de81f Konstantin Khlebnikov 2021-01-21  636   * Move cached negative dentry to the tail of parent->d_subdirs.
65ea9583a5de81f Konstantin Khlebnikov 2021-01-21  637   * This lets walkers skip them all together at first sight.
65ea9583a5de81f Konstantin Khlebnikov 2021-01-21  638   * Must be called at dput of negative dentry.
65ea9583a5de81f Konstantin Khlebnikov 2021-01-21  639   */
65ea9583a5de81f Konstantin Khlebnikov 2021-01-21 @640  static void sweep_negative(struct dentry *dentry)
65ea9583a5de81f Konstantin Khlebnikov 2021-01-21  641  {
65ea9583a5de81f Konstantin Khlebnikov 2021-01-21  642  	struct dentry *parent;
65ea9583a5de81f Konstantin Khlebnikov 2021-01-21  643  
65ea9583a5de81f Konstantin Khlebnikov 2021-01-21  644  	if (!d_is_tail_negative(dentry)) {
65ea9583a5de81f Konstantin Khlebnikov 2021-01-21  645  		parent = lock_parent(dentry);
65ea9583a5de81f Konstantin Khlebnikov 2021-01-21  646  		if (!parent)
65ea9583a5de81f Konstantin Khlebnikov 2021-01-21  647  			return;
65ea9583a5de81f Konstantin Khlebnikov 2021-01-21  648  
65ea9583a5de81f Konstantin Khlebnikov 2021-01-21  649  		if (!d_count(dentry) && d_is_negative(dentry) &&
65ea9583a5de81f Konstantin Khlebnikov 2021-01-21  650  		    !d_is_tail_negative(dentry)) {
65ea9583a5de81f Konstantin Khlebnikov 2021-01-21  651  			dentry->d_flags |= DCACHE_TAIL_NEGATIVE;
65ea9583a5de81f Konstantin Khlebnikov 2021-01-21  652  			list_move_tail(&dentry->d_child, &parent->d_subdirs);
65ea9583a5de81f Konstantin Khlebnikov 2021-01-21  653  		}
65ea9583a5de81f Konstantin Khlebnikov 2021-01-21  654  
65ea9583a5de81f Konstantin Khlebnikov 2021-01-21  655  		spin_unlock(&parent->d_lock);
65ea9583a5de81f Konstantin Khlebnikov 2021-01-21  656  	}
65ea9583a5de81f Konstantin Khlebnikov 2021-01-21  657  }
65ea9583a5de81f Konstantin Khlebnikov 2021-01-21  658  

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

[-- Attachment #2: config.gz --]
[-- Type: application/gzip, Size: 21060 bytes --]