* [patch 00/33] my current vfs scalability patch queue
@ 2009-09-04 6:51 npiggin
2009-09-04 6:51 ` [patch 01/33] fs: no games with DCACHE_UNHASHED npiggin
` (33 more replies)
0 siblings, 34 replies; 64+ messages in thread
From: npiggin @ 2009-09-04 6:51 UTC (permalink / raw)
To: linux-fsdevel; +Cc: linux-kernel
Hi,
Had a bit of time to work on my vfs scalability patches. Since last time: made
some bugfixes, scaled mntget/mntput with per-cpu counter and vfsmount brlock,
and worked on inode cache scalability. This last one is the most interesting...
with my last posting I had got as far as breaking the locks into constituent
parts, but they remained mostly global locks.
- I have now made per-bucket hash lock like the dcache (it still needs to be
made into bitlocks to avoid any bloat, but using spinlocks for now helps eg
with lockdep).
- Made the inode unused lru list into a lazy list like the dcache. This reduces
acquisitions of the lru/writeback list lock.
- Made inode rcu freed. This can enable further optimisations. But it is quite
a big change on its own worth noting.
- RCU freed inode enables the sb_inode_list_lock to be avoided in list walkers,
and therefore allows it to nest within i_lock. This significantly simplifies
the locking and reduces acquisitions of sb_inode_list_lock.
Some remaining obvious issues:
- Not all filesystems are completely audited, especially when it comes to
looking at inode/dentry callbacks now done with locks lifted.
- Global dcache_lru lock. This can be made per-zone which will improve
scalability and enable more efficient targetted reclaim. Needs some of
my old per-zone reclaim shrinker patches.
- inode sb list lock is limiting global rate of inode creation, inode wb
list lock is limiting global rate of inode dirtying and writeback.
- Inode writeback list lock tied with inode lru list lock (they use the same
list head). Could turn them into 2 locks. Then the lru lock can be made
per-zone. The writeback lock I will wait on Jens' writeback work.
- sb_inode_list_lock can be made per-sb. This is a reasonable step, but not
good for single-sb scalability. Could perhaps add some per-cpu magazines or
laziness to reduce some of this locking. Most walkers of this list are
slowpaths, so it could be split into percpu lists or something.
- inode lru lock could also be made per-zone.
- dentries and inodes are now rcu freed, some (most?) nested trylock loops
could be removed in favour of taking the correct lock order and then
re-checking that things haven't changed.
The reason I have had to go on with more changes to locking rather than trying
to get things merged is because it has been difficult to show improvements in
some cases, like for example in the inode cache lock breaking, it first
resulted in actually more global locks for different things so scalability
could be worse in some cases when multiple global locks need to be taken.
But it is now getting to the point where I will need to get some agreement with
the approach.
^ permalink raw reply [flat|nested] 64+ messages in thread
* [patch 01/33] fs: no games with DCACHE_UNHASHED
2009-09-04 6:51 [patch 00/33] my current vfs scalability patch queue npiggin
@ 2009-09-04 6:51 ` npiggin
2009-09-04 6:51 ` [patch 02/33] fs: cleanup files_lock npiggin
` (32 subsequent siblings)
33 siblings, 0 replies; 64+ messages in thread
From: npiggin @ 2009-09-04 6:51 UTC (permalink / raw)
To: linux-fsdevel; +Cc: linux-kernel, Miklos Szeredi
[-- Attachment #1: unhashed-d_delete.patch --]
[-- Type: text/plain, Size: 4438 bytes --]
(this is in -mm)
Filesystems outside the regular namespace do not have to clear DCACHE_UNHASHED
in order to have a working /proc/$pid/fd/XXX. Nothing in proc prevents the
fd link from being used if its dentry is not in the hash.
Also, it does not get put into the dcache hash if DCACHE_UNHASHED is clear;
that depends on the filesystem calling d_add or d_rehash.
So delete the misleading comments and needless code.
Acked-by: Miklos Szeredi <mszeredi@suse.cz>
Signed-off-by: Nick Piggin <npiggin@suse.de>
---
fs/anon_inodes.c | 16 ----------------
fs/pipe.c | 18 ------------------
net/socket.c | 19 -------------------
3 files changed, 53 deletions(-)
Index: linux-2.6/fs/pipe.c
===================================================================
--- linux-2.6.orig/fs/pipe.c
+++ linux-2.6/fs/pipe.c
@@ -887,17 +887,6 @@ void free_pipe_info(struct inode *inode)
}
static struct vfsmount *pipe_mnt __read_mostly;
-static int pipefs_delete_dentry(struct dentry *dentry)
-{
- /*
- * At creation time, we pretended this dentry was hashed
- * (by clearing DCACHE_UNHASHED bit in d_flags)
- * At delete time, we restore the truth : not hashed.
- * (so that dput() can proceed correctly)
- */
- dentry->d_flags |= DCACHE_UNHASHED;
- return 0;
-}
/*
* pipefs_dname() is called from d_path().
@@ -909,7 +898,6 @@ static char *pipefs_dname(struct dentry
}
static const struct dentry_operations pipefs_dentry_operations = {
- .d_delete = pipefs_delete_dentry,
.d_dname = pipefs_dname,
};
@@ -969,12 +957,6 @@ struct file *create_write_pipe(int flags
goto err_inode;
dentry->d_op = &pipefs_dentry_operations;
- /*
- * We dont want to publish this dentry into global dentry hash table.
- * We pretend dentry is already hashed, by unsetting DCACHE_UNHASHED
- * This permits a working /proc/$pid/fd/XXX on pipes
- */
- dentry->d_flags &= ~DCACHE_UNHASHED;
d_instantiate(dentry, inode);
err = -ENFILE;
Index: linux-2.6/net/socket.c
===================================================================
--- linux-2.6.orig/net/socket.c
+++ linux-2.6/net/socket.c
@@ -307,18 +307,6 @@ static struct file_system_type sock_fs_t
.kill_sb = kill_anon_super,
};
-static int sockfs_delete_dentry(struct dentry *dentry)
-{
- /*
- * At creation time, we pretended this dentry was hashed
- * (by clearing DCACHE_UNHASHED bit in d_flags)
- * At delete time, we restore the truth : not hashed.
- * (so that dput() can proceed correctly)
- */
- dentry->d_flags |= DCACHE_UNHASHED;
- return 0;
-}
-
/*
* sockfs_dname() is called from d_path().
*/
@@ -329,7 +317,6 @@ static char *sockfs_dname(struct dentry
}
static const struct dentry_operations sockfs_dentry_operations = {
- .d_delete = sockfs_delete_dentry,
.d_dname = sockfs_dname,
};
@@ -378,12 +365,6 @@ static int sock_attach_fd(struct socket
return -ENOMEM;
dentry->d_op = &sockfs_dentry_operations;
- /*
- * We dont want to push this dentry into global dentry hash table.
- * We pretend dentry is already hashed, by unsetting DCACHE_UNHASHED
- * This permits a working /proc/$pid/fd/XXX on sockets
- */
- dentry->d_flags &= ~DCACHE_UNHASHED;
d_instantiate(dentry, SOCK_INODE(sock));
sock->file = file;
Index: linux-2.6/fs/anon_inodes.c
===================================================================
--- linux-2.6.orig/fs/anon_inodes.c
+++ linux-2.6/fs/anon_inodes.c
@@ -33,24 +33,11 @@ static int anon_inodefs_get_sb(struct fi
mnt);
}
-static int anon_inodefs_delete_dentry(struct dentry *dentry)
-{
- /*
- * We faked vfs to believe the dentry was hashed when we created it.
- * Now we restore the flag so that dput() will work correctly.
- */
- dentry->d_flags |= DCACHE_UNHASHED;
- return 1;
-}
-
static struct file_system_type anon_inode_fs_type = {
.name = "anon_inodefs",
.get_sb = anon_inodefs_get_sb,
.kill_sb = kill_anon_super,
};
-static const struct dentry_operations anon_inodefs_dentry_operations = {
- .d_delete = anon_inodefs_delete_dentry,
-};
/*
* nop .set_page_dirty method so that people can use .page_mkwrite on
@@ -119,9 +106,6 @@ int anon_inode_getfd(const char *name, c
*/
atomic_inc(&anon_inode_inode->i_count);
- dentry->d_op = &anon_inodefs_dentry_operations;
- /* Do not publish this dentry inside the global dentry hash table */
- dentry->d_flags &= ~DCACHE_UNHASHED;
d_instantiate(dentry, anon_inode_inode);
error = -ENFILE;
^ permalink raw reply [flat|nested] 64+ messages in thread
* [patch 02/33] fs: cleanup files_lock
2009-09-04 6:51 [patch 00/33] my current vfs scalability patch queue npiggin
2009-09-04 6:51 ` [patch 01/33] fs: no games with DCACHE_UNHASHED npiggin
@ 2009-09-04 6:51 ` npiggin
2009-09-04 6:51 ` [patch 03/33] fs: scale files_lock npiggin
` (31 subsequent siblings)
33 siblings, 0 replies; 64+ messages in thread
From: npiggin @ 2009-09-04 6:51 UTC (permalink / raw)
To: linux-fsdevel; +Cc: linux-kernel
[-- Attachment #1: fs-files_list-improve.patch --]
[-- Type: text/plain, Size: 9901 bytes --]
Lock tty_files with a new spinlock, tty_files_lock; provide helpers to
manipulate the per-sb files list; unexport the files_lock spinlock.
Signed-off-by: Nick Piggin <npiggin@suse.de>
---
drivers/char/pty.c | 6 +++++-
drivers/char/tty_io.c | 26 ++++++++++++++++++--------
fs/file_table.c | 42 ++++++++++++++++++------------------------
fs/open.c | 4 ++--
include/linux/fs.h | 8 +++-----
include/linux/tty.h | 1 +
security/selinux/hooks.c | 4 ++--
7 files changed, 49 insertions(+), 42 deletions(-)
Index: linux-2.6/drivers/char/pty.c
===================================================================
--- linux-2.6.orig/drivers/char/pty.c
+++ linux-2.6/drivers/char/pty.c
@@ -655,7 +655,11 @@ static int __ptmx_open(struct inode *ino
set_bit(TTY_PTY_LOCK, &tty->flags); /* LOCK THE SLAVE */
filp->private_data = tty;
- file_move(filp, &tty->tty_files);
+
+ file_sb_list_del(filp); /* __dentry_open has put it on the sb list */
+ spin_lock(&tty_files_lock);
+ list_add(&filp->f_u.fu_list, &tty->tty_files);
+ spin_unlock(&tty_files_lock);
retval = devpts_pty_new(inode, tty->link);
if (retval)
Index: linux-2.6/drivers/char/tty_io.c
===================================================================
--- linux-2.6.orig/drivers/char/tty_io.c
+++ linux-2.6/drivers/char/tty_io.c
@@ -136,6 +136,9 @@ LIST_HEAD(tty_drivers); /* linked list
DEFINE_MUTEX(tty_mutex);
EXPORT_SYMBOL(tty_mutex);
+/* Spinlock to protect the tty->tty_files list */
+DEFINE_SPINLOCK(tty_files_lock);
+
static ssize_t tty_read(struct file *, char __user *, size_t, loff_t *);
static ssize_t tty_write(struct file *, const char __user *, size_t, loff_t *);
ssize_t redirected_tty_write(struct file *, const char __user *,
@@ -235,11 +238,11 @@ static int check_tty_count(struct tty_st
struct list_head *p;
int count = 0;
- file_list_lock();
+ spin_lock(&tty_files_lock);
list_for_each(p, &tty->tty_files) {
count++;
}
- file_list_unlock();
+ spin_unlock(&tty_files_lock);
if (tty->driver->type == TTY_DRIVER_TYPE_PTY &&
tty->driver->subtype == PTY_TYPE_SLAVE &&
tty->link && tty->link->count)
@@ -517,7 +520,7 @@ static void do_tty_hangup(struct work_st
spin_unlock(&redirect_lock);
check_tty_count(tty, "do_tty_hangup");
- file_list_lock();
+ spin_lock(&tty_files_lock);
/* This breaks for file handles being sent over AF_UNIX sockets ? */
list_for_each_entry(filp, &tty->tty_files, f_u.fu_list) {
if (filp->f_op->write == redirected_tty_write)
@@ -528,7 +531,7 @@ static void do_tty_hangup(struct work_st
tty_fasync(-1, filp, 0); /* can't block */
filp->f_op = &hung_up_tty_fops;
}
- file_list_unlock();
+ spin_unlock(&tty_files_lock);
tty_ldisc_hangup(tty);
@@ -1400,9 +1403,9 @@ static void release_one_tty(struct kref
tty_driver_kref_put(driver);
module_put(driver->owner);
- file_list_lock();
+ spin_lock(&tty_files_lock);
list_del_init(&tty->tty_files);
- file_list_unlock();
+ spin_unlock(&tty_files_lock);
free_tty_struct(tty);
}
@@ -1611,7 +1614,10 @@ void tty_release_dev(struct file *filp)
* - do_tty_hangup no longer sees this file descriptor as
* something that needs to be handled for hangups.
*/
- file_kill(filp);
+ spin_lock(&tty_files_lock);
+ BUG_ON(list_empty(&filp->f_u.fu_list));
+ list_del_init(&filp->f_u.fu_list);
+ spin_unlock(&tty_files_lock);
filp->private_data = NULL;
/*
@@ -1769,7 +1775,11 @@ got_driver:
return PTR_ERR(tty);
filp->private_data = tty;
- file_move(filp, &tty->tty_files);
+ BUG_ON(list_empty(&filp->f_u.fu_list));
+ file_sb_list_del(filp); /* __dentry_open has put it on the sb list */
+ spin_lock(&tty_files_lock);
+ list_add(&filp->f_u.fu_list, &tty->tty_files);
+ spin_unlock(&tty_files_lock);
check_tty_count(tty, "tty_open");
if (tty->driver->type == TTY_DRIVER_TYPE_PTY &&
tty->driver->subtype == PTY_TYPE_MASTER)
Index: linux-2.6/fs/file_table.c
===================================================================
--- linux-2.6.orig/fs/file_table.c
+++ linux-2.6/fs/file_table.c
@@ -30,8 +30,7 @@ struct files_stat_struct files_stat = {
.max_files = NR_FILE
};
-/* public. Not pretty! */
-__cacheline_aligned_in_smp DEFINE_SPINLOCK(files_lock);
+static __cacheline_aligned_in_smp DEFINE_SPINLOCK(files_lock);
/* SLAB cache for file structures */
static struct kmem_cache *filp_cachep __read_mostly;
@@ -285,7 +284,7 @@ void __fput(struct file *file)
cdev_put(inode->i_cdev);
fops_put(file->f_op);
put_pid(file->f_owner.pid);
- file_kill(file);
+ file_sb_list_del(file);
if (file->f_mode & FMODE_WRITE)
drop_file_write_access(file);
file->f_path.dentry = NULL;
@@ -347,31 +346,29 @@ struct file *fget_light(unsigned int fd,
return file;
}
-
void put_filp(struct file *file)
{
if (atomic_long_dec_and_test(&file->f_count)) {
security_file_free(file);
- file_kill(file);
+ file_sb_list_del(file);
file_free(file);
}
}
-void file_move(struct file *file, struct list_head *list)
+void file_sb_list_add(struct file *file, struct super_block *sb)
{
- if (!list)
- return;
- file_list_lock();
- list_move(&file->f_u.fu_list, list);
- file_list_unlock();
+ spin_lock(&files_lock);
+ BUG_ON(!list_empty(&file->f_u.fu_list));
+ list_add(&file->f_u.fu_list, &sb->s_files);
+ spin_unlock(&files_lock);
}
-void file_kill(struct file *file)
+void file_sb_list_del(struct file *file)
{
if (!list_empty(&file->f_u.fu_list)) {
- file_list_lock();
+ spin_lock(&files_lock);
list_del_init(&file->f_u.fu_list);
- file_list_unlock();
+ spin_unlock(&files_lock);
}
}
@@ -380,7 +377,7 @@ int fs_may_remount_ro(struct super_block
struct file *file;
/* Check that no files are currently opened for writing. */
- file_list_lock();
+ spin_lock(&files_lock);
list_for_each_entry(file, &sb->s_files, f_u.fu_list) {
struct inode *inode = file->f_path.dentry->d_inode;
@@ -392,10 +389,10 @@ int fs_may_remount_ro(struct super_block
if (S_ISREG(inode->i_mode) && (file->f_mode & FMODE_WRITE))
goto too_bad;
}
- file_list_unlock();
+ spin_unlock(&files_lock);
return 1; /* Tis' cool bro. */
too_bad:
- file_list_unlock();
+ spin_unlock(&files_lock);
return 0;
}
@@ -411,7 +408,7 @@ void mark_files_ro(struct super_block *s
struct file *f;
retry:
- file_list_lock();
+ spin_lock(&files_lock);
list_for_each_entry(f, &sb->s_files, f_u.fu_list) {
struct vfsmount *mnt;
if (!S_ISREG(f->f_path.dentry->d_inode->i_mode))
@@ -425,16 +422,13 @@ retry:
continue;
file_release_write(f);
mnt = mntget(f->f_path.mnt);
- file_list_unlock();
- /*
- * This can sleep, so we can't hold
- * the file_list_lock() spinlock.
- */
+ /* This can sleep, so we can't hold the spinlock. */
+ spin_unlock(&files_lock);
mnt_drop_write(mnt);
mntput(mnt);
goto retry;
}
- file_list_unlock();
+ spin_unlock(&files_lock);
}
void __init files_init(unsigned long mempages)
Index: linux-2.6/fs/open.c
===================================================================
--- linux-2.6.orig/fs/open.c
+++ linux-2.6/fs/open.c
@@ -828,7 +828,7 @@ static struct file *__dentry_open(struct
f->f_path.mnt = mnt;
f->f_pos = 0;
f->f_op = fops_get(inode->i_fop);
- file_move(f, &inode->i_sb->s_files);
+ file_sb_list_add(f, inode->i_sb);
error = security_dentry_open(f, cred);
if (error)
@@ -873,7 +873,7 @@ cleanup_all:
mnt_drop_write(mnt);
}
}
- file_kill(f);
+ file_sb_list_del(f);
f->f_path.dentry = NULL;
f->f_path.mnt = NULL;
cleanup_file:
Index: linux-2.6/include/linux/fs.h
===================================================================
--- linux-2.6.orig/include/linux/fs.h
+++ linux-2.6/include/linux/fs.h
@@ -940,9 +940,6 @@ struct file {
unsigned long f_mnt_write_state;
#endif
};
-extern spinlock_t files_lock;
-#define file_list_lock() spin_lock(&files_lock);
-#define file_list_unlock() spin_unlock(&files_lock);
#define get_file(x) atomic_long_inc(&(x)->f_count)
#define file_count(x) atomic_long_read(&(x)->f_count)
@@ -2031,6 +2028,7 @@ extern const struct file_operations read
extern const struct file_operations write_pipefifo_fops;
extern const struct file_operations rdwr_pipefifo_fops;
+extern void mark_files_ro(struct super_block *sb);
extern int fs_may_remount_ro(struct super_block *);
#ifdef CONFIG_BLOCK
@@ -2176,8 +2174,8 @@ static inline void insert_inode_hash(str
}
extern struct file * get_empty_filp(void);
-extern void file_move(struct file *f, struct list_head *list);
-extern void file_kill(struct file *f);
+extern void file_sb_list_add(struct file *f, struct super_block *sb);
+extern void file_sb_list_del(struct file *f);
#ifdef CONFIG_BLOCK
struct bio;
extern void submit_bio(int, struct bio *);
Index: linux-2.6/security/selinux/hooks.c
===================================================================
--- linux-2.6.orig/security/selinux/hooks.c
+++ linux-2.6/security/selinux/hooks.c
@@ -2241,7 +2241,7 @@ static inline void flush_unauthorized_fi
tty = get_current_tty();
if (tty) {
- file_list_lock();
+ spin_lock(&tty_files_lock);
if (!list_empty(&tty->tty_files)) {
struct inode *inode;
@@ -2257,7 +2257,7 @@ static inline void flush_unauthorized_fi
drop_tty = 1;
}
}
- file_list_unlock();
+ spin_unlock(&tty_files_lock);
tty_kref_put(tty);
}
/* Reset controlling tty. */
Index: linux-2.6/include/linux/tty.h
===================================================================
--- linux-2.6.orig/include/linux/tty.h
+++ linux-2.6/include/linux/tty.h
@@ -438,6 +438,7 @@ extern struct tty_struct *tty_pair_get_t
extern struct tty_struct *tty_pair_get_pty(struct tty_struct *tty);
extern struct mutex tty_mutex;
+extern spinlock_t tty_files_lock;
extern void tty_write_unlock(struct tty_struct *tty);
extern int tty_write_lock(struct tty_struct *tty, int ndelay);
^ permalink raw reply [flat|nested] 64+ messages in thread
* [patch 03/33] fs: scale files_lock
2009-09-04 6:51 [patch 00/33] my current vfs scalability patch queue npiggin
2009-09-04 6:51 ` [patch 01/33] fs: no games with DCACHE_UNHASHED npiggin
2009-09-04 6:51 ` [patch 02/33] fs: cleanup files_lock npiggin
@ 2009-09-04 6:51 ` npiggin
2009-09-28 13:22 ` Peter Zijlstra
2009-09-28 13:24 ` Peter Zijlstra
2009-09-04 6:51 ` [patch 04/33] fs: brlock vfsmount_lock npiggin
` (30 subsequent siblings)
33 siblings, 2 replies; 64+ messages in thread
From: npiggin @ 2009-09-04 6:51 UTC (permalink / raw)
To: linux-fsdevel; +Cc: linux-kernel
[-- Attachment #1: fs-files_lock-scale.patch --]
[-- Type: text/plain, Size: 7566 bytes --]
Improve scalability of files_lock by adding per-cpu, per-sb files lists,
protected with per-cpu locking. Effectively turning it into a big-writer
lock.
Signed-off-by: Nick Piggin <npiggin@suse.de>
---
fs/file_table.c | 161 +++++++++++++++++++++++++++++++++++++++--------------
fs/super.c | 16 +++++
include/linux/fs.h | 7 ++
3 files changed, 143 insertions(+), 41 deletions(-)
Index: linux-2.6/fs/file_table.c
===================================================================
--- linux-2.6.orig/fs/file_table.c
+++ linux-2.6/fs/file_table.c
@@ -22,6 +22,7 @@
#include <linux/fsnotify.h>
#include <linux/sysctl.h>
#include <linux/percpu_counter.h>
+#include <linux/percpu.h>
#include <asm/atomic.h>
@@ -30,7 +31,7 @@ struct files_stat_struct files_stat = {
.max_files = NR_FILE
};
-static __cacheline_aligned_in_smp DEFINE_SPINLOCK(files_lock);
+static DEFINE_PER_CPU(spinlock_t, files_cpulock);
/* SLAB cache for file structures */
static struct kmem_cache *filp_cachep __read_mostly;
@@ -124,6 +125,9 @@ struct file *get_empty_filp(void)
goto fail_sec;
INIT_LIST_HEAD(&f->f_u.fu_list);
+#ifdef CONFIG_SMP
+ f->f_sb_list_cpu = -1;
+#endif
atomic_long_set(&f->f_count, 1);
rwlock_init(&f->f_owner.lock);
f->f_cred = get_cred(cred);
@@ -357,42 +361,104 @@ void put_filp(struct file *file)
void file_sb_list_add(struct file *file, struct super_block *sb)
{
- spin_lock(&files_lock);
+ spinlock_t *lock;
+ struct list_head *list;
+#ifdef CONFIG_SMP
+ int cpu;
+#endif
+
+ lock = &get_cpu_var(files_cpulock);
+#ifdef CONFIG_SMP
+ BUG_ON(file->f_sb_list_cpu != -1);
+ cpu = smp_processor_id();
+ list = per_cpu_ptr(sb->s_files, cpu);
+ file->f_sb_list_cpu = cpu;
+#else
+ list = &sb->s_files;
+#endif
+ spin_lock(lock);
BUG_ON(!list_empty(&file->f_u.fu_list));
- list_add(&file->f_u.fu_list, &sb->s_files);
- spin_unlock(&files_lock);
+ list_add(&file->f_u.fu_list, list);
+ spin_unlock(lock);
+ put_cpu_var(files_cpulock);
}
void file_sb_list_del(struct file *file)
{
if (!list_empty(&file->f_u.fu_list)) {
- spin_lock(&files_lock);
+ spinlock_t *lock;
+
+#ifdef CONFIG_SMP
+ BUG_ON(file->f_sb_list_cpu == -1);
+ lock = &per_cpu(files_cpulock, file->f_sb_list_cpu);
+ file->f_sb_list_cpu = -1;
+#else
+ lock = &__get_cpu_var(files_cpulock);
+#endif
+ spin_lock(lock);
list_del_init(&file->f_u.fu_list);
- spin_unlock(&files_lock);
+ spin_unlock(lock);
+ }
+}
+
+static void file_list_lock_all(void)
+{
+ int i;
+ int nr = 0;
+
+ for_each_possible_cpu(i) {
+ spinlock_t *lock;
+
+ lock = &per_cpu(files_cpulock, i);
+ spin_lock_nested(lock, nr);
+ nr++;
+ }
+}
+
+static void file_list_unlock_all(void)
+{
+ int i;
+
+ for_each_possible_cpu(i) {
+ spinlock_t *lock;
+
+ lock = &per_cpu(files_cpulock, i);
+ spin_unlock(lock);
}
}
int fs_may_remount_ro(struct super_block *sb)
{
- struct file *file;
+ int i;
/* Check that no files are currently opened for writing. */
- spin_lock(&files_lock);
- list_for_each_entry(file, &sb->s_files, f_u.fu_list) {
- struct inode *inode = file->f_path.dentry->d_inode;
-
- /* File with pending delete? */
- if (inode->i_nlink == 0)
- goto too_bad;
-
- /* Writeable file? */
- if (S_ISREG(inode->i_mode) && (file->f_mode & FMODE_WRITE))
- goto too_bad;
+ file_list_lock_all();
+ for_each_possible_cpu(i) {
+ struct file *file;
+ struct list_head *list;
+
+#ifdef CONFIG_SMP
+ list = per_cpu_ptr(sb->s_files, i);
+#else
+ list = &sb->s_files;
+#endif
+ list_for_each_entry(file, list, f_u.fu_list) {
+ struct inode *inode = file->f_path.dentry->d_inode;
+
+ /* File with pending delete? */
+ if (inode->i_nlink == 0)
+ goto too_bad;
+
+ /* Writeable file? */
+ if (S_ISREG(inode->i_mode) &&
+ (file->f_mode & FMODE_WRITE))
+ goto too_bad;
+ }
}
- spin_unlock(&files_lock);
+ file_list_unlock_all();
return 1; /* Tis' cool bro. */
too_bad:
- spin_unlock(&files_lock);
+ file_list_unlock_all();
return 0;
}
@@ -405,35 +471,46 @@ too_bad:
*/
void mark_files_ro(struct super_block *sb)
{
- struct file *f;
+ int i;
retry:
- spin_lock(&files_lock);
- list_for_each_entry(f, &sb->s_files, f_u.fu_list) {
- struct vfsmount *mnt;
- if (!S_ISREG(f->f_path.dentry->d_inode->i_mode))
- continue;
- if (!file_count(f))
- continue;
- if (!(f->f_mode & FMODE_WRITE))
- continue;
- f->f_mode &= ~FMODE_WRITE;
- if (file_check_writeable(f) != 0)
- continue;
- file_release_write(f);
- mnt = mntget(f->f_path.mnt);
- /* This can sleep, so we can't hold the spinlock. */
- spin_unlock(&files_lock);
- mnt_drop_write(mnt);
- mntput(mnt);
- goto retry;
+ file_list_lock_all();
+ for_each_possible_cpu(i) {
+ struct file *f;
+ struct list_head *list;
+
+#ifdef CONFIG_SMP
+ list = per_cpu_ptr(sb->s_files, i);
+#else
+ list = &sb->s_files;
+#endif
+ list_for_each_entry(f, list, f_u.fu_list) {
+ struct vfsmount *mnt;
+ if (!S_ISREG(f->f_path.dentry->d_inode->i_mode))
+ continue;
+ if (!file_count(f))
+ continue;
+ if (!(f->f_mode & FMODE_WRITE))
+ continue;
+ f->f_mode &= ~FMODE_WRITE;
+ if (file_check_writeable(f) != 0)
+ continue;
+ file_release_write(f);
+ mnt = mntget(f->f_path.mnt);
+ /* This can sleep, so we can't hold the spinlock. */
+ file_list_unlock_all();
+ mnt_drop_write(mnt);
+ mntput(mnt);
+ goto retry;
+ }
}
- spin_unlock(&files_lock);
+ file_list_unlock_all();
}
void __init files_init(unsigned long mempages)
{
int n;
+ int i;
filp_cachep = kmem_cache_create("filp", sizeof(struct file), 0,
SLAB_HWCACHE_ALIGN | SLAB_PANIC, NULL);
@@ -448,5 +525,7 @@ void __init files_init(unsigned long mem
if (files_stat.max_files < NR_FILE)
files_stat.max_files = NR_FILE;
files_defer_init();
+ for_each_possible_cpu(i)
+ spin_lock_init(&per_cpu(files_cpulock, i));
percpu_counter_init(&nr_files, 0);
}
Index: linux-2.6/fs/super.c
===================================================================
--- linux-2.6.orig/fs/super.c
+++ linux-2.6/fs/super.c
@@ -65,7 +65,23 @@ static struct super_block *alloc_super(s
INIT_LIST_HEAD(&s->s_dirty);
INIT_LIST_HEAD(&s->s_io);
INIT_LIST_HEAD(&s->s_more_io);
+#ifdef CONFIG_SMP
+ s->s_files = alloc_percpu(struct list_head);
+ if (!s->s_files) {
+ security_sb_free(s);
+ kfree(s);
+ s = NULL;
+ goto out;
+ } else {
+ int i;
+
+ for_each_possible_cpu(i)
+ INIT_LIST_HEAD(per_cpu_ptr(s->s_files, i));
+ }
+#else
INIT_LIST_HEAD(&s->s_files);
+#endif
+
INIT_LIST_HEAD(&s->s_instances);
INIT_HLIST_HEAD(&s->s_anon);
INIT_LIST_HEAD(&s->s_inodes);
Index: linux-2.6/include/linux/fs.h
===================================================================
--- linux-2.6.orig/include/linux/fs.h
+++ linux-2.6/include/linux/fs.h
@@ -916,6 +916,9 @@ struct file {
#define f_vfsmnt f_path.mnt
const struct file_operations *f_op;
spinlock_t f_lock; /* f_ep_links, f_flags, no IRQ */
+#ifdef CONFIG_SMP
+ int f_sb_list_cpu;
+#endif
atomic_long_t f_count;
unsigned int f_flags;
fmode_t f_mode;
@@ -1337,7 +1340,11 @@ struct super_block {
struct list_head s_io; /* parked for writeback */
struct list_head s_more_io; /* parked for more writeback */
struct hlist_head s_anon; /* anonymous dentries for (nfs) exporting */
+#ifdef CONFIG_SMP
+ struct list_head *s_files;
+#else
struct list_head s_files;
+#endif
/* s_dentry_lru and s_nr_dentry_unused are protected by dcache_lock */
struct list_head s_dentry_lru; /* unused dentry lru */
int s_nr_dentry_unused; /* # of dentry on lru */
^ permalink raw reply [flat|nested] 64+ messages in thread
* [patch 04/33] fs: brlock vfsmount_lock
2009-09-04 6:51 [patch 00/33] my current vfs scalability patch queue npiggin
` (2 preceding siblings ...)
2009-09-04 6:51 ` [patch 03/33] fs: scale files_lock npiggin
@ 2009-09-04 6:51 ` npiggin
2009-09-04 15:19 ` Jens Axboe
2009-09-22 15:17 ` Al Viro
2009-09-04 6:51 ` [patch 05/33] fs: scale mntget/mntput npiggin
` (29 subsequent siblings)
33 siblings, 2 replies; 64+ messages in thread
From: npiggin @ 2009-09-04 6:51 UTC (permalink / raw)
To: linux-fsdevel; +Cc: linux-kernel
[-- Attachment #1: fs-vfsmount_lock-scale.patch --]
[-- Type: text/plain, Size: 17677 bytes --]
Use a brlock for the vfsmount lock.
---
fs/dcache.c | 4
fs/namei.c | 13 +-
fs/namespace.c | 201 ++++++++++++++++++++++++++++++---------------
fs/pnode.c | 4
fs/proc/base.c | 4
include/linux/mount.h | 6 +
kernel/audit_tree.c | 6 -
security/tomoyo/realpath.c | 4
8 files changed, 161 insertions(+), 81 deletions(-)
Index: linux-2.6/fs/dcache.c
===================================================================
--- linux-2.6.orig/fs/dcache.c
+++ linux-2.6/fs/dcache.c
@@ -1908,7 +1908,7 @@ char *__d_path(const struct path *path,
char *end = buffer + buflen;
char *retval;
- spin_lock(&vfsmount_lock);
+ vfsmount_read_lock();
prepend(&end, &buflen, "\0", 1);
if (d_unlinked(dentry) &&
(prepend(&end, &buflen, " (deleted)", 10) != 0))
@@ -1944,7 +1944,7 @@ char *__d_path(const struct path *path,
}
out:
- spin_unlock(&vfsmount_lock);
+ vfsmount_read_unlock();
return retval;
global_root:
Index: linux-2.6/fs/namei.c
===================================================================
--- linux-2.6.orig/fs/namei.c
+++ linux-2.6/fs/namei.c
@@ -679,15 +679,16 @@ int follow_up(struct path *path)
{
struct vfsmount *parent;
struct dentry *mountpoint;
- spin_lock(&vfsmount_lock);
+
+ vfsmount_read_unlock();
parent = path->mnt->mnt_parent;
if (parent == path->mnt) {
- spin_unlock(&vfsmount_lock);
+ vfsmount_read_unlock();
return 0;
}
mntget(parent);
mountpoint = dget(path->mnt->mnt_mountpoint);
- spin_unlock(&vfsmount_lock);
+ vfsmount_read_unlock();
dput(path->dentry);
path->dentry = mountpoint;
mntput(path->mnt);
@@ -766,15 +767,15 @@ static __always_inline void follow_dotdo
break;
}
spin_unlock(&dcache_lock);
- spin_lock(&vfsmount_lock);
+ vfsmount_read_lock();
parent = nd->path.mnt->mnt_parent;
if (parent == nd->path.mnt) {
- spin_unlock(&vfsmount_lock);
+ vfsmount_read_unlock();
break;
}
mntget(parent);
nd->path.dentry = dget(nd->path.mnt->mnt_mountpoint);
- spin_unlock(&vfsmount_lock);
+ vfsmount_read_unlock();
dput(old);
mntput(nd->path.mnt);
nd->path.mnt = parent;
Index: linux-2.6/fs/namespace.c
===================================================================
--- linux-2.6.orig/fs/namespace.c
+++ linux-2.6/fs/namespace.c
@@ -11,6 +11,8 @@
#include <linux/syscalls.h>
#include <linux/slab.h>
#include <linux/sched.h>
+#include <linux/spinlock.h>
+#include <linux/percpu.h>
#include <linux/smp_lock.h>
#include <linux/init.h>
#include <linux/kernel.h>
@@ -37,12 +39,16 @@
#define HASH_SHIFT ilog2(PAGE_SIZE / sizeof(struct list_head))
#define HASH_SIZE (1UL << HASH_SHIFT)
-/* spinlock for vfsmount related operations, inplace of dcache_lock */
-__cacheline_aligned_in_smp DEFINE_SPINLOCK(vfsmount_lock);
+/*
+ * vfsmount "brlock" style spinlock for vfsmount related operations, use
+ * vfsmount_read_lock/vfsmount_write_lock functions.
+ */
+static DEFINE_PER_CPU(spinlock_t, vfsmount_lock);
static int event;
static DEFINE_IDA(mnt_id_ida);
static DEFINE_IDA(mnt_group_ida);
+static DEFINE_SPINLOCK(mnt_id_lock);
static int mnt_id_start = 0;
static int mnt_group_start = 1;
@@ -54,6 +60,49 @@ static struct rw_semaphore namespace_sem
struct kobject *fs_kobj;
EXPORT_SYMBOL_GPL(fs_kobj);
+void vfsmount_read_lock(void)
+{
+ spinlock_t *lock;
+
+ lock = &get_cpu_var(vfsmount_lock);
+ spin_lock(lock);
+}
+
+void vfsmount_read_unlock(void)
+{
+ spinlock_t *lock;
+
+ lock = &__get_cpu_var(vfsmount_lock);
+ spin_unlock(lock);
+ put_cpu_var(vfsmount_lock);
+}
+
+void vfsmount_write_lock(void)
+{
+ int i;
+ int nr = 0;
+
+ for_each_possible_cpu(i) {
+ spinlock_t *lock;
+
+ lock = &per_cpu(vfsmount_lock, i);
+ spin_lock_nested(lock, nr);
+ nr++;
+ }
+}
+
+void vfsmount_write_unlock(void)
+{
+ int i;
+
+ for_each_possible_cpu(i) {
+ spinlock_t *lock;
+
+ lock = &per_cpu(vfsmount_lock, i);
+ spin_unlock(lock);
+ }
+}
+
static inline unsigned long hash(struct vfsmount *mnt, struct dentry *dentry)
{
unsigned long tmp = ((unsigned long)mnt / L1_CACHE_BYTES);
@@ -64,18 +113,21 @@ static inline unsigned long hash(struct
#define MNT_WRITER_UNDERFLOW_LIMIT -(1<<16)
-/* allocation is serialized by namespace_sem */
+/*
+ * allocation is serialized by namespace_sem, but we need the spinlock to
+ * serialise with freeing.
+ */
static int mnt_alloc_id(struct vfsmount *mnt)
{
int res;
retry:
ida_pre_get(&mnt_id_ida, GFP_KERNEL);
- spin_lock(&vfsmount_lock);
+ spin_lock(&mnt_id_lock);
res = ida_get_new_above(&mnt_id_ida, mnt_id_start, &mnt->mnt_id);
if (!res)
mnt_id_start = mnt->mnt_id + 1;
- spin_unlock(&vfsmount_lock);
+ spin_unlock(&mnt_id_lock);
if (res == -EAGAIN)
goto retry;
@@ -85,11 +137,11 @@ retry:
static void mnt_free_id(struct vfsmount *mnt)
{
int id = mnt->mnt_id;
- spin_lock(&vfsmount_lock);
+ spin_lock(&mnt_id_lock);
ida_remove(&mnt_id_ida, id);
if (mnt_id_start > id)
mnt_id_start = id;
- spin_unlock(&vfsmount_lock);
+ spin_unlock(&mnt_id_lock);
}
/*
@@ -344,7 +396,7 @@ static int mnt_make_readonly(struct vfsm
{
int ret = 0;
- spin_lock(&vfsmount_lock);
+ vfsmount_write_lock();
mnt->mnt_flags |= MNT_WRITE_HOLD;
/*
* After storing MNT_WRITE_HOLD, we'll read the counters. This store
@@ -378,15 +430,15 @@ static int mnt_make_readonly(struct vfsm
*/
smp_wmb();
mnt->mnt_flags &= ~MNT_WRITE_HOLD;
- spin_unlock(&vfsmount_lock);
+ vfsmount_write_unlock();
return ret;
}
static void __mnt_unmake_readonly(struct vfsmount *mnt)
{
- spin_lock(&vfsmount_lock);
+ vfsmount_write_lock();
mnt->mnt_flags &= ~MNT_READONLY;
- spin_unlock(&vfsmount_lock);
+ vfsmount_write_unlock();
}
void simple_set_mnt(struct vfsmount *mnt, struct super_block *sb)
@@ -439,10 +491,11 @@ struct vfsmount *__lookup_mnt(struct vfs
struct vfsmount *lookup_mnt(struct path *path)
{
struct vfsmount *child_mnt;
- spin_lock(&vfsmount_lock);
+
+ vfsmount_read_lock();
if ((child_mnt = __lookup_mnt(path->mnt, path->dentry, 1)))
mntget(child_mnt);
- spin_unlock(&vfsmount_lock);
+ vfsmount_read_unlock();
return child_mnt;
}
@@ -618,40 +671,47 @@ static inline void __mntput(struct vfsmo
void mntput_no_expire(struct vfsmount *mnt)
{
repeat:
- if (atomic_dec_and_lock(&mnt->mnt_count, &vfsmount_lock)) {
- if (likely(!mnt->mnt_pinned)) {
- spin_unlock(&vfsmount_lock);
- __mntput(mnt);
- return;
- }
- atomic_add(mnt->mnt_pinned + 1, &mnt->mnt_count);
- mnt->mnt_pinned = 0;
- spin_unlock(&vfsmount_lock);
- acct_auto_close_mnt(mnt);
- security_sb_umount_close(mnt);
- goto repeat;
+ /* open-code atomic_dec_and_lock for the vfsmount lock */
+ if (atomic_add_unless(&mnt->mnt_count, -1, 1))
+ return;
+ vfsmount_write_lock();
+ if (!atomic_dec_and_test(&mnt->mnt_count)) {
+ vfsmount_write_unlock();
+ return;
}
+
+ if (likely(!mnt->mnt_pinned)) {
+ vfsmount_write_unlock();
+ __mntput(mnt);
+ return;
+ }
+ atomic_add(mnt->mnt_pinned + 1, &mnt->mnt_count);
+ mnt->mnt_pinned = 0;
+ vfsmount_write_unlock();
+ acct_auto_close_mnt(mnt);
+ security_sb_umount_close(mnt);
+ goto repeat;
}
EXPORT_SYMBOL(mntput_no_expire);
void mnt_pin(struct vfsmount *mnt)
{
- spin_lock(&vfsmount_lock);
+ vfsmount_write_lock();
mnt->mnt_pinned++;
- spin_unlock(&vfsmount_lock);
+ vfsmount_write_unlock();
}
EXPORT_SYMBOL(mnt_pin);
void mnt_unpin(struct vfsmount *mnt)
{
- spin_lock(&vfsmount_lock);
+ vfsmount_write_lock();
if (mnt->mnt_pinned) {
atomic_inc(&mnt->mnt_count);
mnt->mnt_pinned--;
}
- spin_unlock(&vfsmount_lock);
+ vfsmount_write_unlock();
}
EXPORT_SYMBOL(mnt_unpin);
@@ -934,12 +994,12 @@ int may_umount_tree(struct vfsmount *mnt
int minimum_refs = 0;
struct vfsmount *p;
- spin_lock(&vfsmount_lock);
+ vfsmount_read_lock();
for (p = mnt; p; p = next_mnt(p, mnt)) {
actual_refs += atomic_read(&p->mnt_count);
minimum_refs += 2;
}
- spin_unlock(&vfsmount_lock);
+ vfsmount_read_unlock();
if (actual_refs > minimum_refs)
return 0;
@@ -965,10 +1025,12 @@ EXPORT_SYMBOL(may_umount_tree);
int may_umount(struct vfsmount *mnt)
{
int ret = 1;
- spin_lock(&vfsmount_lock);
+
+ vfsmount_read_lock();
if (propagate_mount_busy(mnt, 2))
ret = 0;
- spin_unlock(&vfsmount_lock);
+ vfsmount_read_unlock();
+
return ret;
}
@@ -983,13 +1045,14 @@ void release_mounts(struct list_head *he
if (mnt->mnt_parent != mnt) {
struct dentry *dentry;
struct vfsmount *m;
- spin_lock(&vfsmount_lock);
+
+ vfsmount_write_lock();
dentry = mnt->mnt_mountpoint;
m = mnt->mnt_parent;
mnt->mnt_mountpoint = mnt->mnt_root;
mnt->mnt_parent = mnt;
m->mnt_ghosts--;
- spin_unlock(&vfsmount_lock);
+ vfsmount_write_unlock();
dput(dentry);
mntput(m);
}
@@ -1087,7 +1150,7 @@ static int do_umount(struct vfsmount *mn
}
down_write(&namespace_sem);
- spin_lock(&vfsmount_lock);
+ vfsmount_write_lock();
event++;
if (!(flags & MNT_DETACH))
@@ -1099,7 +1162,7 @@ static int do_umount(struct vfsmount *mn
umount_tree(mnt, 1, &umount_list);
retval = 0;
}
- spin_unlock(&vfsmount_lock);
+ vfsmount_write_unlock();
if (retval)
security_sb_umount_busy(mnt);
up_write(&namespace_sem);
@@ -1206,19 +1269,19 @@ struct vfsmount *copy_tree(struct vfsmou
q = clone_mnt(p, p->mnt_root, flag);
if (!q)
goto Enomem;
- spin_lock(&vfsmount_lock);
+ vfsmount_write_lock();
list_add_tail(&q->mnt_list, &res->mnt_list);
attach_mnt(q, &path);
- spin_unlock(&vfsmount_lock);
+ vfsmount_write_unlock();
}
}
return res;
Enomem:
if (res) {
LIST_HEAD(umount_list);
- spin_lock(&vfsmount_lock);
+ vfsmount_write_lock();
umount_tree(res, 0, &umount_list);
- spin_unlock(&vfsmount_lock);
+ vfsmount_write_unlock();
release_mounts(&umount_list);
}
return NULL;
@@ -1237,9 +1300,9 @@ void drop_collected_mounts(struct vfsmou
{
LIST_HEAD(umount_list);
down_write(&namespace_sem);
- spin_lock(&vfsmount_lock);
+ vfsmount_write_lock();
umount_tree(mnt, 0, &umount_list);
- spin_unlock(&vfsmount_lock);
+ vfsmount_write_unlock();
up_write(&namespace_sem);
release_mounts(&umount_list);
}
@@ -1357,7 +1420,7 @@ static int attach_recursive_mnt(struct v
set_mnt_shared(p);
}
- spin_lock(&vfsmount_lock);
+ vfsmount_write_lock();
if (parent_path) {
detach_mnt(source_mnt, parent_path);
attach_mnt(source_mnt, path);
@@ -1371,7 +1434,8 @@ static int attach_recursive_mnt(struct v
list_del_init(&child->mnt_hash);
commit_tree(child);
}
- spin_unlock(&vfsmount_lock);
+ vfsmount_write_unlock();
+
return 0;
out_cleanup_ids:
@@ -1433,10 +1497,10 @@ static int do_change_type(struct path *p
goto out_unlock;
}
- spin_lock(&vfsmount_lock);
+ vfsmount_write_lock();
for (m = mnt; m; m = (recurse ? next_mnt(m, mnt) : NULL))
change_mnt_propagation(m, type);
- spin_unlock(&vfsmount_lock);
+ vfsmount_write_unlock();
out_unlock:
up_write(&namespace_sem);
@@ -1480,9 +1544,10 @@ static int do_loopback(struct path *path
err = graft_tree(mnt, path);
if (err) {
LIST_HEAD(umount_list);
- spin_lock(&vfsmount_lock);
+
+ vfsmount_write_lock();
umount_tree(mnt, 0, &umount_list);
- spin_unlock(&vfsmount_lock);
+ vfsmount_write_unlock();
release_mounts(&umount_list);
}
@@ -1540,9 +1605,9 @@ static int do_remount(struct path *path,
if (!err) {
security_sb_post_remount(path->mnt, flags, data);
- spin_lock(&vfsmount_lock);
+ vfsmount_write_lock();
touch_mnt_namespace(path->mnt->mnt_ns);
- spin_unlock(&vfsmount_lock);
+ vfsmount_write_unlock();
}
return err;
}
@@ -1717,7 +1782,7 @@ void mark_mounts_for_expiry(struct list_
return;
down_write(&namespace_sem);
- spin_lock(&vfsmount_lock);
+ vfsmount_write_lock();
/* extract from the expiration list every vfsmount that matches the
* following criteria:
@@ -1736,7 +1801,7 @@ void mark_mounts_for_expiry(struct list_
touch_mnt_namespace(mnt->mnt_ns);
umount_tree(mnt, 1, &umounts);
}
- spin_unlock(&vfsmount_lock);
+ vfsmount_write_unlock();
up_write(&namespace_sem);
release_mounts(&umounts);
@@ -1996,9 +2061,9 @@ static struct mnt_namespace *dup_mnt_ns(
kfree(new_ns);
return ERR_PTR(-ENOMEM);
}
- spin_lock(&vfsmount_lock);
+ vfsmount_write_lock();
list_add_tail(&new_ns->list, &new_ns->root->mnt_list);
- spin_unlock(&vfsmount_lock);
+ vfsmount_write_unlock();
/*
* Second pass: switch the tsk->fs->* elements and mark new vfsmounts
@@ -2193,7 +2258,7 @@ SYSCALL_DEFINE2(pivot_root, const char _
goto out2; /* not attached */
/* make sure we can reach put_old from new_root */
tmp = old.mnt;
- spin_lock(&vfsmount_lock);
+ vfsmount_write_lock();
if (tmp != new.mnt) {
for (;;) {
if (tmp->mnt_parent == tmp)
@@ -2213,7 +2278,7 @@ SYSCALL_DEFINE2(pivot_root, const char _
/* mount new_root on / */
attach_mnt(new.mnt, &root_parent);
touch_mnt_namespace(current->nsproxy->mnt_ns);
- spin_unlock(&vfsmount_lock);
+ vfsmount_write_unlock();
chroot_fs_refs(&root, &new);
security_sb_post_pivotroot(&root, &new);
error = 0;
@@ -2229,7 +2294,7 @@ out1:
out0:
return error;
out3:
- spin_unlock(&vfsmount_lock);
+ vfsmount_write_unlock();
goto out2;
}
@@ -2259,6 +2324,7 @@ static void __init init_mount_tree(void)
void __init mnt_init(void)
{
unsigned u;
+ int i;
int err;
init_rwsem(&namespace_sem);
@@ -2276,6 +2342,9 @@ void __init mnt_init(void)
for (u = 0; u < HASH_SIZE; u++)
INIT_LIST_HEAD(&mount_hashtable[u]);
+ for_each_possible_cpu(i)
+ spin_lock_init(&per_cpu(vfsmount_lock, i));
+
err = sysfs_init();
if (err)
printk(KERN_WARNING "%s: sysfs_init error: %d\n",
@@ -2291,16 +2360,22 @@ void put_mnt_ns(struct mnt_namespace *ns
{
struct vfsmount *root;
LIST_HEAD(umount_list);
+ spinlock_t *lock;
- if (!atomic_dec_and_lock(&ns->count, &vfsmount_lock))
+ lock = &get_cpu_var(vfsmount_lock);
+ if (!atomic_dec_and_lock(&ns->count, lock)) {
+ put_cpu_var(vfsmount_lock);
return;
+ }
root = ns->root;
ns->root = NULL;
- spin_unlock(&vfsmount_lock);
+ spin_unlock(lock);
+ put_cpu_var(vfsmount_lock);
+
down_write(&namespace_sem);
- spin_lock(&vfsmount_lock);
- umount_tree(root, 0, &umount_list);
- spin_unlock(&vfsmount_lock);
+ vfsmount_write_lock();
+ umount_tree(root, 0, &umount_list);
+ vfsmount_write_unlock();
up_write(&namespace_sem);
release_mounts(&umount_list);
kfree(ns);
Index: linux-2.6/fs/pnode.c
===================================================================
--- linux-2.6.orig/fs/pnode.c
+++ linux-2.6/fs/pnode.c
@@ -264,12 +264,12 @@ int propagate_mnt(struct vfsmount *dest_
prev_src_mnt = child;
}
out:
- spin_lock(&vfsmount_lock);
+ vfsmount_write_lock();
while (!list_empty(&tmp_list)) {
child = list_first_entry(&tmp_list, struct vfsmount, mnt_hash);
umount_tree(child, 0, &umount_list);
}
- spin_unlock(&vfsmount_lock);
+ vfsmount_write_unlock();
release_mounts(&umount_list);
return ret;
}
Index: linux-2.6/fs/proc/base.c
===================================================================
--- linux-2.6.orig/fs/proc/base.c
+++ linux-2.6/fs/proc/base.c
@@ -652,12 +652,12 @@ static unsigned mounts_poll(struct file
poll_wait(file, &ns->poll, wait);
- spin_lock(&vfsmount_lock);
+ vfsmount_read_lock();
if (p->event != ns->event) {
p->event = ns->event;
res |= POLLERR | POLLPRI;
}
- spin_unlock(&vfsmount_lock);
+ vfsmount_read_unlock();
return res;
}
Index: linux-2.6/include/linux/mount.h
===================================================================
--- linux-2.6.orig/include/linux/mount.h
+++ linux-2.6/include/linux/mount.h
@@ -90,6 +90,11 @@ static inline struct vfsmount *mntget(st
struct file; /* forward dec */
+extern void vfsmount_read_lock(void);
+extern void vfsmount_read_unlock(void);
+extern void vfsmount_write_lock(void);
+extern void vfsmount_write_unlock(void);
+
extern int mnt_want_write(struct vfsmount *mnt);
extern int mnt_want_write_file(struct file *file);
extern int mnt_clone_write(struct vfsmount *mnt);
@@ -123,7 +128,6 @@ extern int do_add_mount(struct vfsmount
extern void mark_mounts_for_expiry(struct list_head *mounts);
-extern spinlock_t vfsmount_lock;
extern dev_t name_to_dev_t(char *name);
#endif /* _LINUX_MOUNT_H */
Index: linux-2.6/kernel/audit_tree.c
===================================================================
--- linux-2.6.orig/kernel/audit_tree.c
+++ linux-2.6/kernel/audit_tree.c
@@ -758,15 +758,15 @@ int audit_tag_tree(char *old, char *new)
continue;
}
- spin_lock(&vfsmount_lock);
+ vfsmount_read_lock();
if (!is_under(mnt, dentry, &path)) {
- spin_unlock(&vfsmount_lock);
+ vfsmount_read_unlock();
path_put(&path);
put_tree(tree);
mutex_lock(&audit_filter_mutex);
continue;
}
- spin_unlock(&vfsmount_lock);
+ vfsmount_read_unlock();
path_put(&path);
list_for_each_entry(p, &list, mnt_list) {
Index: linux-2.6/security/tomoyo/realpath.c
===================================================================
--- linux-2.6.orig/security/tomoyo/realpath.c
+++ linux-2.6/security/tomoyo/realpath.c
@@ -96,12 +96,12 @@ int tomoyo_realpath_from_path2(struct pa
root = current->fs->root;
path_get(&root);
read_unlock(¤t->fs->lock);
- spin_lock(&vfsmount_lock);
+ vfsmount_read_lock();
if (root.mnt && root.mnt->mnt_ns)
ns_root.mnt = mntget(root.mnt->mnt_ns->root);
if (ns_root.mnt)
ns_root.dentry = dget(ns_root.mnt->mnt_root);
- spin_unlock(&vfsmount_lock);
+ vfsmount_read_unlock();
spin_lock(&dcache_lock);
tmp = ns_root;
sp = __d_path(path, &tmp, newname, newname_len);
^ permalink raw reply [flat|nested] 64+ messages in thread
* [patch 05/33] fs: scale mntget/mntput
2009-09-04 6:51 [patch 00/33] my current vfs scalability patch queue npiggin
` (3 preceding siblings ...)
2009-09-04 6:51 ` [patch 04/33] fs: brlock vfsmount_lock npiggin
@ 2009-09-04 6:51 ` npiggin
2009-09-07 9:41 ` Nick Piggin
2009-09-04 6:51 ` [patch 06/33] fs: dcache scale hash npiggin
` (28 subsequent siblings)
33 siblings, 1 reply; 64+ messages in thread
From: npiggin @ 2009-09-04 6:51 UTC (permalink / raw)
To: linux-fsdevel; +Cc: linux-kernel
[-- Attachment #1: fs-mntget-scale.patch --]
[-- Type: text/plain, Size: 9164 bytes --]
Improve scalability of mntget/mntput by using per-cpu counters protected
by the reader side of the brlock vfsmount_lock. mnt_mounted keeps track of
whether the vfsmount is actually attached to the tree so we can shortcut
expensive checks in mntput.
---
fs/libfs.c | 1
fs/namespace.c | 122 +++++++++++++++++++++++++++++++++++++++++++-------
fs/pnode.c | 2
include/linux/mount.h | 33 ++++---------
4 files changed, 121 insertions(+), 37 deletions(-)
Index: linux-2.6/fs/namespace.c
===================================================================
--- linux-2.6.orig/fs/namespace.c
+++ linux-2.6/fs/namespace.c
@@ -177,6 +177,49 @@ void mnt_release_group_id(struct vfsmoun
mnt->mnt_group_id = 0;
}
+static inline void add_mnt_count(struct vfsmount *mnt, int n)
+{
+#ifdef CONFIG_SMP
+ (*per_cpu_ptr(mnt->mnt_count, smp_processor_id())) += n;
+#else
+ mnt->mnt_count += n;
+#endif
+}
+
+static inline void inc_mnt_count(struct vfsmount *mnt)
+{
+#ifdef CONFIG_SMP
+ (*per_cpu_ptr(mnt->mnt_count, smp_processor_id()))++;
+#else
+ mnt->mnt_count++;
+#endif
+}
+
+static inline void dec_mnt_count(struct vfsmount *mnt)
+{
+#ifdef CONFIG_SMP
+ (*per_cpu_ptr(mnt->mnt_count, smp_processor_id()))--;
+#else
+ mnt->mnt_count--;
+#endif
+}
+
+unsigned int count_mnt_count(struct vfsmount *mnt)
+{
+#ifdef CONFIG_SMP
+ unsigned int count = 0;
+ int cpu;
+
+ for_each_possible_cpu(cpu) {
+ count += *per_cpu_ptr(mnt->mnt_count, cpu);
+ }
+
+ return count;
+#else
+ return mnt->mnt_count;
+#endif
+}
+
struct vfsmount *alloc_vfsmnt(const char *name)
{
struct vfsmount *mnt = kmem_cache_zalloc(mnt_cache, GFP_KERNEL);
@@ -193,7 +236,13 @@ struct vfsmount *alloc_vfsmnt(const char
goto out_free_id;
}
- atomic_set(&mnt->mnt_count, 1);
+#ifdef CONFIG_SMP
+ mnt->mnt_count = alloc_percpu(int);
+ if (!mnt->mnt_count)
+ goto out_free_devname;
+#else
+ mnt->mnt_count = 0;
+#endif
INIT_LIST_HEAD(&mnt->mnt_hash);
INIT_LIST_HEAD(&mnt->mnt_child);
INIT_LIST_HEAD(&mnt->mnt_mounts);
@@ -205,14 +254,19 @@ struct vfsmount *alloc_vfsmnt(const char
#ifdef CONFIG_SMP
mnt->mnt_writers = alloc_percpu(int);
if (!mnt->mnt_writers)
- goto out_free_devname;
+ goto out_free_mntcount;
#else
mnt->mnt_writers = 0;
#endif
+ preempt_disable();
+ inc_mnt_count(mnt);
+ preempt_enable();
}
return mnt;
#ifdef CONFIG_SMP
+out_free_mntcount:
+ free_percpu(mnt->mnt_count);
out_free_devname:
kfree(mnt->mnt_devname);
#endif
@@ -526,9 +580,11 @@ static void detach_mnt(struct vfsmount *
old_path->mnt = mnt->mnt_parent;
mnt->mnt_parent = mnt;
mnt->mnt_mountpoint = mnt->mnt_root;
- list_del_init(&mnt->mnt_child);
list_del_init(&mnt->mnt_hash);
+ list_del_init(&mnt->mnt_child);
old_path->dentry->d_mounted--;
+ WARN_ON(mnt->mnt_mounted != 1);
+ mnt->mnt_mounted--;
}
void mnt_set_mountpoint(struct vfsmount *mnt, struct dentry *dentry,
@@ -545,6 +601,8 @@ static void attach_mnt(struct vfsmount *
list_add_tail(&mnt->mnt_hash, mount_hashtable +
hash(path->mnt, path->dentry));
list_add_tail(&mnt->mnt_child, &path->mnt->mnt_mounts);
+ WARN_ON(mnt->mnt_mounted != 0);
+ mnt->mnt_mounted++;
}
/*
@@ -567,6 +625,8 @@ static void commit_tree(struct vfsmount
list_add_tail(&mnt->mnt_hash, mount_hashtable +
hash(parent, mnt->mnt_mountpoint));
list_add_tail(&mnt->mnt_child, &parent->mnt_mounts);
+ WARN_ON(mnt->mnt_mounted != 0);
+ mnt->mnt_mounted++;
touch_mnt_namespace(n);
}
@@ -670,50 +730,80 @@ static inline void __mntput(struct vfsmo
void mntput_no_expire(struct vfsmount *mnt)
{
-repeat:
- /* open-code atomic_dec_and_lock for the vfsmount lock */
- if (atomic_add_unless(&mnt->mnt_count, -1, 1))
+ if (likely(mnt->mnt_mounted)) {
+ vfsmount_read_lock();
+ if (unlikely(!mnt->mnt_mounted)) {
+ vfsmount_read_unlock();
+ goto repeat;
+ }
+ dec_mnt_count(mnt);
+ BUG_ON(count_mnt_count(mnt) == 0);
+ vfsmount_read_unlock();
+
return;
+ }
+
+repeat:
vfsmount_write_lock();
- if (!atomic_dec_and_test(&mnt->mnt_count)) {
+ BUG_ON(mnt->mnt_mounted);
+ dec_mnt_count(mnt);
+ if (count_mnt_count(mnt)) {
vfsmount_write_unlock();
return;
}
-
if (likely(!mnt->mnt_pinned)) {
vfsmount_write_unlock();
__mntput(mnt);
return;
}
- atomic_add(mnt->mnt_pinned + 1, &mnt->mnt_count);
+ add_mnt_count(mnt, mnt->mnt_pinned + 1);
mnt->mnt_pinned = 0;
vfsmount_write_unlock();
acct_auto_close_mnt(mnt);
security_sb_umount_close(mnt);
goto repeat;
}
-
EXPORT_SYMBOL(mntput_no_expire);
+void mntput(struct vfsmount *mnt)
+{
+ if (mnt) {
+ /* avoid cacheline pingpong */
+ if (unlikely(mnt->mnt_expiry_mark))
+ mnt->mnt_expiry_mark = 0;
+ mntput_no_expire(mnt);
+ }
+}
+EXPORT_SYMBOL(mntput);
+
+struct vfsmount *mntget(struct vfsmount *mnt)
+{
+ if (mnt) {
+ preempt_disable();
+ inc_mnt_count(mnt);
+ preempt_enable();
+ }
+ return mnt;
+}
+EXPORT_SYMBOL(mntget);
+
void mnt_pin(struct vfsmount *mnt)
{
vfsmount_write_lock();
mnt->mnt_pinned++;
vfsmount_write_unlock();
}
-
EXPORT_SYMBOL(mnt_pin);
void mnt_unpin(struct vfsmount *mnt)
{
vfsmount_write_lock();
if (mnt->mnt_pinned) {
- atomic_inc(&mnt->mnt_count);
+ inc_mnt_count(mnt);
mnt->mnt_pinned--;
}
vfsmount_write_unlock();
}
-
EXPORT_SYMBOL(mnt_unpin);
static inline void mangle(struct seq_file *m, const char *s)
@@ -996,7 +1086,7 @@ int may_umount_tree(struct vfsmount *mnt
vfsmount_read_lock();
for (p = mnt; p; p = next_mnt(p, mnt)) {
- actual_refs += atomic_read(&p->mnt_count);
+ actual_refs += count_mnt_count(p);
minimum_refs += 2;
}
vfsmount_read_unlock();
@@ -1076,6 +1166,8 @@ void umount_tree(struct vfsmount *mnt, i
__touch_mnt_namespace(p->mnt_ns);
p->mnt_ns = NULL;
list_del_init(&p->mnt_child);
+ WARN_ON(p->mnt_mounted != 1);
+ p->mnt_mounted--;
if (p->mnt_parent != p) {
p->mnt_parent->mnt_ghosts++;
p->mnt_mountpoint->d_mounted--;
@@ -1107,7 +1199,7 @@ static int do_umount(struct vfsmount *mn
flags & (MNT_FORCE | MNT_DETACH))
return -EINVAL;
- if (atomic_read(&mnt->mnt_count) != 2)
+ if (count_mnt_count(mnt) != 2)
return -EBUSY;
if (!xchg(&mnt->mnt_expiry_mark, 1))
Index: linux-2.6/include/linux/mount.h
===================================================================
--- linux-2.6.orig/include/linux/mount.h
+++ linux-2.6/include/linux/mount.h
@@ -56,20 +56,20 @@ struct vfsmount {
struct mnt_namespace *mnt_ns; /* containing namespace */
int mnt_id; /* mount identifier */
int mnt_group_id; /* peer group identifier */
- /*
- * We put mnt_count & mnt_expiry_mark at the end of struct vfsmount
- * to let these frequently modified fields in a separate cache line
- * (so that reads of mnt_flags wont ping-pong on SMP machines)
- */
- atomic_t mnt_count;
int mnt_expiry_mark; /* true if marked for expiry */
int mnt_pinned;
int mnt_ghosts;
+ int mnt_mounted;
#ifdef CONFIG_SMP
int *mnt_writers;
#else
int mnt_writers;
#endif
+#ifdef CONFIG_SMP
+ int *mnt_count;
+#else
+ int mnt_count;
+#endif
};
static inline int *get_mnt_writers_ptr(struct vfsmount *mnt)
@@ -81,13 +81,6 @@ static inline int *get_mnt_writers_ptr(s
#endif
}
-static inline struct vfsmount *mntget(struct vfsmount *mnt)
-{
- if (mnt)
- atomic_inc(&mnt->mnt_count);
- return mnt;
-}
-
struct file; /* forward dec */
extern void vfsmount_read_lock(void);
@@ -95,23 +88,21 @@ extern void vfsmount_read_unlock(void);
extern void vfsmount_write_lock(void);
extern void vfsmount_write_unlock(void);
+extern unsigned int count_mnt_count(struct vfsmount *mnt);
+
extern int mnt_want_write(struct vfsmount *mnt);
extern int mnt_want_write_file(struct file *file);
extern int mnt_clone_write(struct vfsmount *mnt);
extern void mnt_drop_write(struct vfsmount *mnt);
+
extern void mntput_no_expire(struct vfsmount *mnt);
+extern struct vfsmount *mntget(struct vfsmount *mnt);
+extern void mntput(struct vfsmount *mnt);
+
extern void mnt_pin(struct vfsmount *mnt);
extern void mnt_unpin(struct vfsmount *mnt);
extern int __mnt_is_readonly(struct vfsmount *mnt);
-static inline void mntput(struct vfsmount *mnt)
-{
- if (mnt) {
- mnt->mnt_expiry_mark = 0;
- mntput_no_expire(mnt);
- }
-}
-
extern struct vfsmount *do_kern_mount(const char *fstype, int flags,
const char *name, void *data);
Index: linux-2.6/fs/pnode.c
===================================================================
--- linux-2.6.orig/fs/pnode.c
+++ linux-2.6/fs/pnode.c
@@ -279,7 +279,7 @@ out:
*/
static inline int do_refcount_check(struct vfsmount *mnt, int count)
{
- int mycount = atomic_read(&mnt->mnt_count) - mnt->mnt_ghosts;
+ int mycount = count_mnt_count(mnt) - mnt->mnt_ghosts;
return (mycount > count);
}
Index: linux-2.6/fs/libfs.c
===================================================================
--- linux-2.6.orig/fs/libfs.c
+++ linux-2.6/fs/libfs.c
@@ -244,6 +244,7 @@ int get_sb_pseudo(struct file_system_typ
d_instantiate(dentry, root);
s->s_root = dentry;
s->s_flags |= MS_ACTIVE;
+ mnt->mnt_mounted++; /* never unmounted, shortcut mntget (XXX: OK?) */
simple_set_mnt(mnt, s);
return 0;
^ permalink raw reply [flat|nested] 64+ messages in thread
* [patch 06/33] fs: dcache scale hash
2009-09-04 6:51 [patch 00/33] my current vfs scalability patch queue npiggin
` (4 preceding siblings ...)
2009-09-04 6:51 ` [patch 05/33] fs: scale mntget/mntput npiggin
@ 2009-09-04 6:51 ` npiggin
2009-09-04 6:51 ` [patch 07/33] fs: dcache scale lru npiggin
` (27 subsequent siblings)
33 siblings, 0 replies; 64+ messages in thread
From: npiggin @ 2009-09-04 6:51 UTC (permalink / raw)
To: linux-fsdevel; +Cc: linux-kernel
[-- Attachment #1: fs-dcache-scale-d_hash.patch --]
[-- Type: text/plain, Size: 3840 bytes --]
Add a new lock, dcache_hash_lock, to protect the dcache hash table from
concurrent modification. d_hash is also protected by d_lock.
---
fs/dcache.c | 35 ++++++++++++++++++++++++-----------
include/linux/dcache.h | 3 +++
2 files changed, 27 insertions(+), 11 deletions(-)
Index: linux-2.6/fs/dcache.c
===================================================================
--- linux-2.6.orig/fs/dcache.c
+++ linux-2.6/fs/dcache.c
@@ -34,12 +34,23 @@
#include <linux/fs_struct.h>
#include "internal.h"
+/*
+ * Usage:
+ * dcache_hash_lock protects dcache hash table
+ *
+ * Ordering:
+ * dcache_lock
+ * dentry->d_lock
+ * dcache_hash_lock
+ */
int sysctl_vfs_cache_pressure __read_mostly = 100;
EXPORT_SYMBOL_GPL(sysctl_vfs_cache_pressure);
- __cacheline_aligned_in_smp DEFINE_SPINLOCK(dcache_lock);
+__cacheline_aligned_in_smp DEFINE_SPINLOCK(dcache_hash_lock);
+__cacheline_aligned_in_smp DEFINE_SPINLOCK(dcache_lock);
__cacheline_aligned_in_smp DEFINE_SEQLOCK(rename_lock);
+EXPORT_SYMBOL(dcache_hash_lock);
EXPORT_SYMBOL(dcache_lock);
static struct kmem_cache *dentry_cache __read_mostly;
@@ -1466,17 +1477,20 @@ int d_validate(struct dentry *dentry, st
goto out;
spin_lock(&dcache_lock);
+ spin_lock(&dcache_hash_lock);
base = d_hash(dparent, dentry->d_name.hash);
hlist_for_each(lhp,base) {
/* hlist_for_each_entry_rcu() not required for d_hash list
* as it is parsed under dcache_lock
*/
if (dentry == hlist_entry(lhp, struct dentry, d_hash)) {
+ spin_unlock(&dcache_hash_lock);
__dget_locked(dentry);
spin_unlock(&dcache_lock);
return 1;
}
}
+ spin_unlock(&dcache_hash_lock);
spin_unlock(&dcache_lock);
out:
return 0;
@@ -1550,7 +1564,9 @@ void d_rehash(struct dentry * entry)
{
spin_lock(&dcache_lock);
spin_lock(&entry->d_lock);
+ spin_lock(&dcache_hash_lock);
_d_rehash(entry);
+ spin_unlock(&dcache_hash_lock);
spin_unlock(&entry->d_lock);
spin_unlock(&dcache_lock);
}
@@ -1629,8 +1645,6 @@ static void switch_names(struct dentry *
*/
static void d_move_locked(struct dentry * dentry, struct dentry * target)
{
- struct hlist_head *list;
-
if (!dentry->d_inode)
printk(KERN_WARNING "VFS: moving negative dcache entry\n");
@@ -1647,14 +1661,11 @@ static void d_move_locked(struct dentry
}
/* Move the dentry to the target hash queue, if on different bucket */
- if (d_unhashed(dentry))
- goto already_unhashed;
-
- hlist_del_rcu(&dentry->d_hash);
-
-already_unhashed:
- list = d_hash(target->d_parent, target->d_name.hash);
- __d_rehash(dentry, list);
+ spin_lock(&dcache_hash_lock);
+ if (!d_unhashed(dentry))
+ hlist_del_rcu(&dentry->d_hash);
+ __d_rehash(dentry, d_hash(target->d_parent, target->d_name.hash));
+ spin_unlock(&dcache_hash_lock);
/* Unhash the target: dput() will then get rid of it */
__d_drop(target);
@@ -1850,7 +1861,9 @@ struct dentry *d_materialise_unique(stru
found_lock:
spin_lock(&actual->d_lock);
found:
+ spin_lock(&dcache_hash_lock);
_d_rehash(actual);
+ spin_unlock(&dcache_hash_lock);
spin_unlock(&actual->d_lock);
spin_unlock(&dcache_lock);
out_nolock:
Index: linux-2.6/include/linux/dcache.h
===================================================================
--- linux-2.6.orig/include/linux/dcache.h
+++ linux-2.6/include/linux/dcache.h
@@ -186,6 +186,7 @@ d_iput: no no no yes
#define DCACHE_FSNOTIFY_PARENT_WATCHED 0x0080 /* Parent inode is watched by some fsnotify listener */
+extern spinlock_t dcache_hash_lock;
extern spinlock_t dcache_lock;
extern seqlock_t rename_lock;
@@ -209,7 +210,9 @@ static inline void __d_drop(struct dentr
{
if (!(dentry->d_flags & DCACHE_UNHASHED)) {
dentry->d_flags |= DCACHE_UNHASHED;
+ spin_lock(&dcache_hash_lock);
hlist_del_rcu(&dentry->d_hash);
+ spin_unlock(&dcache_hash_lock);
}
}
^ permalink raw reply [flat|nested] 64+ messages in thread
* [patch 07/33] fs: dcache scale lru
2009-09-04 6:51 [patch 00/33] my current vfs scalability patch queue npiggin
` (5 preceding siblings ...)
2009-09-04 6:51 ` [patch 06/33] fs: dcache scale hash npiggin
@ 2009-09-04 6:51 ` npiggin
2009-09-04 6:51 ` [patch 08/33] fs: dcache scale nr_dentry npiggin
` (26 subsequent siblings)
33 siblings, 0 replies; 64+ messages in thread
From: npiggin @ 2009-09-04 6:51 UTC (permalink / raw)
To: linux-fsdevel; +Cc: linux-kernel
[-- Attachment #1: fs-dcache-scale-d_lru.patch --]
[-- Type: text/plain, Size: 7884 bytes --]
Add a new lock, dcache_lru_lock, to protect the dcache hash table from
concurrent modification. d_lru is also protected by d_lock.
Move lru scanning out from underneath dcache_lock.
---
fs/dcache.c | 105 ++++++++++++++++++++++++++++++++++++++++++++++++------------
1 file changed, 85 insertions(+), 20 deletions(-)
Index: linux-2.6/fs/dcache.c
===================================================================
--- linux-2.6.orig/fs/dcache.c
+++ linux-2.6/fs/dcache.c
@@ -36,17 +36,26 @@
/*
* Usage:
- * dcache_hash_lock protects dcache hash table
+ * dcache_hash_lock protects:
+ * - the dcache hash table
+ * dcache_lru_lock protects:
+ * - the dcache lru lists and counters
+ * d_lock protects:
+ * - d_flags
+ * - d_name
+ * - d_lru
*
* Ordering:
* dcache_lock
* dentry->d_lock
+ * dcache_lru_lock
* dcache_hash_lock
*/
int sysctl_vfs_cache_pressure __read_mostly = 100;
EXPORT_SYMBOL_GPL(sysctl_vfs_cache_pressure);
__cacheline_aligned_in_smp DEFINE_SPINLOCK(dcache_hash_lock);
+static __cacheline_aligned_in_smp DEFINE_SPINLOCK(dcache_lru_lock);
__cacheline_aligned_in_smp DEFINE_SPINLOCK(dcache_lock);
__cacheline_aligned_in_smp DEFINE_SEQLOCK(rename_lock);
@@ -133,37 +142,56 @@ static void dentry_iput(struct dentry *
}
/*
- * dentry_lru_(add|add_tail|del|del_init) must be called with dcache_lock held.
+ * dentry_lru_(add|add_tail|del|del_init) must be called with d_lock held
+ * to protect list_empty(d_lru) condition.
*/
static void dentry_lru_add(struct dentry *dentry)
{
+ spin_lock(&dcache_lru_lock);
list_add(&dentry->d_lru, &dentry->d_sb->s_dentry_lru);
dentry->d_sb->s_nr_dentry_unused++;
dentry_stat.nr_unused++;
+ spin_unlock(&dcache_lru_lock);
}
static void dentry_lru_add_tail(struct dentry *dentry)
{
+ spin_lock(&dcache_lru_lock);
list_add_tail(&dentry->d_lru, &dentry->d_sb->s_dentry_lru);
dentry->d_sb->s_nr_dentry_unused++;
dentry_stat.nr_unused++;
+ spin_unlock(&dcache_lru_lock);
+}
+
+static void __dentry_lru_del(struct dentry *dentry)
+{
+ list_del(&dentry->d_lru);
+ dentry->d_sb->s_nr_dentry_unused--;
+ dentry_stat.nr_unused--;
+}
+
+static void __dentry_lru_del_init(struct dentry *dentry)
+{
+ list_del_init(&dentry->d_lru);
+ dentry->d_sb->s_nr_dentry_unused--;
+ dentry_stat.nr_unused--;
}
static void dentry_lru_del(struct dentry *dentry)
{
if (!list_empty(&dentry->d_lru)) {
- list_del(&dentry->d_lru);
- dentry->d_sb->s_nr_dentry_unused--;
- dentry_stat.nr_unused--;
+ spin_lock(&dcache_lru_lock);
+ __dentry_lru_del(dentry);
+ spin_unlock(&dcache_lru_lock);
}
}
static void dentry_lru_del_init(struct dentry *dentry)
{
if (likely(!list_empty(&dentry->d_lru))) {
- list_del_init(&dentry->d_lru);
- dentry->d_sb->s_nr_dentry_unused--;
- dentry_stat.nr_unused--;
+ spin_lock(&dcache_lru_lock);
+ __dentry_lru_del_init(dentry);
+ spin_unlock(&dcache_lru_lock);
}
}
@@ -174,6 +202,8 @@ static void dentry_lru_del_init(struct d
* The dentry must already be unhashed and removed from the LRU.
*
* If this is the root of the dentry tree, return NULL.
+ *
+ * dcache_lock and d_lock must be held by caller, are dropped by d_kill.
*/
static struct dentry *d_kill(struct dentry *dentry)
__releases(dentry->d_lock)
@@ -326,11 +356,19 @@ int d_invalidate(struct dentry * dentry)
}
/* This should be called _only_ with dcache_lock held */
+static inline struct dentry * __dget_locked_dlock(struct dentry *dentry)
+{
+ atomic_inc(&dentry->d_count);
+ dentry_lru_del_init(dentry);
+ return dentry;
+}
static inline struct dentry * __dget_locked(struct dentry *dentry)
{
atomic_inc(&dentry->d_count);
+ spin_lock(&dentry->d_lock);
dentry_lru_del_init(dentry);
+ spin_lock(&dentry->d_lock);
return dentry;
}
@@ -407,7 +445,7 @@ restart:
list_for_each_entry(dentry, &inode->i_dentry, d_alias) {
spin_lock(&dentry->d_lock);
if (!atomic_read(&dentry->d_count)) {
- __dget_locked(dentry);
+ __dget_locked_dlock(dentry);
__d_drop(dentry);
spin_unlock(&dentry->d_lock);
spin_unlock(&dcache_lock);
@@ -439,17 +477,18 @@ static void prune_one_dentry(struct dent
* Prune ancestors. Locking is simpler than in dput(),
* because dcache_lock needs to be taken anyway.
*/
- spin_lock(&dcache_lock);
while (dentry) {
- if (!atomic_dec_and_lock(&dentry->d_count, &dentry->d_lock))
+ spin_lock(&dcache_lock);
+ if (!atomic_dec_and_lock(&dentry->d_count, &dentry->d_lock)) {
+ spin_unlock(&dcache_lock);
return;
+ }
if (dentry->d_op && dentry->d_op->d_delete)
dentry->d_op->d_delete(dentry);
dentry_lru_del_init(dentry);
__d_drop(dentry);
dentry = d_kill(dentry);
- spin_lock(&dcache_lock);
}
}
@@ -470,10 +509,11 @@ static void __shrink_dcache_sb(struct su
BUG_ON(!sb);
BUG_ON((flags & DCACHE_REFERENCED) && count == NULL);
- spin_lock(&dcache_lock);
if (count != NULL)
/* called from prune_dcache() and shrink_dcache_parent() */
cnt = *count;
+relock:
+ spin_lock(&dcache_lru_lock);
restart:
if (count == NULL)
list_splice_init(&sb->s_dentry_lru, &tmp);
@@ -483,7 +523,10 @@ restart:
struct dentry, d_lru);
BUG_ON(dentry->d_sb != sb);
- spin_lock(&dentry->d_lock);
+ if (!spin_trylock(&dentry->d_lock)) {
+ spin_unlock(&dcache_lru_lock);
+ goto relock;
+ }
/*
* If we are honouring the DCACHE_REFERENCED flag and
* the dentry has this flag set, don't free it. Clear
@@ -501,13 +544,22 @@ restart:
if (!cnt)
break;
}
- cond_resched_lock(&dcache_lock);
+ cond_resched_lock(&dcache_lru_lock);
}
}
+ spin_unlock(&dcache_lru_lock);
+
+ spin_lock(&dcache_lock);
+again:
+ spin_lock(&dcache_lru_lock); /* lru_lock also protects tmp list */
while (!list_empty(&tmp)) {
dentry = list_entry(tmp.prev, struct dentry, d_lru);
- dentry_lru_del_init(dentry);
- spin_lock(&dentry->d_lock);
+
+ if (!spin_trylock(&dentry->d_lock)) {
+ spin_unlock(&dcache_lru_lock);
+ goto again;
+ }
+ __dentry_lru_del_init(dentry);
/*
* We found an inuse dentry which was not removed from
* the LRU because of laziness during lookup. Do not free
@@ -517,17 +569,22 @@ restart:
spin_unlock(&dentry->d_lock);
continue;
}
+
+ spin_unlock(&dcache_lru_lock);
prune_one_dentry(dentry);
- /* dentry->d_lock was dropped in prune_one_dentry() */
- cond_resched_lock(&dcache_lock);
+ /* dcache_lock and dentry->d_lock dropped */
+ spin_lock(&dcache_lock);
+ spin_lock(&dcache_lru_lock);
}
+ spin_unlock(&dcache_lock);
+
if (count == NULL && !list_empty(&sb->s_dentry_lru))
goto restart;
if (count != NULL)
*count = cnt;
if (!list_empty(&referenced))
list_splice(&referenced, &sb->s_dentry_lru);
- spin_unlock(&dcache_lock);
+ spin_unlock(&dcache_lru_lock);
}
/**
@@ -635,7 +692,9 @@ static void shrink_dcache_for_umount_sub
/* detach this root from the system */
spin_lock(&dcache_lock);
+ spin_lock(&dentry->d_lock);
dentry_lru_del_init(dentry);
+ spin_unlock(&dentry->d_lock);
__d_drop(dentry);
spin_unlock(&dcache_lock);
@@ -649,7 +708,9 @@ static void shrink_dcache_for_umount_sub
spin_lock(&dcache_lock);
list_for_each_entry(loop, &dentry->d_subdirs,
d_u.d_child) {
+ spin_lock(&loop->d_lock);
dentry_lru_del_init(loop);
+ spin_unlock(&loop->d_lock);
__d_drop(loop);
cond_resched_lock(&dcache_lock);
}
@@ -832,13 +893,17 @@ resume:
struct dentry *dentry = list_entry(tmp, struct dentry, d_u.d_child);
next = tmp->next;
+ spin_lock(&dentry->d_lock);
dentry_lru_del_init(dentry);
+ spin_unlock(&dentry->d_lock);
/*
* move only zero ref count dentries to the end
* of the unused list for prune_dcache
*/
if (!atomic_read(&dentry->d_count)) {
+ spin_lock(&dentry->d_lock);
dentry_lru_add_tail(dentry);
+ spin_unlock(&dentry->d_lock);
found++;
}
^ permalink raw reply [flat|nested] 64+ messages in thread
* [patch 08/33] fs: dcache scale nr_dentry
2009-09-04 6:51 [patch 00/33] my current vfs scalability patch queue npiggin
` (6 preceding siblings ...)
2009-09-04 6:51 ` [patch 07/33] fs: dcache scale lru npiggin
@ 2009-09-04 6:51 ` npiggin
2009-09-04 14:41 ` Daniel Walker
2009-09-04 6:51 ` [patch 09/33] fs: dcache scale dentry refcount npiggin
` (25 subsequent siblings)
33 siblings, 1 reply; 64+ messages in thread
From: npiggin @ 2009-09-04 6:51 UTC (permalink / raw)
To: linux-fsdevel; +Cc: linux-kernel
[-- Attachment #1: fs-dcache-scale-nr_dentry.patch --]
[-- Type: text/plain, Size: 3093 bytes --]
Make dentry_stat_t.nr_dentry an atomic_t type, and move it from under
dcache_lock.
---
fs/dcache.c | 20 +++++++++-----------
include/linux/dcache.h | 4 ++--
kernel/sysctl.c | 6 ++++++
3 files changed, 17 insertions(+), 13 deletions(-)
Index: linux-2.6/fs/dcache.c
===================================================================
--- linux-2.6.orig/fs/dcache.c
+++ linux-2.6/fs/dcache.c
@@ -83,6 +83,7 @@ static struct hlist_head *dentry_hashtab
/* Statistics gathering. */
struct dentry_stat_t dentry_stat = {
+ .nr_dentry = ATOMIC_INIT(0),
.age_limit = 45,
};
@@ -101,11 +102,11 @@ static void d_callback(struct rcu_head *
}
/*
- * no dcache_lock, please. The caller must decrement dentry_stat.nr_dentry
- * inside dcache_lock.
+ * no dcache_lock, please.
*/
static void d_free(struct dentry *dentry)
{
+ atomic_dec(&dentry_stat.nr_dentry);
if (dentry->d_op && dentry->d_op->d_release)
dentry->d_op->d_release(dentry);
/* if dentry was never inserted into hash, immediate free is OK */
@@ -212,7 +213,6 @@ static struct dentry *d_kill(struct dent
struct dentry *parent;
list_del(&dentry->d_u.d_child);
- dentry_stat.nr_dentry--; /* For d_free, below */
/*drops the locks, at that point nobody can reach this dentry */
dentry_iput(dentry);
if (IS_ROOT(dentry))
@@ -777,10 +777,7 @@ static void shrink_dcache_for_umount_sub
struct dentry, d_u.d_child);
}
out:
- /* several dentries were freed, need to correct nr_dentry */
- spin_lock(&dcache_lock);
- dentry_stat.nr_dentry -= detached;
- spin_unlock(&dcache_lock);
+ return;
}
/*
@@ -1035,11 +1032,12 @@ struct dentry *d_alloc(struct dentry * p
INIT_LIST_HEAD(&dentry->d_u.d_child);
}
- spin_lock(&dcache_lock);
- if (parent)
+ if (parent) {
+ spin_lock(&dcache_lock);
list_add(&dentry->d_u.d_child, &parent->d_subdirs);
- dentry_stat.nr_dentry++;
- spin_unlock(&dcache_lock);
+ spin_unlock(&dcache_lock);
+ }
+ atomic_inc(&dentry_stat.nr_dentry);
return dentry;
}
Index: linux-2.6/include/linux/dcache.h
===================================================================
--- linux-2.6.orig/include/linux/dcache.h
+++ linux-2.6/include/linux/dcache.h
@@ -37,8 +37,8 @@ struct qstr {
};
struct dentry_stat_t {
- int nr_dentry;
- int nr_unused;
+ atomic_t nr_dentry;
+ int nr_unused; /* protected by dcache_lru_lock */
int age_limit; /* age in seconds */
int want_pages; /* pages requested by system */
int dummy[2];
Index: linux-2.6/kernel/sysctl.c
===================================================================
--- linux-2.6.orig/kernel/sysctl.c
+++ linux-2.6/kernel/sysctl.c
@@ -1413,6 +1413,12 @@ static struct ctl_table fs_table[] = {
.extra2 = &sysctl_nr_open_max,
},
{
+ /*
+ * dentry_stat has an atomic_t member, so this is a bit of
+ * a hack, but it works for the moment, and I won't bother
+ * changing it now because we'll probably want to change to
+ * a more scalable counter anyway.
+ */
.ctl_name = FS_DENTRY,
.procname = "dentry-state",
.data = &dentry_stat,
^ permalink raw reply [flat|nested] 64+ messages in thread
* [patch 09/33] fs: dcache scale dentry refcount
2009-09-04 6:51 [patch 00/33] my current vfs scalability patch queue npiggin
` (7 preceding siblings ...)
2009-09-04 6:51 ` [patch 08/33] fs: dcache scale nr_dentry npiggin
@ 2009-09-04 6:51 ` npiggin
2009-09-06 18:01 ` Eric Paris
2009-09-04 6:51 ` [patch 10/33] fs: dcache scale d_unhashed npiggin
` (24 subsequent siblings)
33 siblings, 1 reply; 64+ messages in thread
From: npiggin @ 2009-09-04 6:51 UTC (permalink / raw)
To: linux-fsdevel; +Cc: linux-kernel
[-- Attachment #1: fs-dcache-scale-d_count.patch --]
[-- Type: text/plain, Size: 25295 bytes --]
Make d_count non-atomic and protect it with d_lock. This allows us to
ensure a 0 refcount dentry remains 0 without dcache_lock. It is also
fairly natural when we start protecting many other dentry members with
d_lock.
---
arch/powerpc/platforms/cell/spufs/inode.c | 2
drivers/infiniband/hw/ipath/ipath_fs.c | 2
fs/autofs4/expire.c | 8 +-
fs/autofs4/root.c | 6 -
fs/coda/dir.c | 2
fs/configfs/dir.c | 3
fs/configfs/inode.c | 2
fs/dcache.c | 103 ++++++++++++++++++++++--------
fs/ecryptfs/inode.c | 2
fs/exportfs/expfs.c | 8 ++
fs/hpfs/namei.c | 2
fs/locks.c | 2
fs/namei.c | 2
fs/nfs/dir.c | 12 +--
fs/nfsd/vfs.c | 5 -
fs/notify/fsnotify.c | 11 ++-
fs/notify/inotify/inotify.c | 12 ++-
fs/smbfs/dir.c | 8 ++
fs/smbfs/proc.c | 8 ++
include/linux/dcache.h | 29 ++++----
kernel/cgroup.c | 2
net/sunrpc/rpc_pipe.c | 2
22 files changed, 156 insertions(+), 77 deletions(-)
Index: linux-2.6/fs/dcache.c
===================================================================
--- linux-2.6.orig/fs/dcache.c
+++ linux-2.6/fs/dcache.c
@@ -107,6 +107,7 @@ static void d_callback(struct rcu_head *
static void d_free(struct dentry *dentry)
{
atomic_dec(&dentry_stat.nr_dentry);
+ BUG_ON(dentry->d_count);
if (dentry->d_op && dentry->d_op->d_release)
dentry->d_op->d_release(dentry);
/* if dentry was never inserted into hash, immediate free is OK */
@@ -258,13 +259,23 @@ void dput(struct dentry *dentry)
return;
repeat:
- if (atomic_read(&dentry->d_count) == 1)
+ if (dentry->d_count == 1)
might_sleep();
- if (!atomic_dec_and_lock(&dentry->d_count, &dcache_lock))
- return;
-
spin_lock(&dentry->d_lock);
- if (atomic_read(&dentry->d_count)) {
+ if (dentry->d_count == 1) {
+ if (!spin_trylock(&dcache_lock)) {
+ /*
+ * Something of a livelock possibility we could avoid
+ * by taking dcache_lock and trying again, but we
+ * want to reduce dcache_lock anyway so this will
+ * get improved.
+ */
+ spin_unlock(&dentry->d_lock);
+ goto repeat;
+ }
+ }
+ dentry->d_count--;
+ if (dentry->d_count) {
spin_unlock(&dentry->d_lock);
spin_unlock(&dcache_lock);
return;
@@ -341,7 +352,7 @@ int d_invalidate(struct dentry * dentry)
* working directory or similar).
*/
spin_lock(&dentry->d_lock);
- if (atomic_read(&dentry->d_count) > 1) {
+ if (dentry->d_count > 1) {
if (dentry->d_inode && S_ISDIR(dentry->d_inode->i_mode)) {
spin_unlock(&dentry->d_lock);
spin_unlock(&dcache_lock);
@@ -355,28 +366,54 @@ int d_invalidate(struct dentry * dentry)
return 0;
}
-/* This should be called _only_ with dcache_lock held */
+/* This should be called _only_ with a lock pinning the dentry */
static inline struct dentry * __dget_locked_dlock(struct dentry *dentry)
{
- atomic_inc(&dentry->d_count);
+ dentry->d_count++;
dentry_lru_del_init(dentry);
return dentry;
}
static inline struct dentry * __dget_locked(struct dentry *dentry)
{
- atomic_inc(&dentry->d_count);
spin_lock(&dentry->d_lock);
- dentry_lru_del_init(dentry);
+ __dget_locked_dlock(dentry);
spin_lock(&dentry->d_lock);
return dentry;
}
+struct dentry * dget_locked_dlock(struct dentry *dentry)
+{
+ return __dget_locked_dlock(dentry);
+}
+
struct dentry * dget_locked(struct dentry *dentry)
{
return __dget_locked(dentry);
}
+struct dentry *dget_parent(struct dentry *dentry)
+{
+ struct dentry *ret;
+
+repeat:
+ spin_lock(&dentry->d_lock);
+ ret = dentry->d_parent;
+ if (!ret)
+ goto out;
+ if (!spin_trylock(&ret->d_lock)) {
+ spin_unlock(&dentry->d_lock);
+ goto repeat;
+ }
+ BUG_ON(!ret->d_count);
+ ret->d_count++;
+ spin_unlock(&ret->d_lock);
+out:
+ spin_unlock(&dentry->d_lock);
+ return ret;
+}
+EXPORT_SYMBOL(dget_parent);
+
/**
* d_find_alias - grab a hashed alias of inode
* @inode: inode in question
@@ -444,7 +481,7 @@ restart:
spin_lock(&dcache_lock);
list_for_each_entry(dentry, &inode->i_dentry, d_alias) {
spin_lock(&dentry->d_lock);
- if (!atomic_read(&dentry->d_count)) {
+ if (!dentry->d_count) {
__dget_locked_dlock(dentry);
__d_drop(dentry);
spin_unlock(&dentry->d_lock);
@@ -479,7 +516,10 @@ static void prune_one_dentry(struct dent
*/
while (dentry) {
spin_lock(&dcache_lock);
- if (!atomic_dec_and_lock(&dentry->d_count, &dentry->d_lock)) {
+ spin_lock(&dentry->d_lock);
+ dentry->d_count--;
+ if (dentry->d_count) {
+ spin_unlock(&dentry->d_lock);
spin_unlock(&dcache_lock);
return;
}
@@ -565,7 +605,7 @@ again:
* the LRU because of laziness during lookup. Do not free
* it - just keep it off the LRU list.
*/
- if (atomic_read(&dentry->d_count)) {
+ if (dentry->d_count) {
spin_unlock(&dentry->d_lock);
continue;
}
@@ -726,7 +766,7 @@ static void shrink_dcache_for_umount_sub
do {
struct inode *inode;
- if (atomic_read(&dentry->d_count) != 0) {
+ if (dentry->d_count != 0) {
printk(KERN_ERR
"BUG: Dentry %p{i=%lx,n=%s}"
" still in use (%d)"
@@ -735,7 +775,7 @@ static void shrink_dcache_for_umount_sub
dentry->d_inode ?
dentry->d_inode->i_ino : 0UL,
dentry->d_name.name,
- atomic_read(&dentry->d_count),
+ dentry->d_count,
dentry->d_sb->s_type->name,
dentry->d_sb->s_id);
BUG();
@@ -745,7 +785,9 @@ static void shrink_dcache_for_umount_sub
parent = NULL;
else {
parent = dentry->d_parent;
- atomic_dec(&parent->d_count);
+ spin_lock(&parent->d_lock);
+ parent->d_count--;
+ spin_unlock(&parent->d_lock);
}
list_del(&dentry->d_u.d_child);
@@ -800,7 +842,9 @@ void shrink_dcache_for_umount(struct sup
dentry = sb->s_root;
sb->s_root = NULL;
- atomic_dec(&dentry->d_count);
+ spin_lock(&dentry->d_lock);
+ dentry->d_count--;
+ spin_unlock(&dentry->d_lock);
shrink_dcache_for_umount_subtree(dentry);
while (!hlist_empty(&sb->s_anon)) {
@@ -892,17 +936,15 @@ resume:
spin_lock(&dentry->d_lock);
dentry_lru_del_init(dentry);
- spin_unlock(&dentry->d_lock);
/*
* move only zero ref count dentries to the end
* of the unused list for prune_dcache
*/
- if (!atomic_read(&dentry->d_count)) {
- spin_lock(&dentry->d_lock);
+ if (!dentry->d_count) {
dentry_lru_add_tail(dentry);
- spin_unlock(&dentry->d_lock);
found++;
}
+ spin_unlock(&dentry->d_lock);
/*
* We can return to the caller if we have found some (this
@@ -1011,7 +1053,7 @@ struct dentry *d_alloc(struct dentry * p
memcpy(dname, name->name, name->len);
dname[name->len] = 0;
- atomic_set(&dentry->d_count, 1);
+ dentry->d_count = 1;
dentry->d_flags = DCACHE_UNHASHED;
spin_lock_init(&dentry->d_lock);
dentry->d_inode = NULL;
@@ -1479,7 +1521,7 @@ struct dentry * __d_lookup(struct dentry
goto next;
}
- atomic_inc(&dentry->d_count);
+ dentry->d_count++;
found = dentry;
spin_unlock(&dentry->d_lock);
break;
@@ -1540,6 +1582,7 @@ int d_validate(struct dentry *dentry, st
goto out;
spin_lock(&dcache_lock);
+ spin_lock(&dentry->d_lock);
spin_lock(&dcache_hash_lock);
base = d_hash(dparent, dentry->d_name.hash);
hlist_for_each(lhp,base) {
@@ -1548,12 +1591,14 @@ int d_validate(struct dentry *dentry, st
*/
if (dentry == hlist_entry(lhp, struct dentry, d_hash)) {
spin_unlock(&dcache_hash_lock);
- __dget_locked(dentry);
+ __dget_locked_dlock(dentry);
+ spin_unlock(&dentry->d_lock);
spin_unlock(&dcache_lock);
return 1;
}
}
spin_unlock(&dcache_hash_lock);
+ spin_unlock(&dentry->d_lock);
spin_unlock(&dcache_lock);
out:
return 0;
@@ -1589,7 +1634,7 @@ void d_delete(struct dentry * dentry)
spin_lock(&dcache_lock);
spin_lock(&dentry->d_lock);
isdir = S_ISDIR(dentry->d_inode->i_mode);
- if (atomic_read(&dentry->d_count) == 1) {
+ if (dentry->d_count == 1) {
dentry_iput(dentry);
fsnotify_nameremove(dentry, isdir);
return;
@@ -2264,11 +2309,15 @@ resume:
this_parent = dentry;
goto repeat;
}
- atomic_dec(&dentry->d_count);
+ spin_lock(&dentry->d_lock);
+ dentry->d_count--;
+ spin_unlock(&dentry->d_lock);
}
if (this_parent != root) {
next = this_parent->d_u.d_child.next;
- atomic_dec(&this_parent->d_count);
+ spin_lock(&this_parent->d_lock);
+ this_parent->d_count--;
+ spin_unlock(&this_parent->d_lock);
this_parent = this_parent->d_parent;
goto resume;
}
Index: linux-2.6/include/linux/dcache.h
===================================================================
--- linux-2.6.orig/include/linux/dcache.h
+++ linux-2.6/include/linux/dcache.h
@@ -87,7 +87,7 @@ full_name_hash(const unsigned char *name
#endif
struct dentry {
- atomic_t d_count;
+ unsigned int d_count; /* protected by d_lock */
unsigned int d_flags; /* protected by d_lock */
spinlock_t d_lock; /* per dentry lock */
int d_mounted;
@@ -332,17 +332,28 @@ extern char *dentry_path(struct dentry *
* needs and they take necessary precautions) you should hold dcache_lock
* and call dget_locked() instead of dget().
*/
-
+static inline struct dentry *dget_dlock(struct dentry *dentry)
+{
+ if (dentry) {
+ BUG_ON(!dentry->d_count);
+ dentry->d_count++;
+ }
+ return dentry;
+}
static inline struct dentry *dget(struct dentry *dentry)
{
if (dentry) {
- BUG_ON(!atomic_read(&dentry->d_count));
- atomic_inc(&dentry->d_count);
+ spin_lock(&dentry->d_lock);
+ dget_dlock(dentry);
+ spin_unlock(&dentry->d_lock);
}
return dentry;
}
extern struct dentry * dget_locked(struct dentry *);
+extern struct dentry * dget_locked_dlock(struct dentry *);
+
+extern struct dentry *dget_parent(struct dentry *dentry);
/**
* d_unhashed - is dentry hashed
@@ -361,16 +372,6 @@ static inline int d_unlinked(struct dent
return d_unhashed(dentry) && !IS_ROOT(dentry);
}
-static inline struct dentry *dget_parent(struct dentry *dentry)
-{
- struct dentry *ret;
-
- spin_lock(&dentry->d_lock);
- ret = dget(dentry->d_parent);
- spin_unlock(&dentry->d_lock);
- return ret;
-}
-
extern void dput(struct dentry *);
static inline int d_mountpoint(struct dentry *dentry)
Index: linux-2.6/fs/configfs/dir.c
===================================================================
--- linux-2.6.orig/fs/configfs/dir.c
+++ linux-2.6/fs/configfs/dir.c
@@ -399,8 +399,7 @@ static void remove_dir(struct dentry * d
if (d->d_inode)
simple_rmdir(parent->d_inode,d);
- pr_debug(" o %s removing done (%d)\n",d->d_name.name,
- atomic_read(&d->d_count));
+ pr_debug(" o %s removing done (%d)\n",d->d_name.name, d->d_count);
dput(parent);
}
Index: linux-2.6/fs/locks.c
===================================================================
--- linux-2.6.orig/fs/locks.c
+++ linux-2.6/fs/locks.c
@@ -1374,7 +1374,7 @@ int generic_setlease(struct file *filp,
if ((arg == F_RDLCK) && (atomic_read(&inode->i_writecount) > 0))
goto out;
if ((arg == F_WRLCK)
- && ((atomic_read(&dentry->d_count) > 1)
+ && (dentry->d_count > 1
|| (atomic_read(&inode->i_count) > 1)))
goto out;
}
Index: linux-2.6/fs/namei.c
===================================================================
--- linux-2.6.orig/fs/namei.c
+++ linux-2.6/fs/namei.c
@@ -2161,7 +2161,7 @@ void dentry_unhash(struct dentry *dentry
shrink_dcache_parent(dentry);
spin_lock(&dcache_lock);
spin_lock(&dentry->d_lock);
- if (atomic_read(&dentry->d_count) == 2)
+ if (dentry->d_count == 2)
__d_drop(dentry);
spin_unlock(&dentry->d_lock);
spin_unlock(&dcache_lock);
Index: linux-2.6/fs/autofs4/expire.c
===================================================================
--- linux-2.6.orig/fs/autofs4/expire.c
+++ linux-2.6/fs/autofs4/expire.c
@@ -198,7 +198,7 @@ static int autofs4_tree_busy(struct vfsm
else
ino_count++;
- if (atomic_read(&p->d_count) > ino_count) {
+ if (p->d_count > ino_count) {
top_ino->last_used = jiffies;
dput(p);
return 1;
@@ -347,7 +347,7 @@ struct dentry *autofs4_expire_indirect(s
/* Path walk currently on this dentry? */
ino_count = atomic_read(&ino->count) + 2;
- if (atomic_read(&dentry->d_count) > ino_count)
+ if (dentry->d_count > ino_count)
goto next;
/* Can we umount this guy */
@@ -369,7 +369,7 @@ struct dentry *autofs4_expire_indirect(s
if (!exp_leaves) {
/* Path walk currently on this dentry? */
ino_count = atomic_read(&ino->count) + 1;
- if (atomic_read(&dentry->d_count) > ino_count)
+ if (dentry->d_count > ino_count)
goto next;
if (!autofs4_tree_busy(mnt, dentry, timeout, do_now)) {
@@ -383,7 +383,7 @@ struct dentry *autofs4_expire_indirect(s
} else {
/* Path walk currently on this dentry? */
ino_count = atomic_read(&ino->count) + 1;
- if (atomic_read(&dentry->d_count) > ino_count)
+ if (dentry->d_count > ino_count)
goto next;
expired = autofs4_check_leaves(mnt, dentry, timeout, do_now);
Index: linux-2.6/fs/autofs4/root.c
===================================================================
--- linux-2.6.orig/fs/autofs4/root.c
+++ linux-2.6/fs/autofs4/root.c
@@ -380,7 +380,7 @@ static struct dentry *autofs4_lookup_act
spin_lock(&dentry->d_lock);
/* Already gone? */
- if (atomic_read(&dentry->d_count) == 0)
+ if (dentry->d_count == 0)
goto next;
qstr = &dentry->d_name;
@@ -396,7 +396,7 @@ static struct dentry *autofs4_lookup_act
goto next;
if (d_unhashed(dentry)) {
- dget(dentry);
+ dget_locked_dlock(dentry);
spin_unlock(&dentry->d_lock);
spin_unlock(&sbi->lookup_lock);
spin_unlock(&dcache_lock);
@@ -448,7 +448,7 @@ static struct dentry *autofs4_lookup_exp
goto next;
if (d_unhashed(dentry)) {
- dget(dentry);
+ dget_locked_dlock(dentry);
spin_unlock(&dentry->d_lock);
spin_unlock(&sbi->lookup_lock);
spin_unlock(&dcache_lock);
Index: linux-2.6/fs/coda/dir.c
===================================================================
--- linux-2.6.orig/fs/coda/dir.c
+++ linux-2.6/fs/coda/dir.c
@@ -611,7 +611,7 @@ static int coda_dentry_revalidate(struct
if (cii->c_flags & C_FLUSH)
coda_flag_inode_children(inode, C_FLUSH);
- if (atomic_read(&de->d_count) > 1)
+ if (de->d_count > 1)
/* pretend it's valid, but don't change the flags */
goto out;
Index: linux-2.6/fs/ecryptfs/inode.c
===================================================================
--- linux-2.6.orig/fs/ecryptfs/inode.c
+++ linux-2.6/fs/ecryptfs/inode.c
@@ -263,7 +263,7 @@ int ecryptfs_lookup_and_interpose_lower(
ecryptfs_dentry->d_parent));
lower_inode = lower_dentry->d_inode;
fsstack_copy_attr_atime(ecryptfs_dir_inode, lower_dir_dentry->d_inode);
- BUG_ON(!atomic_read(&lower_dentry->d_count));
+ BUG_ON(!lower_dentry->d_count);
ecryptfs_set_dentry_private(ecryptfs_dentry,
kmem_cache_alloc(ecryptfs_dentry_info_cache,
GFP_KERNEL));
Index: linux-2.6/fs/hpfs/namei.c
===================================================================
--- linux-2.6.orig/fs/hpfs/namei.c
+++ linux-2.6/fs/hpfs/namei.c
@@ -415,7 +415,7 @@ again:
mutex_unlock(&hpfs_i(inode)->i_parent_mutex);
d_drop(dentry);
spin_lock(&dentry->d_lock);
- if (atomic_read(&dentry->d_count) > 1 ||
+ if (dentry->d_count > 1 ||
generic_permission(inode, MAY_WRITE, NULL) ||
!S_ISREG(inode->i_mode) ||
get_write_access(inode)) {
Index: linux-2.6/fs/nfs/dir.c
===================================================================
--- linux-2.6.orig/fs/nfs/dir.c
+++ linux-2.6/fs/nfs/dir.c
@@ -1325,7 +1325,7 @@ static int nfs_sillyrename(struct inode
dfprintk(VFS, "NFS: silly-rename(%s/%s, ct=%d)\n",
dentry->d_parent->d_name.name, dentry->d_name.name,
- atomic_read(&dentry->d_count));
+ dentry->d_count);
nfs_inc_stats(dir, NFSIOS_SILLYRENAME);
/*
@@ -1434,7 +1434,7 @@ static int nfs_unlink(struct inode *dir,
spin_lock(&dcache_lock);
spin_lock(&dentry->d_lock);
- if (atomic_read(&dentry->d_count) > 1) {
+ if (dentry->d_count > 1) {
spin_unlock(&dentry->d_lock);
spin_unlock(&dcache_lock);
/* Start asynchronous writeout of the inode */
@@ -1589,7 +1589,7 @@ static int nfs_rename(struct inode *old_
dfprintk(VFS, "NFS: rename(%s/%s -> %s/%s, ct=%d)\n",
old_dentry->d_parent->d_name.name, old_dentry->d_name.name,
new_dentry->d_parent->d_name.name, new_dentry->d_name.name,
- atomic_read(&new_dentry->d_count));
+ new_dentry->d_count);
/*
* First check whether the target is busy ... we can't
@@ -1605,7 +1605,7 @@ static int nfs_rename(struct inode *old_
error = -EISDIR;
if (!S_ISDIR(old_inode->i_mode))
goto out;
- } else if (atomic_read(&new_dentry->d_count) > 2) {
+ } else if (new_dentry->d_count > 2) {
int err;
/* copy the target dentry's name */
dentry = d_alloc(new_dentry->d_parent,
@@ -1620,7 +1620,7 @@ static int nfs_rename(struct inode *old_
new_inode = NULL;
/* instantiate the replacement target */
d_instantiate(new_dentry, NULL);
- } else if (atomic_read(&new_dentry->d_count) > 1)
+ } else if (new_dentry->d_count > 1)
/* dentry still busy? */
goto out;
}
@@ -1629,7 +1629,7 @@ go_ahead:
/*
* ... prune child dentries and writebacks if needed.
*/
- if (atomic_read(&old_dentry->d_count) > 1) {
+ if (old_dentry->d_count > 1) {
if (S_ISREG(old_inode->i_mode))
nfs_wb_all(old_inode);
shrink_dcache_parent(old_dentry);
Index: linux-2.6/fs/nfsd/vfs.c
===================================================================
--- linux-2.6.orig/fs/nfsd/vfs.c
+++ linux-2.6/fs/nfsd/vfs.c
@@ -1744,8 +1744,7 @@ nfsd_rename(struct svc_rqst *rqstp, stru
goto out_dput_new;
if (svc_msnfs(ffhp) &&
- ((atomic_read(&odentry->d_count) > 1)
- || (atomic_read(&ndentry->d_count) > 1))) {
+ ((odentry->d_count > 1) || (ndentry->d_count > 1))) {
host_err = -EPERM;
goto out_dput_new;
}
@@ -1831,7 +1830,7 @@ nfsd_unlink(struct svc_rqst *rqstp, stru
if (type != S_IFDIR) { /* It's UNLINK */
#ifdef MSNFS
if ((fhp->fh_export->ex_flags & NFSEXP_MSNFS) &&
- (atomic_read(&rdentry->d_count) > 1)) {
+ (rdentry->d_count > 1)) {
host_err = -EPERM;
} else
#endif
Index: linux-2.6/fs/exportfs/expfs.c
===================================================================
--- linux-2.6.orig/fs/exportfs/expfs.c
+++ linux-2.6/fs/exportfs/expfs.c
@@ -74,11 +74,17 @@ static struct dentry *
find_disconnected_root(struct dentry *dentry)
{
dget(dentry);
+again:
spin_lock(&dentry->d_lock);
while (!IS_ROOT(dentry) &&
(dentry->d_parent->d_flags & DCACHE_DISCONNECTED)) {
struct dentry *parent = dentry->d_parent;
- dget(parent);
+
+ if (!spin_trylock(&parent->d_lock)) {
+ spin_unlock(&dentry->d_lock);
+ goto again;
+ }
+ dget_dlock(parent);
spin_unlock(&dentry->d_lock);
dput(dentry);
dentry = parent;
Index: linux-2.6/fs/notify/inotify/inotify.c
===================================================================
--- linux-2.6.orig/fs/notify/inotify/inotify.c
+++ linux-2.6/fs/notify/inotify/inotify.c
@@ -339,18 +339,26 @@ void inotify_dentry_parent_queue_event(s
if (!(dentry->d_flags & DCACHE_INOTIFY_PARENT_WATCHED))
return;
+again:
spin_lock(&dentry->d_lock);
parent = dentry->d_parent;
+ if (!spin_trylock(&parent->d_lock)) {
+ spin_unlock(&dentry->d_lock);
+ goto again;
+ }
inode = parent->d_inode;
if (inotify_inode_watched(inode)) {
- dget(parent);
+ dget_dlock(parent);
spin_unlock(&dentry->d_lock);
+ spin_unlock(&parent->d_lock);
inotify_inode_queue_event(inode, mask, cookie, name,
dentry->d_inode);
dput(parent);
- } else
+ } else {
spin_unlock(&dentry->d_lock);
+ spin_unlock(&parent->d_lock);
+ }
}
EXPORT_SYMBOL_GPL(inotify_dentry_parent_queue_event);
Index: linux-2.6/fs/smbfs/dir.c
===================================================================
--- linux-2.6.orig/fs/smbfs/dir.c
+++ linux-2.6/fs/smbfs/dir.c
@@ -405,6 +405,7 @@ void
smb_renew_times(struct dentry * dentry)
{
dget(dentry);
+again:
spin_lock(&dentry->d_lock);
for (;;) {
struct dentry *parent;
@@ -413,8 +414,13 @@ smb_renew_times(struct dentry * dentry)
if (IS_ROOT(dentry))
break;
parent = dentry->d_parent;
- dget(parent);
+ if (!spin_trylock(&parent->d_lock)) {
+ spin_unlock(&dentry->d_lock);
+ goto again;
+ }
+ dget_dlock(parent);
spin_unlock(&dentry->d_lock);
+ spin_unlock(&parent->d_lock);
dput(dentry);
dentry = parent;
spin_lock(&dentry->d_lock);
Index: linux-2.6/fs/smbfs/proc.c
===================================================================
--- linux-2.6.orig/fs/smbfs/proc.c
+++ linux-2.6/fs/smbfs/proc.c
@@ -332,6 +332,7 @@ static int smb_build_path(struct smb_sb_
* and store it in reversed order [see reverse_string()]
*/
dget(entry);
+again:
spin_lock(&entry->d_lock);
while (!IS_ROOT(entry)) {
struct dentry *parent;
@@ -350,6 +351,7 @@ static int smb_build_path(struct smb_sb_
dput(entry);
return len;
}
+
reverse_string(path, len);
path += len;
if (unicode) {
@@ -361,7 +363,11 @@ static int smb_build_path(struct smb_sb_
maxlen -= len+1;
parent = entry->d_parent;
- dget(parent);
+ if (!spin_trylock(&parent->d_lock)) {
+ spin_unlock(&entry->d_lock);
+ goto again;
+ }
+ dget_dlock(parent);
spin_unlock(&entry->d_lock);
dput(entry);
entry = parent;
Index: linux-2.6/kernel/cgroup.c
===================================================================
--- linux-2.6.orig/kernel/cgroup.c
+++ linux-2.6/kernel/cgroup.c
@@ -2813,9 +2813,7 @@ again:
list_del(&cgrp->sibling);
cgroup_unlock_hierarchy(cgrp->root);
- spin_lock(&cgrp->dentry->d_lock);
d = dget(cgrp->dentry);
- spin_unlock(&d->d_lock);
cgroup_d_remove_dir(d);
dput(d);
Index: linux-2.6/arch/powerpc/platforms/cell/spufs/inode.c
===================================================================
--- linux-2.6.orig/arch/powerpc/platforms/cell/spufs/inode.c
+++ linux-2.6/arch/powerpc/platforms/cell/spufs/inode.c
@@ -161,7 +161,7 @@ static void spufs_prune_dir(struct dentr
spin_lock(&dcache_lock);
spin_lock(&dentry->d_lock);
if (!(d_unhashed(dentry)) && dentry->d_inode) {
- dget_locked(dentry);
+ dget_locked_dlock(dentry);
__d_drop(dentry);
spin_unlock(&dentry->d_lock);
simple_unlink(dir->d_inode, dentry);
Index: linux-2.6/drivers/infiniband/hw/ipath/ipath_fs.c
===================================================================
--- linux-2.6.orig/drivers/infiniband/hw/ipath/ipath_fs.c
+++ linux-2.6/drivers/infiniband/hw/ipath/ipath_fs.c
@@ -275,7 +275,7 @@ static int remove_file(struct dentry *pa
spin_lock(&dcache_lock);
spin_lock(&tmp->d_lock);
if (!(d_unhashed(tmp) && tmp->d_inode)) {
- dget_locked(tmp);
+ dget_locked_dlock(tmp);
__d_drop(tmp);
spin_unlock(&tmp->d_lock);
spin_unlock(&dcache_lock);
Index: linux-2.6/fs/configfs/inode.c
===================================================================
--- linux-2.6.orig/fs/configfs/inode.c
+++ linux-2.6/fs/configfs/inode.c
@@ -256,7 +256,7 @@ void configfs_drop_dentry(struct configf
spin_lock(&dcache_lock);
spin_lock(&dentry->d_lock);
if (!(d_unhashed(dentry) && dentry->d_inode)) {
- dget_locked(dentry);
+ dget_locked_dlock(dentry);
__d_drop(dentry);
spin_unlock(&dentry->d_lock);
spin_unlock(&dcache_lock);
Index: linux-2.6/net/sunrpc/rpc_pipe.c
===================================================================
--- linux-2.6.orig/net/sunrpc/rpc_pipe.c
+++ linux-2.6/net/sunrpc/rpc_pipe.c
@@ -556,7 +556,7 @@ repeat:
continue;
spin_lock(&dentry->d_lock);
if (!d_unhashed(dentry)) {
- dget_locked(dentry);
+ dget_locked_dlock(dentry);
__d_drop(dentry);
spin_unlock(&dentry->d_lock);
dvec[n++] = dentry;
Index: linux-2.6/fs/notify/fsnotify.c
===================================================================
--- linux-2.6.orig/fs/notify/fsnotify.c
+++ linux-2.6/fs/notify/fsnotify.c
@@ -87,13 +87,18 @@ void __fsnotify_parent(struct dentry *de
if (!(dentry->d_flags & DCACHE_FSNOTIFY_PARENT_WATCHED))
return;
+again:
spin_lock(&dentry->d_lock);
parent = dentry->d_parent;
+ if (parent != dentry && !spin_trylock(&parent->d_lock)) {
+ spin_unlock(&dentry->d_lock);
+ goto again;
+ }
p_inode = parent->d_inode;
if (fsnotify_inode_watches_children(p_inode)) {
if (p_inode->i_fsnotify_mask & mask) {
- dget(parent);
+ dget_dlock(parent);
send = true;
}
} else {
@@ -103,11 +108,13 @@ void __fsnotify_parent(struct dentry *de
* children and update their d_flags to let them know p_inode
* doesn't care about them any more.
*/
- dget(parent);
+ dget_dlock(parent);
should_update_children = true;
}
spin_unlock(&dentry->d_lock);
+ if (parent != dentry)
+ spin_unlock(&parent->d_lock);
if (send) {
/* we are notifying a parent so come up with the new mask which
^ permalink raw reply [flat|nested] 64+ messages in thread
* [patch 10/33] fs: dcache scale d_unhashed
2009-09-04 6:51 [patch 00/33] my current vfs scalability patch queue npiggin
` (8 preceding siblings ...)
2009-09-04 6:51 ` [patch 09/33] fs: dcache scale dentry refcount npiggin
@ 2009-09-04 6:51 ` npiggin
2009-09-04 6:51 ` [patch 11/33] fs: dcache scale subdirs npiggin
` (23 subsequent siblings)
33 siblings, 0 replies; 64+ messages in thread
From: npiggin @ 2009-09-04 6:51 UTC (permalink / raw)
To: linux-fsdevel; +Cc: linux-kernel
[-- Attachment #1: fs-dcache-scale-d_unhashed.patch --]
[-- Type: text/plain, Size: 10675 bytes --]
Protect d_unhashed(dentry) condition with d_lock.
---
arch/powerpc/platforms/cell/spufs/inode.c | 3 ++
fs/configfs/configfs_internal.h | 2 +
fs/dcache.c | 40 +++++++++++++++++++++++-------
fs/libfs.c | 29 +++++++++++++++------
fs/ocfs2/dcache.c | 5 +++
fs/seq_file.c | 3 ++
fs/sysfs/dir.c | 8 +++---
7 files changed, 68 insertions(+), 22 deletions(-)
Index: linux-2.6/fs/sysfs/dir.c
===================================================================
--- linux-2.6.orig/fs/sysfs/dir.c
+++ linux-2.6/fs/sysfs/dir.c
@@ -549,10 +549,12 @@ static void sysfs_drop_dentry(struct sys
repeat:
spin_lock(&dcache_lock);
list_for_each_entry(dentry, &inode->i_dentry, d_alias) {
- if (d_unhashed(dentry))
- continue;
- dget_locked(dentry);
spin_lock(&dentry->d_lock);
+ if (d_unhashed(dentry)) {
+ spin_unlock(&dentry->d_lock);
+ continue;
+ }
+ dget_locked_dlock(dentry);
__d_drop(dentry);
spin_unlock(&dentry->d_lock);
spin_unlock(&dcache_lock);
Index: linux-2.6/fs/libfs.c
===================================================================
--- linux-2.6.orig/fs/libfs.c
+++ linux-2.6/fs/libfs.c
@@ -14,6 +14,11 @@
#include <asm/uaccess.h>
+static inline int simple_positive(struct dentry *dentry)
+{
+ return dentry->d_inode && !d_unhashed(dentry);
+}
+
int simple_getattr(struct vfsmount *mnt, struct dentry *dentry,
struct kstat *stat)
{
@@ -103,8 +108,10 @@ loff_t dcache_dir_lseek(struct file *fil
while (n && p != &file->f_path.dentry->d_subdirs) {
struct dentry *next;
next = list_entry(p, struct dentry, d_u.d_child);
- if (!d_unhashed(next) && next->d_inode)
+ spin_lock(&next->d_lock);
+ if (simple_positive(next))
n--;
+ spin_unlock(&next->d_lock);
p = p->next;
}
list_add_tail(&cursor->d_u.d_child, p);
@@ -158,9 +165,13 @@ int dcache_readdir(struct file * filp, v
for (p=q->next; p != &dentry->d_subdirs; p=p->next) {
struct dentry *next;
next = list_entry(p, struct dentry, d_u.d_child);
- if (d_unhashed(next) || !next->d_inode)
+ spin_lock_nested(&next->d_lock, DENTRY_D_LOCK_NESTED);
+ if (!simple_positive(next)) {
+ spin_unlock(&next->d_lock);
continue;
+ }
+ spin_unlock(&next->d_lock);
spin_unlock(&dcache_lock);
if (filldir(dirent, next->d_name.name,
next->d_name.len, filp->f_pos,
@@ -265,20 +276,20 @@ int simple_link(struct dentry *old_dentr
return 0;
}
-static inline int simple_positive(struct dentry *dentry)
-{
- return dentry->d_inode && !d_unhashed(dentry);
-}
-
int simple_empty(struct dentry *dentry)
{
struct dentry *child;
int ret = 0;
spin_lock(&dcache_lock);
- list_for_each_entry(child, &dentry->d_subdirs, d_u.d_child)
- if (simple_positive(child))
+ list_for_each_entry(child, &dentry->d_subdirs, d_u.d_child) {
+ spin_lock_nested(&child->d_lock, DENTRY_D_LOCK_NESTED);
+ if (simple_positive(child)) {
+ spin_unlock(&child->d_lock);
goto out;
+ }
+ spin_unlock(&child->d_lock);
+ }
ret = 1;
out:
spin_unlock(&dcache_lock);
Index: linux-2.6/fs/seq_file.c
===================================================================
--- linux-2.6.orig/fs/seq_file.c
+++ linux-2.6/fs/seq_file.c
@@ -6,6 +6,7 @@
*/
#include <linux/fs.h>
+#include <linux/mount.h>
#include <linux/module.h>
#include <linux/seq_file.h>
#include <linux/slab.h>
@@ -460,7 +461,9 @@ int seq_path_root(struct seq_file *m, st
char *p;
spin_lock(&dcache_lock);
+ vfsmount_read_lock();
p = __d_path(path, root, s, m->size - m->count);
+ vfsmount_read_unlock();
spin_unlock(&dcache_lock);
err = PTR_ERR(p);
if (!IS_ERR(p)) {
Index: linux-2.6/fs/dcache.c
===================================================================
--- linux-2.6.orig/fs/dcache.c
+++ linux-2.6/fs/dcache.c
@@ -327,7 +327,9 @@ int d_invalidate(struct dentry * dentry)
* If it's already been dropped, return OK.
*/
spin_lock(&dcache_lock);
+ spin_lock(&dentry->d_lock);
if (d_unhashed(dentry)) {
+ spin_unlock(&dentry->d_lock);
spin_unlock(&dcache_lock);
return 0;
}
@@ -336,6 +338,7 @@ int d_invalidate(struct dentry * dentry)
* to get rid of unused child entries.
*/
if (!list_empty(&dentry->d_subdirs)) {
+ spin_unlock(&dentry->d_lock);
spin_unlock(&dcache_lock);
shrink_dcache_parent(dentry);
spin_lock(&dcache_lock);
@@ -443,15 +446,18 @@ static struct dentry * __d_find_alias(st
next = tmp->next;
prefetch(next);
alias = list_entry(tmp, struct dentry, d_alias);
+ spin_lock(&alias->d_lock);
if (S_ISDIR(inode->i_mode) || !d_unhashed(alias)) {
if (IS_ROOT(alias) &&
(alias->d_flags & DCACHE_DISCONNECTED))
discon_alias = alias;
else if (!want_discon) {
- __dget_locked(alias);
+ __dget_locked_dlock(alias);
+ spin_unlock(&alias->d_lock);
return alias;
}
}
+ spin_unlock(&alias->d_lock);
}
if (discon_alias)
__dget_locked(discon_alias);
@@ -734,8 +740,8 @@ static void shrink_dcache_for_umount_sub
spin_lock(&dcache_lock);
spin_lock(&dentry->d_lock);
dentry_lru_del_init(dentry);
- spin_unlock(&dentry->d_lock);
__d_drop(dentry);
+ spin_unlock(&dentry->d_lock);
spin_unlock(&dcache_lock);
for (;;) {
@@ -750,8 +756,8 @@ static void shrink_dcache_for_umount_sub
d_u.d_child) {
spin_lock(&loop->d_lock);
dentry_lru_del_init(loop);
- spin_unlock(&loop->d_lock);
__d_drop(loop);
+ spin_unlock(&loop->d_lock);
cond_resched_lock(&dcache_lock);
}
spin_unlock(&dcache_lock);
@@ -2016,7 +2022,8 @@ static int prepend_name(char **buffer, i
* Returns a pointer into the buffer or an error code if the
* path was too long.
*
- * "buflen" should be positive. Caller holds the dcache_lock.
+ * "buflen" should be positive. Caller holds the dcache_lock and
+ * path->dentry->d_lock.
*
* If path is not reachable from the supplied root, then the value of
* root is changed (without modifying refcounts).
@@ -2029,7 +2036,6 @@ char *__d_path(const struct path *path,
char *end = buffer + buflen;
char *retval;
- vfsmount_read_lock();
prepend(&end, &buflen, "\0", 1);
if (d_unlinked(dentry) &&
(prepend(&end, &buflen, " (deleted)", 10) != 0))
@@ -2065,7 +2071,6 @@ char *__d_path(const struct path *path,
}
out:
- vfsmount_read_unlock();
return retval;
global_root:
@@ -2118,8 +2123,12 @@ char *d_path(const struct path *path, ch
path_get(&root);
read_unlock(¤t->fs->lock);
spin_lock(&dcache_lock);
+ vfsmount_read_lock();
+ spin_lock(&path->dentry->d_lock);
tmp = root;
res = __d_path(path, &tmp, buf, buflen);
+ spin_unlock(&path->dentry->d_lock);
+ vfsmount_read_unlock();
spin_unlock(&dcache_lock);
path_put(&root);
return res;
@@ -2155,6 +2164,7 @@ char *dentry_path(struct dentry *dentry,
char *retval;
spin_lock(&dcache_lock);
+ spin_lock(&dentry->d_lock);
prepend(&end, &buflen, "\0", 1);
if (d_unlinked(dentry) &&
(prepend(&end, &buflen, "//deleted", 9) != 0))
@@ -2176,9 +2186,11 @@ char *dentry_path(struct dentry *dentry,
retval = end;
dentry = parent;
}
+ spin_unlock(&dentry->d_lock);
spin_unlock(&dcache_lock);
return retval;
Elong:
+ spin_unlock(&dentry->d_lock);
spin_unlock(&dcache_lock);
return ERR_PTR(-ENAMETOOLONG);
}
@@ -2219,12 +2231,16 @@ SYSCALL_DEFINE2(getcwd, char __user *, b
error = -ENOENT;
spin_lock(&dcache_lock);
+ vfsmount_read_lock();
+ spin_lock(&pwd.dentry->d_lock);
if (!d_unlinked(pwd.dentry)) {
unsigned long len;
struct path tmp = root;
char * cwd;
cwd = __d_path(&pwd, &tmp, page, PAGE_SIZE);
+ spin_unlock(&pwd.dentry->d_lock);
+ vfsmount_read_unlock();
spin_unlock(&dcache_lock);
error = PTR_ERR(cwd);
@@ -2238,8 +2254,11 @@ SYSCALL_DEFINE2(getcwd, char __user *, b
if (copy_to_user(buf, cwd, len))
error = -EFAULT;
}
- } else
+ } else {
+ spin_unlock(&pwd.dentry->d_lock);
+ vfsmount_read_unlock();
spin_unlock(&dcache_lock);
+ }
out:
path_put(&pwd);
@@ -2303,13 +2322,16 @@ resume:
struct list_head *tmp = next;
struct dentry *dentry = list_entry(tmp, struct dentry, d_u.d_child);
next = tmp->next;
- if (d_unhashed(dentry)||!dentry->d_inode)
+ spin_lock_nested(&dentry->d_lock, DENTRY_D_LOCK_NESTED);
+ if (d_unhashed(dentry) || !dentry->d_inode) {
+ spin_unlock(&dentry->d_lock);
continue;
+ }
if (!list_empty(&dentry->d_subdirs)) {
+ spin_unlock(&dentry->d_lock);
this_parent = dentry;
goto repeat;
}
- spin_lock(&dentry->d_lock);
dentry->d_count--;
spin_unlock(&dentry->d_lock);
}
Index: linux-2.6/arch/powerpc/platforms/cell/spufs/inode.c
===================================================================
--- linux-2.6.orig/arch/powerpc/platforms/cell/spufs/inode.c
+++ linux-2.6/arch/powerpc/platforms/cell/spufs/inode.c
@@ -165,6 +165,9 @@ static void spufs_prune_dir(struct dentr
__d_drop(dentry);
spin_unlock(&dentry->d_lock);
simple_unlink(dir->d_inode, dentry);
+ /* XXX: what is dcache_lock protecting here? Other
+ * filesystems (IB, configfs) release dcache_lock
+ * before unlink */
spin_unlock(&dcache_lock);
dput(dentry);
} else {
Index: linux-2.6/fs/configfs/configfs_internal.h
===================================================================
--- linux-2.6.orig/fs/configfs/configfs_internal.h
+++ linux-2.6/fs/configfs/configfs_internal.h
@@ -121,6 +121,7 @@ static inline struct config_item *config
struct config_item * item = NULL;
spin_lock(&dcache_lock);
+ spin_lock(&dentry->d_lock);
if (!d_unhashed(dentry)) {
struct configfs_dirent * sd = dentry->d_fsdata;
if (sd->s_type & CONFIGFS_ITEM_LINK) {
@@ -129,6 +130,7 @@ static inline struct config_item *config
} else
item = config_item_get(sd->s_element);
}
+ spin_unlock(&dentry->d_lock);
spin_unlock(&dcache_lock);
return item;
Index: linux-2.6/fs/ocfs2/dcache.c
===================================================================
--- linux-2.6.orig/fs/ocfs2/dcache.c
+++ linux-2.6/fs/ocfs2/dcache.c
@@ -145,13 +145,16 @@ struct dentry *ocfs2_find_local_alias(st
list_for_each(p, &inode->i_dentry) {
dentry = list_entry(p, struct dentry, d_alias);
+ spin_lock(&dentry->d_lock);
if (ocfs2_match_dentry(dentry, parent_blkno, skip_unhashed)) {
mlog(0, "dentry found: %.*s\n",
dentry->d_name.len, dentry->d_name.name);
- dget_locked(dentry);
+ dget_locked_dlock(dentry);
+ spin_unlock(&dentry->d_lock);
break;
}
+ spin_unlock(&dentry->d_lock);
dentry = NULL;
}
^ permalink raw reply [flat|nested] 64+ messages in thread
* [patch 11/33] fs: dcache scale subdirs
2009-09-04 6:51 [patch 00/33] my current vfs scalability patch queue npiggin
` (9 preceding siblings ...)
2009-09-04 6:51 ` [patch 10/33] fs: dcache scale d_unhashed npiggin
@ 2009-09-04 6:51 ` npiggin
2010-06-17 15:13 ` Peter Zijlstra
2009-09-04 6:51 ` [patch 12/33] fs: scale inode alias list npiggin
` (22 subsequent siblings)
33 siblings, 1 reply; 64+ messages in thread
From: npiggin @ 2009-09-04 6:51 UTC (permalink / raw)
To: linux-fsdevel; +Cc: linux-kernel
[-- Attachment #1: fs-dcache-scale-d_subdirs.patch --]
[-- Type: text/plain, Size: 32837 bytes --]
Protect d_subdirs and d_child with d_lock, except in filesystems that aren't
using dcache_lock for these anyway (eg. using i_mutex).
XXX: probably don't need parent lock in inotify (because child lock
should stabilize parent). Also, possibly some filesystems don't need so
much locking (eg. of child dentry when modifying d_child, so long as
parent is locked)... but be on the safe side. Hmm, maybe we should just
say d_child list is protected by d_parent->d_lock. d_parent could remain
protected with d_lock.
---
drivers/usb/core/inode.c | 6 +
fs/autofs4/expire.c | 81 ++++++++++++++-------
fs/autofs4/inode.c | 5 +
fs/autofs4/root.c | 9 ++
fs/coda/cache.c | 2
fs/dcache.c | 159 ++++++++++++++++++++++++++++++++++---------
fs/libfs.c | 40 ++++++----
fs/ncpfs/dir.c | 3
fs/ncpfs/ncplib_kernel.h | 4 +
fs/notify/fsnotify.c | 4 -
fs/notify/inotify/inotify.c | 4 -
fs/smbfs/cache.c | 4 +
include/linux/dcache.h | 1
kernel/cgroup.c | 19 ++++-
net/sunrpc/rpc_pipe.c | 2
security/selinux/selinuxfs.c | 12 ++-
16 files changed, 274 insertions(+), 81 deletions(-)
Index: linux-2.6/fs/dcache.c
===================================================================
--- linux-2.6.orig/fs/dcache.c
+++ linux-2.6/fs/dcache.c
@@ -44,6 +44,8 @@
* - d_flags
* - d_name
* - d_lru
+ * - d_unhashed
+ * - d_subdirs and children's d_child
*
* Ordering:
* dcache_lock
@@ -205,7 +207,8 @@ static void dentry_lru_del_init(struct d
*
* If this is the root of the dentry tree, return NULL.
*
- * dcache_lock and d_lock must be held by caller, are dropped by d_kill.
+ * dcache_lock and d_lock and d_parent->d_lock must be held by caller, and
+ * are dropped by d_kill.
*/
static struct dentry *d_kill(struct dentry *dentry)
__releases(dentry->d_lock)
@@ -214,12 +217,14 @@ static struct dentry *d_kill(struct dent
struct dentry *parent;
list_del(&dentry->d_u.d_child);
- /*drops the locks, at that point nobody can reach this dentry */
- dentry_iput(dentry);
+ if (dentry->d_parent && dentry != dentry->d_parent)
+ spin_unlock(&dentry->d_parent->d_lock);
if (IS_ROOT(dentry))
parent = NULL;
else
parent = dentry->d_parent;
+ /*drops the locks, at that point nobody can reach this dentry */
+ dentry_iput(dentry);
d_free(dentry);
return parent;
}
@@ -255,6 +260,7 @@ static struct dentry *d_kill(struct dent
void dput(struct dentry *dentry)
{
+ struct dentry *parent = NULL;
if (!dentry)
return;
@@ -273,6 +279,15 @@ repeat:
spin_unlock(&dentry->d_lock);
goto repeat;
}
+ parent = dentry->d_parent;
+ if (parent) {
+ BUG_ON(parent == dentry);
+ if (!spin_trylock(&parent->d_lock)) {
+ spin_unlock(&dentry->d_lock);
+ spin_unlock(&dcache_lock);
+ goto repeat;
+ }
+ }
}
dentry->d_count--;
if (dentry->d_count) {
@@ -296,6 +311,8 @@ repeat:
dentry_lru_add(dentry);
}
spin_unlock(&dentry->d_lock);
+ if (parent)
+ spin_unlock(&parent->d_lock);
spin_unlock(&dcache_lock);
return;
@@ -521,10 +538,22 @@ static void prune_one_dentry(struct dent
* because dcache_lock needs to be taken anyway.
*/
while (dentry) {
+ struct dentry *parent = NULL;
+
spin_lock(&dcache_lock);
+again:
spin_lock(&dentry->d_lock);
+ if (dentry->d_parent && dentry != dentry->d_parent) {
+ if (!spin_trylock(&dentry->d_parent->d_lock)) {
+ spin_unlock(&dentry->d_lock);
+ goto again;
+ }
+ parent = dentry->d_parent;
+ }
dentry->d_count--;
if (dentry->d_count) {
+ if (parent)
+ spin_unlock(&parent->d_lock);
spin_unlock(&dentry->d_lock);
spin_unlock(&dcache_lock);
return;
@@ -602,20 +631,28 @@ again:
dentry = list_entry(tmp.prev, struct dentry, d_lru);
if (!spin_trylock(&dentry->d_lock)) {
+again1:
spin_unlock(&dcache_lru_lock);
goto again;
}
- __dentry_lru_del_init(dentry);
/*
* We found an inuse dentry which was not removed from
* the LRU because of laziness during lookup. Do not free
* it - just keep it off the LRU list.
*/
if (dentry->d_count) {
+ __dentry_lru_del_init(dentry);
spin_unlock(&dentry->d_lock);
continue;
}
-
+ if (dentry->d_parent) {
+ BUG_ON(dentry == dentry->d_parent);
+ if (!spin_trylock(&dentry->d_parent->d_lock)) {
+ spin_unlock(&dentry->d_lock);
+ goto again1;
+ }
+ }
+ __dentry_lru_del_init(dentry);
spin_unlock(&dcache_lru_lock);
prune_one_dentry(dentry);
/* dcache_lock and dentry->d_lock dropped */
@@ -752,14 +789,15 @@ static void shrink_dcache_for_umount_sub
/* this is a branch with children - detach all of them
* from the system in one go */
spin_lock(&dcache_lock);
+ spin_lock(&dentry->d_lock);
list_for_each_entry(loop, &dentry->d_subdirs,
d_u.d_child) {
- spin_lock(&loop->d_lock);
+ spin_lock_nested(&loop->d_lock, DENTRY_D_LOCK_NESTED);
dentry_lru_del_init(loop);
__d_drop(loop);
spin_unlock(&loop->d_lock);
- cond_resched_lock(&dcache_lock);
}
+ spin_unlock(&dentry->d_lock);
spin_unlock(&dcache_lock);
/* move to the first child */
@@ -787,16 +825,17 @@ static void shrink_dcache_for_umount_sub
BUG();
}
- if (IS_ROOT(dentry))
+ if (IS_ROOT(dentry)) {
parent = NULL;
- else {
+ list_del(&dentry->d_u.d_child);
+ } else {
parent = dentry->d_parent;
spin_lock(&parent->d_lock);
parent->d_count--;
+ list_del(&dentry->d_u.d_child);
spin_unlock(&parent->d_lock);
}
- list_del(&dentry->d_u.d_child);
detached++;
inode = dentry->d_inode;
@@ -881,6 +920,7 @@ int have_submounts(struct dentry *parent
spin_lock(&dcache_lock);
if (d_mountpoint(parent))
goto positive;
+ spin_lock(&this_parent->d_lock);
repeat:
next = this_parent->d_subdirs.next;
resume:
@@ -888,22 +928,34 @@ resume:
struct list_head *tmp = next;
struct dentry *dentry = list_entry(tmp, struct dentry, d_u.d_child);
next = tmp->next;
+
+ spin_lock_nested(&dentry->d_lock, DENTRY_D_LOCK_NESTED);
/* Have we found a mount point ? */
- if (d_mountpoint(dentry))
+ if (d_mountpoint(dentry)) {
+ spin_unlock(&dentry->d_lock);
+ spin_unlock(&this_parent->d_lock);
goto positive;
+ }
if (!list_empty(&dentry->d_subdirs)) {
+ spin_unlock(&this_parent->d_lock);
+ spin_release(&dentry->d_lock.dep_map, 1, _RET_IP_);
this_parent = dentry;
+ spin_acquire(&this_parent->d_lock.dep_map, 0, 1, _RET_IP_);
goto repeat;
}
+ spin_unlock(&dentry->d_lock);
}
/*
* All done at this level ... ascend and resume the search.
*/
if (this_parent != parent) {
next = this_parent->d_u.d_child.next;
+ spin_unlock(&this_parent->d_lock);
this_parent = this_parent->d_parent;
+ spin_lock(&this_parent->d_lock);
goto resume;
}
+ spin_unlock(&this_parent->d_lock);
spin_unlock(&dcache_lock);
return 0; /* No mount points found in tree */
positive:
@@ -932,6 +984,7 @@ static int select_parent(struct dentry *
int found = 0;
spin_lock(&dcache_lock);
+ spin_lock(&this_parent->d_lock);
repeat:
next = this_parent->d_subdirs.next;
resume:
@@ -939,8 +992,9 @@ resume:
struct list_head *tmp = next;
struct dentry *dentry = list_entry(tmp, struct dentry, d_u.d_child);
next = tmp->next;
+ BUG_ON(this_parent == dentry);
- spin_lock(&dentry->d_lock);
+ spin_lock_nested(&dentry->d_lock, DENTRY_D_LOCK_NESTED);
dentry_lru_del_init(dentry);
/*
* move only zero ref count dentries to the end
@@ -950,33 +1004,45 @@ resume:
dentry_lru_add_tail(dentry);
found++;
}
- spin_unlock(&dentry->d_lock);
/*
* We can return to the caller if we have found some (this
* ensures forward progress). We'll be coming back to find
* the rest.
*/
- if (found && need_resched())
+ if (found && need_resched()) {
+ spin_unlock(&dentry->d_lock);
goto out;
+ }
/*
* Descend a level if the d_subdirs list is non-empty.
*/
if (!list_empty(&dentry->d_subdirs)) {
+ spin_unlock(&this_parent->d_lock);
+ spin_release(&dentry->d_lock.dep_map, 1, _RET_IP_);
this_parent = dentry;
+ spin_acquire(&this_parent->d_lock.dep_map, 0, 1, _RET_IP_);
goto repeat;
}
+
+ spin_unlock(&dentry->d_lock);
}
/*
* All done at this level ... ascend and resume the search.
*/
if (this_parent != parent) {
+ struct dentry *tmp;
next = this_parent->d_u.d_child.next;
- this_parent = this_parent->d_parent;
+ tmp = this_parent->d_parent;
+ spin_unlock(&this_parent->d_lock);
+ BUG_ON(tmp == this_parent);
+ this_parent = tmp;
+ spin_lock(&this_parent->d_lock);
goto resume;
}
out:
+ spin_unlock(&this_parent->d_lock);
spin_unlock(&dcache_lock);
return found;
}
@@ -1072,19 +1138,20 @@ struct dentry *d_alloc(struct dentry * p
INIT_LIST_HEAD(&dentry->d_lru);
INIT_LIST_HEAD(&dentry->d_subdirs);
INIT_LIST_HEAD(&dentry->d_alias);
-
- if (parent) {
- dentry->d_parent = dget(parent);
- dentry->d_sb = parent->d_sb;
- } else {
- INIT_LIST_HEAD(&dentry->d_u.d_child);
- }
+ INIT_LIST_HEAD(&dentry->d_u.d_child);
if (parent) {
spin_lock(&dcache_lock);
+ spin_lock(&parent->d_lock);
+ spin_lock_nested(&dentry->d_lock, DENTRY_D_LOCK_NESTED);
+ dentry->d_parent = dget_dlock(parent);
+ dentry->d_sb = parent->d_sb;
list_add(&dentry->d_u.d_child, &parent->d_subdirs);
+ spin_unlock(&dentry->d_lock);
+ spin_unlock(&parent->d_lock);
spin_unlock(&dcache_lock);
}
+
atomic_inc(&dentry_stat.nr_dentry);
return dentry;
@@ -1763,15 +1830,27 @@ static void d_move_locked(struct dentry
printk(KERN_WARNING "VFS: moving negative dcache entry\n");
write_seqlock(&rename_lock);
- /*
- * XXXX: do we really need to take target->d_lock?
- */
+
+ if (target->d_parent != dentry->d_parent) {
+ if (target->d_parent < dentry->d_parent) {
+ spin_lock(&target->d_parent->d_lock);
+ spin_lock_nested(&dentry->d_parent->d_lock,
+ DENTRY_D_LOCK_NESTED);
+ } else {
+ spin_lock(&dentry->d_parent->d_lock);
+ spin_lock_nested(&target->d_parent->d_lock,
+ DENTRY_D_LOCK_NESTED);
+ }
+ } else {
+ spin_lock(&target->d_parent->d_lock);
+ }
+
if (target < dentry) {
- spin_lock(&target->d_lock);
- spin_lock_nested(&dentry->d_lock, DENTRY_D_LOCK_NESTED);
+ spin_lock_nested(&target->d_lock, 2);
+ spin_lock_nested(&dentry->d_lock, 3);
} else {
- spin_lock(&dentry->d_lock);
- spin_lock_nested(&target->d_lock, DENTRY_D_LOCK_NESTED);
+ spin_lock_nested(&dentry->d_lock, 2);
+ spin_lock_nested(&target->d_lock, 3);
}
/* Move the dentry to the target hash queue, if on different bucket */
@@ -1804,7 +1883,10 @@ static void d_move_locked(struct dentry
}
list_add(&dentry->d_u.d_child, &dentry->d_parent->d_subdirs);
+ if (target->d_parent != dentry->d_parent)
+ spin_unlock(&dentry->d_parent->d_lock);
spin_unlock(&target->d_lock);
+ spin_unlock(&target->d_parent->d_lock);
fsnotify_d_move(dentry);
spin_unlock(&dentry->d_lock);
write_sequnlock(&rename_lock);
@@ -1903,6 +1985,12 @@ static void __d_materialise_dentry(struc
dparent = dentry->d_parent;
aparent = anon->d_parent;
+ /* XXX: hack */
+ spin_lock(&aparent->d_lock);
+ spin_lock(&dparent->d_lock);
+ spin_lock(&dentry->d_lock);
+ spin_lock(&anon->d_lock);
+
dentry->d_parent = (aparent == anon) ? dentry : aparent;
list_del(&dentry->d_u.d_child);
if (!IS_ROOT(dentry))
@@ -1917,6 +2005,11 @@ static void __d_materialise_dentry(struc
else
INIT_LIST_HEAD(&anon->d_u.d_child);
+ spin_unlock(&anon->d_lock);
+ spin_unlock(&dentry->d_lock);
+ spin_unlock(&dparent->d_lock);
+ spin_unlock(&aparent->d_lock);
+
anon->d_flags &= ~DCACHE_DISCONNECTED;
}
@@ -2315,6 +2408,7 @@ void d_genocide(struct dentry *root)
struct list_head *next;
spin_lock(&dcache_lock);
+ spin_lock(&this_parent->d_lock);
repeat:
next = this_parent->d_subdirs.next;
resume:
@@ -2328,8 +2422,10 @@ resume:
continue;
}
if (!list_empty(&dentry->d_subdirs)) {
- spin_unlock(&dentry->d_lock);
+ spin_unlock(&this_parent->d_lock);
+ spin_release(&dentry->d_lock.dep_map, 1, _RET_IP_);
this_parent = dentry;
+ spin_acquire(&this_parent->d_lock.dep_map, 0, 1, _RET_IP_);
goto repeat;
}
dentry->d_count--;
@@ -2337,12 +2433,13 @@ resume:
}
if (this_parent != root) {
next = this_parent->d_u.d_child.next;
- spin_lock(&this_parent->d_lock);
this_parent->d_count--;
spin_unlock(&this_parent->d_lock);
this_parent = this_parent->d_parent;
+ spin_lock(&this_parent->d_lock);
goto resume;
}
+ spin_unlock(&this_parent->d_lock);
spin_unlock(&dcache_lock);
}
Index: linux-2.6/fs/libfs.c
===================================================================
--- linux-2.6.orig/fs/libfs.c
+++ linux-2.6/fs/libfs.c
@@ -84,7 +84,8 @@ int dcache_dir_close(struct inode *inode
loff_t dcache_dir_lseek(struct file *file, loff_t offset, int origin)
{
- mutex_lock(&file->f_path.dentry->d_inode->i_mutex);
+ struct dentry *dentry = file->f_path.dentry;
+ mutex_lock(&dentry->d_inode->i_mutex);
switch (origin) {
case 1:
offset += file->f_pos;
@@ -92,7 +93,7 @@ loff_t dcache_dir_lseek(struct file *fil
if (offset >= 0)
break;
default:
- mutex_unlock(&file->f_path.dentry->d_inode->i_mutex);
+ mutex_unlock(&dentry->d_inode->i_mutex);
return -EINVAL;
}
if (offset != file->f_pos) {
@@ -102,23 +103,27 @@ loff_t dcache_dir_lseek(struct file *fil
struct dentry *cursor = file->private_data;
loff_t n = file->f_pos - 2;
- spin_lock(&dcache_lock);
+ spin_lock(&dentry->d_lock);
+ spin_lock_nested(&cursor->d_lock, DENTRY_D_LOCK_NESTED);
list_del(&cursor->d_u.d_child);
- p = file->f_path.dentry->d_subdirs.next;
- while (n && p != &file->f_path.dentry->d_subdirs) {
+ spin_unlock(&cursor->d_lock);
+ p = dentry->d_subdirs.next;
+ while (n && p != &dentry->d_subdirs) {
struct dentry *next;
next = list_entry(p, struct dentry, d_u.d_child);
- spin_lock(&next->d_lock);
+ spin_lock_nested(&next->d_lock, DENTRY_D_LOCK_NESTED);
if (simple_positive(next))
n--;
spin_unlock(&next->d_lock);
p = p->next;
}
+ spin_lock_nested(&cursor->d_lock, DENTRY_D_LOCK_NESTED);
list_add_tail(&cursor->d_u.d_child, p);
- spin_unlock(&dcache_lock);
+ spin_unlock(&cursor->d_lock);
+ spin_unlock(&dentry->d_lock);
}
}
- mutex_unlock(&file->f_path.dentry->d_inode->i_mutex);
+ mutex_unlock(&dentry->d_inode->i_mutex);
return offset;
}
@@ -158,9 +163,12 @@ int dcache_readdir(struct file * filp, v
i++;
/* fallthrough */
default:
- spin_lock(&dcache_lock);
- if (filp->f_pos == 2)
+ spin_lock(&dentry->d_lock);
+ if (filp->f_pos == 2) {
+ spin_lock_nested(&cursor->d_lock, DENTRY_D_LOCK_NESTED);
list_move(q, &dentry->d_subdirs);
+ spin_unlock(&cursor->d_lock);
+ }
for (p=q->next; p != &dentry->d_subdirs; p=p->next) {
struct dentry *next;
@@ -172,19 +180,21 @@ int dcache_readdir(struct file * filp, v
}
spin_unlock(&next->d_lock);
- spin_unlock(&dcache_lock);
+ spin_unlock(&dentry->d_lock);
if (filldir(dirent, next->d_name.name,
next->d_name.len, filp->f_pos,
next->d_inode->i_ino,
dt_type(next->d_inode)) < 0)
return 0;
- spin_lock(&dcache_lock);
+ spin_lock(&dentry->d_lock);
+ spin_lock_nested(&next->d_lock, DENTRY_D_LOCK_NESTED);
/* next is still alive */
list_move(q, p);
+ spin_unlock(&next->d_lock);
p = q;
filp->f_pos++;
}
- spin_unlock(&dcache_lock);
+ spin_unlock(&dentry->d_lock);
}
return 0;
}
@@ -281,7 +291,7 @@ int simple_empty(struct dentry *dentry)
struct dentry *child;
int ret = 0;
- spin_lock(&dcache_lock);
+ spin_lock(&dentry->d_lock);
list_for_each_entry(child, &dentry->d_subdirs, d_u.d_child) {
spin_lock_nested(&child->d_lock, DENTRY_D_LOCK_NESTED);
if (simple_positive(child)) {
@@ -292,7 +302,7 @@ int simple_empty(struct dentry *dentry)
}
ret = 1;
out:
- spin_unlock(&dcache_lock);
+ spin_unlock(&dentry->d_lock);
return ret;
}
Index: linux-2.6/fs/notify/inotify/inotify.c
===================================================================
--- linux-2.6.orig/fs/notify/inotify/inotify.c
+++ linux-2.6/fs/notify/inotify/inotify.c
@@ -189,17 +189,19 @@ static void set_dentry_child_flags(struc
list_for_each_entry(alias, &inode->i_dentry, d_alias) {
struct dentry *child;
+ spin_lock(&alias->d_lock);
list_for_each_entry(child, &alias->d_subdirs, d_u.d_child) {
if (!child->d_inode)
continue;
- spin_lock(&child->d_lock);
+ spin_lock_nested(&child->d_lock, DENTRY_D_LOCK_NESTED);
if (watched)
child->d_flags |= DCACHE_INOTIFY_PARENT_WATCHED;
else
child->d_flags &=~DCACHE_INOTIFY_PARENT_WATCHED;
spin_unlock(&child->d_lock);
}
+ spin_unlock(&alias->d_lock);
}
spin_unlock(&dcache_lock);
}
Index: linux-2.6/include/linux/dcache.h
===================================================================
--- linux-2.6.orig/include/linux/dcache.h
+++ linux-2.6/include/linux/dcache.h
@@ -340,6 +340,7 @@ static inline struct dentry *dget_dlock(
}
return dentry;
}
+
static inline struct dentry *dget(struct dentry *dentry)
{
if (dentry) {
Index: linux-2.6/drivers/usb/core/inode.c
===================================================================
--- linux-2.6.orig/drivers/usb/core/inode.c
+++ linux-2.6/drivers/usb/core/inode.c
@@ -349,16 +349,18 @@ static int usbfs_empty (struct dentry *d
struct list_head *list;
spin_lock(&dcache_lock);
-
+ spin_lock(&dentry->d_lock);
list_for_each(list, &dentry->d_subdirs) {
struct dentry *de = list_entry(list, struct dentry, d_u.d_child);
if (usbfs_positive(de)) {
+ spin_unlock(&dentry->d_lock);
spin_unlock(&dcache_lock);
return 0;
}
}
-
+ spin_unlock(&dentry->d_lock);
spin_unlock(&dcache_lock);
+
return 1;
}
Index: linux-2.6/fs/autofs4/expire.c
===================================================================
--- linux-2.6.orig/fs/autofs4/expire.c
+++ linux-2.6/fs/autofs4/expire.c
@@ -93,22 +93,63 @@ done:
/*
* Calculate next entry in top down tree traversal.
* From next_mnt in namespace.c - elegant.
+ *
+ * How is this supposed to work if we drop dcache_lock between calls anyway?
+ * How does it cope with renames?
+ * And also callers dput the returned dentry before taking dcache_lock again
+ * so what prevents it from being freed??
*/
-static struct dentry *next_dentry(struct dentry *p, struct dentry *root)
+static struct dentry *get_next_positive_dentry(struct dentry *p,
+ struct dentry *root)
{
- struct list_head *next = p->d_subdirs.next;
+ struct list_head *next;
+ struct dentry *ret;
+ spin_lock(&dcache_lock);
+ spin_lock(&p->d_lock);
+again:
+ next = p->d_subdirs.next;
if (next == &p->d_subdirs) {
while (1) {
- if (p == root)
+ struct dentry *parent;
+
+ if (p == root) {
+ spin_unlock(&p->d_lock);
return NULL;
+ }
+
+ parent = p->d_parent;
+ if (!spin_trylock(&parent->d_lock)) {
+ dget_dlock(p);
+ spin_unlock(&p->d_lock);
+ parent = dget_parent(p);
+ spin_unlock(&dcache_lock);
+ dput(p);
+ spin_lock(&dcache_lock);
+ spin_lock(&parent->d_lock);
+ } else
+ spin_unlock(&p->d_lock);
next = p->d_u.d_child.next;
- if (next != &p->d_parent->d_subdirs)
+ p = parent;
+ if (next != &parent->d_subdirs)
break;
- p = p->d_parent;
}
}
- return list_entry(next, struct dentry, d_u.d_child);
+ ret = list_entry(next, struct dentry, d_u.d_child);
+
+ spin_lock(&ret->d_lock);
+ /* Negative dentry - give up */
+ if (!simple_positive(ret)) {
+ spin_unlock(&ret->d_lock);
+ p = ret;
+ goto again;
+ }
+ dget_dlock(ret);
+ spin_unlock(&ret->d_lock);
+
+ spin_unlock(&dcache_lock);
+
+ return ret;
}
/*
@@ -158,18 +199,11 @@ static int autofs4_tree_busy(struct vfsm
if (!simple_positive(top))
return 1;
- spin_lock(&dcache_lock);
- for (p = top; p; p = next_dentry(p, top)) {
- /* Negative dentry - give up */
- if (!simple_positive(p))
- continue;
+ for (p = top; p; p = get_next_positive_dentry(p, top)) {
DPRINTK("dentry %p %.*s",
p, (int) p->d_name.len, p->d_name.name);
- p = dget(p);
- spin_unlock(&dcache_lock);
-
/*
* Is someone visiting anywhere in the subtree ?
* If there's no mount we need to check the usage
@@ -205,9 +239,7 @@ static int autofs4_tree_busy(struct vfsm
}
}
dput(p);
- spin_lock(&dcache_lock);
}
- spin_unlock(&dcache_lock);
/* Timeout of a tree mount is ultimately determined by its top dentry */
if (!autofs4_can_expire(top, timeout, do_now))
@@ -226,18 +258,11 @@ static struct dentry *autofs4_check_leav
DPRINTK("parent %p %.*s",
parent, (int)parent->d_name.len, parent->d_name.name);
- spin_lock(&dcache_lock);
- for (p = parent; p; p = next_dentry(p, parent)) {
- /* Negative dentry - give up */
- if (!simple_positive(p))
- continue;
+ for (p = parent; p; p = get_next_positive_dentry(p, parent)) {
DPRINTK("dentry %p %.*s",
p, (int) p->d_name.len, p->d_name.name);
- p = dget(p);
- spin_unlock(&dcache_lock);
-
if (d_mountpoint(p)) {
/* Can we umount this guy */
if (autofs4_mount_busy(mnt, p))
@@ -249,9 +274,7 @@ static struct dentry *autofs4_check_leav
}
cont:
dput(p);
- spin_lock(&dcache_lock);
}
- spin_unlock(&dcache_lock);
return NULL;
}
@@ -316,6 +339,7 @@ struct dentry *autofs4_expire_indirect(s
timeout = sbi->exp_timeout;
spin_lock(&dcache_lock);
+ spin_lock(&root->d_lock);
next = root->d_subdirs.next;
/* On exit from the loop expire is set to a dgot dentry
@@ -330,6 +354,7 @@ struct dentry *autofs4_expire_indirect(s
}
dentry = dget(dentry);
+ spin_unlock(&root->d_lock);
spin_unlock(&dcache_lock);
spin_lock(&sbi->fs_lock);
@@ -396,8 +421,10 @@ next:
spin_unlock(&sbi->fs_lock);
dput(dentry);
spin_lock(&dcache_lock);
+ spin_lock(&root->d_lock);
next = next->next;
}
+ spin_unlock(&root->d_lock);
spin_unlock(&dcache_lock);
return NULL;
@@ -409,7 +436,9 @@ found:
init_completion(&ino->expire_complete);
spin_unlock(&sbi->fs_lock);
spin_lock(&dcache_lock);
+ spin_lock(&expired->d_parent->d_lock);
list_move(&expired->d_parent->d_subdirs, &expired->d_u.d_child);
+ spin_unlock(&expired->d_parent->d_lock);
spin_unlock(&dcache_lock);
return expired;
}
Index: linux-2.6/fs/autofs4/inode.c
===================================================================
--- linux-2.6.orig/fs/autofs4/inode.c
+++ linux-2.6/fs/autofs4/inode.c
@@ -111,6 +111,7 @@ static void autofs4_force_release(struct
spin_lock(&dcache_lock);
repeat:
+ spin_lock(&this_parent->d_lock);
next = this_parent->d_subdirs.next;
resume:
while (next != &this_parent->d_subdirs) {
@@ -128,6 +129,7 @@ resume:
}
next = next->next;
+ spin_unlock(&this_parent->d_lock);
spin_unlock(&dcache_lock);
DPRINTK("dentry %p %.*s",
@@ -141,15 +143,18 @@ resume:
struct dentry *dentry = this_parent;
next = this_parent->d_u.d_child.next;
+ spin_unlock(&this_parent->d_lock);
this_parent = this_parent->d_parent;
spin_unlock(&dcache_lock);
DPRINTK("parent dentry %p %.*s",
dentry, (int)dentry->d_name.len, dentry->d_name.name);
dput(dentry);
spin_lock(&dcache_lock);
+ spin_lock(&this_parent->d_lock);
goto resume;
}
spin_unlock(&dcache_lock);
+ spin_unlock(&this_parent->d_lock);
}
void autofs4_kill_sb(struct super_block *sb)
Index: linux-2.6/fs/autofs4/root.c
===================================================================
--- linux-2.6.orig/fs/autofs4/root.c
+++ linux-2.6/fs/autofs4/root.c
@@ -93,10 +93,13 @@ static int autofs4_dir_open(struct inode
* it.
*/
spin_lock(&dcache_lock);
+ spin_lock(&dentry->d_lock);
if (!d_mountpoint(dentry) && __simple_empty(dentry)) {
+ spin_unlock(&dentry->d_lock);
spin_unlock(&dcache_lock);
return -ENOENT;
}
+ spin_unlock(&dentry->d_lock);
spin_unlock(&dcache_lock);
out:
@@ -212,8 +215,10 @@ static void *autofs4_follow_link(struct
* mount it again.
*/
spin_lock(&dcache_lock);
+ spin_lock(&dentry->d_lock);
if (dentry->d_flags & DCACHE_AUTOFS_PENDING ||
(!d_mountpoint(dentry) && __simple_empty(dentry))) {
+ spin_unlock(&dentry->d_lock);
spin_unlock(&dcache_lock);
status = try_to_fill_dentry(dentry, 0);
@@ -222,6 +227,7 @@ static void *autofs4_follow_link(struct
goto follow;
}
+ spin_unlock(&dentry->d_lock);
spin_unlock(&dcache_lock);
follow:
/*
@@ -730,7 +736,9 @@ static int autofs4_dir_rmdir(struct inod
return -EACCES;
spin_lock(&dcache_lock);
+ spin_lock(&dentry->d_lock);
if (!list_empty(&dentry->d_subdirs)) {
+ spin_unlock(&dentry->d_lock);
spin_unlock(&dcache_lock);
return -ENOTEMPTY;
}
@@ -738,7 +746,6 @@ static int autofs4_dir_rmdir(struct inod
if (list_empty(&ino->expiring))
list_add(&ino->expiring, &sbi->expiring_list);
spin_unlock(&sbi->lookup_lock);
- spin_lock(&dentry->d_lock);
__d_drop(dentry);
spin_unlock(&dentry->d_lock);
spin_unlock(&dcache_lock);
Index: linux-2.6/fs/coda/cache.c
===================================================================
--- linux-2.6.orig/fs/coda/cache.c
+++ linux-2.6/fs/coda/cache.c
@@ -87,6 +87,7 @@ static void coda_flag_children(struct de
struct dentry *de;
spin_lock(&dcache_lock);
+ spin_lock(&parent->d_lock);
list_for_each(child, &parent->d_subdirs)
{
de = list_entry(child, struct dentry, d_u.d_child);
@@ -95,6 +96,7 @@ static void coda_flag_children(struct de
continue;
coda_flag_inode(de->d_inode, flag);
}
+ spin_unlock(&parent->d_lock);
spin_unlock(&dcache_lock);
return;
}
Index: linux-2.6/fs/ncpfs/dir.c
===================================================================
--- linux-2.6.orig/fs/ncpfs/dir.c
+++ linux-2.6/fs/ncpfs/dir.c
@@ -365,6 +365,7 @@ ncp_dget_fpos(struct dentry *dentry, str
/* If a pointer is invalid, we search the dentry. */
spin_lock(&dcache_lock);
+ spin_lock(&parent->d_lock);
next = parent->d_subdirs.next;
while (next != &parent->d_subdirs) {
dent = list_entry(next, struct dentry, d_u.d_child);
@@ -373,11 +374,13 @@ ncp_dget_fpos(struct dentry *dentry, str
dget_locked(dent);
else
dent = NULL;
+ spin_unlock(&parent->d_lock);
spin_unlock(&dcache_lock);
goto out;
}
next = next->next;
}
+ spin_unlock(&parent->d_lock);
spin_unlock(&dcache_lock);
return NULL;
Index: linux-2.6/fs/ncpfs/ncplib_kernel.h
===================================================================
--- linux-2.6.orig/fs/ncpfs/ncplib_kernel.h
+++ linux-2.6/fs/ncpfs/ncplib_kernel.h
@@ -193,6 +193,7 @@ ncp_renew_dentries(struct dentry *parent
struct dentry *dentry;
spin_lock(&dcache_lock);
+ spin_lock(&parent->d_lock);
next = parent->d_subdirs.next;
while (next != &parent->d_subdirs) {
dentry = list_entry(next, struct dentry, d_u.d_child);
@@ -204,6 +205,7 @@ ncp_renew_dentries(struct dentry *parent
next = next->next;
}
+ spin_unlock(&parent->d_lock);
spin_unlock(&dcache_lock);
}
@@ -215,6 +217,7 @@ ncp_invalidate_dircache_entries(struct d
struct dentry *dentry;
spin_lock(&dcache_lock);
+ spin_lock(&parent->d_lock);
next = parent->d_subdirs.next;
while (next != &parent->d_subdirs) {
dentry = list_entry(next, struct dentry, d_u.d_child);
@@ -222,6 +225,7 @@ ncp_invalidate_dircache_entries(struct d
ncp_age_dentry(server, dentry);
next = next->next;
}
+ spin_unlock(&parent->d_lock);
spin_unlock(&dcache_lock);
}
Index: linux-2.6/fs/smbfs/cache.c
===================================================================
--- linux-2.6.orig/fs/smbfs/cache.c
+++ linux-2.6/fs/smbfs/cache.c
@@ -63,6 +63,7 @@ smb_invalidate_dircache_entries(struct d
struct dentry *dentry;
spin_lock(&dcache_lock);
+ spin_lock(&parent->d_lock);
next = parent->d_subdirs.next;
while (next != &parent->d_subdirs) {
dentry = list_entry(next, struct dentry, d_u.d_child);
@@ -70,6 +71,7 @@ smb_invalidate_dircache_entries(struct d
smb_age_dentry(server, dentry);
next = next->next;
}
+ spin_unlock(&parent->d_lock);
spin_unlock(&dcache_lock);
}
@@ -97,6 +99,7 @@ smb_dget_fpos(struct dentry *dentry, str
/* If a pointer is invalid, we search the dentry. */
spin_lock(&dcache_lock);
+ spin_lock(&parent->d_lock);
next = parent->d_subdirs.next;
while (next != &parent->d_subdirs) {
dent = list_entry(next, struct dentry, d_u.d_child);
@@ -111,6 +114,7 @@ smb_dget_fpos(struct dentry *dentry, str
}
dent = NULL;
out_unlock:
+ spin_unlock(&parent->d_lock);
spin_unlock(&dcache_lock);
return dent;
}
Index: linux-2.6/kernel/cgroup.c
===================================================================
--- linux-2.6.orig/kernel/cgroup.c
+++ linux-2.6/kernel/cgroup.c
@@ -696,23 +696,31 @@ static void cgroup_clear_directory(struc
BUG_ON(!mutex_is_locked(&dentry->d_inode->i_mutex));
spin_lock(&dcache_lock);
+ spin_lock(&dentry->d_lock);
node = dentry->d_subdirs.next;
while (node != &dentry->d_subdirs) {
struct dentry *d = list_entry(node, struct dentry, d_u.d_child);
+
+ spin_lock_nested(&d->d_lock, DENTRY_D_LOCK_NESTED);
list_del_init(node);
if (d->d_inode) {
/* This should never be called on a cgroup
* directory with child cgroups */
BUG_ON(d->d_inode->i_mode & S_IFDIR);
- d = dget_locked(d);
+ dget_locked_dlock(d);
+ spin_unlock(&d->d_lock);
+ spin_unlock(&dentry->d_lock);
spin_unlock(&dcache_lock);
d_delete(d);
simple_unlink(dentry->d_inode, d);
dput(d);
spin_lock(&dcache_lock);
- }
+ spin_lock(&dentry->d_lock);
+ } else
+ spin_unlock(&d->d_lock);
node = dentry->d_subdirs.next;
}
+ spin_unlock(&dentry->d_lock);
spin_unlock(&dcache_lock);
}
@@ -721,10 +729,17 @@ static void cgroup_clear_directory(struc
*/
static void cgroup_d_remove_dir(struct dentry *dentry)
{
+ struct dentry *parent;
+
cgroup_clear_directory(dentry);
spin_lock(&dcache_lock);
+ parent = dentry->d_parent;
+ spin_lock(&parent->d_lock);
+ spin_lock(&dentry->d_lock);
list_del_init(&dentry->d_u.d_child);
+ spin_unlock(&dentry->d_lock);
+ spin_unlock(&parent->d_lock);
spin_unlock(&dcache_lock);
remove_dir(dentry);
}
Index: linux-2.6/net/sunrpc/rpc_pipe.c
===================================================================
--- linux-2.6.orig/net/sunrpc/rpc_pipe.c
+++ linux-2.6/net/sunrpc/rpc_pipe.c
@@ -548,6 +548,7 @@ static void rpc_depopulate(struct dentry
mutex_lock_nested(&dir->i_mutex, I_MUTEX_CHILD);
repeat:
spin_lock(&dcache_lock);
+ spin_lock(&parent->d_lock);
list_for_each_safe(pos, next, &parent->d_subdirs) {
dentry = list_entry(pos, struct dentry, d_u.d_child);
if (!dentry->d_inode ||
@@ -565,6 +566,7 @@ repeat:
} else
spin_unlock(&dentry->d_lock);
}
+ spin_unlock(&parent->d_lock);
spin_unlock(&dcache_lock);
if (n) {
do {
Index: linux-2.6/security/selinux/selinuxfs.c
===================================================================
--- linux-2.6.orig/security/selinux/selinuxfs.c
+++ linux-2.6/security/selinux/selinuxfs.c
@@ -944,22 +944,30 @@ static void sel_remove_entries(struct de
struct list_head *node;
spin_lock(&dcache_lock);
+ spin_lock(&de->d_lock);
node = de->d_subdirs.next;
while (node != &de->d_subdirs) {
struct dentry *d = list_entry(node, struct dentry, d_u.d_child);
+
+ spin_lock_nested(&d->d_lock, DENTRY_D_LOCK_NESTED);
list_del_init(node);
if (d->d_inode) {
- d = dget_locked(d);
+ dget_locked_dlock(d);
+ spin_unlock(&de->d_lock);
+ spin_unlock(&d->d_lock);
spin_unlock(&dcache_lock);
d_delete(d);
simple_unlink(de->d_inode, d);
dput(d);
spin_lock(&dcache_lock);
- }
+ spin_lock(&de->d_lock);
+ } else
+ spin_unlock(&d->d_lock);
node = de->d_subdirs.next;
}
+ spin_unlock(&de->d_lock);
spin_unlock(&dcache_lock);
}
Index: linux-2.6/fs/notify/fsnotify.c
===================================================================
--- linux-2.6.orig/fs/notify/fsnotify.c
+++ linux-2.6/fs/notify/fsnotify.c
@@ -61,17 +61,19 @@ void __fsnotify_update_child_dentry_flag
/* run all of the children of the original inode and fix their
* d_flags to indicate parental interest (their parent is the
* original inode) */
+ spin_lock(&alias->d_lock);
list_for_each_entry(child, &alias->d_subdirs, d_u.d_child) {
if (!child->d_inode)
continue;
- spin_lock(&child->d_lock);
+ spin_lock_nested(&child->d_lock, DENTRY_D_LOCK_NESTED);
if (watched)
child->d_flags |= DCACHE_FSNOTIFY_PARENT_WATCHED;
else
child->d_flags &= ~DCACHE_FSNOTIFY_PARENT_WATCHED;
spin_unlock(&child->d_lock);
}
+ spin_unlock(&alias->d_lock);
}
spin_unlock(&dcache_lock);
}
^ permalink raw reply [flat|nested] 64+ messages in thread
* [patch 12/33] fs: scale inode alias list
2009-09-04 6:51 [patch 00/33] my current vfs scalability patch queue npiggin
` (10 preceding siblings ...)
2009-09-04 6:51 ` [patch 11/33] fs: dcache scale subdirs npiggin
@ 2009-09-04 6:51 ` npiggin
2009-09-04 6:51 ` [patch 13/33] fs: use RCU / seqlock logic for reverse and multi-step operaitons npiggin
` (21 subsequent siblings)
33 siblings, 0 replies; 64+ messages in thread
From: npiggin @ 2009-09-04 6:51 UTC (permalink / raw)
To: linux-fsdevel; +Cc: linux-kernel
[-- Attachment #1: fs-dcache-scale-i_dentry.patch --]
[-- Type: text/plain, Size: 14042 bytes --]
Add a new lock, dcache_inode_lock, to protect the inode's i_dentry list
from concurrent modification. d_alias is also protected by d_lock.
---
fs/affs/amigaffs.c | 2 +
fs/dcache.c | 56 +++++++++++++++++++++++++++++++++++++++-----
fs/exportfs/expfs.c | 4 +++
fs/nfs/getroot.c | 4 +++
fs/notify/fsnotify.c | 2 +
fs/notify/inotify/inotify.c | 2 +
fs/ocfs2/dcache.c | 3 +-
fs/sysfs/dir.c | 3 ++
include/linux/dcache.h | 1
9 files changed, 70 insertions(+), 7 deletions(-)
Index: linux-2.6/fs/dcache.c
===================================================================
--- linux-2.6.orig/fs/dcache.c
+++ linux-2.6/fs/dcache.c
@@ -36,6 +36,8 @@
/*
* Usage:
+ * dcache_inode_lock protects:
+ * - the inode alias lists, d_inode
* dcache_hash_lock protects:
* - the dcache hash table
* dcache_lru_lock protects:
@@ -49,18 +51,21 @@
*
* Ordering:
* dcache_lock
- * dentry->d_lock
- * dcache_lru_lock
- * dcache_hash_lock
+ * dcache_inode_lock
+ * dentry->d_lock
+ * dcache_lru_lock
+ * dcache_hash_lock
*/
int sysctl_vfs_cache_pressure __read_mostly = 100;
EXPORT_SYMBOL_GPL(sysctl_vfs_cache_pressure);
+__cacheline_aligned_in_smp DEFINE_SPINLOCK(dcache_inode_lock);
__cacheline_aligned_in_smp DEFINE_SPINLOCK(dcache_hash_lock);
static __cacheline_aligned_in_smp DEFINE_SPINLOCK(dcache_lru_lock);
__cacheline_aligned_in_smp DEFINE_SPINLOCK(dcache_lock);
__cacheline_aligned_in_smp DEFINE_SEQLOCK(rename_lock);
+EXPORT_SYMBOL(dcache_inode_lock);
EXPORT_SYMBOL(dcache_hash_lock);
EXPORT_SYMBOL(dcache_lock);
@@ -125,6 +130,7 @@ static void d_free(struct dentry *dentry
*/
static void dentry_iput(struct dentry * dentry)
__releases(dentry->d_lock)
+ __releases(dcache_inode_lock)
__releases(dcache_lock)
{
struct inode *inode = dentry->d_inode;
@@ -132,6 +138,7 @@ static void dentry_iput(struct dentry *
dentry->d_inode = NULL;
list_del_init(&dentry->d_alias);
spin_unlock(&dentry->d_lock);
+ spin_unlock(&dcache_inode_lock);
spin_unlock(&dcache_lock);
if (!inode->i_nlink)
fsnotify_inoderemove(inode);
@@ -141,6 +148,7 @@ static void dentry_iput(struct dentry *
iput(inode);
} else {
spin_unlock(&dentry->d_lock);
+ spin_unlock(&dcache_inode_lock);
spin_unlock(&dcache_lock);
}
}
@@ -212,6 +220,7 @@ static void dentry_lru_del_init(struct d
*/
static struct dentry *d_kill(struct dentry *dentry)
__releases(dentry->d_lock)
+ __releases(dcache_inode_lock)
__releases(dcache_lock)
{
struct dentry *parent;
@@ -276,16 +285,21 @@ repeat:
* want to reduce dcache_lock anyway so this will
* get improved.
*/
+drop1:
spin_unlock(&dentry->d_lock);
goto repeat;
}
+ if (!spin_trylock(&dcache_inode_lock)) {
+drop2:
+ spin_unlock(&dcache_lock);
+ goto drop1;
+ }
parent = dentry->d_parent;
if (parent) {
BUG_ON(parent == dentry);
if (!spin_trylock(&parent->d_lock)) {
- spin_unlock(&dentry->d_lock);
- spin_unlock(&dcache_lock);
- goto repeat;
+ spin_unlock(&dcache_inode_lock);
+ goto drop2;
}
}
}
@@ -313,6 +327,7 @@ repeat:
spin_unlock(&dentry->d_lock);
if (parent)
spin_unlock(&parent->d_lock);
+ spin_unlock(&dcache_inode_lock);
spin_unlock(&dcache_lock);
return;
@@ -487,7 +502,9 @@ struct dentry * d_find_alias(struct inod
if (!list_empty(&inode->i_dentry)) {
spin_lock(&dcache_lock);
+ spin_lock(&dcache_inode_lock);
de = __d_find_alias(inode, 0);
+ spin_unlock(&dcache_inode_lock);
spin_unlock(&dcache_lock);
}
return de;
@@ -502,18 +519,21 @@ void d_prune_aliases(struct inode *inode
struct dentry *dentry;
restart:
spin_lock(&dcache_lock);
+ spin_lock(&dcache_inode_lock);
list_for_each_entry(dentry, &inode->i_dentry, d_alias) {
spin_lock(&dentry->d_lock);
if (!dentry->d_count) {
__dget_locked_dlock(dentry);
__d_drop(dentry);
spin_unlock(&dentry->d_lock);
+ spin_unlock(&dcache_inode_lock);
spin_unlock(&dcache_lock);
dput(dentry);
goto restart;
}
spin_unlock(&dentry->d_lock);
}
+ spin_unlock(&dcache_inode_lock);
spin_unlock(&dcache_lock);
}
@@ -541,6 +561,7 @@ static void prune_one_dentry(struct dent
struct dentry *parent = NULL;
spin_lock(&dcache_lock);
+ spin_lock(&dcache_inode_lock);
again:
spin_lock(&dentry->d_lock);
if (dentry->d_parent && dentry != dentry->d_parent) {
@@ -555,6 +576,7 @@ again:
if (parent)
spin_unlock(&parent->d_lock);
spin_unlock(&dentry->d_lock);
+ spin_unlock(&dcache_inode_lock);
spin_unlock(&dcache_lock);
return;
}
@@ -625,6 +647,7 @@ restart:
spin_unlock(&dcache_lru_lock);
spin_lock(&dcache_lock);
+ spin_lock(&dcache_inode_lock);
again:
spin_lock(&dcache_lru_lock); /* lru_lock also protects tmp list */
while (!list_empty(&tmp)) {
@@ -657,8 +680,10 @@ again1:
prune_one_dentry(dentry);
/* dcache_lock and dentry->d_lock dropped */
spin_lock(&dcache_lock);
+ spin_lock(&dcache_inode_lock);
spin_lock(&dcache_lru_lock);
}
+ spin_unlock(&dcache_inode_lock);
spin_unlock(&dcache_lock);
if (count == NULL && !list_empty(&sb->s_dentry_lru))
@@ -1195,7 +1220,9 @@ void d_instantiate(struct dentry *entry,
{
BUG_ON(!list_empty(&entry->d_alias));
spin_lock(&dcache_lock);
+ spin_lock(&dcache_inode_lock);
__d_instantiate(entry, inode);
+ spin_unlock(&dcache_inode_lock);
spin_unlock(&dcache_lock);
security_d_instantiate(entry, inode);
}
@@ -1255,7 +1282,9 @@ struct dentry *d_instantiate_unique(stru
BUG_ON(!list_empty(&entry->d_alias));
spin_lock(&dcache_lock);
+ spin_lock(&dcache_inode_lock);
result = __d_instantiate_unique(entry, inode);
+ spin_unlock(&dcache_inode_lock);
spin_unlock(&dcache_lock);
if (!result) {
@@ -1345,8 +1374,10 @@ struct dentry *d_obtain_alias(struct ino
tmp->d_parent = tmp; /* make sure dput doesn't croak */
spin_lock(&dcache_lock);
+ spin_lock(&dcache_inode_lock);
res = __d_find_alias(inode, 0);
if (res) {
+ spin_unlock(&dcache_inode_lock);
spin_unlock(&dcache_lock);
dput(tmp);
goto out_iput;
@@ -1361,6 +1392,7 @@ struct dentry *d_obtain_alias(struct ino
list_add(&tmp->d_alias, &inode->i_dentry);
hlist_add_head(&tmp->d_hash, &inode->i_sb->s_anon);
spin_unlock(&tmp->d_lock);
+ spin_unlock(&dcache_inode_lock);
spin_unlock(&dcache_lock);
return tmp;
@@ -1393,9 +1425,11 @@ struct dentry *d_splice_alias(struct ino
if (inode && S_ISDIR(inode->i_mode)) {
spin_lock(&dcache_lock);
+ spin_lock(&dcache_inode_lock);
new = __d_find_alias(inode, 1);
if (new) {
BUG_ON(!(new->d_flags & DCACHE_DISCONNECTED));
+ spin_unlock(&dcache_inode_lock);
spin_unlock(&dcache_lock);
security_d_instantiate(new, inode);
d_rehash(dentry);
@@ -1404,6 +1438,7 @@ struct dentry *d_splice_alias(struct ino
} else {
/* already taking dcache_lock, so d_add() by hand */
__d_instantiate(dentry, inode);
+ spin_unlock(&dcache_inode_lock);
spin_unlock(&dcache_lock);
security_d_instantiate(dentry, inode);
d_rehash(dentry);
@@ -1477,8 +1512,10 @@ struct dentry *d_add_ci(struct dentry *d
* already has a dentry.
*/
spin_lock(&dcache_lock);
+ spin_lock(&dcache_inode_lock);
if (!S_ISDIR(inode->i_mode) || list_empty(&inode->i_dentry)) {
__d_instantiate(found, inode);
+ spin_unlock(&dcache_inode_lock);
spin_unlock(&dcache_lock);
security_d_instantiate(found, inode);
return found;
@@ -1490,6 +1527,7 @@ struct dentry *d_add_ci(struct dentry *d
*/
new = list_entry(inode->i_dentry.next, struct dentry, d_alias);
dget_locked(new);
+ spin_unlock(&dcache_inode_lock);
spin_unlock(&dcache_lock);
security_d_instantiate(found, inode);
d_move(new, found);
@@ -1705,6 +1743,7 @@ void d_delete(struct dentry * dentry)
* Are we the only user?
*/
spin_lock(&dcache_lock);
+ spin_lock(&dcache_inode_lock);
spin_lock(&dentry->d_lock);
isdir = S_ISDIR(dentry->d_inode->i_mode);
if (dentry->d_count == 1) {
@@ -1717,6 +1756,7 @@ void d_delete(struct dentry * dentry)
__d_drop(dentry);
spin_unlock(&dentry->d_lock);
+ spin_unlock(&dcache_inode_lock);
spin_unlock(&dcache_lock);
fsnotify_nameremove(dentry, isdir);
@@ -1963,6 +2003,7 @@ out_unalias:
d_move_locked(alias, dentry);
ret = alias;
out_err:
+ spin_unlock(&dcache_inode_lock);
spin_unlock(&dcache_lock);
if (m2)
mutex_unlock(m2);
@@ -2028,6 +2069,7 @@ struct dentry *d_materialise_unique(stru
BUG_ON(!d_unhashed(dentry));
spin_lock(&dcache_lock);
+ spin_lock(&dcache_inode_lock);
if (!inode) {
actual = dentry;
@@ -2072,6 +2114,7 @@ found:
_d_rehash(actual);
spin_unlock(&dcache_hash_lock);
spin_unlock(&actual->d_lock);
+ spin_unlock(&dcache_inode_lock);
spin_unlock(&dcache_lock);
out_nolock:
if (actual == dentry) {
@@ -2083,6 +2126,7 @@ out_nolock:
return actual;
shouldnt_be_hashed:
+ spin_unlock(&dcache_inode_lock);
spin_unlock(&dcache_lock);
BUG();
}
Index: linux-2.6/fs/sysfs/dir.c
===================================================================
--- linux-2.6.orig/fs/sysfs/dir.c
+++ linux-2.6/fs/sysfs/dir.c
@@ -548,6 +548,7 @@ static void sysfs_drop_dentry(struct sys
*/
repeat:
spin_lock(&dcache_lock);
+ spin_lock(&dcache_inode_lock);
list_for_each_entry(dentry, &inode->i_dentry, d_alias) {
spin_lock(&dentry->d_lock);
if (d_unhashed(dentry)) {
@@ -557,10 +558,12 @@ repeat:
dget_locked_dlock(dentry);
__d_drop(dentry);
spin_unlock(&dentry->d_lock);
+ spin_unlock(&dcache_inode_lock);
spin_unlock(&dcache_lock);
dput(dentry);
goto repeat;
}
+ spin_unlock(&dcache_inode_lock);
spin_unlock(&dcache_lock);
/* adjust nlink and update timestamp */
Index: linux-2.6/include/linux/dcache.h
===================================================================
--- linux-2.6.orig/include/linux/dcache.h
+++ linux-2.6/include/linux/dcache.h
@@ -186,6 +186,7 @@ d_iput: no no no yes
#define DCACHE_FSNOTIFY_PARENT_WATCHED 0x0080 /* Parent inode is watched by some fsnotify listener */
+extern spinlock_t dcache_inode_lock;
extern spinlock_t dcache_hash_lock;
extern spinlock_t dcache_lock;
extern seqlock_t rename_lock;
Index: linux-2.6/fs/notify/inotify/inotify.c
===================================================================
--- linux-2.6.orig/fs/notify/inotify/inotify.c
+++ linux-2.6/fs/notify/inotify/inotify.c
@@ -186,6 +186,7 @@ static void set_dentry_child_flags(struc
struct dentry *alias;
spin_lock(&dcache_lock);
+ spin_lock(&dcache_inode_lock);
list_for_each_entry(alias, &inode->i_dentry, d_alias) {
struct dentry *child;
@@ -203,6 +204,7 @@ static void set_dentry_child_flags(struc
}
spin_unlock(&alias->d_lock);
}
+ spin_unlock(&dcache_inode_lock);
spin_unlock(&dcache_lock);
}
Index: linux-2.6/fs/exportfs/expfs.c
===================================================================
--- linux-2.6.orig/fs/exportfs/expfs.c
+++ linux-2.6/fs/exportfs/expfs.c
@@ -48,8 +48,10 @@ find_acceptable_alias(struct dentry *res
return result;
spin_lock(&dcache_lock);
+ spin_lock(&dcache_inode_lock);
list_for_each_entry(dentry, &result->d_inode->i_dentry, d_alias) {
dget_locked(dentry);
+ spin_unlock(&dcache_inode_lock);
spin_unlock(&dcache_lock);
if (toput)
dput(toput);
@@ -58,8 +60,10 @@ find_acceptable_alias(struct dentry *res
return dentry;
}
spin_lock(&dcache_lock);
+ spin_lock(&dcache_inode_lock);
toput = dentry;
}
+ spin_unlock(&dcache_inode_lock);
spin_unlock(&dcache_lock);
if (toput)
Index: linux-2.6/fs/affs/amigaffs.c
===================================================================
--- linux-2.6.orig/fs/affs/amigaffs.c
+++ linux-2.6/fs/affs/amigaffs.c
@@ -129,6 +129,7 @@ affs_fix_dcache(struct dentry *dentry, u
struct list_head *head, *next;
spin_lock(&dcache_lock);
+ spin_lock(&dcache_inode_lock);
head = &inode->i_dentry;
next = head->next;
while (next != head) {
@@ -139,6 +140,7 @@ affs_fix_dcache(struct dentry *dentry, u
}
next = next->next;
}
+ spin_unlock(&dcache_inode_lock);
spin_unlock(&dcache_lock);
}
Index: linux-2.6/fs/ocfs2/dcache.c
===================================================================
--- linux-2.6.orig/fs/ocfs2/dcache.c
+++ linux-2.6/fs/ocfs2/dcache.c
@@ -141,7 +141,7 @@ struct dentry *ocfs2_find_local_alias(st
struct dentry *dentry = NULL;
spin_lock(&dcache_lock);
-
+ spin_lock(&dcache_inode_lock);
list_for_each(p, &inode->i_dentry) {
dentry = list_entry(p, struct dentry, d_alias);
@@ -159,6 +159,7 @@ struct dentry *ocfs2_find_local_alias(st
dentry = NULL;
}
+ spin_unlock(&dcache_inode_lock);
spin_unlock(&dcache_lock);
return dentry;
Index: linux-2.6/fs/nfs/getroot.c
===================================================================
--- linux-2.6.orig/fs/nfs/getroot.c
+++ linux-2.6/fs/nfs/getroot.c
@@ -65,7 +65,11 @@ static int nfs_superblock_set_dummy_root
* Oops, since the test for IS_ROOT() will fail.
*/
spin_lock(&dcache_lock);
+ spin_lock(&dcache_inode_lock);
+ spin_lock(&sb->s_root->d_lock);
list_del_init(&sb->s_root->d_alias);
+ spin_unlock(&sb->s_root->d_lock);
+ spin_unlock(&dcache_inode_lock);
spin_unlock(&dcache_lock);
}
return 0;
Index: linux-2.6/fs/notify/fsnotify.c
===================================================================
--- linux-2.6.orig/fs/notify/fsnotify.c
+++ linux-2.6/fs/notify/fsnotify.c
@@ -53,6 +53,7 @@ void __fsnotify_update_child_dentry_flag
watched = fsnotify_inode_watches_children(inode);
spin_lock(&dcache_lock);
+ spin_lock(&dcache_inode_lock);
/* run all of the dentries associated with this inode. Since this is a
* directory, there damn well better only be one item on this list */
list_for_each_entry(alias, &inode->i_dentry, d_alias) {
@@ -75,6 +76,7 @@ void __fsnotify_update_child_dentry_flag
}
spin_unlock(&alias->d_lock);
}
+ spin_unlock(&dcache_inode_lock);
spin_unlock(&dcache_lock);
}
^ permalink raw reply [flat|nested] 64+ messages in thread
* [patch 13/33] fs: use RCU / seqlock logic for reverse and multi-step operaitons
2009-09-04 6:51 [patch 00/33] my current vfs scalability patch queue npiggin
` (11 preceding siblings ...)
2009-09-04 6:51 ` [patch 12/33] fs: scale inode alias list npiggin
@ 2009-09-04 6:51 ` npiggin
2009-09-04 6:51 ` [patch 14/33] fs: dcache remove dcache_lock npiggin
` (20 subsequent siblings)
33 siblings, 0 replies; 64+ messages in thread
From: npiggin @ 2009-09-04 6:51 UTC (permalink / raw)
To: linux-fsdevel; +Cc: linux-kernel
[-- Attachment #1: fs-dcache_lock-multi-step.patch --]
[-- Type: text/plain, Size: 11532 bytes --]
The remaining usages for dcache_lock is to allow atomic, multi-step read-side
operations over the directory tree by excluding modifications to the tree.
Also, to walk in the leaf->root direction in the tree where we don't have
a natural d_lock ordering.
This could be accomplished by taking every d_lock, but this would mean a
huge number of locks and actually gets very tricky.
Solve this instead by using the rename seqlock for multi-step read-side
operations. Insert operations are not serialised. Delete operations are
tricky when walking up the directory our parent might have been deleted
when dropping locks so also need to check and retry for that.
XXX: hmm, we could of course just take the rename lock if there is any worry
about livelock. Most of these are slow paths.
---
drivers/staging/pohmelfs/path_entry.c | 7 ++
fs/autofs4/waitq.c | 10 ++
fs/dcache.c | 116 +++++++++++++++++++++++++++++-----
fs/nfs/namespace.c | 10 ++
fs/seq_file.c | 6 +
5 files changed, 134 insertions(+), 15 deletions(-)
Index: linux-2.6/fs/dcache.c
===================================================================
--- linux-2.6.orig/fs/dcache.c
+++ linux-2.6/fs/dcache.c
@@ -936,11 +936,15 @@ void shrink_dcache_for_umount(struct sup
* Return true if the parent or its subdirectories contain
* a mount point
*/
-
int have_submounts(struct dentry *parent)
{
- struct dentry *this_parent = parent;
+ struct dentry *this_parent;
struct list_head *next;
+ unsigned seq;
+
+rename_retry:
+ this_parent = parent;
+ seq = read_seqbegin(&rename_lock);
spin_lock(&dcache_lock);
if (d_mountpoint(parent))
@@ -974,17 +978,38 @@ resume:
* All done at this level ... ascend and resume the search.
*/
if (this_parent != parent) {
- next = this_parent->d_u.d_child.next;
+ struct dentry *tmp;
+ struct dentry *child;
+
+ tmp = this_parent->d_parent;
+ rcu_read_lock();
spin_unlock(&this_parent->d_lock);
- this_parent = this_parent->d_parent;
+ child = this_parent;
+ this_parent = tmp;
spin_lock(&this_parent->d_lock);
+ /* might go back up the wrong parent if we have had a rename
+ * or deletion */
+ if (this_parent != child->d_parent ||
+ // d_unlinked(this_parent) || XXX
+ read_seqretry(&rename_lock, seq)) {
+ spin_unlock(&this_parent->d_lock);
+ spin_unlock(&dcache_lock);
+ rcu_read_unlock();
+ goto rename_retry;
+ }
+ rcu_read_unlock();
+ next = child->d_u.d_child.next;
goto resume;
}
spin_unlock(&this_parent->d_lock);
spin_unlock(&dcache_lock);
+ if (read_seqretry(&rename_lock, seq))
+ goto rename_retry;
return 0; /* No mount points found in tree */
positive:
spin_unlock(&dcache_lock);
+ if (read_seqretry(&rename_lock, seq))
+ goto rename_retry;
return 1;
}
@@ -1004,10 +1029,15 @@ positive:
*/
static int select_parent(struct dentry * parent)
{
- struct dentry *this_parent = parent;
+ struct dentry *this_parent;
struct list_head *next;
+ unsigned seq;
int found = 0;
+rename_retry:
+ this_parent = parent;
+ seq = read_seqbegin(&rename_lock);
+
spin_lock(&dcache_lock);
spin_lock(&this_parent->d_lock);
repeat:
@@ -1017,7 +1047,6 @@ resume:
struct list_head *tmp = next;
struct dentry *dentry = list_entry(tmp, struct dentry, d_u.d_child);
next = tmp->next;
- BUG_ON(this_parent == dentry);
spin_lock_nested(&dentry->d_lock, DENTRY_D_LOCK_NESTED);
dentry_lru_del_init(dentry);
@@ -1058,17 +1087,33 @@ resume:
*/
if (this_parent != parent) {
struct dentry *tmp;
- next = this_parent->d_u.d_child.next;
+ struct dentry *child;
+
tmp = this_parent->d_parent;
+ rcu_read_lock();
spin_unlock(&this_parent->d_lock);
- BUG_ON(tmp == this_parent);
+ child = this_parent;
this_parent = tmp;
spin_lock(&this_parent->d_lock);
+ /* might go back up the wrong parent if we have had a rename
+ * or deletion */
+ if (this_parent != child->d_parent ||
+ // d_unlinked(this_parent) || XXX
+ read_seqretry(&rename_lock, seq)) {
+ spin_unlock(&this_parent->d_lock);
+ spin_unlock(&dcache_lock);
+ rcu_read_unlock();
+ goto rename_retry;
+ }
+ rcu_read_unlock();
+ next = child->d_u.d_child.next;
goto resume;
}
out:
spin_unlock(&this_parent->d_lock);
spin_unlock(&dcache_lock);
+ if (read_seqretry(&rename_lock, seq))
+ goto rename_retry;
return found;
}
@@ -2173,6 +2218,7 @@ char *__d_path(const struct path *path,
char *end = buffer + buflen;
char *retval;
+ rcu_read_lock();
prepend(&end, &buflen, "\0", 1);
if (d_unlinked(dentry) &&
(prepend(&end, &buflen, " (deleted)", 10) != 0))
@@ -2208,6 +2254,7 @@ char *__d_path(const struct path *path,
}
out:
+ rcu_read_unlock();
return retval;
global_root:
@@ -2244,6 +2291,7 @@ char *d_path(const struct path *path, ch
char *res;
struct path root;
struct path tmp;
+ unsigned seq;
/*
* We have various synthetic filesystems that never get mounted. On
@@ -2259,6 +2307,9 @@ char *d_path(const struct path *path, ch
root = current->fs->root;
path_get(&root);
read_unlock(¤t->fs->lock);
+
+rename_retry:
+ seq = read_seqbegin(&rename_lock);
spin_lock(&dcache_lock);
vfsmount_read_lock();
spin_lock(&path->dentry->d_lock);
@@ -2267,6 +2318,9 @@ char *d_path(const struct path *path, ch
spin_unlock(&path->dentry->d_lock);
vfsmount_read_unlock();
spin_unlock(&dcache_lock);
+ if (read_seqretry(&rename_lock, seq))
+ goto rename_retry;
+
path_put(&root);
return res;
}
@@ -2297,9 +2351,14 @@ char *dynamic_dname(struct dentry *dentr
*/
char *dentry_path(struct dentry *dentry, char *buf, int buflen)
{
- char *end = buf + buflen;
+ char *end;
char *retval;
+ unsigned seq;
+rename_retry:
+ end = buf + buflen;
+ seq = read_seqbegin(&rename_lock);
+ rcu_read_lock(); /* protect parent */
spin_lock(&dcache_lock);
spin_lock(&dentry->d_lock);
prepend(&end, &buflen, "\0", 1);
@@ -2323,13 +2382,16 @@ char *dentry_path(struct dentry *dentry,
retval = end;
dentry = parent;
}
+out:
spin_unlock(&dentry->d_lock);
spin_unlock(&dcache_lock);
+ rcu_read_unlock();
+ if (read_seqretry(&rename_lock, seq))
+ goto rename_retry;
return retval;
Elong:
- spin_unlock(&dentry->d_lock);
- spin_unlock(&dcache_lock);
- return ERR_PTR(-ENAMETOOLONG);
+ retval = ERR_PTR(-ENAMETOOLONG);
+ goto out;
}
/*
@@ -2448,9 +2510,13 @@ int is_subdir(struct dentry *new_dentry,
void d_genocide(struct dentry *root)
{
- struct dentry *this_parent = root;
+ struct dentry *this_parent;
struct list_head *next;
+ unsigned seq;
+rename_retry:
+ this_parent = root;
+ seq = read_seqbegin(&rename_lock);
spin_lock(&dcache_lock);
spin_lock(&this_parent->d_lock);
repeat:
@@ -2460,6 +2526,7 @@ resume:
struct list_head *tmp = next;
struct dentry *dentry = list_entry(tmp, struct dentry, d_u.d_child);
next = tmp->next;
+
spin_lock_nested(&dentry->d_lock, DENTRY_D_LOCK_NESTED);
if (d_unhashed(dentry) || !dentry->d_inode) {
spin_unlock(&dentry->d_lock);
@@ -2476,15 +2543,34 @@ resume:
spin_unlock(&dentry->d_lock);
}
if (this_parent != root) {
- next = this_parent->d_u.d_child.next;
+ struct dentry *tmp;
+ struct dentry *child;
+
+ tmp = this_parent->d_parent;
this_parent->d_count--;
+ rcu_read_lock();
spin_unlock(&this_parent->d_lock);
- this_parent = this_parent->d_parent;
+ child = this_parent;
+ this_parent = tmp;
spin_lock(&this_parent->d_lock);
+ /* might go back up the wrong parent if we have had a rename
+ * or deletion */
+ if (this_parent != child->d_parent ||
+ // d_unlinked(this_parent) || XXX
+ read_seqretry(&rename_lock, seq)) {
+ spin_unlock(&this_parent->d_lock);
+ spin_unlock(&dcache_lock);
+ rcu_read_unlock();
+ goto rename_retry;
+ }
+ rcu_read_unlock();
+ next = child->d_u.d_child.next;
goto resume;
}
spin_unlock(&this_parent->d_lock);
spin_unlock(&dcache_lock);
+ if (read_seqretry(&rename_lock, seq))
+ goto rename_retry;
}
/**
Index: linux-2.6/fs/seq_file.c
===================================================================
--- linux-2.6.orig/fs/seq_file.c
+++ linux-2.6/fs/seq_file.c
@@ -459,12 +459,18 @@ int seq_path_root(struct seq_file *m, st
if (m->count < m->size) {
char *s = m->buf + m->count;
char *p;
+ unsigned seq;
+rename_retry:
+ seq = read_seqbegin(&rename_lock);
spin_lock(&dcache_lock);
vfsmount_read_lock();
p = __d_path(path, root, s, m->size - m->count);
vfsmount_read_unlock();
spin_unlock(&dcache_lock);
+ if (read_seqretry(&rename_lock, seq))
+ goto rename_retry;
+
err = PTR_ERR(p);
if (!IS_ERR(p)) {
s = mangle_path(s, p, esc);
Index: linux-2.6/drivers/staging/pohmelfs/path_entry.c
===================================================================
--- linux-2.6.orig/drivers/staging/pohmelfs/path_entry.c
+++ linux-2.6/drivers/staging/pohmelfs/path_entry.c
@@ -85,6 +85,7 @@ int pohmelfs_path_length(struct pohmelfs
{
struct dentry *d, *root, *first;
int len = 1; /* Root slash */
+ unsigned seq;
first = d = d_find_alias(&pi->vfs_inode);
if (!d) {
@@ -96,6 +97,9 @@ int pohmelfs_path_length(struct pohmelfs
root = dget(current->fs->root.dentry);
read_unlock(¤t->fs->lock);
+ rcu_read_lock();
+rename_retry:
+ seq = read_seqbegin(&rename_lock);
spin_lock(&dcache_lock);
if (!IS_ROOT(d) && d_unhashed(d))
@@ -106,6 +110,9 @@ int pohmelfs_path_length(struct pohmelfs
d = d->d_parent;
}
spin_unlock(&dcache_lock);
+ if (read_seqretry(&rename_lock, seq))
+ goto rename_retry;
+ rcu_read_unlock();
dput(root);
dput(first);
Index: linux-2.6/fs/autofs4/waitq.c
===================================================================
--- linux-2.6.orig/fs/autofs4/waitq.c
+++ linux-2.6/fs/autofs4/waitq.c
@@ -189,13 +189,20 @@ static int autofs4_getpath(struct autofs
char *buf = *name;
char *p;
int len = 0;
+ unsigned seq;
+ rcu_read_lock();
+rename_retry:
+ seq = read_seqbegin(&rename_lock);
spin_lock(&dcache_lock);
for (tmp = dentry ; tmp != root ; tmp = tmp->d_parent)
len += tmp->d_name.len + 1;
if (!len || --len > NAME_MAX) {
spin_unlock(&dcache_lock);
+ if (read_seqretry(&rename_lock, seq))
+ goto rename_retry;
+ rcu_read_unlock();
return 0;
}
@@ -209,6 +216,9 @@ static int autofs4_getpath(struct autofs
strncpy(p, tmp->d_name.name, tmp->d_name.len);
}
spin_unlock(&dcache_lock);
+ if (read_seqretry(&rename_lock, seq))
+ goto rename_retry;
+ rcu_read_unlock();
return len;
}
Index: linux-2.6/fs/nfs/namespace.c
===================================================================
--- linux-2.6.orig/fs/nfs/namespace.c
+++ linux-2.6/fs/nfs/namespace.c
@@ -50,9 +50,13 @@ char *nfs_path(const char *base,
{
char *end = buffer+buflen;
int namelen;
+ unsigned seq;
*--end = '\0';
buflen--;
+ rcu_read_lock();
+rename_retry:
+ seq = read_seqbegin(&rename_lock);
spin_lock(&dcache_lock);
while (!IS_ROOT(dentry) && dentry != droot) {
namelen = dentry->d_name.len;
@@ -65,6 +69,9 @@ char *nfs_path(const char *base,
dentry = dentry->d_parent;
}
spin_unlock(&dcache_lock);
+ if (read_seqretry(&rename_lock, seq))
+ goto rename_retry;
+ rcu_read_unlock();
if (*end != '/') {
if (--buflen < 0)
goto Elong;
@@ -82,6 +89,9 @@ char *nfs_path(const char *base,
return end;
Elong_unlock:
spin_unlock(&dcache_lock);
+ if (read_seqretry(&rename_lock, seq))
+ goto rename_retry;
+ rcu_read_unlock();
Elong:
return ERR_PTR(-ENAMETOOLONG);
}
^ permalink raw reply [flat|nested] 64+ messages in thread
* [patch 14/33] fs: dcache remove dcache_lock
2009-09-04 6:51 [patch 00/33] my current vfs scalability patch queue npiggin
` (12 preceding siblings ...)
2009-09-04 6:51 ` [patch 13/33] fs: use RCU / seqlock logic for reverse and multi-step operaitons npiggin
@ 2009-09-04 6:51 ` npiggin
2009-09-04 6:51 ` [patch 15/33] fs: dcache reduce dput locking npiggin
` (19 subsequent siblings)
33 siblings, 0 replies; 64+ messages in thread
From: npiggin @ 2009-09-04 6:51 UTC (permalink / raw)
To: linux-fsdevel; +Cc: linux-kernel
[-- Attachment #1: fs-dcache_lock-remove.patch --]
[-- Type: text/plain, Size: 52694 bytes --]
dcache_lock no longer protects anything (I hope). remove it.
This breaks a lot of the tree where I haven't thought about the problem,
but it simplifies the dcache.c code quite a bit (and it's also probably
a good thing to break unconverted code). So I include this here before
making further changes to the locking.
---
Documentation/filesystems/Locking | 2
arch/powerpc/platforms/cell/spufs/inode.c | 5 -
drivers/infiniband/hw/ipath/ipath_fs.c | 6 -
drivers/staging/pohmelfs/path_entry.c | 2
drivers/usb/core/inode.c | 3
fs/affs/amigaffs.c | 2
fs/autofs4/expire.c | 11 --
fs/autofs4/inode.c | 6 -
fs/autofs4/root.c | 20 ----
fs/autofs4/waitq.c | 3
fs/coda/cache.c | 2
fs/configfs/configfs_internal.h | 2
fs/configfs/inode.c | 6 -
fs/dcache.c | 131 ++++--------------------------
fs/exportfs/expfs.c | 4
fs/namei.c | 5 -
fs/ncpfs/dir.c | 3
fs/ncpfs/ncplib_kernel.h | 4
fs/nfs/dir.c | 3
fs/nfs/getroot.c | 2
fs/nfs/namespace.c | 3
fs/notify/fsnotify.c | 2
fs/notify/inotify/inotify.c | 4
fs/ocfs2/dcache.c | 2
fs/seq_file.c | 2
fs/smbfs/cache.c | 4
fs/sysfs/dir.c | 3
include/linux/dcache.h | 17 +--
include/linux/fsnotify_backend.h | 5 -
kernel/cgroup.c | 6 -
net/sunrpc/rpc_pipe.c | 11 +-
security/selinux/selinuxfs.c | 4
security/tomoyo/realpath.c | 2
33 files changed, 37 insertions(+), 250 deletions(-)
Index: linux-2.6/fs/dcache.c
===================================================================
--- linux-2.6.orig/fs/dcache.c
+++ linux-2.6/fs/dcache.c
@@ -50,11 +50,10 @@
* - d_subdirs and children's d_child
*
* Ordering:
- * dcache_lock
- * dcache_inode_lock
- * dentry->d_lock
- * dcache_lru_lock
- * dcache_hash_lock
+ * dcache_inode_lock
+ * dentry->d_lock
+ * dcache_lru_lock
+ * dcache_hash_lock
*/
int sysctl_vfs_cache_pressure __read_mostly = 100;
EXPORT_SYMBOL_GPL(sysctl_vfs_cache_pressure);
@@ -62,12 +61,10 @@ EXPORT_SYMBOL_GPL(sysctl_vfs_cache_press
__cacheline_aligned_in_smp DEFINE_SPINLOCK(dcache_inode_lock);
__cacheline_aligned_in_smp DEFINE_SPINLOCK(dcache_hash_lock);
static __cacheline_aligned_in_smp DEFINE_SPINLOCK(dcache_lru_lock);
-__cacheline_aligned_in_smp DEFINE_SPINLOCK(dcache_lock);
__cacheline_aligned_in_smp DEFINE_SEQLOCK(rename_lock);
EXPORT_SYMBOL(dcache_inode_lock);
EXPORT_SYMBOL(dcache_hash_lock);
-EXPORT_SYMBOL(dcache_lock);
static struct kmem_cache *dentry_cache __read_mostly;
@@ -109,7 +106,7 @@ static void d_callback(struct rcu_head *
}
/*
- * no dcache_lock, please.
+ * no locks, please.
*/
static void d_free(struct dentry *dentry)
{
@@ -131,7 +128,6 @@ static void d_free(struct dentry *dentry
static void dentry_iput(struct dentry * dentry)
__releases(dentry->d_lock)
__releases(dcache_inode_lock)
- __releases(dcache_lock)
{
struct inode *inode = dentry->d_inode;
if (inode) {
@@ -139,7 +135,6 @@ static void dentry_iput(struct dentry *
list_del_init(&dentry->d_alias);
spin_unlock(&dentry->d_lock);
spin_unlock(&dcache_inode_lock);
- spin_unlock(&dcache_lock);
if (!inode->i_nlink)
fsnotify_inoderemove(inode);
if (dentry->d_op && dentry->d_op->d_iput)
@@ -149,7 +144,6 @@ static void dentry_iput(struct dentry *
} else {
spin_unlock(&dentry->d_lock);
spin_unlock(&dcache_inode_lock);
- spin_unlock(&dcache_lock);
}
}
@@ -215,13 +209,12 @@ static void dentry_lru_del_init(struct d
*
* If this is the root of the dentry tree, return NULL.
*
- * dcache_lock and d_lock and d_parent->d_lock must be held by caller, and
+ * d_lock and d_parent->d_lock must be held by caller, and
* are dropped by d_kill.
*/
static struct dentry *d_kill(struct dentry *dentry)
__releases(dentry->d_lock)
__releases(dcache_inode_lock)
- __releases(dcache_lock)
{
struct dentry *parent;
@@ -278,21 +271,10 @@ repeat:
might_sleep();
spin_lock(&dentry->d_lock);
if (dentry->d_count == 1) {
- if (!spin_trylock(&dcache_lock)) {
- /*
- * Something of a livelock possibility we could avoid
- * by taking dcache_lock and trying again, but we
- * want to reduce dcache_lock anyway so this will
- * get improved.
- */
-drop1:
- spin_unlock(&dentry->d_lock);
- goto repeat;
- }
if (!spin_trylock(&dcache_inode_lock)) {
drop2:
- spin_unlock(&dcache_lock);
- goto drop1;
+ spin_unlock(&dentry->d_lock);
+ goto repeat;
}
parent = dentry->d_parent;
if (parent) {
@@ -306,7 +288,6 @@ drop2:
dentry->d_count--;
if (dentry->d_count) {
spin_unlock(&dentry->d_lock);
- spin_unlock(&dcache_lock);
return;
}
@@ -328,7 +309,6 @@ drop2:
if (parent)
spin_unlock(&parent->d_lock);
spin_unlock(&dcache_inode_lock);
- spin_unlock(&dcache_lock);
return;
unhash_it:
@@ -358,11 +338,9 @@ int d_invalidate(struct dentry * dentry)
/*
* If it's already been dropped, return OK.
*/
- spin_lock(&dcache_lock);
spin_lock(&dentry->d_lock);
if (d_unhashed(dentry)) {
spin_unlock(&dentry->d_lock);
- spin_unlock(&dcache_lock);
return 0;
}
/*
@@ -371,9 +349,7 @@ int d_invalidate(struct dentry * dentry)
*/
if (!list_empty(&dentry->d_subdirs)) {
spin_unlock(&dentry->d_lock);
- spin_unlock(&dcache_lock);
shrink_dcache_parent(dentry);
- spin_lock(&dcache_lock);
}
/*
@@ -390,14 +366,12 @@ int d_invalidate(struct dentry * dentry)
if (dentry->d_count > 1) {
if (dentry->d_inode && S_ISDIR(dentry->d_inode->i_mode)) {
spin_unlock(&dentry->d_lock);
- spin_unlock(&dcache_lock);
return -EBUSY;
}
}
__d_drop(dentry);
spin_unlock(&dentry->d_lock);
- spin_unlock(&dcache_lock);
return 0;
}
@@ -501,11 +475,9 @@ struct dentry * d_find_alias(struct inod
struct dentry *de = NULL;
if (!list_empty(&inode->i_dentry)) {
- spin_lock(&dcache_lock);
spin_lock(&dcache_inode_lock);
de = __d_find_alias(inode, 0);
spin_unlock(&dcache_inode_lock);
- spin_unlock(&dcache_lock);
}
return de;
}
@@ -518,7 +490,6 @@ void d_prune_aliases(struct inode *inode
{
struct dentry *dentry;
restart:
- spin_lock(&dcache_lock);
spin_lock(&dcache_inode_lock);
list_for_each_entry(dentry, &inode->i_dentry, d_alias) {
spin_lock(&dentry->d_lock);
@@ -527,14 +498,12 @@ restart:
__d_drop(dentry);
spin_unlock(&dentry->d_lock);
spin_unlock(&dcache_inode_lock);
- spin_unlock(&dcache_lock);
dput(dentry);
goto restart;
}
spin_unlock(&dentry->d_lock);
}
spin_unlock(&dcache_inode_lock);
- spin_unlock(&dcache_lock);
}
/*
@@ -547,20 +516,16 @@ restart:
*/
static void prune_one_dentry(struct dentry * dentry)
__releases(dentry->d_lock)
- __releases(dcache_lock)
- __acquires(dcache_lock)
{
__d_drop(dentry);
dentry = d_kill(dentry);
/*
- * Prune ancestors. Locking is simpler than in dput(),
- * because dcache_lock needs to be taken anyway.
+ * Prune ancestors.
*/
while (dentry) {
struct dentry *parent = NULL;
- spin_lock(&dcache_lock);
spin_lock(&dcache_inode_lock);
again:
spin_lock(&dentry->d_lock);
@@ -577,7 +542,6 @@ again:
spin_unlock(&parent->d_lock);
spin_unlock(&dentry->d_lock);
spin_unlock(&dcache_inode_lock);
- spin_unlock(&dcache_lock);
return;
}
@@ -646,7 +610,6 @@ restart:
}
spin_unlock(&dcache_lru_lock);
- spin_lock(&dcache_lock);
spin_lock(&dcache_inode_lock);
again:
spin_lock(&dcache_lru_lock); /* lru_lock also protects tmp list */
@@ -677,14 +640,13 @@ again1:
}
__dentry_lru_del_init(dentry);
spin_unlock(&dcache_lru_lock);
+
prune_one_dentry(dentry);
- /* dcache_lock and dentry->d_lock dropped */
- spin_lock(&dcache_lock);
+ /* dentry->d_lock dropped */
spin_lock(&dcache_inode_lock);
spin_lock(&dcache_lru_lock);
}
spin_unlock(&dcache_inode_lock);
- spin_unlock(&dcache_lock);
if (count == NULL && !list_empty(&sb->s_dentry_lru))
goto restart;
@@ -714,7 +676,6 @@ static void prune_dcache(int count)
if (unused == 0 || count == 0)
return;
- spin_lock(&dcache_lock);
restart:
if (count >= unused)
prune_ratio = 1;
@@ -750,11 +711,9 @@ restart:
if (down_read_trylock(&sb->s_umount)) {
if ((sb->s_root != NULL) &&
(!list_empty(&sb->s_dentry_lru))) {
- spin_unlock(&dcache_lock);
__shrink_dcache_sb(sb, &w_count,
DCACHE_REFERENCED);
pruned -= w_count;
- spin_lock(&dcache_lock);
}
up_read(&sb->s_umount);
}
@@ -770,7 +729,6 @@ restart:
}
}
spin_unlock(&sb_lock);
- spin_unlock(&dcache_lock);
}
/**
@@ -799,12 +757,10 @@ static void shrink_dcache_for_umount_sub
BUG_ON(!IS_ROOT(dentry));
/* detach this root from the system */
- spin_lock(&dcache_lock);
spin_lock(&dentry->d_lock);
dentry_lru_del_init(dentry);
__d_drop(dentry);
spin_unlock(&dentry->d_lock);
- spin_unlock(&dcache_lock);
for (;;) {
/* descend to the first leaf in the current subtree */
@@ -813,7 +769,6 @@ static void shrink_dcache_for_umount_sub
/* this is a branch with children - detach all of them
* from the system in one go */
- spin_lock(&dcache_lock);
spin_lock(&dentry->d_lock);
list_for_each_entry(loop, &dentry->d_subdirs,
d_u.d_child) {
@@ -823,7 +778,6 @@ static void shrink_dcache_for_umount_sub
spin_unlock(&loop->d_lock);
}
spin_unlock(&dentry->d_lock);
- spin_unlock(&dcache_lock);
/* move to the first child */
dentry = list_entry(dentry->d_subdirs.next,
@@ -894,8 +848,7 @@ out:
/*
* destroy the dentries attached to a superblock on unmounting
- * - we don't need to use dentry->d_lock, and only need dcache_lock when
- * removing the dentry from the system lists and hashes because:
+ * - we don't need to use dentry->d_lock because:
* - the superblock is detached from all mountings and open files, so the
* dentry trees will not be rearranged by the VFS
* - s_umount is write-locked, so the memory pressure shrinker will ignore
@@ -946,7 +899,6 @@ rename_retry:
this_parent = parent;
seq = read_seqbegin(&rename_lock);
- spin_lock(&dcache_lock);
if (d_mountpoint(parent))
goto positive;
spin_lock(&this_parent->d_lock);
@@ -993,7 +945,6 @@ resume:
// d_unlinked(this_parent) || XXX
read_seqretry(&rename_lock, seq)) {
spin_unlock(&this_parent->d_lock);
- spin_unlock(&dcache_lock);
rcu_read_unlock();
goto rename_retry;
}
@@ -1002,12 +953,10 @@ resume:
goto resume;
}
spin_unlock(&this_parent->d_lock);
- spin_unlock(&dcache_lock);
if (read_seqretry(&rename_lock, seq))
goto rename_retry;
return 0; /* No mount points found in tree */
positive:
- spin_unlock(&dcache_lock);
if (read_seqretry(&rename_lock, seq))
goto rename_retry;
return 1;
@@ -1038,7 +987,6 @@ rename_retry:
this_parent = parent;
seq = read_seqbegin(&rename_lock);
- spin_lock(&dcache_lock);
spin_lock(&this_parent->d_lock);
repeat:
next = this_parent->d_subdirs.next;
@@ -1101,7 +1049,6 @@ resume:
// d_unlinked(this_parent) || XXX
read_seqretry(&rename_lock, seq)) {
spin_unlock(&this_parent->d_lock);
- spin_unlock(&dcache_lock);
rcu_read_unlock();
goto rename_retry;
}
@@ -1111,7 +1058,6 @@ resume:
}
out:
spin_unlock(&this_parent->d_lock);
- spin_unlock(&dcache_lock);
if (read_seqretry(&rename_lock, seq))
goto rename_retry;
return found;
@@ -1211,7 +1157,6 @@ struct dentry *d_alloc(struct dentry * p
INIT_LIST_HEAD(&dentry->d_u.d_child);
if (parent) {
- spin_lock(&dcache_lock);
spin_lock(&parent->d_lock);
spin_lock_nested(&dentry->d_lock, DENTRY_D_LOCK_NESTED);
dentry->d_parent = dget_dlock(parent);
@@ -1219,7 +1164,6 @@ struct dentry *d_alloc(struct dentry * p
list_add(&dentry->d_u.d_child, &parent->d_subdirs);
spin_unlock(&dentry->d_lock);
spin_unlock(&parent->d_lock);
- spin_unlock(&dcache_lock);
}
atomic_inc(&dentry_stat.nr_dentry);
@@ -1237,7 +1181,6 @@ struct dentry *d_alloc_name(struct dentr
return d_alloc(parent, &q);
}
-/* the caller must hold dcache_lock */
static void __d_instantiate(struct dentry *dentry, struct inode *inode)
{
if (inode)
@@ -1264,11 +1207,9 @@ static void __d_instantiate(struct dentr
void d_instantiate(struct dentry *entry, struct inode * inode)
{
BUG_ON(!list_empty(&entry->d_alias));
- spin_lock(&dcache_lock);
spin_lock(&dcache_inode_lock);
__d_instantiate(entry, inode);
spin_unlock(&dcache_inode_lock);
- spin_unlock(&dcache_lock);
security_d_instantiate(entry, inode);
}
@@ -1326,11 +1267,9 @@ struct dentry *d_instantiate_unique(stru
BUG_ON(!list_empty(&entry->d_alias));
- spin_lock(&dcache_lock);
spin_lock(&dcache_inode_lock);
result = __d_instantiate_unique(entry, inode);
spin_unlock(&dcache_inode_lock);
- spin_unlock(&dcache_lock);
if (!result) {
security_d_instantiate(entry, inode);
@@ -1418,12 +1357,10 @@ struct dentry *d_obtain_alias(struct ino
}
tmp->d_parent = tmp; /* make sure dput doesn't croak */
- spin_lock(&dcache_lock);
spin_lock(&dcache_inode_lock);
res = __d_find_alias(inode, 0);
if (res) {
spin_unlock(&dcache_inode_lock);
- spin_unlock(&dcache_lock);
dput(tmp);
goto out_iput;
}
@@ -1439,7 +1376,6 @@ struct dentry *d_obtain_alias(struct ino
spin_unlock(&tmp->d_lock);
spin_unlock(&dcache_inode_lock);
- spin_unlock(&dcache_lock);
return tmp;
out_iput:
@@ -1469,22 +1405,19 @@ struct dentry *d_splice_alias(struct ino
struct dentry *new = NULL;
if (inode && S_ISDIR(inode->i_mode)) {
- spin_lock(&dcache_lock);
spin_lock(&dcache_inode_lock);
new = __d_find_alias(inode, 1);
if (new) {
BUG_ON(!(new->d_flags & DCACHE_DISCONNECTED));
spin_unlock(&dcache_inode_lock);
- spin_unlock(&dcache_lock);
security_d_instantiate(new, inode);
d_rehash(dentry);
d_move(new, dentry);
iput(inode);
} else {
- /* already taking dcache_lock, so d_add() by hand */
+ /* already taken dcache_inode_lock, d_add() by hand */
__d_instantiate(dentry, inode);
spin_unlock(&dcache_inode_lock);
- spin_unlock(&dcache_lock);
security_d_instantiate(dentry, inode);
d_rehash(dentry);
}
@@ -1556,12 +1489,10 @@ struct dentry *d_add_ci(struct dentry *d
* Negative dentry: instantiate it unless the inode is a directory and
* already has a dentry.
*/
- spin_lock(&dcache_lock);
spin_lock(&dcache_inode_lock);
if (!S_ISDIR(inode->i_mode) || list_empty(&inode->i_dentry)) {
__d_instantiate(found, inode);
spin_unlock(&dcache_inode_lock);
- spin_unlock(&dcache_lock);
security_d_instantiate(found, inode);
return found;
}
@@ -1573,7 +1504,6 @@ struct dentry *d_add_ci(struct dentry *d
new = list_entry(inode->i_dentry.next, struct dentry, d_alias);
dget_locked(new);
spin_unlock(&dcache_inode_lock);
- spin_unlock(&dcache_lock);
security_d_instantiate(found, inode);
d_move(new, found);
iput(inode);
@@ -1595,7 +1525,7 @@ err_out:
* is returned. The caller must use dput to free the entry when it has
* finished using it. %NULL is returned on failure.
*
- * __d_lookup is dcache_lock free. The hash list is protected using RCU.
+ * __d_lookup is global lock free. The hash list is protected using RCU.
* Memory barriers are used while updating and doing lockless traversal.
* To avoid races with d_move while rename is happening, d_lock is used.
*
@@ -1607,7 +1537,7 @@ err_out:
*
* The dentry unused LRU is not updated even if lookup finds the required dentry
* in there. It is updated in places such as prune_dcache, shrink_dcache_sb,
- * select_parent and __dget_locked. This laziness saves lookup from dcache_lock
+ * select_parent and __dget_locked. This laziness saves lookup from LRU lock
* acquisition.
*
* d_lookup() is protected against the concurrent renames in some unrelated
@@ -1737,25 +1667,22 @@ int d_validate(struct dentry *dentry, st
if (dentry->d_parent != dparent)
goto out;
- spin_lock(&dcache_lock);
spin_lock(&dentry->d_lock);
spin_lock(&dcache_hash_lock);
base = d_hash(dparent, dentry->d_name.hash);
hlist_for_each(lhp,base) {
/* hlist_for_each_entry_rcu() not required for d_hash list
- * as it is parsed under dcache_lock
+ * as it is parsed under dcache_hash_lock
*/
if (dentry == hlist_entry(lhp, struct dentry, d_hash)) {
spin_unlock(&dcache_hash_lock);
__dget_locked_dlock(dentry);
spin_unlock(&dentry->d_lock);
- spin_unlock(&dcache_lock);
return 1;
}
}
spin_unlock(&dcache_hash_lock);
spin_unlock(&dentry->d_lock);
- spin_unlock(&dcache_lock);
out:
return 0;
}
@@ -1787,7 +1714,6 @@ void d_delete(struct dentry * dentry)
/*
* Are we the only user?
*/
- spin_lock(&dcache_lock);
spin_lock(&dcache_inode_lock);
spin_lock(&dentry->d_lock);
isdir = S_ISDIR(dentry->d_inode->i_mode);
@@ -1802,7 +1728,6 @@ void d_delete(struct dentry * dentry)
spin_unlock(&dentry->d_lock);
spin_unlock(&dcache_inode_lock);
- spin_unlock(&dcache_lock);
fsnotify_nameremove(dentry, isdir);
}
@@ -1828,13 +1753,11 @@ static void _d_rehash(struct dentry * en
void d_rehash(struct dentry * entry)
{
- spin_lock(&dcache_lock);
spin_lock(&entry->d_lock);
spin_lock(&dcache_hash_lock);
_d_rehash(entry);
spin_unlock(&dcache_hash_lock);
spin_unlock(&entry->d_lock);
- spin_unlock(&dcache_lock);
}
/*
@@ -1988,9 +1911,7 @@ static void d_move_locked(struct dentry
void d_move(struct dentry * dentry, struct dentry * target)
{
- spin_lock(&dcache_lock);
d_move_locked(dentry, target);
- spin_unlock(&dcache_lock);
}
/**
@@ -2016,13 +1937,12 @@ struct dentry *d_ancestor(struct dentry
* This helper attempts to cope with remotely renamed directories
*
* It assumes that the caller is already holding
- * dentry->d_parent->d_inode->i_mutex and the dcache_lock
+ * dentry->d_parent->d_inode->i_mutex
*
* Note: If ever the locking in lock_rename() changes, then please
* remember to update this too...
*/
static struct dentry *__d_unalias(struct dentry *dentry, struct dentry *alias)
- __releases(dcache_lock)
{
struct mutex *m1 = NULL, *m2 = NULL;
struct dentry *ret;
@@ -2049,7 +1969,6 @@ out_unalias:
ret = alias;
out_err:
spin_unlock(&dcache_inode_lock);
- spin_unlock(&dcache_lock);
if (m2)
mutex_unlock(m2);
if (m1)
@@ -2113,7 +2032,6 @@ struct dentry *d_materialise_unique(stru
BUG_ON(!d_unhashed(dentry));
- spin_lock(&dcache_lock);
spin_lock(&dcache_inode_lock);
if (!inode) {
@@ -2160,7 +2078,6 @@ found:
spin_unlock(&dcache_hash_lock);
spin_unlock(&actual->d_lock);
spin_unlock(&dcache_inode_lock);
- spin_unlock(&dcache_lock);
out_nolock:
if (actual == dentry) {
security_d_instantiate(dentry, inode);
@@ -2172,7 +2089,6 @@ out_nolock:
shouldnt_be_hashed:
spin_unlock(&dcache_inode_lock);
- spin_unlock(&dcache_lock);
BUG();
}
@@ -2204,8 +2120,7 @@ static int prepend_name(char **buffer, i
* Returns a pointer into the buffer or an error code if the
* path was too long.
*
- * "buflen" should be positive. Caller holds the dcache_lock and
- * path->dentry->d_lock.
+ * "buflen" should be positive. Caller holds the path->dentry->d_lock.
*
* If path is not reachable from the supplied root, then the value of
* root is changed (without modifying refcounts).
@@ -2310,14 +2225,12 @@ char *d_path(const struct path *path, ch
rename_retry:
seq = read_seqbegin(&rename_lock);
- spin_lock(&dcache_lock);
vfsmount_read_lock();
spin_lock(&path->dentry->d_lock);
tmp = root;
res = __d_path(path, &tmp, buf, buflen);
spin_unlock(&path->dentry->d_lock);
vfsmount_read_unlock();
- spin_unlock(&dcache_lock);
if (read_seqretry(&rename_lock, seq))
goto rename_retry;
@@ -2359,7 +2272,6 @@ rename_retry:
end = buf + buflen;
seq = read_seqbegin(&rename_lock);
rcu_read_lock(); /* protect parent */
- spin_lock(&dcache_lock);
spin_lock(&dentry->d_lock);
prepend(&end, &buflen, "\0", 1);
if (d_unlinked(dentry) &&
@@ -2384,7 +2296,6 @@ rename_retry:
}
out:
spin_unlock(&dentry->d_lock);
- spin_unlock(&dcache_lock);
rcu_read_unlock();
if (read_seqretry(&rename_lock, seq))
goto rename_retry;
@@ -2429,7 +2340,6 @@ SYSCALL_DEFINE2(getcwd, char __user *, b
read_unlock(¤t->fs->lock);
error = -ENOENT;
- spin_lock(&dcache_lock);
vfsmount_read_lock();
spin_lock(&pwd.dentry->d_lock);
if (!d_unlinked(pwd.dentry)) {
@@ -2440,7 +2350,6 @@ SYSCALL_DEFINE2(getcwd, char __user *, b
cwd = __d_path(&pwd, &tmp, page, PAGE_SIZE);
spin_unlock(&pwd.dentry->d_lock);
vfsmount_read_unlock();
- spin_unlock(&dcache_lock);
error = PTR_ERR(cwd);
if (IS_ERR(cwd))
@@ -2456,7 +2365,6 @@ SYSCALL_DEFINE2(getcwd, char __user *, b
} else {
spin_unlock(&pwd.dentry->d_lock);
vfsmount_read_unlock();
- spin_unlock(&dcache_lock);
}
out:
@@ -2517,7 +2425,6 @@ void d_genocide(struct dentry *root)
rename_retry:
this_parent = root;
seq = read_seqbegin(&rename_lock);
- spin_lock(&dcache_lock);
spin_lock(&this_parent->d_lock);
repeat:
next = this_parent->d_subdirs.next;
@@ -2559,7 +2466,6 @@ resume:
// d_unlinked(this_parent) || XXX
read_seqretry(&rename_lock, seq)) {
spin_unlock(&this_parent->d_lock);
- spin_unlock(&dcache_lock);
rcu_read_unlock();
goto rename_retry;
}
@@ -2568,7 +2474,6 @@ resume:
goto resume;
}
spin_unlock(&this_parent->d_lock);
- spin_unlock(&dcache_lock);
if (read_seqretry(&rename_lock, seq))
goto rename_retry;
}
Index: linux-2.6/fs/namei.c
===================================================================
--- linux-2.6.orig/fs/namei.c
+++ linux-2.6/fs/namei.c
@@ -759,14 +759,11 @@ static __always_inline void follow_dotdo
nd->path.mnt == nd->root.mnt) {
break;
}
- spin_lock(&dcache_lock);
if (nd->path.dentry != nd->path.mnt->mnt_root) {
nd->path.dentry = dget(nd->path.dentry->d_parent);
- spin_unlock(&dcache_lock);
dput(old);
break;
}
- spin_unlock(&dcache_lock);
vfsmount_read_lock();
parent = nd->path.mnt->mnt_parent;
if (parent == nd->path.mnt) {
@@ -2159,12 +2156,10 @@ void dentry_unhash(struct dentry *dentry
{
dget(dentry);
shrink_dcache_parent(dentry);
- spin_lock(&dcache_lock);
spin_lock(&dentry->d_lock);
if (dentry->d_count == 2)
__d_drop(dentry);
spin_unlock(&dentry->d_lock);
- spin_unlock(&dcache_lock);
}
int vfs_rmdir(struct inode *dir, struct dentry *dentry)
Index: linux-2.6/fs/seq_file.c
===================================================================
--- linux-2.6.orig/fs/seq_file.c
+++ linux-2.6/fs/seq_file.c
@@ -463,11 +463,9 @@ int seq_path_root(struct seq_file *m, st
rename_retry:
seq = read_seqbegin(&rename_lock);
- spin_lock(&dcache_lock);
vfsmount_read_lock();
p = __d_path(path, root, s, m->size - m->count);
vfsmount_read_unlock();
- spin_unlock(&dcache_lock);
if (read_seqretry(&rename_lock, seq))
goto rename_retry;
Index: linux-2.6/fs/sysfs/dir.c
===================================================================
--- linux-2.6.orig/fs/sysfs/dir.c
+++ linux-2.6/fs/sysfs/dir.c
@@ -547,7 +547,6 @@ static void sysfs_drop_dentry(struct sys
* dput to immediately free the dentry if it is not in use.
*/
repeat:
- spin_lock(&dcache_lock);
spin_lock(&dcache_inode_lock);
list_for_each_entry(dentry, &inode->i_dentry, d_alias) {
spin_lock(&dentry->d_lock);
@@ -559,12 +558,10 @@ repeat:
__d_drop(dentry);
spin_unlock(&dentry->d_lock);
spin_unlock(&dcache_inode_lock);
- spin_unlock(&dcache_lock);
dput(dentry);
goto repeat;
}
spin_unlock(&dcache_inode_lock);
- spin_unlock(&dcache_lock);
/* adjust nlink and update timestamp */
mutex_lock(&inode->i_mutex);
Index: linux-2.6/include/linux/dcache.h
===================================================================
--- linux-2.6.orig/include/linux/dcache.h
+++ linux-2.6/include/linux/dcache.h
@@ -150,13 +150,13 @@ struct dentry_operations {
/*
locking rules:
- big lock dcache_lock d_lock may block
-d_revalidate: no no no yes
-d_hash no no no yes
-d_compare: no yes yes no
-d_delete: no yes no no
-d_release: no no no yes
-d_iput: no no no yes
+ big lock d_lock may block
+d_revalidate: no no yes
+d_hash no no yes
+d_compare: no yes no
+d_delete: no no no
+d_release: no no yes
+d_iput: no no yes
*/
/* d_flags entries */
@@ -188,7 +188,6 @@ d_iput: no no no yes
extern spinlock_t dcache_inode_lock;
extern spinlock_t dcache_hash_lock;
-extern spinlock_t dcache_lock;
extern seqlock_t rename_lock;
/**
@@ -219,11 +218,9 @@ static inline void __d_drop(struct dentr
static inline void d_drop(struct dentry *dentry)
{
- spin_lock(&dcache_lock);
spin_lock(&dentry->d_lock);
__d_drop(dentry);
spin_unlock(&dentry->d_lock);
- spin_unlock(&dcache_lock);
}
static inline int dname_external(struct dentry *dentry)
Index: linux-2.6/fs/notify/inotify/inotify.c
===================================================================
--- linux-2.6.orig/fs/notify/inotify/inotify.c
+++ linux-2.6/fs/notify/inotify/inotify.c
@@ -185,7 +185,6 @@ static void set_dentry_child_flags(struc
{
struct dentry *alias;
- spin_lock(&dcache_lock);
spin_lock(&dcache_inode_lock);
list_for_each_entry(alias, &inode->i_dentry, d_alias) {
struct dentry *child;
@@ -205,7 +204,6 @@ static void set_dentry_child_flags(struc
spin_unlock(&alias->d_lock);
}
spin_unlock(&dcache_inode_lock);
- spin_unlock(&dcache_lock);
}
/*
@@ -273,6 +271,7 @@ void inotify_d_instantiate(struct dentry
if (!inode)
return;
+ /* XXX: need parent lock in place of dcache_lock? */
spin_lock(&entry->d_lock);
parent = entry->d_parent;
if (parent->d_inode && inotify_inode_watched(parent->d_inode))
@@ -287,6 +286,7 @@ void inotify_d_move(struct dentry *entry
{
struct dentry *parent;
+ /* XXX: need parent lock in place of dcache_lock? */
parent = entry->d_parent;
if (inotify_inode_watched(parent->d_inode))
entry->d_flags |= DCACHE_INOTIFY_PARENT_WATCHED;
Index: linux-2.6/fs/exportfs/expfs.c
===================================================================
--- linux-2.6.orig/fs/exportfs/expfs.c
+++ linux-2.6/fs/exportfs/expfs.c
@@ -47,24 +47,20 @@ find_acceptable_alias(struct dentry *res
if (acceptable(context, result))
return result;
- spin_lock(&dcache_lock);
spin_lock(&dcache_inode_lock);
list_for_each_entry(dentry, &result->d_inode->i_dentry, d_alias) {
dget_locked(dentry);
spin_unlock(&dcache_inode_lock);
- spin_unlock(&dcache_lock);
if (toput)
dput(toput);
if (dentry != result && acceptable(context, dentry)) {
dput(result);
return dentry;
}
- spin_lock(&dcache_lock);
spin_lock(&dcache_inode_lock);
toput = dentry;
}
spin_unlock(&dcache_inode_lock);
- spin_unlock(&dcache_lock);
if (toput)
dput(toput);
Index: linux-2.6/Documentation/filesystems/Locking
===================================================================
--- linux-2.6.orig/Documentation/filesystems/Locking
+++ linux-2.6/Documentation/filesystems/Locking
@@ -17,7 +17,7 @@ prototypes:
void (*d_iput)(struct dentry *, struct inode *);
char *(*d_dname)((struct dentry *dentry, char *buffer, int buflen);
-locking rules:
+locking rules: XXX: update these!!
none have BKL
dcache_lock rename_lock ->d_lock may block
d_revalidate: no no no yes
Index: linux-2.6/arch/powerpc/platforms/cell/spufs/inode.c
===================================================================
--- linux-2.6.orig/arch/powerpc/platforms/cell/spufs/inode.c
+++ linux-2.6/arch/powerpc/platforms/cell/spufs/inode.c
@@ -158,21 +158,18 @@ static void spufs_prune_dir(struct dentr
mutex_lock(&dir->d_inode->i_mutex);
list_for_each_entry_safe(dentry, tmp, &dir->d_subdirs, d_u.d_child) {
- spin_lock(&dcache_lock);
spin_lock(&dentry->d_lock);
if (!(d_unhashed(dentry)) && dentry->d_inode) {
dget_locked_dlock(dentry);
__d_drop(dentry);
spin_unlock(&dentry->d_lock);
simple_unlink(dir->d_inode, dentry);
- /* XXX: what is dcache_lock protecting here? Other
+ /* XXX: what was dcache_lock protecting here? Other
* filesystems (IB, configfs) release dcache_lock
* before unlink */
- spin_unlock(&dcache_lock);
dput(dentry);
} else {
spin_unlock(&dentry->d_lock);
- spin_unlock(&dcache_lock);
}
}
shrink_dcache_parent(dir);
Index: linux-2.6/drivers/infiniband/hw/ipath/ipath_fs.c
===================================================================
--- linux-2.6.orig/drivers/infiniband/hw/ipath/ipath_fs.c
+++ linux-2.6/drivers/infiniband/hw/ipath/ipath_fs.c
@@ -272,18 +272,14 @@ static int remove_file(struct dentry *pa
goto bail;
}
- spin_lock(&dcache_lock);
spin_lock(&tmp->d_lock);
if (!(d_unhashed(tmp) && tmp->d_inode)) {
dget_locked_dlock(tmp);
__d_drop(tmp);
spin_unlock(&tmp->d_lock);
- spin_unlock(&dcache_lock);
simple_unlink(parent->d_inode, tmp);
- } else {
+ } else
spin_unlock(&tmp->d_lock);
- spin_unlock(&dcache_lock);
- }
ret = 0;
bail:
Index: linux-2.6/drivers/usb/core/inode.c
===================================================================
--- linux-2.6.orig/drivers/usb/core/inode.c
+++ linux-2.6/drivers/usb/core/inode.c
@@ -348,18 +348,15 @@ static int usbfs_empty (struct dentry *d
{
struct list_head *list;
- spin_lock(&dcache_lock);
spin_lock(&dentry->d_lock);
list_for_each(list, &dentry->d_subdirs) {
struct dentry *de = list_entry(list, struct dentry, d_u.d_child);
if (usbfs_positive(de)) {
spin_unlock(&dentry->d_lock);
- spin_unlock(&dcache_lock);
return 0;
}
}
spin_unlock(&dentry->d_lock);
- spin_unlock(&dcache_lock);
return 1;
}
Index: linux-2.6/fs/affs/amigaffs.c
===================================================================
--- linux-2.6.orig/fs/affs/amigaffs.c
+++ linux-2.6/fs/affs/amigaffs.c
@@ -128,7 +128,6 @@ affs_fix_dcache(struct dentry *dentry, u
void *data = dentry->d_fsdata;
struct list_head *head, *next;
- spin_lock(&dcache_lock);
spin_lock(&dcache_inode_lock);
head = &inode->i_dentry;
next = head->next;
@@ -141,7 +140,6 @@ affs_fix_dcache(struct dentry *dentry, u
next = next->next;
}
spin_unlock(&dcache_inode_lock);
- spin_unlock(&dcache_lock);
}
Index: linux-2.6/fs/autofs4/expire.c
===================================================================
--- linux-2.6.orig/fs/autofs4/expire.c
+++ linux-2.6/fs/autofs4/expire.c
@@ -105,7 +105,6 @@ static struct dentry *get_next_positive_
struct list_head *next;
struct dentry *ret;
- spin_lock(&dcache_lock);
spin_lock(&p->d_lock);
again:
next = p->d_subdirs.next;
@@ -123,9 +122,7 @@ again:
dget_dlock(p);
spin_unlock(&p->d_lock);
parent = dget_parent(p);
- spin_unlock(&dcache_lock);
dput(p);
- spin_lock(&dcache_lock);
spin_lock(&parent->d_lock);
} else
spin_unlock(&p->d_lock);
@@ -147,8 +144,6 @@ again:
dget_dlock(ret);
spin_unlock(&ret->d_lock);
- spin_unlock(&dcache_lock);
-
return ret;
}
@@ -338,7 +333,6 @@ struct dentry *autofs4_expire_indirect(s
now = jiffies;
timeout = sbi->exp_timeout;
- spin_lock(&dcache_lock);
spin_lock(&root->d_lock);
next = root->d_subdirs.next;
@@ -355,7 +349,6 @@ struct dentry *autofs4_expire_indirect(s
dentry = dget(dentry);
spin_unlock(&root->d_lock);
- spin_unlock(&dcache_lock);
spin_lock(&sbi->fs_lock);
ino = autofs4_dentry_ino(dentry);
@@ -420,12 +413,10 @@ struct dentry *autofs4_expire_indirect(s
next:
spin_unlock(&sbi->fs_lock);
dput(dentry);
- spin_lock(&dcache_lock);
spin_lock(&root->d_lock);
next = next->next;
}
spin_unlock(&root->d_lock);
- spin_unlock(&dcache_lock);
return NULL;
found:
@@ -435,11 +426,9 @@ found:
ino->flags |= AUTOFS_INF_EXPIRING;
init_completion(&ino->expire_complete);
spin_unlock(&sbi->fs_lock);
- spin_lock(&dcache_lock);
spin_lock(&expired->d_parent->d_lock);
list_move(&expired->d_parent->d_subdirs, &expired->d_u.d_child);
spin_unlock(&expired->d_parent->d_lock);
- spin_unlock(&dcache_lock);
return expired;
}
Index: linux-2.6/fs/autofs4/inode.c
===================================================================
--- linux-2.6.orig/fs/autofs4/inode.c
+++ linux-2.6/fs/autofs4/inode.c
@@ -109,7 +109,6 @@ static void autofs4_force_release(struct
if (!sbi->sb->s_root)
return;
- spin_lock(&dcache_lock);
repeat:
spin_lock(&this_parent->d_lock);
next = this_parent->d_subdirs.next;
@@ -130,13 +129,11 @@ resume:
next = next->next;
spin_unlock(&this_parent->d_lock);
- spin_unlock(&dcache_lock);
DPRINTK("dentry %p %.*s",
dentry, (int)dentry->d_name.len, dentry->d_name.name);
dput(dentry);
- spin_lock(&dcache_lock);
}
if (this_parent != sbi->sb->s_root) {
@@ -145,15 +142,12 @@ resume:
next = this_parent->d_u.d_child.next;
spin_unlock(&this_parent->d_lock);
this_parent = this_parent->d_parent;
- spin_unlock(&dcache_lock);
DPRINTK("parent dentry %p %.*s",
dentry, (int)dentry->d_name.len, dentry->d_name.name);
dput(dentry);
- spin_lock(&dcache_lock);
spin_lock(&this_parent->d_lock);
goto resume;
}
- spin_unlock(&dcache_lock);
spin_unlock(&this_parent->d_lock);
}
Index: linux-2.6/fs/autofs4/root.c
===================================================================
--- linux-2.6.orig/fs/autofs4/root.c
+++ linux-2.6/fs/autofs4/root.c
@@ -92,15 +92,12 @@ static int autofs4_dir_open(struct inode
* autofs file system so just let the libfs routines handle
* it.
*/
- spin_lock(&dcache_lock);
spin_lock(&dentry->d_lock);
if (!d_mountpoint(dentry) && __simple_empty(dentry)) {
spin_unlock(&dentry->d_lock);
- spin_unlock(&dcache_lock);
return -ENOENT;
}
spin_unlock(&dentry->d_lock);
- spin_unlock(&dcache_lock);
out:
return dcache_dir_open(inode, file);
@@ -214,12 +211,10 @@ static void *autofs4_follow_link(struct
* multi-mount with no root mount offset. So don't try to
* mount it again.
*/
- spin_lock(&dcache_lock);
spin_lock(&dentry->d_lock);
if (dentry->d_flags & DCACHE_AUTOFS_PENDING ||
(!d_mountpoint(dentry) && __simple_empty(dentry))) {
spin_unlock(&dentry->d_lock);
- spin_unlock(&dcache_lock);
status = try_to_fill_dentry(dentry, 0);
if (status)
@@ -228,7 +223,6 @@ static void *autofs4_follow_link(struct
goto follow;
}
spin_unlock(&dentry->d_lock);
- spin_unlock(&dcache_lock);
follow:
/*
* If there is no root mount it must be an autofs
@@ -298,13 +292,11 @@ static int autofs4_revalidate(struct den
return 0;
/* Check for a non-mountpoint directory with no contents */
- spin_lock(&dcache_lock);
if (S_ISDIR(dentry->d_inode->i_mode) &&
!d_mountpoint(dentry) &&
__simple_empty(dentry)) {
DPRINTK("dentry=%p %.*s, emptydir",
dentry, dentry->d_name.len, dentry->d_name.name);
- spin_unlock(&dcache_lock);
/* The daemon never causes a mount to trigger */
if (oz_mode)
@@ -320,7 +312,6 @@ static int autofs4_revalidate(struct den
return status;
}
- spin_unlock(&dcache_lock);
return 1;
}
@@ -372,7 +363,6 @@ static struct dentry *autofs4_lookup_act
const unsigned char *str = name->name;
struct list_head *p, *head;
- spin_lock(&dcache_lock);
spin_lock(&sbi->lookup_lock);
head = &sbi->active_list;
list_for_each(p, head) {
@@ -405,14 +395,12 @@ static struct dentry *autofs4_lookup_act
dget_locked_dlock(dentry);
spin_unlock(&dentry->d_lock);
spin_unlock(&sbi->lookup_lock);
- spin_unlock(&dcache_lock);
return dentry;
}
next:
spin_unlock(&dentry->d_lock);
}
spin_unlock(&sbi->lookup_lock);
- spin_unlock(&dcache_lock);
return NULL;
}
@@ -424,7 +412,6 @@ static struct dentry *autofs4_lookup_exp
const unsigned char *str = name->name;
struct list_head *p, *head;
- spin_lock(&dcache_lock);
spin_lock(&sbi->lookup_lock);
head = &sbi->expiring_list;
list_for_each(p, head) {
@@ -457,14 +444,12 @@ static struct dentry *autofs4_lookup_exp
dget_locked_dlock(dentry);
spin_unlock(&dentry->d_lock);
spin_unlock(&sbi->lookup_lock);
- spin_unlock(&dcache_lock);
return dentry;
}
next:
spin_unlock(&dentry->d_lock);
}
spin_unlock(&sbi->lookup_lock);
- spin_unlock(&dcache_lock);
return NULL;
}
@@ -710,7 +695,6 @@ static int autofs4_dir_unlink(struct ino
dir->i_mtime = CURRENT_TIME;
- spin_lock(&dcache_lock);
spin_lock(&sbi->lookup_lock);
if (list_empty(&ino->expiring))
list_add(&ino->expiring, &sbi->expiring_list);
@@ -718,7 +702,6 @@ static int autofs4_dir_unlink(struct ino
spin_lock(&dentry->d_lock);
__d_drop(dentry);
spin_unlock(&dentry->d_lock);
- spin_unlock(&dcache_lock);
return 0;
}
@@ -735,11 +718,9 @@ static int autofs4_dir_rmdir(struct inod
if (!autofs4_oz_mode(sbi))
return -EACCES;
- spin_lock(&dcache_lock);
spin_lock(&dentry->d_lock);
if (!list_empty(&dentry->d_subdirs)) {
spin_unlock(&dentry->d_lock);
- spin_unlock(&dcache_lock);
return -ENOTEMPTY;
}
spin_lock(&sbi->lookup_lock);
@@ -748,7 +729,6 @@ static int autofs4_dir_rmdir(struct inod
spin_unlock(&sbi->lookup_lock);
__d_drop(dentry);
spin_unlock(&dentry->d_lock);
- spin_unlock(&dcache_lock);
if (atomic_dec_and_test(&ino->count)) {
p_ino = autofs4_dentry_ino(dentry->d_parent);
Index: linux-2.6/fs/coda/cache.c
===================================================================
--- linux-2.6.orig/fs/coda/cache.c
+++ linux-2.6/fs/coda/cache.c
@@ -86,7 +86,6 @@ static void coda_flag_children(struct de
struct list_head *child;
struct dentry *de;
- spin_lock(&dcache_lock);
spin_lock(&parent->d_lock);
list_for_each(child, &parent->d_subdirs)
{
@@ -97,7 +96,6 @@ static void coda_flag_children(struct de
coda_flag_inode(de->d_inode, flag);
}
spin_unlock(&parent->d_lock);
- spin_unlock(&dcache_lock);
return;
}
Index: linux-2.6/fs/configfs/configfs_internal.h
===================================================================
--- linux-2.6.orig/fs/configfs/configfs_internal.h
+++ linux-2.6/fs/configfs/configfs_internal.h
@@ -120,7 +120,6 @@ static inline struct config_item *config
{
struct config_item * item = NULL;
- spin_lock(&dcache_lock);
spin_lock(&dentry->d_lock);
if (!d_unhashed(dentry)) {
struct configfs_dirent * sd = dentry->d_fsdata;
@@ -131,7 +130,6 @@ static inline struct config_item *config
item = config_item_get(sd->s_element);
}
spin_unlock(&dentry->d_lock);
- spin_unlock(&dcache_lock);
return item;
}
Index: linux-2.6/fs/configfs/inode.c
===================================================================
--- linux-2.6.orig/fs/configfs/inode.c
+++ linux-2.6/fs/configfs/inode.c
@@ -253,18 +253,14 @@ void configfs_drop_dentry(struct configf
struct dentry * dentry = sd->s_dentry;
if (dentry) {
- spin_lock(&dcache_lock);
spin_lock(&dentry->d_lock);
if (!(d_unhashed(dentry) && dentry->d_inode)) {
dget_locked_dlock(dentry);
__d_drop(dentry);
spin_unlock(&dentry->d_lock);
- spin_unlock(&dcache_lock);
simple_unlink(parent->d_inode, dentry);
- } else {
+ } else
spin_unlock(&dentry->d_lock);
- spin_unlock(&dcache_lock);
- }
}
}
Index: linux-2.6/fs/ncpfs/dir.c
===================================================================
--- linux-2.6.orig/fs/ncpfs/dir.c
+++ linux-2.6/fs/ncpfs/dir.c
@@ -364,7 +364,6 @@ ncp_dget_fpos(struct dentry *dentry, str
}
/* If a pointer is invalid, we search the dentry. */
- spin_lock(&dcache_lock);
spin_lock(&parent->d_lock);
next = parent->d_subdirs.next;
while (next != &parent->d_subdirs) {
@@ -375,13 +374,11 @@ ncp_dget_fpos(struct dentry *dentry, str
else
dent = NULL;
spin_unlock(&parent->d_lock);
- spin_unlock(&dcache_lock);
goto out;
}
next = next->next;
}
spin_unlock(&parent->d_lock);
- spin_unlock(&dcache_lock);
return NULL;
out:
Index: linux-2.6/fs/ncpfs/ncplib_kernel.h
===================================================================
--- linux-2.6.orig/fs/ncpfs/ncplib_kernel.h
+++ linux-2.6/fs/ncpfs/ncplib_kernel.h
@@ -192,7 +192,6 @@ ncp_renew_dentries(struct dentry *parent
struct list_head *next;
struct dentry *dentry;
- spin_lock(&dcache_lock);
spin_lock(&parent->d_lock);
next = parent->d_subdirs.next;
while (next != &parent->d_subdirs) {
@@ -206,7 +205,6 @@ ncp_renew_dentries(struct dentry *parent
next = next->next;
}
spin_unlock(&parent->d_lock);
- spin_unlock(&dcache_lock);
}
static inline void
@@ -216,7 +214,6 @@ ncp_invalidate_dircache_entries(struct d
struct list_head *next;
struct dentry *dentry;
- spin_lock(&dcache_lock);
spin_lock(&parent->d_lock);
next = parent->d_subdirs.next;
while (next != &parent->d_subdirs) {
@@ -226,7 +223,6 @@ ncp_invalidate_dircache_entries(struct d
next = next->next;
}
spin_unlock(&parent->d_lock);
- spin_unlock(&dcache_lock);
}
struct ncp_cache_head {
Index: linux-2.6/fs/nfs/dir.c
===================================================================
--- linux-2.6.orig/fs/nfs/dir.c
+++ linux-2.6/fs/nfs/dir.c
@@ -1432,11 +1432,9 @@ static int nfs_unlink(struct inode *dir,
dfprintk(VFS, "NFS: unlink(%s/%ld, %s)\n", dir->i_sb->s_id,
dir->i_ino, dentry->d_name.name);
- spin_lock(&dcache_lock);
spin_lock(&dentry->d_lock);
if (dentry->d_count > 1) {
spin_unlock(&dentry->d_lock);
- spin_unlock(&dcache_lock);
/* Start asynchronous writeout of the inode */
write_inode_now(dentry->d_inode, 0);
error = nfs_sillyrename(dir, dentry);
@@ -1447,7 +1445,6 @@ static int nfs_unlink(struct inode *dir,
need_rehash = 1;
}
spin_unlock(&dentry->d_lock);
- spin_unlock(&dcache_lock);
error = nfs_safe_remove(dentry);
if (!error || error == -ENOENT) {
nfs_set_verifier(dentry, nfs_save_change_attribute(dir));
Index: linux-2.6/fs/ocfs2/dcache.c
===================================================================
--- linux-2.6.orig/fs/ocfs2/dcache.c
+++ linux-2.6/fs/ocfs2/dcache.c
@@ -140,7 +140,6 @@ struct dentry *ocfs2_find_local_alias(st
struct list_head *p;
struct dentry *dentry = NULL;
- spin_lock(&dcache_lock);
spin_lock(&dcache_inode_lock);
list_for_each(p, &inode->i_dentry) {
dentry = list_entry(p, struct dentry, d_alias);
@@ -160,7 +159,6 @@ struct dentry *ocfs2_find_local_alias(st
}
spin_unlock(&dcache_inode_lock);
- spin_unlock(&dcache_lock);
return dentry;
}
Index: linux-2.6/fs/smbfs/cache.c
===================================================================
--- linux-2.6.orig/fs/smbfs/cache.c
+++ linux-2.6/fs/smbfs/cache.c
@@ -62,7 +62,6 @@ smb_invalidate_dircache_entries(struct d
struct list_head *next;
struct dentry *dentry;
- spin_lock(&dcache_lock);
spin_lock(&parent->d_lock);
next = parent->d_subdirs.next;
while (next != &parent->d_subdirs) {
@@ -72,7 +71,6 @@ smb_invalidate_dircache_entries(struct d
next = next->next;
}
spin_unlock(&parent->d_lock);
- spin_unlock(&dcache_lock);
}
/*
@@ -98,7 +96,6 @@ smb_dget_fpos(struct dentry *dentry, str
}
/* If a pointer is invalid, we search the dentry. */
- spin_lock(&dcache_lock);
spin_lock(&parent->d_lock);
next = parent->d_subdirs.next;
while (next != &parent->d_subdirs) {
@@ -115,7 +112,6 @@ smb_dget_fpos(struct dentry *dentry, str
dent = NULL;
out_unlock:
spin_unlock(&parent->d_lock);
- spin_unlock(&dcache_lock);
return dent;
}
Index: linux-2.6/kernel/cgroup.c
===================================================================
--- linux-2.6.orig/kernel/cgroup.c
+++ linux-2.6/kernel/cgroup.c
@@ -695,7 +695,6 @@ static void cgroup_clear_directory(struc
struct list_head *node;
BUG_ON(!mutex_is_locked(&dentry->d_inode->i_mutex));
- spin_lock(&dcache_lock);
spin_lock(&dentry->d_lock);
node = dentry->d_subdirs.next;
while (node != &dentry->d_subdirs) {
@@ -710,18 +709,15 @@ static void cgroup_clear_directory(struc
dget_locked_dlock(d);
spin_unlock(&d->d_lock);
spin_unlock(&dentry->d_lock);
- spin_unlock(&dcache_lock);
d_delete(d);
simple_unlink(dentry->d_inode, d);
dput(d);
- spin_lock(&dcache_lock);
spin_lock(&dentry->d_lock);
} else
spin_unlock(&d->d_lock);
node = dentry->d_subdirs.next;
}
spin_unlock(&dentry->d_lock);
- spin_unlock(&dcache_lock);
}
/*
@@ -733,14 +729,12 @@ static void cgroup_d_remove_dir(struct d
cgroup_clear_directory(dentry);
- spin_lock(&dcache_lock);
parent = dentry->d_parent;
spin_lock(&parent->d_lock);
spin_lock(&dentry->d_lock);
list_del_init(&dentry->d_u.d_child);
spin_unlock(&dentry->d_lock);
spin_unlock(&parent->d_lock);
- spin_unlock(&dcache_lock);
remove_dir(dentry);
}
Index: linux-2.6/net/sunrpc/rpc_pipe.c
===================================================================
--- linux-2.6.orig/net/sunrpc/rpc_pipe.c
+++ linux-2.6/net/sunrpc/rpc_pipe.c
@@ -547,15 +547,14 @@ static void rpc_depopulate(struct dentry
mutex_lock_nested(&dir->i_mutex, I_MUTEX_CHILD);
repeat:
- spin_lock(&dcache_lock);
spin_lock(&parent->d_lock);
list_for_each_safe(pos, next, &parent->d_subdirs) {
dentry = list_entry(pos, struct dentry, d_u.d_child);
+ spin_lock(&dentry->d_lock);
if (!dentry->d_inode ||
dentry->d_inode->i_ino < start ||
dentry->d_inode->i_ino >= eof)
- continue;
- spin_lock(&dentry->d_lock);
+ goto next;
if (!d_unhashed(dentry)) {
dget_locked_dlock(dentry);
__d_drop(dentry);
@@ -563,11 +562,11 @@ repeat:
dvec[n++] = dentry;
if (n == ARRAY_SIZE(dvec))
break;
- } else
- spin_unlock(&dentry->d_lock);
+ }
+next:
+ spin_unlock(&dentry->d_lock);
}
spin_unlock(&parent->d_lock);
- spin_unlock(&dcache_lock);
if (n) {
do {
dentry = dvec[--n];
Index: linux-2.6/security/selinux/selinuxfs.c
===================================================================
--- linux-2.6.orig/security/selinux/selinuxfs.c
+++ linux-2.6/security/selinux/selinuxfs.c
@@ -943,7 +943,6 @@ static void sel_remove_entries(struct de
{
struct list_head *node;
- spin_lock(&dcache_lock);
spin_lock(&de->d_lock);
node = de->d_subdirs.next;
while (node != &de->d_subdirs) {
@@ -956,11 +955,9 @@ static void sel_remove_entries(struct de
dget_locked_dlock(d);
spin_unlock(&de->d_lock);
spin_unlock(&d->d_lock);
- spin_unlock(&dcache_lock);
d_delete(d);
simple_unlink(de->d_inode, d);
dput(d);
- spin_lock(&dcache_lock);
spin_lock(&de->d_lock);
} else
spin_unlock(&d->d_lock);
@@ -968,7 +965,6 @@ static void sel_remove_entries(struct de
}
spin_unlock(&de->d_lock);
- spin_unlock(&dcache_lock);
}
#define BOOL_DIR_NAME "booleans"
Index: linux-2.6/security/tomoyo/realpath.c
===================================================================
--- linux-2.6.orig/security/tomoyo/realpath.c
+++ linux-2.6/security/tomoyo/realpath.c
@@ -102,10 +102,8 @@ int tomoyo_realpath_from_path2(struct pa
if (ns_root.mnt)
ns_root.dentry = dget(ns_root.mnt->mnt_root);
vfsmount_read_unlock();
- spin_lock(&dcache_lock);
tmp = ns_root;
sp = __d_path(path, &tmp, newname, newname_len);
- spin_unlock(&dcache_lock);
path_put(&root);
path_put(&ns_root);
}
Index: linux-2.6/fs/nfs/getroot.c
===================================================================
--- linux-2.6.orig/fs/nfs/getroot.c
+++ linux-2.6/fs/nfs/getroot.c
@@ -64,13 +64,11 @@ static int nfs_superblock_set_dummy_root
* This again causes shrink_dcache_for_umount_subtree() to
* Oops, since the test for IS_ROOT() will fail.
*/
- spin_lock(&dcache_lock);
spin_lock(&dcache_inode_lock);
spin_lock(&sb->s_root->d_lock);
list_del_init(&sb->s_root->d_alias);
spin_unlock(&sb->s_root->d_lock);
spin_unlock(&dcache_inode_lock);
- spin_unlock(&dcache_lock);
}
return 0;
}
Index: linux-2.6/drivers/staging/pohmelfs/path_entry.c
===================================================================
--- linux-2.6.orig/drivers/staging/pohmelfs/path_entry.c
+++ linux-2.6/drivers/staging/pohmelfs/path_entry.c
@@ -100,7 +100,6 @@ int pohmelfs_path_length(struct pohmelfs
rcu_read_lock();
rename_retry:
seq = read_seqbegin(&rename_lock);
- spin_lock(&dcache_lock);
if (!IS_ROOT(d) && d_unhashed(d))
len += UNHASHED_OBSCURE_STRING_SIZE; /* Obscure " (deleted)" string */
@@ -109,7 +108,6 @@ rename_retry:
len += d->d_name.len + 1; /* Plus slash */
d = d->d_parent;
}
- spin_unlock(&dcache_lock);
if (read_seqretry(&rename_lock, seq))
goto rename_retry;
rcu_read_unlock();
Index: linux-2.6/fs/autofs4/waitq.c
===================================================================
--- linux-2.6.orig/fs/autofs4/waitq.c
+++ linux-2.6/fs/autofs4/waitq.c
@@ -194,12 +194,10 @@ static int autofs4_getpath(struct autofs
rcu_read_lock();
rename_retry:
seq = read_seqbegin(&rename_lock);
- spin_lock(&dcache_lock);
for (tmp = dentry ; tmp != root ; tmp = tmp->d_parent)
len += tmp->d_name.len + 1;
if (!len || --len > NAME_MAX) {
- spin_unlock(&dcache_lock);
if (read_seqretry(&rename_lock, seq))
goto rename_retry;
rcu_read_unlock();
@@ -215,7 +213,6 @@ rename_retry:
p -= tmp->d_name.len;
strncpy(p, tmp->d_name.name, tmp->d_name.len);
}
- spin_unlock(&dcache_lock);
if (read_seqretry(&rename_lock, seq))
goto rename_retry;
rcu_read_unlock();
Index: linux-2.6/fs/nfs/namespace.c
===================================================================
--- linux-2.6.orig/fs/nfs/namespace.c
+++ linux-2.6/fs/nfs/namespace.c
@@ -57,7 +57,6 @@ char *nfs_path(const char *base,
rcu_read_lock();
rename_retry:
seq = read_seqbegin(&rename_lock);
- spin_lock(&dcache_lock);
while (!IS_ROOT(dentry) && dentry != droot) {
namelen = dentry->d_name.len;
buflen -= namelen + 1;
@@ -68,7 +67,6 @@ rename_retry:
*--end = '/';
dentry = dentry->d_parent;
}
- spin_unlock(&dcache_lock);
if (read_seqretry(&rename_lock, seq))
goto rename_retry;
rcu_read_unlock();
@@ -88,7 +86,6 @@ rename_retry:
memcpy(end, base, namelen);
return end;
Elong_unlock:
- spin_unlock(&dcache_lock);
if (read_seqretry(&rename_lock, seq))
goto rename_retry;
rcu_read_unlock();
Index: linux-2.6/include/linux/fsnotify_backend.h
===================================================================
--- linux-2.6.orig/include/linux/fsnotify_backend.h
+++ linux-2.6/include/linux/fsnotify_backend.h
@@ -276,10 +276,10 @@ static inline void __fsnotify_update_dca
{
struct dentry *parent;
- assert_spin_locked(&dcache_lock);
assert_spin_locked(&dentry->d_lock);
parent = dentry->d_parent;
+ /* XXX: after dcache_lock removal, there is a race with parent->d_inode and fsnotify_inode_watches_children. must fix */
if (parent->d_inode && fsnotify_inode_watches_children(parent->d_inode))
dentry->d_flags |= DCACHE_FSNOTIFY_PARENT_WATCHED;
else
@@ -288,15 +288,12 @@ static inline void __fsnotify_update_dca
/*
* fsnotify_d_instantiate - instantiate a dentry for inode
- * Called with dcache_lock held.
*/
static inline void __fsnotify_d_instantiate(struct dentry *dentry, struct inode *inode)
{
if (!inode)
return;
- assert_spin_locked(&dcache_lock);
-
spin_lock(&dentry->d_lock);
__fsnotify_update_dcache_flags(dentry);
spin_unlock(&dentry->d_lock);
Index: linux-2.6/fs/notify/fsnotify.c
===================================================================
--- linux-2.6.orig/fs/notify/fsnotify.c
+++ linux-2.6/fs/notify/fsnotify.c
@@ -52,7 +52,6 @@ void __fsnotify_update_child_dentry_flag
/* determine if the children should tell inode about their events */
watched = fsnotify_inode_watches_children(inode);
- spin_lock(&dcache_lock);
spin_lock(&dcache_inode_lock);
/* run all of the dentries associated with this inode. Since this is a
* directory, there damn well better only be one item on this list */
@@ -77,7 +76,6 @@ void __fsnotify_update_child_dentry_flag
spin_unlock(&alias->d_lock);
}
spin_unlock(&dcache_inode_lock);
- spin_unlock(&dcache_lock);
}
/* Notify this dentry's parent about a child's events. */
^ permalink raw reply [flat|nested] 64+ messages in thread
* [patch 15/33] fs: dcache reduce dput locking
2009-09-04 6:51 [patch 00/33] my current vfs scalability patch queue npiggin
` (13 preceding siblings ...)
2009-09-04 6:51 ` [patch 14/33] fs: dcache remove dcache_lock npiggin
@ 2009-09-04 6:51 ` npiggin
2009-09-04 6:51 ` [patch 16/33] fs: dcache per-bucket dcache hash locking npiggin
` (18 subsequent siblings)
33 siblings, 0 replies; 64+ messages in thread
From: npiggin @ 2009-09-04 6:51 UTC (permalink / raw)
To: linux-fsdevel; +Cc: linux-kernel
[-- Attachment #1: dcache-dput-less-dcache_lock.patch --]
[-- Type: text/plain, Size: 2605 bytes --]
It is possible to run dput without taking locks up-front. In many cases
where we don't kill the dentry anyway, these locks are not required.
(I think... need to think about it more). Further changes ->d_delete
locking which is not all audited.
---
fs/dcache.c | 59 ++++++++++++++++++++++++++++++++---------------------------
1 file changed, 32 insertions(+), 27 deletions(-)
Index: linux-2.6/fs/dcache.c
===================================================================
--- linux-2.6.orig/fs/dcache.c
+++ linux-2.6/fs/dcache.c
@@ -262,7 +262,8 @@ static struct dentry *d_kill(struct dent
void dput(struct dentry *dentry)
{
- struct dentry *parent = NULL;
+ struct dentry *parent;
+
if (!dentry)
return;
@@ -270,23 +271,9 @@ repeat:
if (dentry->d_count == 1)
might_sleep();
spin_lock(&dentry->d_lock);
- if (dentry->d_count == 1) {
- if (!spin_trylock(&dcache_inode_lock)) {
-drop2:
- spin_unlock(&dentry->d_lock);
- goto repeat;
- }
- parent = dentry->d_parent;
- if (parent) {
- BUG_ON(parent == dentry);
- if (!spin_trylock(&parent->d_lock)) {
- spin_unlock(&dcache_inode_lock);
- goto drop2;
- }
- }
- }
- dentry->d_count--;
- if (dentry->d_count) {
+ BUG_ON(!dentry->d_count);
+ if (dentry->d_count > 1) {
+ dentry->d_count--;
spin_unlock(&dentry->d_lock);
return;
}
@@ -295,8 +282,10 @@ drop2:
* AV: ->d_delete() is _NOT_ allowed to block now.
*/
if (dentry->d_op && dentry->d_op->d_delete) {
- if (dentry->d_op->d_delete(dentry))
- goto unhash_it;
+ if (dentry->d_op->d_delete(dentry)) {
+ __d_drop(dentry);
+ goto kill_it;
+ }
}
/* Unreachable? Get rid of it */
if (d_unhashed(dentry))
@@ -305,15 +294,31 @@ drop2:
dentry->d_flags |= DCACHE_REFERENCED;
dentry_lru_add(dentry);
}
- spin_unlock(&dentry->d_lock);
- if (parent)
- spin_unlock(&parent->d_lock);
- spin_unlock(&dcache_inode_lock);
- return;
+ dentry->d_count--;
+ spin_unlock(&dentry->d_lock);
+ return;
-unhash_it:
- __d_drop(dentry);
kill_it:
+ spin_unlock(&dentry->d_lock);
+ spin_lock(&dcache_inode_lock);
+relock:
+ spin_lock(&dentry->d_lock);
+ parent = dentry->d_parent;
+ if (parent) {
+ BUG_ON(parent == dentry);
+ if (!spin_trylock(&parent->d_lock)) {
+ spin_unlock(&dentry->d_lock);
+ goto relock;
+ }
+ }
+ dentry->d_count--;
+ if (dentry->d_count) {
+ /* This case should be fine */
+ spin_unlock(&dentry->d_lock);
+ spin_unlock(&parent->d_lock);
+ spin_unlock(&dcache_inode_lock);
+ return;
+ }
/* if dentry was on the d_lru list delete it from there */
dentry_lru_del(dentry);
dentry = d_kill(dentry);
^ permalink raw reply [flat|nested] 64+ messages in thread
* [patch 16/33] fs: dcache per-bucket dcache hash locking
2009-09-04 6:51 [patch 00/33] my current vfs scalability patch queue npiggin
` (14 preceding siblings ...)
2009-09-04 6:51 ` [patch 15/33] fs: dcache reduce dput locking npiggin
@ 2009-09-04 6:51 ` npiggin
2009-09-04 14:51 ` Daniel Walker
2009-09-04 6:51 ` [patch 17/33] fs: dcache reduce dcache_inode_lock npiggin
` (17 subsequent siblings)
33 siblings, 1 reply; 64+ messages in thread
From: npiggin @ 2009-09-04 6:51 UTC (permalink / raw)
To: linux-fsdevel; +Cc: linux-kernel
[-- Attachment #1: dcache-chain-hashlock.patch --]
[-- Type: text/plain, Size: 11208 bytes --]
We can turn the dcache hash locking from a global dcache_hash_lock into
per-bucket locking.
XXX: should probably use a bit lock in the first bit of the hash pointers
to avoid any space bloating (non-atomic unlock means no extra atomics either)
---
fs/dcache.c | 197 ++++++++++++++++++++++++++++---------------------
include/linux/dcache.h | 20 ----
2 files changed, 115 insertions(+), 102 deletions(-)
Index: linux-2.6/fs/dcache.c
===================================================================
--- linux-2.6.orig/fs/dcache.c
+++ linux-2.6/fs/dcache.c
@@ -38,7 +38,7 @@
* Usage:
* dcache_inode_lock protects:
* - the inode alias lists, d_inode
- * dcache_hash_lock protects:
+ * dcache_hash_bucket->lock protects:
* - the dcache hash table
* dcache_lru_lock protects:
* - the dcache lru lists and counters
@@ -53,18 +53,16 @@
* dcache_inode_lock
* dentry->d_lock
* dcache_lru_lock
- * dcache_hash_lock
+ * dcache_hash_bucket->lock
*/
int sysctl_vfs_cache_pressure __read_mostly = 100;
EXPORT_SYMBOL_GPL(sysctl_vfs_cache_pressure);
__cacheline_aligned_in_smp DEFINE_SPINLOCK(dcache_inode_lock);
-__cacheline_aligned_in_smp DEFINE_SPINLOCK(dcache_hash_lock);
static __cacheline_aligned_in_smp DEFINE_SPINLOCK(dcache_lru_lock);
__cacheline_aligned_in_smp DEFINE_SEQLOCK(rename_lock);
EXPORT_SYMBOL(dcache_inode_lock);
-EXPORT_SYMBOL(dcache_hash_lock);
static struct kmem_cache *dentry_cache __read_mostly;
@@ -83,7 +81,12 @@ static struct kmem_cache *dentry_cache _
static unsigned int d_hash_mask __read_mostly;
static unsigned int d_hash_shift __read_mostly;
-static struct hlist_head *dentry_hashtable __read_mostly;
+
+struct dcache_hash_bucket {
+ spinlock_t lock;
+ struct hlist_head head;
+};
+static struct dcache_hash_bucket *dentry_hashtable __read_mostly;
/* Statistics gathering. */
struct dentry_stat_t dentry_stat = {
@@ -91,6 +94,14 @@ struct dentry_stat_t dentry_stat = {
.age_limit = 45,
};
+static inline struct dcache_hash_bucket *d_hash(struct dentry *parent,
+ unsigned long hash)
+{
+ hash += ((unsigned long) parent ^ GOLDEN_RATIO_PRIME) / L1_CACHE_BYTES;
+ hash = hash ^ ((hash ^ GOLDEN_RATIO_PRIME) >> D_HASHBITS);
+ return dentry_hashtable + (hash & D_HASHMASK);
+}
+
static void __d_free(struct dentry *dentry)
{
WARN_ON(!list_empty(&dentry->d_alias));
@@ -231,6 +242,73 @@ static struct dentry *d_kill(struct dent
return parent;
}
+void __d_drop(struct dentry *dentry)
+{
+ if (!(dentry->d_flags & DCACHE_UNHASHED)) {
+ struct dcache_hash_bucket *b;
+ b = d_hash(dentry->d_parent, dentry->d_name.hash);
+ dentry->d_flags |= DCACHE_UNHASHED;
+ spin_lock(&b->lock);
+ hlist_del_rcu(&dentry->d_hash);
+ spin_unlock(&b->lock);
+ }
+}
+
+void d_drop(struct dentry *dentry)
+{
+ spin_lock(&dentry->d_lock);
+ __d_drop(dentry);
+ spin_unlock(&dentry->d_lock);
+}
+
+/* This should be called _only_ with a lock pinning the dentry */
+static inline struct dentry * __dget_locked_dlock(struct dentry *dentry)
+{
+ dentry->d_count++;
+ dentry_lru_del_init(dentry);
+ return dentry;
+}
+
+static inline struct dentry * __dget_locked(struct dentry *dentry)
+{
+ spin_lock(&dentry->d_lock);
+ __dget_locked_dlock(dentry);
+ spin_lock(&dentry->d_lock);
+ return dentry;
+}
+
+struct dentry * dget_locked_dlock(struct dentry *dentry)
+{
+ return __dget_locked_dlock(dentry);
+}
+
+struct dentry * dget_locked(struct dentry *dentry)
+{
+ return __dget_locked(dentry);
+}
+
+struct dentry *dget_parent(struct dentry *dentry)
+{
+ struct dentry *ret;
+
+repeat:
+ spin_lock(&dentry->d_lock);
+ ret = dentry->d_parent;
+ if (!ret)
+ goto out;
+ if (!spin_trylock(&ret->d_lock)) {
+ spin_unlock(&dentry->d_lock);
+ goto repeat;
+ }
+ BUG_ON(!ret->d_count);
+ ret->d_count++;
+ spin_unlock(&ret->d_lock);
+out:
+ spin_unlock(&dentry->d_lock);
+ return ret;
+}
+EXPORT_SYMBOL(dget_parent);
+
/*
* This is dput
*
@@ -380,54 +458,6 @@ int d_invalidate(struct dentry * dentry)
return 0;
}
-/* This should be called _only_ with a lock pinning the dentry */
-static inline struct dentry * __dget_locked_dlock(struct dentry *dentry)
-{
- dentry->d_count++;
- dentry_lru_del_init(dentry);
- return dentry;
-}
-
-static inline struct dentry * __dget_locked(struct dentry *dentry)
-{
- spin_lock(&dentry->d_lock);
- __dget_locked_dlock(dentry);
- spin_lock(&dentry->d_lock);
- return dentry;
-}
-
-struct dentry * dget_locked_dlock(struct dentry *dentry)
-{
- return __dget_locked_dlock(dentry);
-}
-
-struct dentry * dget_locked(struct dentry *dentry)
-{
- return __dget_locked(dentry);
-}
-
-struct dentry *dget_parent(struct dentry *dentry)
-{
- struct dentry *ret;
-
-repeat:
- spin_lock(&dentry->d_lock);
- ret = dentry->d_parent;
- if (!ret)
- goto out;
- if (!spin_trylock(&ret->d_lock)) {
- spin_unlock(&dentry->d_lock);
- goto repeat;
- }
- BUG_ON(!ret->d_count);
- ret->d_count++;
- spin_unlock(&ret->d_lock);
-out:
- spin_unlock(&dentry->d_lock);
- return ret;
-}
-EXPORT_SYMBOL(dget_parent);
-
/**
* d_find_alias - grab a hashed alias of inode
* @inode: inode in question
@@ -1314,14 +1344,6 @@ struct dentry * d_alloc_root(struct inod
return res;
}
-static inline struct hlist_head *d_hash(struct dentry *parent,
- unsigned long hash)
-{
- hash += ((unsigned long) parent ^ GOLDEN_RATIO_PRIME) / L1_CACHE_BYTES;
- hash = hash ^ ((hash ^ GOLDEN_RATIO_PRIME) >> D_HASHBITS);
- return dentry_hashtable + (hash & D_HASHMASK);
-}
-
/**
* d_obtain_alias - find or allocate a dentry for a given inode
* @inode: inode to allocate the dentry for
@@ -1568,7 +1590,8 @@ struct dentry * __d_lookup(struct dentry
unsigned int len = name->len;
unsigned int hash = name->hash;
const unsigned char *str = name->name;
- struct hlist_head *head = d_hash(parent,hash);
+ struct dcache_hash_bucket *b = d_hash(parent, hash);
+ struct hlist_head *head = &b->head;
struct dentry *found = NULL;
struct hlist_node *node;
struct dentry *dentry;
@@ -1662,6 +1685,7 @@ out:
int d_validate(struct dentry *dentry, struct dentry *dparent)
{
+ struct dcache_hash_bucket *b;
struct hlist_head *base;
struct hlist_node *lhp;
@@ -1673,20 +1697,21 @@ int d_validate(struct dentry *dentry, st
goto out;
spin_lock(&dentry->d_lock);
- spin_lock(&dcache_hash_lock);
- base = d_hash(dparent, dentry->d_name.hash);
- hlist_for_each(lhp,base) {
+ b = d_hash(dparent, dentry->d_name.hash);
+ base = &b->head;
+ spin_lock(&b->lock);
+ hlist_for_each(lhp, base) {
/* hlist_for_each_entry_rcu() not required for d_hash list
- * as it is parsed under dcache_hash_lock
+ * as it is parsed under dcache_hash_bucket->lock
*/
if (dentry == hlist_entry(lhp, struct dentry, d_hash)) {
- spin_unlock(&dcache_hash_lock);
+ spin_unlock(&b->lock);
__dget_locked_dlock(dentry);
spin_unlock(&dentry->d_lock);
return 1;
}
}
- spin_unlock(&dcache_hash_lock);
+ spin_unlock(&b->lock);
spin_unlock(&dentry->d_lock);
out:
return 0;
@@ -1737,11 +1762,12 @@ void d_delete(struct dentry * dentry)
fsnotify_nameremove(dentry, isdir);
}
-static void __d_rehash(struct dentry * entry, struct hlist_head *list)
+static void __d_rehash(struct dentry * entry, struct dcache_hash_bucket *b)
{
-
entry->d_flags &= ~DCACHE_UNHASHED;
- hlist_add_head_rcu(&entry->d_hash, list);
+ spin_lock(&b->lock);
+ hlist_add_head_rcu(&entry->d_hash, &b->head);
+ spin_unlock(&b->lock);
}
static void _d_rehash(struct dentry * entry)
@@ -1759,9 +1785,7 @@ static void _d_rehash(struct dentry * en
void d_rehash(struct dentry * entry)
{
spin_lock(&entry->d_lock);
- spin_lock(&dcache_hash_lock);
_d_rehash(entry);
- spin_unlock(&dcache_hash_lock);
spin_unlock(&entry->d_lock);
}
@@ -1839,6 +1863,7 @@ static void switch_names(struct dentry *
*/
static void d_move_locked(struct dentry * dentry, struct dentry * target)
{
+ struct dcache_hash_bucket *b;
if (!dentry->d_inode)
printk(KERN_WARNING "VFS: moving negative dcache entry\n");
@@ -1867,11 +1892,13 @@ static void d_move_locked(struct dentry
}
/* Move the dentry to the target hash queue, if on different bucket */
- spin_lock(&dcache_hash_lock);
- if (!d_unhashed(dentry))
+ if (!d_unhashed(dentry)) {
+ b = d_hash(dentry->d_parent, dentry->d_name.hash);
+ spin_lock(&b->lock);
hlist_del_rcu(&dentry->d_hash);
+ spin_unlock(&b->lock);
+ }
__d_rehash(dentry, d_hash(target->d_parent, target->d_name.hash));
- spin_unlock(&dcache_hash_lock);
/* Unhash the target: dput() will then get rid of it */
__d_drop(target);
@@ -2078,9 +2105,7 @@ struct dentry *d_materialise_unique(stru
found_lock:
spin_lock(&actual->d_lock);
found:
- spin_lock(&dcache_hash_lock);
_d_rehash(actual);
- spin_unlock(&dcache_hash_lock);
spin_unlock(&actual->d_lock);
spin_unlock(&dcache_inode_lock);
out_nolock:
@@ -2533,7 +2558,7 @@ static void __init dcache_init_early(voi
dentry_hashtable =
alloc_large_system_hash("Dentry cache",
- sizeof(struct hlist_head),
+ sizeof(struct dcache_hash_bucket),
dhash_entries,
13,
HASH_EARLY,
@@ -2541,8 +2566,10 @@ static void __init dcache_init_early(voi
&d_hash_mask,
0);
- for (loop = 0; loop < (1 << d_hash_shift); loop++)
- INIT_HLIST_HEAD(&dentry_hashtable[loop]);
+ for (loop = 0; loop < (1 << d_hash_shift); loop++) {
+ spin_lock_init(&dentry_hashtable[loop].lock);
+ INIT_HLIST_HEAD(&dentry_hashtable[loop].head);
+ }
}
static void __init dcache_init(void)
@@ -2565,7 +2592,7 @@ static void __init dcache_init(void)
dentry_hashtable =
alloc_large_system_hash("Dentry cache",
- sizeof(struct hlist_head),
+ sizeof(struct dcache_hash_bucket),
dhash_entries,
13,
0,
@@ -2573,8 +2600,10 @@ static void __init dcache_init(void)
&d_hash_mask,
0);
- for (loop = 0; loop < (1 << d_hash_shift); loop++)
- INIT_HLIST_HEAD(&dentry_hashtable[loop]);
+ for (loop = 0; loop < (1 << d_hash_shift); loop++) {
+ spin_lock_init(&dentry_hashtable[loop].lock);
+ INIT_HLIST_HEAD(&dentry_hashtable[loop].head);
+ }
}
/* SLAB cache for __getname() consumers */
Index: linux-2.6/include/linux/dcache.h
===================================================================
--- linux-2.6.orig/include/linux/dcache.h
+++ linux-2.6/include/linux/dcache.h
@@ -187,7 +187,6 @@ d_iput: no no yes
#define DCACHE_FSNOTIFY_PARENT_WATCHED 0x0080 /* Parent inode is watched by some fsnotify listener */
extern spinlock_t dcache_inode_lock;
-extern spinlock_t dcache_hash_lock;
extern seqlock_t rename_lock;
/**
@@ -205,23 +204,8 @@ extern seqlock_t rename_lock;
*
* __d_drop requires dentry->d_lock.
*/
-
-static inline void __d_drop(struct dentry *dentry)
-{
- if (!(dentry->d_flags & DCACHE_UNHASHED)) {
- dentry->d_flags |= DCACHE_UNHASHED;
- spin_lock(&dcache_hash_lock);
- hlist_del_rcu(&dentry->d_hash);
- spin_unlock(&dcache_hash_lock);
- }
-}
-
-static inline void d_drop(struct dentry *dentry)
-{
- spin_lock(&dentry->d_lock);
- __d_drop(dentry);
- spin_unlock(&dentry->d_lock);
-}
+void d_drop(struct dentry *dentry);
+void __d_drop(struct dentry *dentry);
static inline int dname_external(struct dentry *dentry)
{
^ permalink raw reply [flat|nested] 64+ messages in thread
* [patch 17/33] fs: dcache reduce dcache_inode_lock
2009-09-04 6:51 [patch 00/33] my current vfs scalability patch queue npiggin
` (15 preceding siblings ...)
2009-09-04 6:51 ` [patch 16/33] fs: dcache per-bucket dcache hash locking npiggin
@ 2009-09-04 6:51 ` npiggin
2009-09-04 6:51 ` [patch 18/33] fs: dcache per-inode inode alias locking npiggin
` (16 subsequent siblings)
33 siblings, 0 replies; 64+ messages in thread
From: npiggin @ 2009-09-04 6:51 UTC (permalink / raw)
To: linux-fsdevel; +Cc: linux-kernel
[-- Attachment #1: fs-dcache-d_delete-less-lock.patch --]
[-- Type: text/plain, Size: 1893 bytes --]
dcache_inode_lock can be avoided in d_delete() and d_materialise_unique()
in cases where it is not required.
---
fs/dcache.c | 23 +++++++++++------------
1 file changed, 11 insertions(+), 12 deletions(-)
Index: linux-2.6/fs/dcache.c
===================================================================
--- linux-2.6.orig/fs/dcache.c
+++ linux-2.6/fs/dcache.c
@@ -1744,10 +1744,14 @@ void d_delete(struct dentry * dentry)
/*
* Are we the only user?
*/
- spin_lock(&dcache_inode_lock);
+again:
spin_lock(&dentry->d_lock);
isdir = S_ISDIR(dentry->d_inode->i_mode);
if (dentry->d_count == 1) {
+ if (!spin_trylock(&dcache_inode_lock)) {
+ spin_unlock(&dentry->d_lock);
+ goto again;
+ }
dentry_iput(dentry);
fsnotify_nameremove(dentry, isdir);
return;
@@ -1757,7 +1761,6 @@ void d_delete(struct dentry * dentry)
__d_drop(dentry);
spin_unlock(&dentry->d_lock);
- spin_unlock(&dcache_inode_lock);
fsnotify_nameremove(dentry, isdir);
}
@@ -2064,14 +2067,15 @@ struct dentry *d_materialise_unique(stru
BUG_ON(!d_unhashed(dentry));
- spin_lock(&dcache_inode_lock);
-
if (!inode) {
actual = dentry;
__d_instantiate(dentry, NULL);
- goto found_lock;
+ d_rehash(actual);
+ goto out_nolock;
}
+ spin_lock(&dcache_inode_lock);
+
if (S_ISDIR(inode->i_mode)) {
struct dentry *alias;
@@ -2099,10 +2103,9 @@ struct dentry *d_materialise_unique(stru
actual = __d_instantiate_unique(dentry, inode);
if (!actual)
actual = dentry;
- else if (unlikely(!d_unhashed(actual)))
- goto shouldnt_be_hashed;
+ else
+ BUG_ON(!d_unhashed(actual));
-found_lock:
spin_lock(&actual->d_lock);
found:
_d_rehash(actual);
@@ -2116,10 +2119,6 @@ out_nolock:
iput(inode);
return actual;
-
-shouldnt_be_hashed:
- spin_unlock(&dcache_inode_lock);
- BUG();
}
static int prepend(char **buffer, int *buflen, const char *str, int namelen)
^ permalink raw reply [flat|nested] 64+ messages in thread
* [patch 18/33] fs: dcache per-inode inode alias locking
2009-09-04 6:51 [patch 00/33] my current vfs scalability patch queue npiggin
` (16 preceding siblings ...)
2009-09-04 6:51 ` [patch 17/33] fs: dcache reduce dcache_inode_lock npiggin
@ 2009-09-04 6:51 ` npiggin
2009-09-04 6:52 ` [patch 19/33] fs: icache lock s_inodes list npiggin
` (15 subsequent siblings)
33 siblings, 0 replies; 64+ messages in thread
From: npiggin @ 2009-09-04 6:51 UTC (permalink / raw)
To: linux-fsdevel; +Cc: linux-kernel
[-- Attachment #1: dcache-split-inode_lock.patch --]
[-- Type: text/plain, Size: 16634 bytes --]
dcache_inode_lock can be replaced with per-inode locking. Use existing
inode->i_lock for this. This is slightly non-trivial because we sometimes
need to find the inode from the dentry, which requires d_inode to be
stabilised (either with refcount or d_lock).
---
fs/affs/amigaffs.c | 4 -
fs/dcache.c | 118 +++++++++++++++++++++++++-------------------
fs/exportfs/expfs.c | 12 ++--
fs/nfs/getroot.c | 4 -
fs/notify/fsnotify.c | 4 -
fs/notify/inotify/inotify.c | 4 -
fs/ocfs2/dcache.c | 4 -
fs/sysfs/dir.c | 6 +-
include/linux/dcache.h | 1
9 files changed, 89 insertions(+), 68 deletions(-)
Index: linux-2.6/fs/dcache.c
===================================================================
--- linux-2.6.orig/fs/dcache.c
+++ linux-2.6/fs/dcache.c
@@ -36,7 +36,7 @@
/*
* Usage:
- * dcache_inode_lock protects:
+ * dcache->d_inode->i_lock protects:
* - the inode alias lists, d_inode
* dcache_hash_bucket->lock protects:
* - the dcache hash table
@@ -50,7 +50,7 @@
* - d_subdirs and children's d_child
*
* Ordering:
- * dcache_inode_lock
+ * dcache->d_inode->i_lock
* dentry->d_lock
* dcache_lru_lock
* dcache_hash_bucket->lock
@@ -58,12 +58,9 @@
int sysctl_vfs_cache_pressure __read_mostly = 100;
EXPORT_SYMBOL_GPL(sysctl_vfs_cache_pressure);
-__cacheline_aligned_in_smp DEFINE_SPINLOCK(dcache_inode_lock);
static __cacheline_aligned_in_smp DEFINE_SPINLOCK(dcache_lru_lock);
__cacheline_aligned_in_smp DEFINE_SEQLOCK(rename_lock);
-EXPORT_SYMBOL(dcache_inode_lock);
-
static struct kmem_cache *dentry_cache __read_mostly;
#define DNAME_INLINE_LEN (sizeof(struct dentry)-offsetof(struct dentry,d_iname))
@@ -138,14 +135,13 @@ static void d_free(struct dentry *dentry
*/
static void dentry_iput(struct dentry * dentry)
__releases(dentry->d_lock)
- __releases(dcache_inode_lock)
{
struct inode *inode = dentry->d_inode;
if (inode) {
dentry->d_inode = NULL;
list_del_init(&dentry->d_alias);
spin_unlock(&dentry->d_lock);
- spin_unlock(&dcache_inode_lock);
+ spin_unlock(&inode->i_lock);
if (!inode->i_nlink)
fsnotify_inoderemove(inode);
if (dentry->d_op && dentry->d_op->d_iput)
@@ -154,7 +150,6 @@ static void dentry_iput(struct dentry *
iput(inode);
} else {
spin_unlock(&dentry->d_lock);
- spin_unlock(&dcache_inode_lock);
}
}
@@ -225,7 +220,6 @@ static void dentry_lru_del_init(struct d
*/
static struct dentry *d_kill(struct dentry *dentry)
__releases(dentry->d_lock)
- __releases(dcache_inode_lock)
{
struct dentry *parent;
@@ -341,6 +335,7 @@ EXPORT_SYMBOL(dget_parent);
void dput(struct dentry *dentry)
{
struct dentry *parent;
+ struct inode *inode;
if (!dentry)
return;
@@ -376,17 +371,24 @@ repeat:
spin_unlock(&dentry->d_lock);
return;
-kill_it:
- spin_unlock(&dentry->d_lock);
- spin_lock(&dcache_inode_lock);
-relock:
+relock1:
spin_lock(&dentry->d_lock);
+kill_it:
+ inode = dentry->d_inode;
+ if (inode) {
+ if (!spin_trylock(&inode->i_lock)) {
+relock2:
+ spin_unlock(&dentry->d_lock);
+ goto relock1;
+ }
+ }
parent = dentry->d_parent;
if (parent) {
BUG_ON(parent == dentry);
if (!spin_trylock(&parent->d_lock)) {
- spin_unlock(&dentry->d_lock);
- goto relock;
+ if (inode)
+ spin_unlock(&inode->i_lock);
+ goto relock2;
}
}
dentry->d_count--;
@@ -394,7 +396,8 @@ relock:
/* This case should be fine */
spin_unlock(&dentry->d_lock);
spin_unlock(&parent->d_lock);
- spin_unlock(&dcache_inode_lock);
+ if (inode)
+ spin_unlock(&inode->i_lock);
return;
}
/* if dentry was on the d_lru list delete it from there */
@@ -510,9 +513,9 @@ struct dentry * d_find_alias(struct inod
struct dentry *de = NULL;
if (!list_empty(&inode->i_dentry)) {
- spin_lock(&dcache_inode_lock);
+ spin_lock(&inode->i_lock);
de = __d_find_alias(inode, 0);
- spin_unlock(&dcache_inode_lock);
+ spin_unlock(&inode->i_lock);
}
return de;
}
@@ -525,20 +528,20 @@ void d_prune_aliases(struct inode *inode
{
struct dentry *dentry;
restart:
- spin_lock(&dcache_inode_lock);
+ spin_lock(&inode->i_lock);
list_for_each_entry(dentry, &inode->i_dentry, d_alias) {
spin_lock(&dentry->d_lock);
if (!dentry->d_count) {
__dget_locked_dlock(dentry);
__d_drop(dentry);
spin_unlock(&dentry->d_lock);
- spin_unlock(&dcache_inode_lock);
+ spin_unlock(&inode->i_lock);
dput(dentry);
goto restart;
}
spin_unlock(&dentry->d_lock);
}
- spin_unlock(&dcache_inode_lock);
+ spin_unlock(&inode->i_lock);
}
/*
@@ -560,8 +563,10 @@ static void prune_one_dentry(struct dent
*/
while (dentry) {
struct dentry *parent = NULL;
+ struct inode *inode = dentry->d_inode;
- spin_lock(&dcache_inode_lock);
+ if (inode)
+ spin_lock(&inode->i_lock);
again:
spin_lock(&dentry->d_lock);
if (dentry->d_parent && dentry != dentry->d_parent) {
@@ -576,7 +581,8 @@ again:
if (parent)
spin_unlock(&parent->d_lock);
spin_unlock(&dentry->d_lock);
- spin_unlock(&dcache_inode_lock);
+ if (inode)
+ spin_unlock(&inode->i_lock);
return;
}
@@ -645,10 +651,11 @@ restart:
}
spin_unlock(&dcache_lru_lock);
- spin_lock(&dcache_inode_lock);
again:
spin_lock(&dcache_lru_lock); /* lru_lock also protects tmp list */
while (!list_empty(&tmp)) {
+ struct inode *inode;
+
dentry = list_entry(tmp.prev, struct dentry, d_lru);
if (!spin_trylock(&dentry->d_lock)) {
@@ -666,11 +673,18 @@ again1:
spin_unlock(&dentry->d_lock);
continue;
}
+ inode = dentry->d_inode;
+ if (inode && !spin_trylock(&inode->i_lock)) {
+again2:
+ spin_unlock(&dentry->d_lock);
+ goto again1;
+ }
if (dentry->d_parent) {
BUG_ON(dentry == dentry->d_parent);
if (!spin_trylock(&dentry->d_parent->d_lock)) {
- spin_unlock(&dentry->d_lock);
- goto again1;
+ if (inode)
+ spin_unlock(&inode->i_lock);
+ goto again2;
}
}
__dentry_lru_del_init(dentry);
@@ -678,10 +692,8 @@ again1:
prune_one_dentry(dentry);
/* dentry->d_lock dropped */
- spin_lock(&dcache_inode_lock);
spin_lock(&dcache_lru_lock);
}
- spin_unlock(&dcache_inode_lock);
if (count == NULL && !list_empty(&sb->s_dentry_lru))
goto restart;
@@ -1242,9 +1254,11 @@ static void __d_instantiate(struct dentr
void d_instantiate(struct dentry *entry, struct inode * inode)
{
BUG_ON(!list_empty(&entry->d_alias));
- spin_lock(&dcache_inode_lock);
+ if (inode)
+ spin_lock(&inode->i_lock);
__d_instantiate(entry, inode);
- spin_unlock(&dcache_inode_lock);
+ if (inode)
+ spin_unlock(&inode->i_lock);
security_d_instantiate(entry, inode);
}
@@ -1302,9 +1316,11 @@ struct dentry *d_instantiate_unique(stru
BUG_ON(!list_empty(&entry->d_alias));
- spin_lock(&dcache_inode_lock);
+ if (inode)
+ spin_lock(&inode->i_lock);
result = __d_instantiate_unique(entry, inode);
- spin_unlock(&dcache_inode_lock);
+ if (inode)
+ spin_unlock(&inode->i_lock);
if (!result) {
security_d_instantiate(entry, inode);
@@ -1384,10 +1400,10 @@ struct dentry *d_obtain_alias(struct ino
}
tmp->d_parent = tmp; /* make sure dput doesn't croak */
- spin_lock(&dcache_inode_lock);
+ spin_lock(&inode->i_lock);
res = __d_find_alias(inode, 0);
if (res) {
- spin_unlock(&dcache_inode_lock);
+ spin_unlock(&inode->i_lock);
dput(tmp);
goto out_iput;
}
@@ -1401,7 +1417,7 @@ struct dentry *d_obtain_alias(struct ino
list_add(&tmp->d_alias, &inode->i_dentry);
hlist_add_head(&tmp->d_hash, &inode->i_sb->s_anon);
spin_unlock(&tmp->d_lock);
- spin_unlock(&dcache_inode_lock);
+ spin_unlock(&inode->i_lock);
return tmp;
@@ -1432,19 +1448,19 @@ struct dentry *d_splice_alias(struct ino
struct dentry *new = NULL;
if (inode && S_ISDIR(inode->i_mode)) {
- spin_lock(&dcache_inode_lock);
+ spin_lock(&inode->i_lock);
new = __d_find_alias(inode, 1);
if (new) {
BUG_ON(!(new->d_flags & DCACHE_DISCONNECTED));
- spin_unlock(&dcache_inode_lock);
+ spin_unlock(&inode->i_lock);
security_d_instantiate(new, inode);
d_rehash(dentry);
d_move(new, dentry);
iput(inode);
} else {
- /* already taken dcache_inode_lock, d_add() by hand */
+ /* already taken inode->i_lock, d_add() by hand */
__d_instantiate(dentry, inode);
- spin_unlock(&dcache_inode_lock);
+ spin_unlock(&inode->i_lock);
security_d_instantiate(dentry, inode);
d_rehash(dentry);
}
@@ -1516,10 +1532,10 @@ struct dentry *d_add_ci(struct dentry *d
* Negative dentry: instantiate it unless the inode is a directory and
* already has a dentry.
*/
- spin_lock(&dcache_inode_lock);
+ spin_lock(&inode->i_lock);
if (!S_ISDIR(inode->i_mode) || list_empty(&inode->i_dentry)) {
__d_instantiate(found, inode);
- spin_unlock(&dcache_inode_lock);
+ spin_unlock(&inode->i_lock);
security_d_instantiate(found, inode);
return found;
}
@@ -1530,7 +1546,7 @@ struct dentry *d_add_ci(struct dentry *d
*/
new = list_entry(inode->i_dentry.next, struct dentry, d_alias);
dget_locked(new);
- spin_unlock(&dcache_inode_lock);
+ spin_unlock(&inode->i_lock);
security_d_instantiate(found, inode);
d_move(new, found);
iput(inode);
@@ -1740,15 +1756,17 @@ out:
void d_delete(struct dentry * dentry)
{
+ struct inode *inode;
int isdir = 0;
/*
* Are we the only user?
*/
again:
spin_lock(&dentry->d_lock);
- isdir = S_ISDIR(dentry->d_inode->i_mode);
+ inode = dentry->d_inode;
+ isdir = S_ISDIR(inode->i_mode);
if (dentry->d_count == 1) {
- if (!spin_trylock(&dcache_inode_lock)) {
+ if (inode && !spin_trylock(&inode->i_lock)) {
spin_unlock(&dentry->d_lock);
goto again;
}
@@ -1981,6 +1999,7 @@ static struct dentry *__d_unalias(struct
{
struct mutex *m1 = NULL, *m2 = NULL;
struct dentry *ret;
+ struct inode *inode;
/* If alias and dentry share a parent, then no extra locks required */
if (alias->d_parent == dentry->d_parent)
@@ -1996,14 +2015,15 @@ static struct dentry *__d_unalias(struct
if (!mutex_trylock(&dentry->d_sb->s_vfs_rename_mutex))
goto out_err;
m1 = &dentry->d_sb->s_vfs_rename_mutex;
- if (!mutex_trylock(&alias->d_parent->d_inode->i_mutex))
+ inode = alias->d_parent->d_inode;
+ if (!mutex_trylock(&inode->i_mutex))
goto out_err;
- m2 = &alias->d_parent->d_inode->i_mutex;
+ m2 = &inode->i_mutex;
out_unalias:
d_move_locked(alias, dentry);
ret = alias;
out_err:
- spin_unlock(&dcache_inode_lock);
+ spin_unlock(&inode->i_lock);
if (m2)
mutex_unlock(m2);
if (m1)
@@ -2074,7 +2094,7 @@ struct dentry *d_materialise_unique(stru
goto out_nolock;
}
- spin_lock(&dcache_inode_lock);
+ spin_lock(&inode->i_lock);
if (S_ISDIR(inode->i_mode)) {
struct dentry *alias;
@@ -2110,7 +2130,7 @@ struct dentry *d_materialise_unique(stru
found:
_d_rehash(actual);
spin_unlock(&actual->d_lock);
- spin_unlock(&dcache_inode_lock);
+ spin_unlock(&inode->i_lock);
out_nolock:
if (actual == dentry) {
security_d_instantiate(dentry, inode);
Index: linux-2.6/fs/sysfs/dir.c
===================================================================
--- linux-2.6.orig/fs/sysfs/dir.c
+++ linux-2.6/fs/sysfs/dir.c
@@ -547,7 +547,7 @@ static void sysfs_drop_dentry(struct sys
* dput to immediately free the dentry if it is not in use.
*/
repeat:
- spin_lock(&dcache_inode_lock);
+ spin_lock(&inode->i_lock);
list_for_each_entry(dentry, &inode->i_dentry, d_alias) {
spin_lock(&dentry->d_lock);
if (d_unhashed(dentry)) {
@@ -557,11 +557,11 @@ repeat:
dget_locked_dlock(dentry);
__d_drop(dentry);
spin_unlock(&dentry->d_lock);
- spin_unlock(&dcache_inode_lock);
+ spin_unlock(&inode->i_lock);
dput(dentry);
goto repeat;
}
- spin_unlock(&dcache_inode_lock);
+ spin_unlock(&inode->i_lock);
/* adjust nlink and update timestamp */
mutex_lock(&inode->i_mutex);
Index: linux-2.6/include/linux/dcache.h
===================================================================
--- linux-2.6.orig/include/linux/dcache.h
+++ linux-2.6/include/linux/dcache.h
@@ -186,7 +186,6 @@ d_iput: no no yes
#define DCACHE_FSNOTIFY_PARENT_WATCHED 0x0080 /* Parent inode is watched by some fsnotify listener */
-extern spinlock_t dcache_inode_lock;
extern seqlock_t rename_lock;
/**
Index: linux-2.6/fs/notify/inotify/inotify.c
===================================================================
--- linux-2.6.orig/fs/notify/inotify/inotify.c
+++ linux-2.6/fs/notify/inotify/inotify.c
@@ -185,7 +185,7 @@ static void set_dentry_child_flags(struc
{
struct dentry *alias;
- spin_lock(&dcache_inode_lock);
+ spin_lock(&inode->i_lock);
list_for_each_entry(alias, &inode->i_dentry, d_alias) {
struct dentry *child;
@@ -203,7 +203,7 @@ static void set_dentry_child_flags(struc
}
spin_unlock(&alias->d_lock);
}
- spin_unlock(&dcache_inode_lock);
+ spin_unlock(&inode->i_lock);
}
/*
Index: linux-2.6/fs/exportfs/expfs.c
===================================================================
--- linux-2.6.orig/fs/exportfs/expfs.c
+++ linux-2.6/fs/exportfs/expfs.c
@@ -43,24 +43,26 @@ find_acceptable_alias(struct dentry *res
void *context)
{
struct dentry *dentry, *toput = NULL;
+ struct inode *inode;
if (acceptable(context, result))
return result;
- spin_lock(&dcache_inode_lock);
- list_for_each_entry(dentry, &result->d_inode->i_dentry, d_alias) {
+ inode = result->d_inode;
+ spin_lock(&inode->i_lock);
+ list_for_each_entry(dentry, &inode->i_dentry, d_alias) {
dget_locked(dentry);
- spin_unlock(&dcache_inode_lock);
+ spin_unlock(&inode->i_lock);
if (toput)
dput(toput);
if (dentry != result && acceptable(context, dentry)) {
dput(result);
return dentry;
}
- spin_lock(&dcache_inode_lock);
+ spin_lock(&inode->i_lock);
toput = dentry;
}
- spin_unlock(&dcache_inode_lock);
+ spin_unlock(&inode->i_lock);
if (toput)
dput(toput);
Index: linux-2.6/fs/affs/amigaffs.c
===================================================================
--- linux-2.6.orig/fs/affs/amigaffs.c
+++ linux-2.6/fs/affs/amigaffs.c
@@ -128,7 +128,7 @@ affs_fix_dcache(struct dentry *dentry, u
void *data = dentry->d_fsdata;
struct list_head *head, *next;
- spin_lock(&dcache_inode_lock);
+ spin_lock(&inode->i_lock);
head = &inode->i_dentry;
next = head->next;
while (next != head) {
@@ -139,7 +139,7 @@ affs_fix_dcache(struct dentry *dentry, u
}
next = next->next;
}
- spin_unlock(&dcache_inode_lock);
+ spin_unlock(&inode->i_lock);
}
Index: linux-2.6/fs/nfs/getroot.c
===================================================================
--- linux-2.6.orig/fs/nfs/getroot.c
+++ linux-2.6/fs/nfs/getroot.c
@@ -64,11 +64,11 @@ static int nfs_superblock_set_dummy_root
* This again causes shrink_dcache_for_umount_subtree() to
* Oops, since the test for IS_ROOT() will fail.
*/
- spin_lock(&dcache_inode_lock);
+ spin_lock(&sb->s_root->d_inode->i_lock);
spin_lock(&sb->s_root->d_lock);
list_del_init(&sb->s_root->d_alias);
spin_unlock(&sb->s_root->d_lock);
- spin_unlock(&dcache_inode_lock);
+ spin_unlock(&sb->s_root->d_inode->i_lock);
}
return 0;
}
Index: linux-2.6/fs/ocfs2/dcache.c
===================================================================
--- linux-2.6.orig/fs/ocfs2/dcache.c
+++ linux-2.6/fs/ocfs2/dcache.c
@@ -140,7 +140,7 @@ struct dentry *ocfs2_find_local_alias(st
struct list_head *p;
struct dentry *dentry = NULL;
- spin_lock(&dcache_inode_lock);
+ spin_lock(&inode->i_lock);
list_for_each(p, &inode->i_dentry) {
dentry = list_entry(p, struct dentry, d_alias);
@@ -158,7 +158,7 @@ struct dentry *ocfs2_find_local_alias(st
dentry = NULL;
}
- spin_unlock(&dcache_inode_lock);
+ spin_unlock(&inode->i_lock);
return dentry;
}
Index: linux-2.6/fs/notify/fsnotify.c
===================================================================
--- linux-2.6.orig/fs/notify/fsnotify.c
+++ linux-2.6/fs/notify/fsnotify.c
@@ -52,7 +52,7 @@ void __fsnotify_update_child_dentry_flag
/* determine if the children should tell inode about their events */
watched = fsnotify_inode_watches_children(inode);
- spin_lock(&dcache_inode_lock);
+ spin_lock(&inode->i_lock);
/* run all of the dentries associated with this inode. Since this is a
* directory, there damn well better only be one item on this list */
list_for_each_entry(alias, &inode->i_dentry, d_alias) {
@@ -75,7 +75,7 @@ void __fsnotify_update_child_dentry_flag
}
spin_unlock(&alias->d_lock);
}
- spin_unlock(&dcache_inode_lock);
+ spin_unlock(&inode->i_lock);
}
/* Notify this dentry's parent about a child's events. */
^ permalink raw reply [flat|nested] 64+ messages in thread
* [patch 19/33] fs: icache lock s_inodes list
2009-09-04 6:51 [patch 00/33] my current vfs scalability patch queue npiggin
` (17 preceding siblings ...)
2009-09-04 6:51 ` [patch 18/33] fs: dcache per-inode inode alias locking npiggin
@ 2009-09-04 6:52 ` npiggin
2009-09-04 6:52 ` [patch 20/33] fs: icache lock inode hash npiggin
` (14 subsequent siblings)
33 siblings, 0 replies; 64+ messages in thread
From: npiggin @ 2009-09-04 6:52 UTC (permalink / raw)
To: linux-fsdevel; +Cc: linux-kernel
[-- Attachment #1: fs-inode_lock-scale.patch --]
[-- Type: text/plain, Size: 8176 bytes --]
Protect sb->s_inodes with a new lock, sb_inode_list_lock.
---
fs/drop_caches.c | 4 ++++
fs/fs-writeback.c | 4 ++++
fs/hugetlbfs/inode.c | 2 ++
fs/inode.c | 12 ++++++++++++
fs/notify/inode_mark.c | 2 ++
fs/notify/inotify/inotify.c | 2 ++
fs/quota/dquot.c | 6 ++++++
include/linux/writeback.h | 1 +
8 files changed, 33 insertions(+)
Index: linux-2.6/fs/drop_caches.c
===================================================================
--- linux-2.6.orig/fs/drop_caches.c
+++ linux-2.6/fs/drop_caches.c
@@ -17,18 +17,22 @@ static void drop_pagecache_sb(struct sup
struct inode *inode, *toput_inode = NULL;
spin_lock(&inode_lock);
+ spin_lock(&sb_inode_list_lock);
list_for_each_entry(inode, &sb->s_inodes, i_sb_list) {
if (inode->i_state & (I_FREEING|I_CLEAR|I_WILL_FREE|I_NEW))
continue;
if (inode->i_mapping->nrpages == 0)
continue;
__iget(inode);
+ spin_unlock(&sb_inode_list_lock);
spin_unlock(&inode_lock);
invalidate_mapping_pages(inode->i_mapping, 0, -1);
iput(toput_inode);
toput_inode = inode;
spin_lock(&inode_lock);
+ spin_lock(&sb_inode_list_lock);
}
+ spin_unlock(&sb_inode_list_lock);
spin_unlock(&inode_lock);
iput(toput_inode);
}
Index: linux-2.6/fs/fs-writeback.c
===================================================================
--- linux-2.6.orig/fs/fs-writeback.c
+++ linux-2.6/fs/fs-writeback.c
@@ -558,6 +558,7 @@ void generic_sync_sb_inodes(struct super
* In which case, the inode may not be on the dirty list, but
* we still have to wait for that writeout.
*/
+ spin_lock(&sb_inode_list_lock);
list_for_each_entry(inode, &sb->s_inodes, i_sb_list) {
struct address_space *mapping;
@@ -568,6 +569,7 @@ void generic_sync_sb_inodes(struct super
if (mapping->nrpages == 0)
continue;
__iget(inode);
+ spin_unlock(&sb_inode_list_lock);
spin_unlock(&inode_lock);
/*
* We hold a reference to 'inode' so it couldn't have
@@ -585,7 +587,9 @@ void generic_sync_sb_inodes(struct super
cond_resched();
spin_lock(&inode_lock);
+ spin_lock(&sb_inode_list_lock);
}
+ spin_unlock(&sb_inode_list_lock);
spin_unlock(&inode_lock);
iput(old_inode);
} else
Index: linux-2.6/fs/inode.c
===================================================================
--- linux-2.6.orig/fs/inode.c
+++ linux-2.6/fs/inode.c
@@ -85,6 +85,7 @@ static struct hlist_head *inode_hashtabl
* the i_state of an inode while it is in use..
*/
DEFINE_SPINLOCK(inode_lock);
+DEFINE_SPINLOCK(sb_inode_list_lock);
/*
* iprune_mutex provides exclusion between the kswapd or try_to_free_pages
@@ -355,7 +356,9 @@ static void dispose_list(struct list_hea
spin_lock(&inode_lock);
hlist_del_init(&inode->i_hash);
+ spin_lock(&sb_inode_list_lock);
list_del_init(&inode->i_sb_list);
+ spin_unlock(&sb_inode_list_lock);
spin_unlock(&inode_lock);
wake_up_inode(inode);
@@ -387,6 +390,7 @@ static int invalidate_list(struct list_h
* shrink_icache_memory() away.
*/
cond_resched_lock(&inode_lock);
+ cond_resched_lock(&sb_inode_list_lock);
next = next->next;
if (tmp == head)
@@ -424,9 +428,11 @@ int invalidate_inodes(struct super_block
mutex_lock(&iprune_mutex);
spin_lock(&inode_lock);
+ spin_lock(&sb_inode_list_lock);
inotify_unmount_inodes(&sb->s_inodes);
fsnotify_unmount_inodes(&sb->s_inodes);
busy = invalidate_list(&sb->s_inodes, &throw_away);
+ spin_unlock(&sb_inode_list_lock);
spin_unlock(&inode_lock);
dispose_list(&throw_away);
@@ -614,7 +620,9 @@ __inode_add_to_lists(struct super_block
{
inodes_stat.nr_inodes++;
list_add(&inode->i_list, &inode_in_use);
+ spin_lock(&sb_inode_list_lock);
list_add(&inode->i_sb_list, &sb->s_inodes);
+ spin_unlock(&sb_inode_list_lock);
if (head)
hlist_add_head(&inode->i_hash, head);
}
@@ -1206,7 +1214,9 @@ void generic_delete_inode(struct inode *
const struct super_operations *op = inode->i_sb->s_op;
list_del_init(&inode->i_list);
+ spin_lock(&sb_inode_list_lock);
list_del_init(&inode->i_sb_list);
+ spin_unlock(&sb_inode_list_lock);
WARN_ON(inode->i_state & I_NEW);
inode->i_state |= I_FREEING;
inodes_stat.nr_inodes--;
@@ -1259,7 +1269,9 @@ static void generic_forget_inode(struct
hlist_del_init(&inode->i_hash);
}
list_del_init(&inode->i_list);
+ spin_lock(&sb_inode_list_lock);
list_del_init(&inode->i_sb_list);
+ spin_unlock(&sb_inode_list_lock);
WARN_ON(inode->i_state & I_NEW);
inode->i_state |= I_FREEING;
inodes_stat.nr_inodes--;
Index: linux-2.6/fs/notify/inotify/inotify.c
===================================================================
--- linux-2.6.orig/fs/notify/inotify/inotify.c
+++ linux-2.6/fs/notify/inotify/inotify.c
@@ -431,6 +431,7 @@ void inotify_unmount_inodes(struct list_
* will be added since the umount has begun. Finally,
* iprune_mutex keeps shrink_icache_memory() away.
*/
+ spin_unlock(&sb_inode_list_lock);
spin_unlock(&inode_lock);
if (need_iput_tmp)
@@ -453,6 +454,7 @@ void inotify_unmount_inodes(struct list_
iput(inode);
spin_lock(&inode_lock);
+ spin_lock(&sb_inode_list_lock);
}
}
EXPORT_SYMBOL_GPL(inotify_unmount_inodes);
Index: linux-2.6/fs/quota/dquot.c
===================================================================
--- linux-2.6.orig/fs/quota/dquot.c
+++ linux-2.6/fs/quota/dquot.c
@@ -822,6 +822,7 @@ static void add_dquot_ref(struct super_b
struct inode *inode, *old_inode = NULL;
spin_lock(&inode_lock);
+ spin_lock(&sb_inode_list_lock);
list_for_each_entry(inode, &sb->s_inodes, i_sb_list) {
if (inode->i_state & (I_FREEING|I_CLEAR|I_WILL_FREE|I_NEW))
continue;
@@ -831,6 +832,7 @@ static void add_dquot_ref(struct super_b
continue;
__iget(inode);
+ spin_unlock(&sb_inode_list_lock);
spin_unlock(&inode_lock);
iput(old_inode);
@@ -842,7 +844,9 @@ static void add_dquot_ref(struct super_b
* keep the reference and iput it later. */
old_inode = inode;
spin_lock(&inode_lock);
+ spin_lock(&sb_inode_list_lock);
}
+ spin_unlock(&sb_inode_list_lock);
spin_unlock(&inode_lock);
iput(old_inode);
}
@@ -914,6 +918,7 @@ static void remove_dquot_ref(struct supe
struct inode *inode;
spin_lock(&inode_lock);
+ spin_lock(&sb_inode_list_lock);
list_for_each_entry(inode, &sb->s_inodes, i_sb_list) {
/*
* We have to scan also I_NEW inodes because they can already
@@ -924,6 +929,7 @@ static void remove_dquot_ref(struct supe
if (!IS_NOQUOTA(inode))
remove_inode_dquot_ref(inode, type, tofree_head);
}
+ spin_unlock(&sb_inode_list_lock);
spin_unlock(&inode_lock);
}
Index: linux-2.6/include/linux/writeback.h
===================================================================
--- linux-2.6.orig/include/linux/writeback.h
+++ linux-2.6/include/linux/writeback.h
@@ -10,6 +10,7 @@
struct backing_dev_info;
extern spinlock_t inode_lock;
+extern spinlock_t sb_inode_list_lock;
extern struct list_head inode_in_use;
extern struct list_head inode_unused;
Index: linux-2.6/fs/hugetlbfs/inode.c
===================================================================
--- linux-2.6.orig/fs/hugetlbfs/inode.c
+++ linux-2.6/fs/hugetlbfs/inode.c
@@ -404,7 +404,9 @@ static void hugetlbfs_forget_inode(struc
hlist_del_init(&inode->i_hash);
}
list_del_init(&inode->i_list);
+ spin_lock(&sb_inode_list_lock);
list_del_init(&inode->i_sb_list);
+ spin_unlock(&sb_inode_list_lock);
inode->i_state |= I_FREEING;
inodes_stat.nr_inodes--;
spin_unlock(&inode_lock);
Index: linux-2.6/fs/notify/inode_mark.c
===================================================================
--- linux-2.6.orig/fs/notify/inode_mark.c
+++ linux-2.6/fs/notify/inode_mark.c
@@ -409,6 +409,7 @@ void fsnotify_unmount_inodes(struct list
* will be added since the umount has begun. Finally,
* iprune_mutex keeps shrink_icache_memory() away.
*/
+ spin_unlock(&sb_inode_list_lock);
spin_unlock(&inode_lock);
if (need_iput_tmp)
@@ -422,5 +423,6 @@ void fsnotify_unmount_inodes(struct list
iput(inode);
spin_lock(&inode_lock);
+ spin_lock(&sb_inode_list_lock);
}
}
^ permalink raw reply [flat|nested] 64+ messages in thread
* [patch 20/33] fs: icache lock inode hash
2009-09-04 6:51 [patch 00/33] my current vfs scalability patch queue npiggin
` (18 preceding siblings ...)
2009-09-04 6:52 ` [patch 19/33] fs: icache lock s_inodes list npiggin
@ 2009-09-04 6:52 ` npiggin
2009-09-04 6:52 ` [patch 21/33] fs: icache lock i_state npiggin
` (13 subsequent siblings)
33 siblings, 0 replies; 64+ messages in thread
From: npiggin @ 2009-09-04 6:52 UTC (permalink / raw)
To: linux-fsdevel; +Cc: linux-kernel
[-- Attachment #1: fs-inode_lock-scale-2.patch --]
[-- Type: text/plain, Size: 5693 bytes --]
Add a new lock, inode_hash_lock, to protect the inode hash table lists.
---
fs/hugetlbfs/inode.c | 2 ++
fs/inode.c | 29 ++++++++++++++++++++++++++++-
include/linux/writeback.h | 1 +
3 files changed, 31 insertions(+), 1 deletion(-)
Index: linux-2.6/fs/hugetlbfs/inode.c
===================================================================
--- linux-2.6.orig/fs/hugetlbfs/inode.c
+++ linux-2.6/fs/hugetlbfs/inode.c
@@ -401,7 +401,9 @@ static void hugetlbfs_forget_inode(struc
spin_lock(&inode_lock);
inode->i_state &= ~I_WILL_FREE;
inodes_stat.nr_unused--;
+ spin_lock(&inode_hash_lock);
hlist_del_init(&inode->i_hash);
+ spin_unlock(&inode_hash_lock);
}
list_del_init(&inode->i_list);
spin_lock(&sb_inode_list_lock);
Index: linux-2.6/fs/inode.c
===================================================================
--- linux-2.6.orig/fs/inode.c
+++ linux-2.6/fs/inode.c
@@ -86,6 +86,7 @@ static struct hlist_head *inode_hashtabl
*/
DEFINE_SPINLOCK(inode_lock);
DEFINE_SPINLOCK(sb_inode_list_lock);
+DEFINE_SPINLOCK(inode_hash_lock);
/*
* iprune_mutex provides exclusion between the kswapd or try_to_free_pages
@@ -355,7 +356,9 @@ static void dispose_list(struct list_hea
clear_inode(inode);
spin_lock(&inode_lock);
+ spin_lock(&inode_hash_lock);
hlist_del_init(&inode->i_hash);
+ spin_unlock(&inode_hash_lock);
spin_lock(&sb_inode_list_lock);
list_del_init(&inode->i_sb_list);
spin_unlock(&sb_inode_list_lock);
@@ -565,17 +568,20 @@ static struct inode *find_inode(struct s
struct inode *inode = NULL;
repeat:
+ spin_lock(&inode_hash_lock);
hlist_for_each_entry(inode, node, head, i_hash) {
if (inode->i_sb != sb)
continue;
if (!test(inode, data))
continue;
if (inode->i_state & (I_FREEING|I_CLEAR|I_WILL_FREE)) {
+ spin_unlock(&inode_hash_lock);
__wait_on_freeing_inode(inode);
goto repeat;
}
break;
}
+ spin_unlock(&inode_hash_lock);
return node ? inode : NULL;
}
@@ -590,17 +596,20 @@ static struct inode *find_inode_fast(str
struct inode *inode = NULL;
repeat:
+ spin_lock(&inode_hash_lock);
hlist_for_each_entry(inode, node, head, i_hash) {
if (inode->i_ino != ino)
continue;
if (inode->i_sb != sb)
continue;
if (inode->i_state & (I_FREEING|I_CLEAR|I_WILL_FREE)) {
+ spin_unlock(&inode_hash_lock);
__wait_on_freeing_inode(inode);
goto repeat;
}
break;
}
+ spin_unlock(&inode_hash_lock);
return node ? inode : NULL;
}
@@ -623,8 +632,11 @@ __inode_add_to_lists(struct super_block
spin_lock(&sb_inode_list_lock);
list_add(&inode->i_sb_list, &sb->s_inodes);
spin_unlock(&sb_inode_list_lock);
- if (head)
+ if (head) {
+ spin_lock(&inode_hash_lock);
hlist_add_head(&inode->i_hash, head);
+ spin_unlock(&inode_hash_lock);
+ }
}
/**
@@ -1100,7 +1112,9 @@ int insert_inode_locked(struct inode *in
while (1) {
struct hlist_node *node;
struct inode *old = NULL;
+
spin_lock(&inode_lock);
+ spin_lock(&inode_hash_lock);
hlist_for_each_entry(old, node, head, i_hash) {
if (old->i_ino != ino)
continue;
@@ -1112,9 +1126,11 @@ int insert_inode_locked(struct inode *in
}
if (likely(!node)) {
hlist_add_head(&inode->i_hash, head);
+ spin_unlock(&inode_hash_lock);
spin_unlock(&inode_lock);
return 0;
}
+ spin_unlock(&inode_hash_lock);
__iget(old);
spin_unlock(&inode_lock);
wait_on_inode(old);
@@ -1140,6 +1156,7 @@ int insert_inode_locked4(struct inode *i
struct inode *old = NULL;
spin_lock(&inode_lock);
+ spin_lock(&inode_hash_lock);
hlist_for_each_entry(old, node, head, i_hash) {
if (old->i_sb != sb)
continue;
@@ -1151,9 +1168,11 @@ int insert_inode_locked4(struct inode *i
}
if (likely(!node)) {
hlist_add_head(&inode->i_hash, head);
+ spin_unlock(&inode_hash_lock);
spin_unlock(&inode_lock);
return 0;
}
+ spin_unlock(&inode_hash_lock);
__iget(old);
spin_unlock(&inode_lock);
wait_on_inode(old);
@@ -1178,7 +1197,9 @@ void __insert_inode_hash(struct inode *i
{
struct hlist_head *head = inode_hashtable + hash(inode->i_sb, hashval);
spin_lock(&inode_lock);
+ spin_lock(&inode_hash_lock);
hlist_add_head(&inode->i_hash, head);
+ spin_unlock(&inode_hash_lock);
spin_unlock(&inode_lock);
}
EXPORT_SYMBOL(__insert_inode_hash);
@@ -1192,7 +1213,9 @@ EXPORT_SYMBOL(__insert_inode_hash);
void remove_inode_hash(struct inode *inode)
{
spin_lock(&inode_lock);
+ spin_lock(&inode_hash_lock);
hlist_del_init(&inode->i_hash);
+ spin_unlock(&inode_hash_lock);
spin_unlock(&inode_lock);
}
EXPORT_SYMBOL(remove_inode_hash);
@@ -1238,7 +1261,9 @@ void generic_delete_inode(struct inode *
clear_inode(inode);
}
spin_lock(&inode_lock);
+ spin_lock(&inode_hash_lock);
hlist_del_init(&inode->i_hash);
+ spin_unlock(&inode_hash_lock);
spin_unlock(&inode_lock);
wake_up_inode(inode);
BUG_ON(inode->i_state != I_CLEAR);
@@ -1266,7 +1291,9 @@ static void generic_forget_inode(struct
WARN_ON(inode->i_state & I_NEW);
inode->i_state &= ~I_WILL_FREE;
inodes_stat.nr_unused--;
+ spin_lock(&inode_hash_lock);
hlist_del_init(&inode->i_hash);
+ spin_unlock(&inode_hash_lock);
}
list_del_init(&inode->i_list);
spin_lock(&sb_inode_list_lock);
Index: linux-2.6/include/linux/writeback.h
===================================================================
--- linux-2.6.orig/include/linux/writeback.h
+++ linux-2.6/include/linux/writeback.h
@@ -11,6 +11,7 @@ struct backing_dev_info;
extern spinlock_t inode_lock;
extern spinlock_t sb_inode_list_lock;
+extern spinlock_t inode_hash_lock;
extern struct list_head inode_in_use;
extern struct list_head inode_unused;
^ permalink raw reply [flat|nested] 64+ messages in thread
* [patch 21/33] fs: icache lock i_state
2009-09-04 6:51 [patch 00/33] my current vfs scalability patch queue npiggin
` (19 preceding siblings ...)
2009-09-04 6:52 ` [patch 20/33] fs: icache lock inode hash npiggin
@ 2009-09-04 6:52 ` npiggin
2009-09-04 6:52 ` [patch 22/33] fs: icache lock i_count npiggin
` (12 subsequent siblings)
33 siblings, 0 replies; 64+ messages in thread
From: npiggin @ 2009-09-04 6:52 UTC (permalink / raw)
To: linux-fsdevel; +Cc: linux-kernel
[-- Attachment #1: fs-inode_lock-scale-3.patch --]
[-- Type: text/plain, Size: 18056 bytes --]
Protect i_state updates with i_lock
---
fs/drop_caches.c | 9 +++--
fs/fs-writeback.c | 45 ++++++++++++++++++++--------
fs/hugetlbfs/inode.c | 6 +++
fs/inode.c | 81 ++++++++++++++++++++++++++++++++++++++++++++-------
fs/nilfs2/gcdat.c | 1
fs/quota/dquot.c | 14 ++++++--
6 files changed, 127 insertions(+), 29 deletions(-)
Index: linux-2.6/fs/drop_caches.c
===================================================================
--- linux-2.6.orig/fs/drop_caches.c
+++ linux-2.6/fs/drop_caches.c
@@ -19,11 +19,14 @@ static void drop_pagecache_sb(struct sup
spin_lock(&inode_lock);
spin_lock(&sb_inode_list_lock);
list_for_each_entry(inode, &sb->s_inodes, i_sb_list) {
- if (inode->i_state & (I_FREEING|I_CLEAR|I_WILL_FREE|I_NEW))
- continue;
- if (inode->i_mapping->nrpages == 0)
+ spin_lock(&inode->i_lock);
+ if (inode->i_state & (I_FREEING|I_CLEAR|I_WILL_FREE|I_NEW)
+ || inode->i_mapping->nrpages == 0) {
+ spin_unlock(&inode->i_lock);
continue;
+ }
__iget(inode);
+ spin_unlock(&inode->i_lock);
spin_unlock(&sb_inode_list_lock);
spin_unlock(&inode_lock);
invalidate_mapping_pages(inode->i_mapping, 0, -1);
Index: linux-2.6/fs/fs-writeback.c
===================================================================
--- linux-2.6.orig/fs/fs-writeback.c
+++ linux-2.6/fs/fs-writeback.c
@@ -140,6 +140,7 @@ void __mark_inode_dirty(struct inode *in
block_dump___mark_inode_dirty(inode);
spin_lock(&inode_lock);
+ spin_lock(&inode->i_lock);
if ((inode->i_state & flags) != flags) {
const int was_dirty = inode->i_state & I_DIRTY;
@@ -174,6 +175,7 @@ void __mark_inode_dirty(struct inode *in
}
}
out:
+ spin_unlock(&inode->i_lock);
spin_unlock(&inode_lock);
}
@@ -287,9 +289,11 @@ static void inode_wait_for_writeback(str
wqh = bit_waitqueue(&inode->i_state, __I_SYNC);
do {
+ spin_unlock(&inode->i_lock);
spin_unlock(&inode_lock);
__wait_on_bit(wqh, &wq, inode_wait, TASK_UNINTERRUPTIBLE);
spin_lock(&inode_lock);
+ spin_lock(&inode->i_lock);
} while (inode->i_state & I_SYNC);
}
@@ -346,6 +350,7 @@ writeback_single_inode(struct inode *ino
inode->i_state |= I_SYNC;
inode->i_state &= ~I_DIRTY;
+ spin_unlock(&inode->i_lock);
spin_unlock(&inode_lock);
ret = do_writepages(mapping, wbc);
@@ -364,6 +369,7 @@ writeback_single_inode(struct inode *ino
}
spin_lock(&inode_lock);
+ spin_lock(&inode->i_lock);
inode->i_state &= ~I_SYNC;
if (!(inode->i_state & (I_FREEING | I_CLEAR))) {
if (!(inode->i_state & I_DIRTY) &&
@@ -492,11 +498,6 @@ void generic_sync_sb_inodes(struct super
break;
}
- if (inode->i_state & (I_NEW | I_WILL_FREE)) {
- requeue_io(inode);
- continue;
- }
-
if (wbc->nonblocking && bdi_write_congested(bdi)) {
wbc->encountered_congestion = 1;
if (!sb_is_blkdev_sb(sb))
@@ -512,16 +513,27 @@ void generic_sync_sb_inodes(struct super
continue; /* blockdev has wrong queue */
}
+ spin_lock(&inode->i_lock);
+ if (inode->i_state & (I_NEW | I_WILL_FREE)) {
+ spin_unlock(&inode->i_lock);
+ requeue_io(inode);
+ continue;
+ }
+
/*
* Was this inode dirtied after sync_sb_inodes was called?
* This keeps sync from extra jobs and livelock.
*/
- if (inode_dirtied_after(inode, start))
+ if (inode_dirtied_after(inode, start)) {
+ spin_unlock(&inode->i_lock);
break;
+ }
/* Is another pdflush already flushing this queue? */
- if (current_is_pdflush() && !writeback_acquire(bdi))
+ if (current_is_pdflush() && !writeback_acquire(bdi)) {
+ spin_unlock(&inode->i_lock);
break;
+ }
BUG_ON(inode->i_state & (I_FREEING | I_CLEAR));
__iget(inode);
@@ -536,6 +548,7 @@ void generic_sync_sb_inodes(struct super
*/
redirty_tail(inode);
}
+ spin_unlock(&inode->i_lock);
spin_unlock(&inode_lock);
iput(inode);
cond_resched();
@@ -560,15 +573,17 @@ void generic_sync_sb_inodes(struct super
*/
spin_lock(&sb_inode_list_lock);
list_for_each_entry(inode, &sb->s_inodes, i_sb_list) {
- struct address_space *mapping;
+ struct address_space *mapping = inode->i_mapping;
+ spin_lock(&inode->i_lock);
if (inode->i_state &
- (I_FREEING|I_CLEAR|I_WILL_FREE|I_NEW))
- continue;
- mapping = inode->i_mapping;
- if (mapping->nrpages == 0)
+ (I_FREEING|I_CLEAR|I_WILL_FREE|I_NEW)
+ || mapping->nrpages == 0) {
+ spin_unlock(&inode->i_lock);
continue;
+ }
__iget(inode);
+ spin_unlock(&inode->i_lock);
spin_unlock(&sb_inode_list_lock);
spin_unlock(&inode_lock);
/*
@@ -712,7 +727,9 @@ int write_inode_now(struct inode *inode,
might_sleep();
spin_lock(&inode_lock);
+ spin_lock(&inode->i_lock);
ret = writeback_single_inode(inode, &wbc);
+ spin_unlock(&inode->i_lock);
spin_unlock(&inode_lock);
if (sync)
inode_sync_wait(inode);
@@ -736,7 +753,9 @@ int sync_inode(struct inode *inode, stru
int ret;
spin_lock(&inode_lock);
+ spin_lock(&inode->i_lock);
ret = writeback_single_inode(inode, wbc);
+ spin_unlock(&inode->i_lock);
spin_unlock(&inode_lock);
return ret;
}
@@ -779,9 +798,11 @@ int generic_osync_inode(struct inode *in
}
spin_lock(&inode_lock);
+ spin_lock(&inode->i_lock);
if ((inode->i_state & I_DIRTY) &&
((what & OSYNC_INODE) || (inode->i_state & I_DIRTY_DATASYNC)))
need_write_inode_now = 1;
+ spin_unlock(&inode->i_lock);
spin_unlock(&inode_lock);
if (need_write_inode_now) {
Index: linux-2.6/fs/inode.c
===================================================================
--- linux-2.6.orig/fs/inode.c
+++ linux-2.6/fs/inode.c
@@ -296,6 +296,7 @@ static void init_once(void *foo)
*/
void __iget(struct inode *inode)
{
+ assert_spin_locked(&inode->i_lock);
if (atomic_read(&inode->i_count)) {
atomic_inc(&inode->i_count);
return;
@@ -399,16 +400,21 @@ static int invalidate_list(struct list_h
if (tmp == head)
break;
inode = list_entry(tmp, struct inode, i_sb_list);
- if (inode->i_state & I_NEW)
+ spin_lock(&inode->i_lock);
+ if (inode->i_state & I_NEW) {
+ spin_unlock(&inode->i_lock);
continue;
+ }
invalidate_inode_buffers(inode);
if (!atomic_read(&inode->i_count)) {
list_move(&inode->i_list, dispose);
WARN_ON(inode->i_state & I_NEW);
inode->i_state |= I_FREEING;
+ spin_unlock(&inode->i_lock);
count++;
continue;
}
+ spin_unlock(&inode->i_lock);
busy = 1;
}
/* only unused inodes may be cached with i_count zero */
@@ -488,12 +494,15 @@ static void prune_icache(int nr_to_scan)
inode = list_entry(inode_unused.prev, struct inode, i_list);
+ spin_lock(&inode->i_lock);
if (inode->i_state || atomic_read(&inode->i_count)) {
list_move(&inode->i_list, &inode_unused);
+ spin_unlock(&inode->i_lock);
continue;
}
if (inode_has_buffers(inode) || inode->i_data.nrpages) {
__iget(inode);
+ spin_unlock(&inode->i_lock);
spin_unlock(&inode_lock);
if (remove_inode_buffers(inode))
reap += invalidate_mapping_pages(&inode->i_data,
@@ -504,12 +513,16 @@ static void prune_icache(int nr_to_scan)
if (inode != list_entry(inode_unused.next,
struct inode, i_list))
continue; /* wrong inode or list_empty */
- if (!can_unuse(inode))
+ spin_lock(&inode->i_lock);
+ if (!can_unuse(inode)) {
+ spin_unlock(&inode->i_lock);
continue;
+ }
}
list_move(&inode->i_list, &freeable);
WARN_ON(inode->i_state & I_NEW);
inode->i_state |= I_FREEING;
+ spin_unlock(&inode->i_lock);
nr_pruned++;
}
inodes_stat.nr_unused -= nr_pruned;
@@ -572,8 +585,14 @@ repeat:
hlist_for_each_entry(inode, node, head, i_hash) {
if (inode->i_sb != sb)
continue;
- if (!test(inode, data))
+ if (!spin_trylock(&inode->i_lock)) {
+ spin_unlock(&inode_hash_lock);
+ goto repeat;
+ }
+ if (!test(inode, data)) {
+ spin_unlock(&inode->i_lock);
continue;
+ }
if (inode->i_state & (I_FREEING|I_CLEAR|I_WILL_FREE)) {
spin_unlock(&inode_hash_lock);
__wait_on_freeing_inode(inode);
@@ -602,6 +621,10 @@ repeat:
continue;
if (inode->i_sb != sb)
continue;
+ if (!spin_trylock(&inode->i_lock)) {
+ spin_unlock(&inode_hash_lock);
+ goto repeat;
+ }
if (inode->i_state & (I_FREEING|I_CLEAR|I_WILL_FREE)) {
spin_unlock(&inode_hash_lock);
__wait_on_freeing_inode(inode);
@@ -628,10 +651,10 @@ __inode_add_to_lists(struct super_block
struct inode *inode)
{
inodes_stat.nr_inodes++;
- list_add(&inode->i_list, &inode_in_use);
spin_lock(&sb_inode_list_lock);
list_add(&inode->i_sb_list, &sb->s_inodes);
spin_unlock(&sb_inode_list_lock);
+ list_add(&inode->i_list, &inode_in_use);
if (head) {
spin_lock(&inode_hash_lock);
hlist_add_head(&inode->i_hash, head);
@@ -688,9 +711,9 @@ struct inode *new_inode(struct super_blo
inode = alloc_inode(sb);
if (inode) {
spin_lock(&inode_lock);
- __inode_add_to_lists(sb, NULL, inode);
inode->i_ino = ++last_ino;
inode->i_state = 0;
+ __inode_add_to_lists(sb, NULL, inode);
spin_unlock(&inode_lock);
}
return inode;
@@ -755,8 +778,8 @@ static struct inode *get_new_inode(struc
if (set(inode, data))
goto set_failed;
- __inode_add_to_lists(sb, head, inode);
inode->i_state = I_LOCK|I_NEW;
+ __inode_add_to_lists(sb, head, inode);
spin_unlock(&inode_lock);
/* Return the locked inode with I_NEW set, the
@@ -771,6 +794,7 @@ static struct inode *get_new_inode(struc
* allocated.
*/
__iget(old);
+ spin_unlock(&old->i_lock);
spin_unlock(&inode_lock);
destroy_inode(inode);
inode = old;
@@ -779,6 +803,7 @@ static struct inode *get_new_inode(struc
return inode;
set_failed:
+ spin_unlock(&inode->i_lock);
spin_unlock(&inode_lock);
destroy_inode(inode);
return NULL;
@@ -802,8 +827,8 @@ static struct inode *get_new_inode_fast(
old = find_inode_fast(sb, head, ino);
if (!old) {
inode->i_ino = ino;
- __inode_add_to_lists(sb, head, inode);
inode->i_state = I_LOCK|I_NEW;
+ __inode_add_to_lists(sb, head, inode);
spin_unlock(&inode_lock);
/* Return the locked inode with I_NEW set, the
@@ -818,6 +843,7 @@ static struct inode *get_new_inode_fast(
* allocated.
*/
__iget(old);
+ spin_unlock(&old->i_lock);
spin_unlock(&inode_lock);
destroy_inode(inode);
inode = old;
@@ -859,6 +885,7 @@ ino_t iunique(struct super_block *sb, in
res = counter++;
head = inode_hashtable + hash(sb, res);
inode = find_inode_fast(sb, head, res);
+ spin_unlock(&inode->i_lock);
} while (inode != NULL);
spin_unlock(&inode_lock);
@@ -868,7 +895,10 @@ EXPORT_SYMBOL(iunique);
struct inode *igrab(struct inode *inode)
{
+ struct inode *ret = inode;
+
spin_lock(&inode_lock);
+ spin_lock(&inode->i_lock);
if (!(inode->i_state & (I_FREEING|I_CLEAR|I_WILL_FREE)))
__iget(inode);
else
@@ -877,9 +907,11 @@ struct inode *igrab(struct inode *inode)
* called yet, and somebody is calling igrab
* while the inode is getting freed.
*/
- inode = NULL;
+ ret = NULL;
+ spin_unlock(&inode->i_lock);
spin_unlock(&inode_lock);
- return inode;
+
+ return ret;
}
EXPORT_SYMBOL(igrab);
@@ -912,6 +944,7 @@ static struct inode *ifind(struct super_
inode = find_inode(sb, head, test, data);
if (inode) {
__iget(inode);
+ spin_unlock(&inode->i_lock);
spin_unlock(&inode_lock);
if (likely(wait))
wait_on_inode(inode);
@@ -945,6 +978,7 @@ static struct inode *ifind_fast(struct s
inode = find_inode_fast(sb, head, ino);
if (inode) {
__iget(inode);
+ spin_unlock(&inode->i_lock);
spin_unlock(&inode_lock);
wait_on_inode(inode);
return inode;
@@ -1114,6 +1148,7 @@ int insert_inode_locked(struct inode *in
struct inode *old = NULL;
spin_lock(&inode_lock);
+repeat:
spin_lock(&inode_hash_lock);
hlist_for_each_entry(old, node, head, i_hash) {
if (old->i_ino != ino)
@@ -1122,6 +1157,10 @@ int insert_inode_locked(struct inode *in
continue;
if (old->i_state & (I_FREEING|I_CLEAR|I_WILL_FREE))
continue;
+ if (!spin_trylock(&old->i_lock)) {
+ spin_unlock(&inode_hash_lock);
+ goto repeat;
+ }
break;
}
if (likely(!node)) {
@@ -1132,6 +1171,7 @@ int insert_inode_locked(struct inode *in
}
spin_unlock(&inode_hash_lock);
__iget(old);
+ spin_unlock(&old->i_lock);
spin_unlock(&inode_lock);
wait_on_inode(old);
if (unlikely(!hlist_unhashed(&old->i_hash))) {
@@ -1156,6 +1196,7 @@ int insert_inode_locked4(struct inode *i
struct inode *old = NULL;
spin_lock(&inode_lock);
+repeat:
spin_lock(&inode_hash_lock);
hlist_for_each_entry(old, node, head, i_hash) {
if (old->i_sb != sb)
@@ -1164,6 +1205,10 @@ int insert_inode_locked4(struct inode *i
continue;
if (old->i_state & (I_FREEING|I_CLEAR|I_WILL_FREE))
continue;
+ if (!spin_trylock(&old->i_lock)) {
+ spin_unlock(&inode_hash_lock);
+ goto repeat;
+ }
break;
}
if (likely(!node)) {
@@ -1174,6 +1219,7 @@ int insert_inode_locked4(struct inode *i
}
spin_unlock(&inode_hash_lock);
__iget(old);
+ spin_unlock(&old->i_lock);
spin_unlock(&inode_lock);
wait_on_inode(old);
if (unlikely(!hlist_unhashed(&old->i_hash))) {
@@ -1236,12 +1282,14 @@ void generic_delete_inode(struct inode *
{
const struct super_operations *op = inode->i_sb->s_op;
- list_del_init(&inode->i_list);
spin_lock(&sb_inode_list_lock);
+ spin_lock(&inode->i_lock);
+ list_del_init(&inode->i_list);
list_del_init(&inode->i_sb_list);
spin_unlock(&sb_inode_list_lock);
WARN_ON(inode->i_state & I_NEW);
inode->i_state |= I_FREEING;
+ spin_unlock(&inode->i_lock);
inodes_stat.nr_inodes--;
spin_unlock(&inode_lock);
@@ -1275,19 +1323,27 @@ static void generic_forget_inode(struct
{
struct super_block *sb = inode->i_sb;
+ spin_lock(&sb_inode_list_lock);
+ spin_lock(&inode->i_lock);
if (!hlist_unhashed(&inode->i_hash)) {
if (!(inode->i_state & (I_DIRTY|I_SYNC)))
list_move(&inode->i_list, &inode_unused);
inodes_stat.nr_unused++;
if (sb->s_flags & MS_ACTIVE) {
+ spin_unlock(&inode->i_lock);
+ spin_unlock(&sb_inode_list_lock);
spin_unlock(&inode_lock);
return;
}
WARN_ON(inode->i_state & I_NEW);
inode->i_state |= I_WILL_FREE;
+ spin_unlock(&inode->i_lock);
+ spin_unlock(&sb_inode_list_lock);
spin_unlock(&inode_lock);
write_inode_now(inode, 1);
spin_lock(&inode_lock);
+ spin_lock(&sb_inode_list_lock);
+ spin_lock(&inode->i_lock);
WARN_ON(inode->i_state & I_NEW);
inode->i_state &= ~I_WILL_FREE;
inodes_stat.nr_unused--;
@@ -1296,12 +1352,12 @@ static void generic_forget_inode(struct
spin_unlock(&inode_hash_lock);
}
list_del_init(&inode->i_list);
- spin_lock(&sb_inode_list_lock);
list_del_init(&inode->i_sb_list);
spin_unlock(&sb_inode_list_lock);
WARN_ON(inode->i_state & I_NEW);
inode->i_state |= I_FREEING;
inodes_stat.nr_inodes--;
+ spin_unlock(&inode->i_lock);
spin_unlock(&inode_lock);
if (inode->i_data.nrpages)
truncate_inode_pages(&inode->i_data, 0);
@@ -1538,6 +1594,8 @@ EXPORT_SYMBOL(inode_wait);
* wake_up_inode() after removing from the hash list will DTRT.
*
* This is called with inode_lock held.
+ *
+ * Called with i_lock held and returns with it dropped.
*/
static void __wait_on_freeing_inode(struct inode *inode)
{
@@ -1545,6 +1603,7 @@ static void __wait_on_freeing_inode(stru
DEFINE_WAIT_BIT(wait, &inode->i_state, __I_LOCK);
wq = bit_waitqueue(&inode->i_state, __I_LOCK);
prepare_to_wait(wq, &wait.wait, TASK_UNINTERRUPTIBLE);
+ spin_unlock(&inode->i_lock);
spin_unlock(&inode_lock);
schedule();
finish_wait(wq, &wait.wait);
Index: linux-2.6/fs/hugetlbfs/inode.c
===================================================================
--- linux-2.6.orig/fs/hugetlbfs/inode.c
+++ linux-2.6/fs/hugetlbfs/inode.c
@@ -391,7 +391,9 @@ static void hugetlbfs_forget_inode(struc
spin_unlock(&inode_lock);
return;
}
+ spin_lock(&inode->i_lock);
inode->i_state |= I_WILL_FREE;
+ spin_unlock(&inode->i_lock);
spin_unlock(&inode_lock);
/*
* write_inode_now is a noop as we set BDI_CAP_NO_WRITEBACK
@@ -399,7 +401,9 @@ static void hugetlbfs_forget_inode(struc
*/
write_inode_now(inode, 1);
spin_lock(&inode_lock);
+ spin_lock(&inode->i_lock);
inode->i_state &= ~I_WILL_FREE;
+ spin_unlock(&inode->i_lock);
inodes_stat.nr_unused--;
spin_lock(&inode_hash_lock);
hlist_del_init(&inode->i_hash);
@@ -409,7 +413,9 @@ static void hugetlbfs_forget_inode(struc
spin_lock(&sb_inode_list_lock);
list_del_init(&inode->i_sb_list);
spin_unlock(&sb_inode_list_lock);
+ spin_lock(&inode->i_lock);
inode->i_state |= I_FREEING;
+ spin_unlock(&inode->i_lock);
inodes_stat.nr_inodes--;
spin_unlock(&inode_lock);
truncate_hugepages(inode, 0);
Index: linux-2.6/fs/nilfs2/gcdat.c
===================================================================
--- linux-2.6.orig/fs/nilfs2/gcdat.c
+++ linux-2.6/fs/nilfs2/gcdat.c
@@ -27,6 +27,7 @@
#include "page.h"
#include "mdt.h"
+/* XXX: what protects i_state? */
int nilfs_init_gcdat_inode(struct the_nilfs *nilfs)
{
struct inode *dat = nilfs->ns_dat, *gcdat = nilfs->ns_gc_dat;
Index: linux-2.6/fs/quota/dquot.c
===================================================================
--- linux-2.6.orig/fs/quota/dquot.c
+++ linux-2.6/fs/quota/dquot.c
@@ -824,14 +824,22 @@ static void add_dquot_ref(struct super_b
spin_lock(&inode_lock);
spin_lock(&sb_inode_list_lock);
list_for_each_entry(inode, &sb->s_inodes, i_sb_list) {
- if (inode->i_state & (I_FREEING|I_CLEAR|I_WILL_FREE|I_NEW))
+ spin_lock(&inode->i_lock);
+ if (inode->i_state & (I_FREEING|I_CLEAR|I_WILL_FREE|I_NEW)) {
+ spin_unlock(&inode->i_lock);
continue;
- if (!atomic_read(&inode->i_writecount))
+ }
+ if (!atomic_read(&inode->i_writecount)) {
+ spin_unlock(&inode->i_lock);
continue;
- if (!dqinit_needed(inode, type))
+ }
+ if (!dqinit_needed(inode, type)) {
+ spin_unlock(&inode->i_lock);
continue;
+ }
__iget(inode);
+ spin_unlock(&inode->i_lock);
spin_unlock(&sb_inode_list_lock);
spin_unlock(&inode_lock);
^ permalink raw reply [flat|nested] 64+ messages in thread
* [patch 22/33] fs: icache lock i_count
2009-09-04 6:51 [patch 00/33] my current vfs scalability patch queue npiggin
` (20 preceding siblings ...)
2009-09-04 6:52 ` [patch 21/33] fs: icache lock i_state npiggin
@ 2009-09-04 6:52 ` npiggin
2009-09-04 6:52 ` [patch 23/33] fs: icache atomic inodes_stat npiggin
` (11 subsequent siblings)
33 siblings, 0 replies; 64+ messages in thread
From: npiggin @ 2009-09-04 6:52 UTC (permalink / raw)
To: linux-fsdevel; +Cc: linux-kernel
[-- Attachment #1: fs-inode_lock-scale-4.patch --]
[-- Type: text/plain, Size: 35502 bytes --]
Protect inode->i_count with i_lock, rather than having it atomic.
Next step should also be to move things together (eg. the refcount increment
into d_instantiate, which will remove a lock/unlock cycle on i_lock).
---
arch/powerpc/platforms/cell/spufs/file.c | 2 -
fs/affs/inode.c | 4 ++-
fs/afs/dir.c | 4 ++-
fs/anon_inodes.c | 4 ++-
fs/bfs/dir.c | 4 ++-
fs/block_dev.c | 15 ++++++++++--
fs/btrfs/inode.c | 4 ++-
fs/coda/dir.c | 4 ++-
fs/exofs/inode.c | 12 +++++++---
fs/exofs/namei.c | 4 ++-
fs/ext2/namei.c | 4 ++-
fs/ext3/ialloc.c | 4 +--
fs/ext3/namei.c | 4 ++-
fs/ext4/ialloc.c | 4 +--
fs/ext4/namei.c | 4 ++-
fs/fs-writeback.c | 4 +--
fs/gfs2/ops_inode.c | 4 ++-
fs/hfsplus/dir.c | 4 ++-
fs/hpfs/inode.c | 2 -
fs/inode.c | 37 ++++++++++++++++++++-----------
fs/jffs2/dir.c | 8 +++++-
fs/jfs/jfs_txnmgr.c | 4 ++-
fs/jfs/namei.c | 4 ++-
fs/libfs.c | 4 ++-
fs/locks.c | 3 --
fs/minix/namei.c | 4 ++-
fs/namei.c | 7 ++++-
fs/nfs/dir.c | 4 ++-
fs/nfs/getroot.c | 4 ++-
fs/nfs/inode.c | 4 +--
fs/nilfs2/mdt.c | 2 -
fs/nilfs2/namei.c | 4 ++-
fs/notify/inode_mark.c | 22 +++++++++++-------
fs/notify/inotify/inotify.c | 28 +++++++++++++----------
fs/ntfs/super.c | 4 ++-
fs/ocfs2/namei.c | 4 ++-
fs/reiserfs/file.c | 4 +--
fs/reiserfs/namei.c | 4 ++-
fs/reiserfs/stree.c | 2 -
fs/sysv/namei.c | 4 ++-
fs/ubifs/dir.c | 4 ++-
fs/ubifs/super.c | 2 -
fs/udf/namei.c | 4 ++-
fs/ufs/namei.c | 4 ++-
fs/xfs/linux-2.6/xfs_iops.c | 4 ++-
fs/xfs/xfs_iget.c | 2 -
fs/xfs/xfs_inode.h | 6 +++--
include/linux/fs.h | 2 -
ipc/mqueue.c | 7 ++++-
kernel/futex.c | 4 ++-
mm/shmem.c | 4 ++-
51 files changed, 200 insertions(+), 95 deletions(-)
Index: linux-2.6/arch/powerpc/platforms/cell/spufs/file.c
===================================================================
--- linux-2.6.orig/arch/powerpc/platforms/cell/spufs/file.c
+++ linux-2.6/arch/powerpc/platforms/cell/spufs/file.c
@@ -1548,7 +1548,7 @@ static int spufs_mfc_open(struct inode *
if (ctx->owner != current->mm)
return -EINVAL;
- if (atomic_read(&inode->i_count) != 1)
+ if (inode->i_count != 1)
return -EBUSY;
mutex_lock(&ctx->mapping_lock);
Index: linux-2.6/fs/affs/inode.c
===================================================================
--- linux-2.6.orig/fs/affs/inode.c
+++ linux-2.6/fs/affs/inode.c
@@ -379,7 +379,9 @@ affs_add_entry(struct inode *dir, struct
affs_adjust_checksum(inode_bh, block - be32_to_cpu(chain));
mark_buffer_dirty_inode(inode_bh, inode);
inode->i_nlink = 2;
- atomic_inc(&inode->i_count);
+ spin_lock(&inode->i_lock);
+ inode->i_count++;
+ spin_unlock(&inode->i_lock);
}
affs_fix_checksum(sb, bh);
mark_buffer_dirty_inode(bh, inode);
Index: linux-2.6/fs/afs/dir.c
===================================================================
--- linux-2.6.orig/fs/afs/dir.c
+++ linux-2.6/fs/afs/dir.c
@@ -1007,7 +1007,9 @@ static int afs_link(struct dentry *from,
if (ret < 0)
goto link_error;
- atomic_inc(&vnode->vfs_inode.i_count);
+ spin_lock(&vnode->vfs_inode.i_lock);
+ vnode->vfs_inode.i_count++;
+ spin_unlock(&vnode->vfs_inode.i_lock);
d_instantiate(dentry, &vnode->vfs_inode);
key_put(key);
_leave(" = 0");
Index: linux-2.6/fs/anon_inodes.c
===================================================================
--- linux-2.6.orig/fs/anon_inodes.c
+++ linux-2.6/fs/anon_inodes.c
@@ -104,7 +104,9 @@ int anon_inode_getfd(const char *name, c
* so we can avoid doing an igrab() and we can use an open-coded
* atomic_inc().
*/
- atomic_inc(&anon_inode_inode->i_count);
+ spin_lock(&anon_inode_inode->i_lock);
+ anon_inode_inode->i_count++;
+ spin_unlock(&anon_inode_inode->i_lock);
d_instantiate(dentry, anon_inode_inode);
Index: linux-2.6/fs/block_dev.c
===================================================================
--- linux-2.6.orig/fs/block_dev.c
+++ linux-2.6/fs/block_dev.c
@@ -570,7 +570,12 @@ EXPORT_SYMBOL(bdget);
*/
struct block_device *bdgrab(struct block_device *bdev)
{
- atomic_inc(&bdev->bd_inode->i_count);
+ struct inode *inode = bdev->bd_inode;
+
+ spin_lock(&inode->i_lock);
+ inode->i_count++;
+ spin_unlock(&inode->i_lock);
+
return bdev;
}
@@ -600,7 +605,9 @@ static struct block_device *bd_acquire(s
spin_lock(&bdev_lock);
bdev = inode->i_bdev;
if (bdev) {
- atomic_inc(&bdev->bd_inode->i_count);
+ spin_lock(&inode->i_lock);
+ bdev->bd_inode->i_count++;
+ spin_unlock(&inode->i_lock);
spin_unlock(&bdev_lock);
return bdev;
}
@@ -616,7 +623,9 @@ static struct block_device *bd_acquire(s
* So, we can access it via ->i_mapping always
* without igrab().
*/
- atomic_inc(&bdev->bd_inode->i_count);
+ spin_lock(&inode->i_lock);
+ bdev->bd_inode->i_count++;
+ spin_unlock(&inode->i_lock);
inode->i_bdev = bdev;
inode->i_mapping = bdev->bd_inode->i_mapping;
list_add(&inode->i_devices, &bdev->bd_inodes);
Index: linux-2.6/fs/ext2/namei.c
===================================================================
--- linux-2.6.orig/fs/ext2/namei.c
+++ linux-2.6/fs/ext2/namei.c
@@ -196,7 +196,9 @@ static int ext2_link (struct dentry * ol
inode->i_ctime = CURRENT_TIME_SEC;
inode_inc_link_count(inode);
- atomic_inc(&inode->i_count);
+ spin_lock(&inode->i_lock);
+ inode->i_count++;
+ spin_unlock(&inode->i_lock);
err = ext2_add_link(dentry, inode);
if (!err) {
Index: linux-2.6/fs/ext3/ialloc.c
===================================================================
--- linux-2.6.orig/fs/ext3/ialloc.c
+++ linux-2.6/fs/ext3/ialloc.c
@@ -100,9 +100,9 @@ void ext3_free_inode (handle_t *handle,
struct ext3_sb_info *sbi;
int fatal = 0, err;
- if (atomic_read(&inode->i_count) > 1) {
+ if (inode->i_count > 1) {
printk ("ext3_free_inode: inode has count=%d\n",
- atomic_read(&inode->i_count));
+ inode->i_count);
return;
}
if (inode->i_nlink) {
Index: linux-2.6/fs/ext3/namei.c
===================================================================
--- linux-2.6.orig/fs/ext3/namei.c
+++ linux-2.6/fs/ext3/namei.c
@@ -2244,7 +2244,9 @@ retry:
inode->i_ctime = CURRENT_TIME_SEC;
inc_nlink(inode);
- atomic_inc(&inode->i_count);
+ spin_lock(&inode->i_lock);
+ inode->i_count++;
+ spin_unlock(&inode->i_lock);
err = ext3_add_entry(handle, dentry, inode);
if (!err) {
Index: linux-2.6/fs/fs-writeback.c
===================================================================
--- linux-2.6.orig/fs/fs-writeback.c
+++ linux-2.6/fs/fs-writeback.c
@@ -318,7 +318,7 @@ writeback_single_inode(struct inode *ino
unsigned dirty;
int ret;
- if (!atomic_read(&inode->i_count))
+ if (!inode->i_count)
WARN_ON(!(inode->i_state & (I_WILL_FREE|I_FREEING)));
else
WARN_ON(inode->i_state & I_WILL_FREE);
@@ -423,7 +423,7 @@ writeback_single_inode(struct inode *ino
* the pages.
*/
redirty_tail(inode);
- } else if (atomic_read(&inode->i_count)) {
+ } else if (inode->i_count) {
/*
* The inode is clean, inuse
*/
Index: linux-2.6/fs/inode.c
===================================================================
--- linux-2.6.orig/fs/inode.c
+++ linux-2.6/fs/inode.c
@@ -132,7 +132,7 @@ int inode_init_always(struct super_block
inode->i_sb = sb;
inode->i_blkbits = sb->s_blocksize_bits;
inode->i_flags = 0;
- atomic_set(&inode->i_count, 1);
+ inode->i_count = 1;
inode->i_op = &empty_iops;
inode->i_fop = &empty_fops;
inode->i_nlink = 1;
@@ -297,11 +297,10 @@ static void init_once(void *foo)
void __iget(struct inode *inode)
{
assert_spin_locked(&inode->i_lock);
- if (atomic_read(&inode->i_count)) {
- atomic_inc(&inode->i_count);
+ inode->i_count++;
+ if (inode->i_count > 1)
return;
- }
- atomic_inc(&inode->i_count);
+
if (!(inode->i_state & (I_DIRTY|I_SYNC)))
list_move(&inode->i_list, &inode_in_use);
inodes_stat.nr_unused--;
@@ -406,7 +405,7 @@ static int invalidate_list(struct list_h
continue;
}
invalidate_inode_buffers(inode);
- if (!atomic_read(&inode->i_count)) {
+ if (!inode->i_count) {
list_move(&inode->i_list, dispose);
WARN_ON(inode->i_state & I_NEW);
inode->i_state |= I_FREEING;
@@ -457,7 +456,7 @@ static int can_unuse(struct inode *inode
return 0;
if (inode_has_buffers(inode))
return 0;
- if (atomic_read(&inode->i_count))
+ if (inode->i_count)
return 0;
if (inode->i_data.nrpages)
return 0;
@@ -495,7 +494,7 @@ static void prune_icache(int nr_to_scan)
inode = list_entry(inode_unused.prev, struct inode, i_list);
spin_lock(&inode->i_lock);
- if (inode->i_state || atomic_read(&inode->i_count)) {
+ if (inode->i_state || inode->i_count) {
list_move(&inode->i_list, &inode_unused);
spin_unlock(&inode->i_lock);
continue;
@@ -1282,8 +1281,6 @@ void generic_delete_inode(struct inode *
{
const struct super_operations *op = inode->i_sb->s_op;
- spin_lock(&sb_inode_list_lock);
- spin_lock(&inode->i_lock);
list_del_init(&inode->i_list);
list_del_init(&inode->i_sb_list);
spin_unlock(&sb_inode_list_lock);
@@ -1323,8 +1320,6 @@ static void generic_forget_inode(struct
{
struct super_block *sb = inode->i_sb;
- spin_lock(&sb_inode_list_lock);
- spin_lock(&inode->i_lock);
if (!hlist_unhashed(&inode->i_hash)) {
if (!(inode->i_state & (I_DIRTY|I_SYNC)))
list_move(&inode->i_list, &inode_unused);
@@ -1415,8 +1410,24 @@ void iput(struct inode *inode)
if (inode) {
BUG_ON(inode->i_state == I_CLEAR);
- if (atomic_dec_and_lock(&inode->i_count, &inode_lock))
+retry:
+ spin_lock(&inode->i_lock);
+ if (inode->i_count == 1) {
+ if (!spin_trylock(&inode_lock)) {
+ spin_unlock(&inode->i_lock);
+ goto retry;
+ }
+ if (!spin_trylock(&sb_inode_list_lock)) {
+ spin_unlock(&inode_lock);
+ spin_unlock(&inode->i_lock);
+ goto retry;
+ }
+ inode->i_count--;
iput_final(inode);
+ } else {
+ inode->i_count--;
+ spin_unlock(&inode->i_lock);
+ }
}
}
EXPORT_SYMBOL(iput);
Index: linux-2.6/fs/libfs.c
===================================================================
--- linux-2.6.orig/fs/libfs.c
+++ linux-2.6/fs/libfs.c
@@ -280,7 +280,9 @@ int simple_link(struct dentry *old_dentr
inode->i_ctime = dir->i_ctime = dir->i_mtime = CURRENT_TIME;
inc_nlink(inode);
- atomic_inc(&inode->i_count);
+ spin_lock(&inode->i_lock);
+ inode->i_count++;
+ spin_unlock(&inode->i_lock);
dget(dentry);
d_instantiate(dentry, inode);
return 0;
Index: linux-2.6/fs/locks.c
===================================================================
--- linux-2.6.orig/fs/locks.c
+++ linux-2.6/fs/locks.c
@@ -1374,8 +1374,7 @@ int generic_setlease(struct file *filp,
if ((arg == F_RDLCK) && (atomic_read(&inode->i_writecount) > 0))
goto out;
if ((arg == F_WRLCK)
- && (dentry->d_count > 1
- || (atomic_read(&inode->i_count) > 1)))
+ && (dentry->d_count > 1 || inode->i_count > 1))
goto out;
}
Index: linux-2.6/fs/namei.c
===================================================================
--- linux-2.6.orig/fs/namei.c
+++ linux-2.6/fs/namei.c
@@ -2312,8 +2312,11 @@ static long do_unlinkat(int dfd, const c
if (nd.last.name[nd.last.len])
goto slashes;
inode = dentry->d_inode;
- if (inode)
- atomic_inc(&inode->i_count);
+ if (inode) {
+ spin_lock(&inode->i_lock);
+ inode->i_count++;
+ spin_unlock(&inode->i_lock);
+ }
error = mnt_want_write(nd.path.mnt);
if (error)
goto exit2;
Index: linux-2.6/fs/nfs/dir.c
===================================================================
--- linux-2.6.orig/fs/nfs/dir.c
+++ linux-2.6/fs/nfs/dir.c
@@ -1536,7 +1536,9 @@ nfs_link(struct dentry *old_dentry, stru
d_drop(dentry);
error = NFS_PROTO(dir)->link(inode, dir, &dentry->d_name);
if (error == 0) {
- atomic_inc(&inode->i_count);
+ spin_lock(&inode->i_lock);
+ inode->i_count++;
+ spin_unlock(&inode->i_lock);
d_add(dentry, inode);
}
return error;
Index: linux-2.6/fs/nfs/getroot.c
===================================================================
--- linux-2.6.orig/fs/nfs/getroot.c
+++ linux-2.6/fs/nfs/getroot.c
@@ -55,7 +55,9 @@ static int nfs_superblock_set_dummy_root
return -ENOMEM;
}
/* Circumvent igrab(): we know the inode is not being freed */
- atomic_inc(&inode->i_count);
+ spin_lock(&inode->i_lock);
+ inode->i_count++;
+ spin_unlock(&inode->i_lock);
/*
* Ensure that this dentry is invisible to d_find_alias().
* Otherwise, it may be spliced into the tree by
Index: linux-2.6/fs/notify/inotify/inotify.c
===================================================================
--- linux-2.6.orig/fs/notify/inotify/inotify.c
+++ linux-2.6/fs/notify/inotify/inotify.c
@@ -406,23 +406,28 @@ void inotify_unmount_inodes(struct list_
* evict all inodes with zero i_count from icache which is
* unnecessarily violent and may in fact be illegal to do.
*/
- if (!atomic_read(&inode->i_count))
+ if (!inode->i_count)
continue;
need_iput_tmp = need_iput;
need_iput = NULL;
/* In case inotify_remove_watch_locked() drops a reference. */
- if (inode != need_iput_tmp)
+ if (inode != need_iput_tmp) {
+ spin_lock(&inode->i_lock);
__iget(inode);
- else
+ spin_unlock(&inode->i_lock);
+ } else
need_iput_tmp = NULL;
/* In case the dropping of a reference would nuke next_i. */
- if ((&next_i->i_sb_list != list) &&
- atomic_read(&next_i->i_count) &&
- !(next_i->i_state & (I_CLEAR | I_FREEING |
- I_WILL_FREE))) {
- __iget(next_i);
- need_iput = next_i;
+ if (&next_i->i_sb_list != list) {
+ spin_lock(&next_i->i_lock);
+ if (next_i->i_count &&
+ !(next_i->i_state &
+ (I_CLEAR|I_FREEING|I_WILL_FREE))) {
+ __iget(next_i);
+ need_iput = next_i;
+ }
+ spin_unlock(&next_i->i_lock);
}
/*
@@ -441,11 +446,10 @@ void inotify_unmount_inodes(struct list_
mutex_lock(&inode->inotify_mutex);
watches = &inode->inotify_watches;
list_for_each_entry_safe(watch, next_w, watches, i_list) {
- struct inotify_handle *ih= watch->ih;
+ struct inotify_handle *ih = watch->ih;
get_inotify_watch(watch);
mutex_lock(&ih->mutex);
- ih->in_ops->handle_event(watch, watch->wd, IN_UNMOUNT, 0,
- NULL, NULL);
+ ih->in_ops->handle_event(watch, watch->wd, IN_UNMOUNT, 0, NULL, NULL);
inotify_remove_watch_locked(ih, watch);
mutex_unlock(&ih->mutex);
put_inotify_watch(watch);
Index: linux-2.6/fs/xfs/linux-2.6/xfs_iops.c
===================================================================
--- linux-2.6.orig/fs/xfs/linux-2.6/xfs_iops.c
+++ linux-2.6/fs/xfs/linux-2.6/xfs_iops.c
@@ -358,7 +358,9 @@ xfs_vn_link(
if (unlikely(error))
return -error;
- atomic_inc(&inode->i_count);
+ spin_lock(&inode->i_lock);
+ inode->i_count++;
+ spin_unlock(&inode->i_lock);
d_instantiate(dentry, inode);
return 0;
}
Index: linux-2.6/fs/xfs/xfs_iget.c
===================================================================
--- linux-2.6.orig/fs/xfs/xfs_iget.c
+++ linux-2.6/fs/xfs/xfs_iget.c
@@ -827,7 +827,7 @@ xfs_isilocked(
/* 0 */ (void *)(__psint_t)(vk), \
/* 1 */ (void *)(s), \
/* 2 */ (void *)(__psint_t) line, \
-/* 3 */ (void *)(__psint_t)atomic_read(&VFS_I(ip)->i_count), \
+/* 3 */ (void *)(__psint_t)&VFS_I(ip)->i_count, \
/* 4 */ (void *)(ra), \
/* 5 */ NULL, \
/* 6 */ (void *)(__psint_t)current_cpu(), \
Index: linux-2.6/fs/xfs/xfs_inode.h
===================================================================
--- linux-2.6.orig/fs/xfs/xfs_inode.h
+++ linux-2.6/fs/xfs/xfs_inode.h
@@ -545,8 +545,10 @@ extern void xfs_itrace_rele(struct xfs_i
#define IHOLD(ip) \
do { \
- ASSERT(atomic_read(&VFS_I(ip)->i_count) > 0) ; \
- atomic_inc(&(VFS_I(ip)->i_count)); \
+ spin_lock(&VFS_I(ip)->i_lock); \
+ ASSERT(&VFS_I(ip)->i_count > 0); \
+ VFS_I(ip)->i_count++; \
+ spin_unlock(&VFS_I(ip)->i_lock); \
xfs_itrace_hold((ip), __FILE__, __LINE__, (inst_t *)__return_address); \
} while (0)
Index: linux-2.6/include/linux/fs.h
===================================================================
--- linux-2.6.orig/include/linux/fs.h
+++ linux-2.6/include/linux/fs.h
@@ -719,7 +719,7 @@ struct inode {
struct list_head i_sb_list;
struct list_head i_dentry;
unsigned long i_ino;
- atomic_t i_count;
+ unsigned int i_count;
unsigned int i_nlink;
uid_t i_uid;
gid_t i_gid;
Index: linux-2.6/ipc/mqueue.c
===================================================================
--- linux-2.6.orig/ipc/mqueue.c
+++ linux-2.6/ipc/mqueue.c
@@ -779,8 +779,11 @@ SYSCALL_DEFINE1(mq_unlink, const char __
}
inode = dentry->d_inode;
- if (inode)
- atomic_inc(&inode->i_count);
+ if (inode) {
+ spin_lock(&inode->i_lock);
+ inode->i_count++;
+ spin_unlock(&inode->i_lock);
+ }
err = mnt_want_write(ipc_ns->mq_mnt);
if (err)
goto out_err;
Index: linux-2.6/kernel/futex.c
===================================================================
--- linux-2.6.orig/kernel/futex.c
+++ linux-2.6/kernel/futex.c
@@ -164,7 +164,9 @@ static void get_futex_key_refs(union fut
switch (key->both.offset & (FUT_OFF_INODE|FUT_OFF_MMSHARED)) {
case FUT_OFF_INODE:
- atomic_inc(&key->shared.inode->i_count);
+ spin_lock(&key->shared.inode->i_lock);
+ key->shared.inode->i_count++;
+ spin_unlock(&key->shared.inode->i_lock);
break;
case FUT_OFF_MMSHARED:
atomic_inc(&key->private.mm->mm_count);
Index: linux-2.6/mm/shmem.c
===================================================================
--- linux-2.6.orig/mm/shmem.c
+++ linux-2.6/mm/shmem.c
@@ -1867,7 +1867,9 @@ static int shmem_link(struct dentry *old
dir->i_size += BOGO_DIRENT_SIZE;
inode->i_ctime = dir->i_ctime = dir->i_mtime = CURRENT_TIME;
inc_nlink(inode);
- atomic_inc(&inode->i_count); /* New dentry reference */
+ spin_lock(&inode->i_lock);
+ inode->i_count++; /* New dentry reference */
+ spin_unlock(&inode->i_lock);
dget(dentry); /* Extra pinning count for the created dentry */
d_instantiate(dentry, inode);
out:
Index: linux-2.6/fs/bfs/dir.c
===================================================================
--- linux-2.6.orig/fs/bfs/dir.c
+++ linux-2.6/fs/bfs/dir.c
@@ -178,7 +178,9 @@ static int bfs_link(struct dentry *old,
inc_nlink(inode);
inode->i_ctime = CURRENT_TIME_SEC;
mark_inode_dirty(inode);
- atomic_inc(&inode->i_count);
+ spin_lock(&inode->i_lock);
+ inode->i_count++;
+ spin_unlock(&inode->i_lock);
d_instantiate(new, inode);
mutex_unlock(&info->bfs_lock);
return 0;
Index: linux-2.6/fs/btrfs/inode.c
===================================================================
--- linux-2.6.orig/fs/btrfs/inode.c
+++ linux-2.6/fs/btrfs/inode.c
@@ -3871,7 +3871,9 @@ static int btrfs_link(struct dentry *old
trans = btrfs_start_transaction(root, 1);
btrfs_set_trans_block_group(trans, dir);
- atomic_inc(&inode->i_count);
+ spin_lock(&inode->i_lock);
+ inode->i_count++;
+ spin_unlock(&inode->i_lock);
err = btrfs_add_nondir(trans, dentry, inode, 1, index);
Index: linux-2.6/fs/coda/dir.c
===================================================================
--- linux-2.6.orig/fs/coda/dir.c
+++ linux-2.6/fs/coda/dir.c
@@ -302,7 +302,9 @@ static int coda_link(struct dentry *sour
}
coda_dir_update_mtime(dir_inode);
- atomic_inc(&inode->i_count);
+ spin_lock(&inode->i_lock);
+ inode->i_count++;
+ spin_unlock(&inode->i_lock);
d_instantiate(de, inode);
inc_nlink(inode);
Index: linux-2.6/fs/exofs/inode.c
===================================================================
--- linux-2.6.orig/fs/exofs/inode.c
+++ linux-2.6/fs/exofs/inode.c
@@ -1038,7 +1038,9 @@ static void create_done(struct osd_reque
} else
set_obj_created(oi);
- atomic_dec(&inode->i_count);
+ spin_lock(&inode->i_lock);
+ inode->i_count--;
+ spin_unlock(&inode->i_lock);
wake_up(&oi->i_wq);
}
@@ -1104,11 +1106,15 @@ struct inode *exofs_new_inode(struct ino
/* increment the refcount so that the inode will still be around when we
* reach the callback
*/
- atomic_inc(&inode->i_count);
+ spin_lock(&inode->i_lock);
+ inode->i_count++;
+ spin_unlock(&inode->i_lock);
ret = exofs_async_op(or, create_done, inode, oi->i_cred);
if (ret) {
- atomic_dec(&inode->i_count);
+ spin_lock(&inode->i_lock);
+ inode->i_count--;
+ spin_unlock(&inode->i_lock);
osd_end_request(or);
return ERR_PTR(-EIO);
}
Index: linux-2.6/fs/exofs/namei.c
===================================================================
--- linux-2.6.orig/fs/exofs/namei.c
+++ linux-2.6/fs/exofs/namei.c
@@ -153,7 +153,9 @@ static int exofs_link(struct dentry *old
inode->i_ctime = CURRENT_TIME;
inode_inc_link_count(inode);
- atomic_inc(&inode->i_count);
+ spin_lock(&inode->i_lock);
+ inode->i_count++;
+ spin_unlock(&inode->i_lock);
return exofs_add_nondir(dentry, inode);
}
Index: linux-2.6/fs/ext4/ialloc.c
===================================================================
--- linux-2.6.orig/fs/ext4/ialloc.c
+++ linux-2.6/fs/ext4/ialloc.c
@@ -192,9 +192,9 @@ void ext4_free_inode(handle_t *handle, s
struct ext4_sb_info *sbi;
int fatal = 0, err, count, cleared;
- if (atomic_read(&inode->i_count) > 1) {
+ if (inode->i_count > 1) {
printk(KERN_ERR "ext4_free_inode: inode has count=%d\n",
- atomic_read(&inode->i_count));
+ inode->i_count);
return;
}
if (inode->i_nlink) {
Index: linux-2.6/fs/ext4/namei.c
===================================================================
--- linux-2.6.orig/fs/ext4/namei.c
+++ linux-2.6/fs/ext4/namei.c
@@ -2331,7 +2331,9 @@ retry:
inode->i_ctime = ext4_current_time(inode);
ext4_inc_count(handle, inode);
- atomic_inc(&inode->i_count);
+ spin_lock(&inode->i_lock);
+ inode->i_count++;
+ spin_unlock(&inode->i_lock);
err = ext4_add_entry(handle, dentry, inode);
if (!err) {
Index: linux-2.6/fs/gfs2/ops_inode.c
===================================================================
--- linux-2.6.orig/fs/gfs2/ops_inode.c
+++ linux-2.6/fs/gfs2/ops_inode.c
@@ -255,7 +255,9 @@ out_parent:
gfs2_holder_uninit(ghs);
gfs2_holder_uninit(ghs + 1);
if (!error) {
- atomic_inc(&inode->i_count);
+ spin_lock(&inode->i_lock);
+ inode->i_count++;
+ spin_unlock(&inode->i_lock);
d_instantiate(dentry, inode);
mark_inode_dirty(inode);
}
Index: linux-2.6/fs/hfsplus/dir.c
===================================================================
--- linux-2.6.orig/fs/hfsplus/dir.c
+++ linux-2.6/fs/hfsplus/dir.c
@@ -301,7 +301,9 @@ static int hfsplus_link(struct dentry *s
inc_nlink(inode);
hfsplus_instantiate(dst_dentry, inode, cnid);
- atomic_inc(&inode->i_count);
+ spin_lock(&inode->i_lock);
+ inode->i_count++;
+ spin_unlock(&inode->i_lock);
inode->i_ctime = CURRENT_TIME_SEC;
mark_inode_dirty(inode);
HFSPLUS_SB(sb).file_count++;
Index: linux-2.6/fs/hpfs/inode.c
===================================================================
--- linux-2.6.orig/fs/hpfs/inode.c
+++ linux-2.6/fs/hpfs/inode.c
@@ -182,7 +182,7 @@ void hpfs_write_inode(struct inode *i)
struct hpfs_inode_info *hpfs_inode = hpfs_i(i);
struct inode *parent;
if (i->i_ino == hpfs_sb(i->i_sb)->sb_root) return;
- if (hpfs_inode->i_rddir_off && !atomic_read(&i->i_count)) {
+ if (hpfs_inode->i_rddir_off && !i->i_count) {
if (*hpfs_inode->i_rddir_off) printk("HPFS: write_inode: some position still there\n");
kfree(hpfs_inode->i_rddir_off);
hpfs_inode->i_rddir_off = NULL;
Index: linux-2.6/fs/jffs2/dir.c
===================================================================
--- linux-2.6.orig/fs/jffs2/dir.c
+++ linux-2.6/fs/jffs2/dir.c
@@ -287,7 +287,9 @@ static int jffs2_link (struct dentry *ol
mutex_unlock(&f->sem);
d_instantiate(dentry, old_dentry->d_inode);
dir_i->i_mtime = dir_i->i_ctime = ITIME(now);
- atomic_inc(&old_dentry->d_inode->i_count);
+ spin_lock(&old_dentry->d_inode->i_lock);
+ old_dentry->d_inode->i_count++;
+ spin_unlock(&old_dentry->d_inode->i_lock);
}
return ret;
}
@@ -866,7 +868,9 @@ static int jffs2_rename (struct inode *o
printk(KERN_NOTICE "jffs2_rename(): Link succeeded, unlink failed (err %d). You now have a hard link\n", ret);
/* Might as well let the VFS know */
d_instantiate(new_dentry, old_dentry->d_inode);
- atomic_inc(&old_dentry->d_inode->i_count);
+ spin_lock(&old_dentry->d_inode->i_lock);
+ old_dentry->d_inode->i_count++;
+ spin_unlock(&old_dentry->d_inode->i_lock);
new_dir_i->i_mtime = new_dir_i->i_ctime = ITIME(now);
return ret;
}
Index: linux-2.6/fs/jfs/jfs_txnmgr.c
===================================================================
--- linux-2.6.orig/fs/jfs/jfs_txnmgr.c
+++ linux-2.6/fs/jfs/jfs_txnmgr.c
@@ -1279,7 +1279,9 @@ int txCommit(tid_t tid, /* transaction
* lazy commit thread finishes processing
*/
if (tblk->xflag & COMMIT_DELETE) {
- atomic_inc(&tblk->u.ip->i_count);
+ spin_lock(&tblk->u.ip->i_lock);
+ tblk->u.ip->i_count++;
+ spin_unlock(&tblk->u.ip->i_lock);
/*
* Avoid a rare deadlock
*
Index: linux-2.6/fs/jfs/namei.c
===================================================================
--- linux-2.6.orig/fs/jfs/namei.c
+++ linux-2.6/fs/jfs/namei.c
@@ -831,7 +831,9 @@ static int jfs_link(struct dentry *old_d
ip->i_ctime = CURRENT_TIME;
dir->i_ctime = dir->i_mtime = CURRENT_TIME;
mark_inode_dirty(dir);
- atomic_inc(&ip->i_count);
+ spin_lock(&ip->i_lock);
+ ip->i_count++;
+ spin_unlock(&ip->i_lock);
iplist[0] = ip;
iplist[1] = dir;
Index: linux-2.6/fs/minix/namei.c
===================================================================
--- linux-2.6.orig/fs/minix/namei.c
+++ linux-2.6/fs/minix/namei.c
@@ -103,7 +103,9 @@ static int minix_link(struct dentry * ol
inode->i_ctime = CURRENT_TIME_SEC;
inode_inc_link_count(inode);
- atomic_inc(&inode->i_count);
+ spin_lock(&inode->i_lock);
+ inode->i_count++;
+ spin_unlock(&inode->i_lock);
return add_nondir(dentry, inode);
}
Index: linux-2.6/fs/nfs/inode.c
===================================================================
--- linux-2.6.orig/fs/nfs/inode.c
+++ linux-2.6/fs/nfs/inode.c
@@ -364,7 +364,7 @@ nfs_fhget(struct super_block *sb, struct
dprintk("NFS: nfs_fhget(%s/%Ld ct=%d)\n",
inode->i_sb->s_id,
(long long)NFS_FILEID(inode),
- atomic_read(&inode->i_count));
+ inode->i_count);
out:
return inode;
@@ -1148,7 +1148,7 @@ static int nfs_update_inode(struct inode
dfprintk(VFS, "NFS: %s(%s/%ld ct=%d info=0x%x)\n",
__func__, inode->i_sb->s_id, inode->i_ino,
- atomic_read(&inode->i_count), fattr->valid);
+ inode->i_count, fattr->valid);
if ((fattr->valid & NFS_ATTR_FATTR_FILEID) && nfsi->fileid != fattr->fileid)
goto out_fileid;
Index: linux-2.6/fs/nilfs2/mdt.c
===================================================================
--- linux-2.6.orig/fs/nilfs2/mdt.c
+++ linux-2.6/fs/nilfs2/mdt.c
@@ -470,7 +470,7 @@ nilfs_mdt_new_common(struct the_nilfs *n
inode->i_sb = sb; /* sb may be NULL for some meta data files */
inode->i_blkbits = nilfs->ns_blocksize_bits;
inode->i_flags = 0;
- atomic_set(&inode->i_count, 1);
+ inode->i_count = 1;
inode->i_nlink = 1;
inode->i_ino = ino;
inode->i_mode = S_IFREG;
Index: linux-2.6/fs/nilfs2/namei.c
===================================================================
--- linux-2.6.orig/fs/nilfs2/namei.c
+++ linux-2.6/fs/nilfs2/namei.c
@@ -221,7 +221,9 @@ static int nilfs_link(struct dentry *old
inode->i_ctime = CURRENT_TIME;
inode_inc_link_count(inode);
- atomic_inc(&inode->i_count);
+ spin_lock(&inode->i_lock);
+ inode->i_count++;
+ spin_unlock(&inode->i_lock);
err = nilfs_add_nondir(dentry, inode);
if (!err)
Index: linux-2.6/fs/ocfs2/namei.c
===================================================================
--- linux-2.6.orig/fs/ocfs2/namei.c
+++ linux-2.6/fs/ocfs2/namei.c
@@ -719,7 +719,9 @@ static int ocfs2_link(struct dentry *old
goto out_commit;
}
- atomic_inc(&inode->i_count);
+ spin_lock(&inode->i_lock);
+ inode->i_count++;
+ spin_unlock(&inode->i_lock);
dentry->d_op = &ocfs2_dentry_ops;
d_instantiate(dentry, inode);
Index: linux-2.6/fs/reiserfs/file.c
===================================================================
--- linux-2.6.orig/fs/reiserfs/file.c
+++ linux-2.6/fs/reiserfs/file.c
@@ -39,7 +39,7 @@ static int reiserfs_file_release(struct
BUG_ON(!S_ISREG(inode->i_mode));
/* fast out for when nothing needs to be done */
- if ((atomic_read(&inode->i_count) > 1 ||
+ if ((inode->i_count > 1 ||
!(REISERFS_I(inode)->i_flags & i_pack_on_close_mask) ||
!tail_has_to_be_packed(inode)) &&
REISERFS_I(inode)->i_prealloc_count <= 0) {
@@ -94,7 +94,7 @@ static int reiserfs_file_release(struct
if (!err)
err = jbegin_failure;
- if (!err && atomic_read(&inode->i_count) <= 1 &&
+ if (!err && inode->i_count <= 1 &&
(REISERFS_I(inode)->i_flags & i_pack_on_close_mask) &&
tail_has_to_be_packed(inode)) {
/* if regular file is released by last holder and it has been
Index: linux-2.6/fs/reiserfs/namei.c
===================================================================
--- linux-2.6.orig/fs/reiserfs/namei.c
+++ linux-2.6/fs/reiserfs/namei.c
@@ -1142,7 +1142,9 @@ static int reiserfs_link(struct dentry *
inode->i_ctime = CURRENT_TIME_SEC;
reiserfs_update_sd(&th, inode);
- atomic_inc(&inode->i_count);
+ spin_lock(&inode->i_lock);
+ inode->i_count++;
+ spin_unlock(&inode->i_lock);
d_instantiate(dentry, inode);
retval = journal_end(&th, dir->i_sb, jbegin_count);
reiserfs_write_unlock(dir->i_sb);
Index: linux-2.6/fs/reiserfs/stree.c
===================================================================
--- linux-2.6.orig/fs/reiserfs/stree.c
+++ linux-2.6/fs/reiserfs/stree.c
@@ -1440,7 +1440,7 @@ static int maybe_indirect_to_direct(stru
** reading in the last block. The user will hit problems trying to
** read the file, but for now we just skip the indirect2direct
*/
- if (atomic_read(&inode->i_count) > 1 ||
+ if (inode->i_count > 1 ||
!tail_has_to_be_packed(inode) ||
!page || (REISERFS_I(inode)->i_flags & i_nopack_mask)) {
/* leave tail in an unformatted node */
Index: linux-2.6/fs/sysv/namei.c
===================================================================
--- linux-2.6.orig/fs/sysv/namei.c
+++ linux-2.6/fs/sysv/namei.c
@@ -126,7 +126,9 @@ static int sysv_link(struct dentry * old
inode->i_ctime = CURRENT_TIME_SEC;
inode_inc_link_count(inode);
- atomic_inc(&inode->i_count);
+ spin_lock(&inode->i_lock);
+ inode->i_count++;
+ spin_unlock(&inode->i_lock);
return add_nondir(dentry, inode);
}
Index: linux-2.6/fs/ubifs/dir.c
===================================================================
--- linux-2.6.orig/fs/ubifs/dir.c
+++ linux-2.6/fs/ubifs/dir.c
@@ -557,7 +557,9 @@ static int ubifs_link(struct dentry *old
lock_2_inodes(dir, inode);
inc_nlink(inode);
- atomic_inc(&inode->i_count);
+ spin_lock(&inode->i_lock);
+ inode->i_count++;
+ spin_unlock(&inode->i_lock);
inode->i_ctime = ubifs_current_time(inode);
dir->i_size += sz_change;
dir_ui->ui_size = dir->i_size;
Index: linux-2.6/fs/ubifs/super.c
===================================================================
--- linux-2.6.orig/fs/ubifs/super.c
+++ linux-2.6/fs/ubifs/super.c
@@ -341,7 +341,7 @@ static void ubifs_delete_inode(struct in
goto out;
dbg_gen("inode %lu, mode %#x", inode->i_ino, (int)inode->i_mode);
- ubifs_assert(!atomic_read(&inode->i_count));
+ ubifs_assert(!inode->i_count);
ubifs_assert(inode->i_nlink == 0);
truncate_inode_pages(&inode->i_data, 0);
Index: linux-2.6/fs/udf/namei.c
===================================================================
--- linux-2.6.orig/fs/udf/namei.c
+++ linux-2.6/fs/udf/namei.c
@@ -1091,7 +1091,9 @@ static int udf_link(struct dentry *old_d
inc_nlink(inode);
inode->i_ctime = current_fs_time(inode->i_sb);
mark_inode_dirty(inode);
- atomic_inc(&inode->i_count);
+ spin_lock(&inode->i_lock);
+ inode->i_count++;
+ spin_unlock(&inode->i_lock);
d_instantiate(dentry, inode);
unlock_kernel();
Index: linux-2.6/fs/ufs/namei.c
===================================================================
--- linux-2.6.orig/fs/ufs/namei.c
+++ linux-2.6/fs/ufs/namei.c
@@ -178,7 +178,9 @@ static int ufs_link (struct dentry * old
inode->i_ctime = CURRENT_TIME_SEC;
inode_inc_link_count(inode);
- atomic_inc(&inode->i_count);
+ spin_lock(&inode->i_lock);
+ inode->i_count++;
+ spin_unlock(&inode->i_lock);
error = ufs_add_nondir(dentry, inode);
unlock_kernel();
Index: linux-2.6/fs/notify/inode_mark.c
===================================================================
--- linux-2.6.orig/fs/notify/inode_mark.c
+++ linux-2.6/fs/notify/inode_mark.c
@@ -383,24 +383,30 @@ void fsnotify_unmount_inodes(struct list
* evict all inodes with zero i_count from icache which is
* unnecessarily violent and may in fact be illegal to do.
*/
- if (!atomic_read(&inode->i_count))
+ if (!inode->i_count)
continue;
need_iput_tmp = need_iput;
need_iput = NULL;
/* In case fsnotify_inode_delete() drops a reference. */
- if (inode != need_iput_tmp)
+ if (inode != need_iput_tmp) {
+ spin_lock(&inode->i_lock);
__iget(inode);
- else
+ spin_unlock(&inode->i_lock);
+ } else
need_iput_tmp = NULL;
/* In case the dropping of a reference would nuke next_i. */
- if ((&next_i->i_sb_list != list) &&
- atomic_read(&next_i->i_count) &&
- !(next_i->i_state & (I_CLEAR | I_FREEING | I_WILL_FREE))) {
- __iget(next_i);
- need_iput = next_i;
+ if (&next_i->i_sb_list != list) {
+ spin_lock(&next_i->i_lock);
+ if (next_i->i_count &&
+ !(next_i->i_state &
+ (I_CLEAR | I_FREEING | I_WILL_FREE))) {
+ __iget(next_i);
+ need_iput = next_i;
+ }
+ spin_unlock(&next_i->i_lock);
}
/*
Index: linux-2.6/fs/ntfs/super.c
===================================================================
--- linux-2.6.orig/fs/ntfs/super.c
+++ linux-2.6/fs/ntfs/super.c
@@ -2923,7 +2923,9 @@ static int ntfs_fill_super(struct super_
}
if ((sb->s_root = d_alloc_root(vol->root_ino))) {
/* We increment i_count simulating an ntfs_iget(). */
- atomic_inc(&vol->root_ino->i_count);
+ spin_lock(&vol->root_ino->i_lock);
+ vol->root_ino->i_count++;
+ spin_unlock(&vol->root_ino->i_lock);
ntfs_debug("Exiting, status successful.");
/* Release the default upcase if it has no users. */
mutex_lock(&ntfs_lock);
^ permalink raw reply [flat|nested] 64+ messages in thread
* [patch 23/33] fs: icache atomic inodes_stat
2009-09-04 6:51 [patch 00/33] my current vfs scalability patch queue npiggin
` (21 preceding siblings ...)
2009-09-04 6:52 ` [patch 22/33] fs: icache lock i_count npiggin
@ 2009-09-04 6:52 ` npiggin
2009-09-04 6:52 ` [patch 24/33] fs: icache lock lru/writeback lists npiggin
` (10 subsequent siblings)
33 siblings, 0 replies; 64+ messages in thread
From: npiggin @ 2009-09-04 6:52 UTC (permalink / raw)
To: linux-fsdevel; +Cc: linux-kernel
[-- Attachment #1: fs-inode_lock-scale-5.patch --]
[-- Type: text/plain, Size: 7135 bytes --]
Protect inodes_stat statistics with atomic ops rather than inode_lock.
---
fs/cifs/inode.c | 2 +-
fs/fs-writeback.c | 3 ++-
fs/hugetlbfs/inode.c | 6 +++---
fs/inode.c | 28 +++++++++++++++-------------
include/linux/fs.h | 5 +++--
mm/page-writeback.c | 3 ++-
6 files changed, 26 insertions(+), 21 deletions(-)
Index: linux-2.6/fs/fs-writeback.c
===================================================================
--- linux-2.6.orig/fs/fs-writeback.c
+++ linux-2.6/fs/fs-writeback.c
@@ -695,7 +695,8 @@ void sync_inodes_sb(struct super_block *
unsigned long nr_unstable = global_page_state(NR_UNSTABLE_NFS);
wbc.nr_to_write = nr_dirty + nr_unstable +
- (inodes_stat.nr_inodes - inodes_stat.nr_unused);
+ (atomic_read(&inodes_stat.nr_inodes) -
+ atomic_read(&inodes_stat.nr_unused));
} else
wbc.nr_to_write = LONG_MAX; /* doesn't actually matter */
Index: linux-2.6/fs/hugetlbfs/inode.c
===================================================================
--- linux-2.6.orig/fs/hugetlbfs/inode.c
+++ linux-2.6/fs/hugetlbfs/inode.c
@@ -386,7 +386,7 @@ static void hugetlbfs_forget_inode(struc
if (!hlist_unhashed(&inode->i_hash)) {
if (!(inode->i_state & (I_DIRTY|I_SYNC)))
list_move(&inode->i_list, &inode_unused);
- inodes_stat.nr_unused++;
+ atomic_inc(&inodes_stat.nr_unused);
if (!sb || (sb->s_flags & MS_ACTIVE)) {
spin_unlock(&inode_lock);
return;
@@ -404,7 +404,7 @@ static void hugetlbfs_forget_inode(struc
spin_lock(&inode->i_lock);
inode->i_state &= ~I_WILL_FREE;
spin_unlock(&inode->i_lock);
- inodes_stat.nr_unused--;
+ atomic_dec(&inodes_stat.nr_unused);
spin_lock(&inode_hash_lock);
hlist_del_init(&inode->i_hash);
spin_unlock(&inode_hash_lock);
@@ -416,7 +416,7 @@ static void hugetlbfs_forget_inode(struc
spin_lock(&inode->i_lock);
inode->i_state |= I_FREEING;
spin_unlock(&inode->i_lock);
- inodes_stat.nr_inodes--;
+ atomic_dec(&inodes_stat.nr_unused);
spin_unlock(&inode_lock);
truncate_hugepages(inode, 0);
clear_inode(inode);
Index: linux-2.6/fs/inode.c
===================================================================
--- linux-2.6.orig/fs/inode.c
+++ linux-2.6/fs/inode.c
@@ -101,7 +101,10 @@ static DEFINE_MUTEX(iprune_mutex);
/*
* Statistics gathering..
*/
-struct inodes_stat_t inodes_stat;
+struct inodes_stat_t inodes_stat = {
+ .nr_inodes = ATOMIC_INIT(0),
+ .nr_unused = ATOMIC_INIT(0),
+};
static struct kmem_cache *inode_cachep __read_mostly;
@@ -303,7 +306,7 @@ void __iget(struct inode *inode)
if (!(inode->i_state & (I_DIRTY|I_SYNC)))
list_move(&inode->i_list, &inode_in_use);
- inodes_stat.nr_unused--;
+ atomic_dec(&inodes_stat.nr_unused);
}
/**
@@ -368,9 +371,7 @@ static void dispose_list(struct list_hea
destroy_inode(inode);
nr_disposed++;
}
- spin_lock(&inode_lock);
- inodes_stat.nr_inodes -= nr_disposed;
- spin_unlock(&inode_lock);
+ atomic_sub(nr_disposed, &inodes_stat.nr_inodes);
}
/*
@@ -417,7 +418,7 @@ static int invalidate_list(struct list_h
busy = 1;
}
/* only unused inodes may be cached with i_count zero */
- inodes_stat.nr_unused -= count;
+ atomic_sub(count, &inodes_stat.nr_unused);
return busy;
}
@@ -524,7 +525,7 @@ static void prune_icache(int nr_to_scan)
spin_unlock(&inode->i_lock);
nr_pruned++;
}
- inodes_stat.nr_unused -= nr_pruned;
+ atomic_sub(nr_pruned, &inodes_stat.nr_unused);
if (current_is_kswapd())
__count_vm_events(KSWAPD_INODESTEAL, reap);
else
@@ -556,7 +557,8 @@ static int shrink_icache_memory(int nr,
return -1;
prune_icache(nr);
}
- return (inodes_stat.nr_unused / 100) * sysctl_vfs_cache_pressure;
+ return (atomic_read(&inodes_stat.nr_unused) / 100) *
+ sysctl_vfs_cache_pressure;
}
static struct shrinker icache_shrinker = {
@@ -649,7 +651,7 @@ static inline void
__inode_add_to_lists(struct super_block *sb, struct hlist_head *head,
struct inode *inode)
{
- inodes_stat.nr_inodes++;
+ atomic_inc(&inodes_stat.nr_inodes);
spin_lock(&sb_inode_list_lock);
list_add(&inode->i_sb_list, &sb->s_inodes);
spin_unlock(&sb_inode_list_lock);
@@ -1287,7 +1289,7 @@ void generic_delete_inode(struct inode *
WARN_ON(inode->i_state & I_NEW);
inode->i_state |= I_FREEING;
spin_unlock(&inode->i_lock);
- inodes_stat.nr_inodes--;
+ atomic_dec(&inodes_stat.nr_inodes);
spin_unlock(&inode_lock);
security_inode_delete(inode);
@@ -1323,7 +1325,7 @@ static void generic_forget_inode(struct
if (!hlist_unhashed(&inode->i_hash)) {
if (!(inode->i_state & (I_DIRTY|I_SYNC)))
list_move(&inode->i_list, &inode_unused);
- inodes_stat.nr_unused++;
+ atomic_inc(&inodes_stat.nr_unused);
if (sb->s_flags & MS_ACTIVE) {
spin_unlock(&inode->i_lock);
spin_unlock(&sb_inode_list_lock);
@@ -1341,7 +1343,7 @@ static void generic_forget_inode(struct
spin_lock(&inode->i_lock);
WARN_ON(inode->i_state & I_NEW);
inode->i_state &= ~I_WILL_FREE;
- inodes_stat.nr_unused--;
+ atomic_dec(&inodes_stat.nr_unused);
spin_lock(&inode_hash_lock);
hlist_del_init(&inode->i_hash);
spin_unlock(&inode_hash_lock);
@@ -1351,7 +1353,7 @@ static void generic_forget_inode(struct
spin_unlock(&sb_inode_list_lock);
WARN_ON(inode->i_state & I_NEW);
inode->i_state |= I_FREEING;
- inodes_stat.nr_inodes--;
+ atomic_dec(&inodes_stat.nr_inodes);
spin_unlock(&inode->i_lock);
spin_unlock(&inode_lock);
if (inode->i_data.nrpages)
Index: linux-2.6/include/linux/fs.h
===================================================================
--- linux-2.6.orig/include/linux/fs.h
+++ linux-2.6/include/linux/fs.h
@@ -8,6 +8,7 @@
#include <linux/limits.h>
#include <linux/ioctl.h>
+#include <asm/atomic.h>
/*
* It's silly to have NR_OPEN bigger than NR_FILE, but you can change
@@ -39,8 +40,8 @@ struct files_stat_struct {
};
struct inodes_stat_t {
- int nr_inodes;
- int nr_unused;
+ atomic_t nr_inodes;
+ atomic_t nr_unused;
int dummy[5]; /* padding for sysctl ABI compatibility */
};
Index: linux-2.6/mm/page-writeback.c
===================================================================
--- linux-2.6.orig/mm/page-writeback.c
+++ linux-2.6/mm/page-writeback.c
@@ -779,7 +779,8 @@ static void wb_kupdate(unsigned long arg
next_jif = start_jif + msecs_to_jiffies(dirty_writeback_interval * 10);
nr_to_write = global_page_state(NR_FILE_DIRTY) +
global_page_state(NR_UNSTABLE_NFS) +
- (inodes_stat.nr_inodes - inodes_stat.nr_unused);
+ (atomic_read(&inodes_stat.nr_inodes) -
+ atomic_read(&inodes_stat.nr_unused));
while (nr_to_write > 0) {
wbc.more_io = 0;
wbc.encountered_congestion = 0;
Index: linux-2.6/fs/cifs/inode.c
===================================================================
--- linux-2.6.orig/fs/cifs/inode.c
+++ linux-2.6/fs/cifs/inode.c
@@ -1428,7 +1428,7 @@ int cifs_revalidate(struct dentry *diren
}
cFYI(1, ("Revalidate: %s inode 0x%p count %d dentry: 0x%p d_time %ld "
"jiffies %ld", full_path, direntry->d_inode,
- direntry->d_inode->i_count.counter, direntry,
+ direntry->d_inode->i_count, direntry,
direntry->d_time, jiffies));
if (cifsInode->time == 0) {
^ permalink raw reply [flat|nested] 64+ messages in thread
* [patch 24/33] fs: icache lock lru/writeback lists
2009-09-04 6:51 [patch 00/33] my current vfs scalability patch queue npiggin
` (22 preceding siblings ...)
2009-09-04 6:52 ` [patch 23/33] fs: icache atomic inodes_stat npiggin
@ 2009-09-04 6:52 ` npiggin
2009-09-04 6:52 ` [patch 25/33] fs: icache protect inode state npiggin
` (9 subsequent siblings)
33 siblings, 0 replies; 64+ messages in thread
From: npiggin @ 2009-09-04 6:52 UTC (permalink / raw)
To: linux-fsdevel; +Cc: linux-kernel
[-- Attachment #1: fs-inode_lock-scale-6.patch --]
[-- Type: text/plain, Size: 12731 bytes --]
Add a new lock, wb_inode_list_lock, to protect i_list and various lists
which the inode can be put onto.
XXX: haven't audited ocfs2
---
fs/fs-writeback.c | 41 ++++++++++++++++++++++++++++++++++------
fs/hugetlbfs/inode.c | 11 +++++++---
fs/inode.c | 47 ++++++++++++++++++++++++++++++++++++----------
include/linux/writeback.h | 1
4 files changed, 81 insertions(+), 19 deletions(-)
Index: linux-2.6/fs/fs-writeback.c
===================================================================
--- linux-2.6.orig/fs/fs-writeback.c
+++ linux-2.6/fs/fs-writeback.c
@@ -171,7 +171,9 @@ void __mark_inode_dirty(struct inode *in
*/
if (!was_dirty) {
inode->dirtied_when = jiffies;
+ spin_lock(&wb_inode_list_lock);
list_move(&inode->i_list, &sb->s_dirty);
+ spin_unlock(&wb_inode_list_lock);
}
}
out:
@@ -201,12 +203,12 @@ static void redirty_tail(struct inode *i
{
struct super_block *sb = inode->i_sb;
+ assert_spin_locked(&wb_inode_list_lock);
if (!list_empty(&sb->s_dirty)) {
struct inode *tail_inode;
tail_inode = list_entry(sb->s_dirty.next, struct inode, i_list);
- if (time_before(inode->dirtied_when,
- tail_inode->dirtied_when))
+ if (time_before(inode->dirtied_when, tail_inode->dirtied_when))
inode->dirtied_when = jiffies;
}
list_move(&inode->i_list, &sb->s_dirty);
@@ -217,6 +219,7 @@ static void redirty_tail(struct inode *i
*/
static void requeue_io(struct inode *inode)
{
+ assert_spin_locked(&wb_inode_list_lock);
list_move(&inode->i_list, &inode->i_sb->s_more_io);
}
@@ -251,6 +254,7 @@ static void move_expired_inodes(struct l
struct list_head *dispatch_queue,
unsigned long *older_than_this)
{
+ assert_spin_locked(&wb_inode_list_lock);
while (!list_empty(delaying_queue)) {
struct inode *inode = list_entry(delaying_queue->prev,
struct inode, i_list);
@@ -289,11 +293,13 @@ static void inode_wait_for_writeback(str
wqh = bit_waitqueue(&inode->i_state, __I_SYNC);
do {
+ spin_unlock(&wb_inode_list_lock);
spin_unlock(&inode->i_lock);
spin_unlock(&inode_lock);
__wait_on_bit(wqh, &wq, inode_wait, TASK_UNINTERRUPTIBLE);
spin_lock(&inode_lock);
spin_lock(&inode->i_lock);
+ spin_lock(&wb_inode_list_lock);
} while (inode->i_state & I_SYNC);
}
@@ -350,6 +356,7 @@ writeback_single_inode(struct inode *ino
inode->i_state |= I_SYNC;
inode->i_state &= ~I_DIRTY;
+ spin_unlock(&wb_inode_list_lock);
spin_unlock(&inode->i_lock);
spin_unlock(&inode_lock);
@@ -370,6 +377,7 @@ writeback_single_inode(struct inode *ino
spin_lock(&inode_lock);
spin_lock(&inode->i_lock);
+ spin_lock(&wb_inode_list_lock);
inode->i_state &= ~I_SYNC;
if (!(inode->i_state & (I_FREEING | I_CLEAR))) {
if (!(inode->i_state & I_DIRTY) &&
@@ -471,6 +479,8 @@ void generic_sync_sb_inodes(struct super
int sync = wbc->sync_mode == WB_SYNC_ALL;
spin_lock(&inode_lock);
+again:
+ spin_lock(&wb_inode_list_lock);
if (!wbc->for_kupdate || list_empty(&sb->s_io))
queue_io(sb, wbc->older_than_this);
@@ -481,6 +491,11 @@ void generic_sync_sb_inodes(struct super
struct backing_dev_info *bdi = mapping->backing_dev_info;
long pages_skipped;
+ if (!spin_trylock(&inode->i_lock)) {
+ spin_unlock(&wb_inode_list_lock);
+ goto again;
+ }
+
if (!bdi_cap_writeback_dirty(bdi)) {
redirty_tail(inode);
if (sb_is_blkdev_sb(sb)) {
@@ -488,6 +503,7 @@ void generic_sync_sb_inodes(struct super
* Dirty memory-backed blockdev: the ramdisk
* driver does this. Skip just this inode
*/
+ spin_unlock(&inode->i_lock);
continue;
}
/*
@@ -495,28 +511,34 @@ void generic_sync_sb_inodes(struct super
* than the kernel-internal bdev filesystem. Skip the
* entire superblock.
*/
+ spin_unlock(&inode->i_lock);
break;
}
if (wbc->nonblocking && bdi_write_congested(bdi)) {
wbc->encountered_congestion = 1;
- if (!sb_is_blkdev_sb(sb))
+ if (!sb_is_blkdev_sb(sb)) {
+ spin_unlock(&inode->i_lock);
break; /* Skip a congested fs */
+ }
requeue_io(inode);
+ spin_unlock(&inode->i_lock);
continue; /* Skip a congested blockdev */
}
if (wbc->bdi && bdi != wbc->bdi) {
- if (!sb_is_blkdev_sb(sb))
+ if (!sb_is_blkdev_sb(sb)) {
+ spin_unlock(&inode->i_lock);
break; /* fs has the wrong queue */
+ }
requeue_io(inode);
+ spin_unlock(&inode->i_lock);
continue; /* blockdev has wrong queue */
}
- spin_lock(&inode->i_lock);
if (inode->i_state & (I_NEW | I_WILL_FREE)) {
- spin_unlock(&inode->i_lock);
requeue_io(inode);
+ spin_unlock(&inode->i_lock);
continue;
}
@@ -548,11 +570,13 @@ void generic_sync_sb_inodes(struct super
*/
redirty_tail(inode);
}
+ spin_unlock(&wb_inode_list_lock);
spin_unlock(&inode->i_lock);
spin_unlock(&inode_lock);
iput(inode);
cond_resched();
spin_lock(&inode_lock);
+ spin_lock(&wb_inode_list_lock);
if (wbc->nr_to_write <= 0) {
wbc->more_io = 1;
break;
@@ -560,6 +584,7 @@ void generic_sync_sb_inodes(struct super
if (!list_empty(&sb->s_more_io))
wbc->more_io = 1;
}
+ spin_unlock(&wb_inode_list_lock);
if (sync) {
struct inode *inode, *old_inode = NULL;
@@ -729,7 +754,9 @@ int write_inode_now(struct inode *inode,
might_sleep();
spin_lock(&inode_lock);
spin_lock(&inode->i_lock);
+ spin_lock(&wb_inode_list_lock);
ret = writeback_single_inode(inode, &wbc);
+ spin_unlock(&wb_inode_list_lock);
spin_unlock(&inode->i_lock);
spin_unlock(&inode_lock);
if (sync)
@@ -755,7 +782,9 @@ int sync_inode(struct inode *inode, stru
spin_lock(&inode_lock);
spin_lock(&inode->i_lock);
+ spin_lock(&wb_inode_list_lock);
ret = writeback_single_inode(inode, wbc);
+ spin_unlock(&wb_inode_list_lock);
spin_unlock(&inode->i_lock);
spin_unlock(&inode_lock);
return ret;
Index: linux-2.6/fs/hugetlbfs/inode.c
===================================================================
--- linux-2.6.orig/fs/hugetlbfs/inode.c
+++ linux-2.6/fs/hugetlbfs/inode.c
@@ -384,8 +384,11 @@ static void hugetlbfs_forget_inode(struc
struct super_block *sb = inode->i_sb;
if (!hlist_unhashed(&inode->i_hash)) {
- if (!(inode->i_state & (I_DIRTY|I_SYNC)))
+ if (!(inode->i_state & (I_DIRTY|I_SYNC))) {
+ spin_lock(&wb_inode_list_lock);
list_move(&inode->i_list, &inode_unused);
+ spin_unlock(&wb_inode_list_lock);
+ }
atomic_inc(&inodes_stat.nr_unused);
if (!sb || (sb->s_flags & MS_ACTIVE)) {
spin_unlock(&inode_lock);
@@ -403,13 +406,15 @@ static void hugetlbfs_forget_inode(struc
spin_lock(&inode_lock);
spin_lock(&inode->i_lock);
inode->i_state &= ~I_WILL_FREE;
- spin_unlock(&inode->i_lock);
- atomic_dec(&inodes_stat.nr_unused);
spin_lock(&inode_hash_lock);
hlist_del_init(&inode->i_hash);
spin_unlock(&inode_hash_lock);
+ spin_unlock(&inode->i_lock);
+ atomic_dec(&inodes_stat.nr_unused);
}
+ spin_lock(&wb_inode_list_lock);
list_del_init(&inode->i_list);
+ spin_unlock(&wb_inode_list_lock);
spin_lock(&sb_inode_list_lock);
list_del_init(&inode->i_sb_list);
spin_unlock(&sb_inode_list_lock);
Index: linux-2.6/fs/inode.c
===================================================================
--- linux-2.6.orig/fs/inode.c
+++ linux-2.6/fs/inode.c
@@ -86,6 +86,7 @@ static struct hlist_head *inode_hashtabl
*/
DEFINE_SPINLOCK(inode_lock);
DEFINE_SPINLOCK(sb_inode_list_lock);
+DEFINE_SPINLOCK(wb_inode_list_lock);
DEFINE_SPINLOCK(inode_hash_lock);
/*
@@ -304,8 +305,11 @@ void __iget(struct inode *inode)
if (inode->i_count > 1)
return;
- if (!(inode->i_state & (I_DIRTY|I_SYNC)))
+ if (!(inode->i_state & (I_DIRTY|I_SYNC))) {
+ spin_lock(&wb_inode_list_lock);
list_move(&inode->i_list, &inode_in_use);
+ spin_unlock(&wb_inode_list_lock);
+ }
atomic_dec(&inodes_stat.nr_unused);
}
@@ -407,7 +411,9 @@ static int invalidate_list(struct list_h
}
invalidate_inode_buffers(inode);
if (!inode->i_count) {
+ spin_lock(&wb_inode_list_lock);
list_move(&inode->i_list, dispose);
+ spin_unlock(&wb_inode_list_lock);
WARN_ON(inode->i_state & I_NEW);
inode->i_state |= I_FREEING;
spin_unlock(&inode->i_lock);
@@ -486,6 +492,8 @@ static void prune_icache(int nr_to_scan)
mutex_lock(&iprune_mutex);
spin_lock(&inode_lock);
+again:
+ spin_lock(&wb_inode_list_lock);
for (nr_scanned = 0; nr_scanned < nr_to_scan; nr_scanned++) {
struct inode *inode;
@@ -494,13 +502,17 @@ static void prune_icache(int nr_to_scan)
inode = list_entry(inode_unused.prev, struct inode, i_list);
- spin_lock(&inode->i_lock);
+ if (!spin_trylock(&inode->i_lock)) {
+ spin_unlock(&wb_inode_list_lock);
+ goto again;
+ }
if (inode->i_state || inode->i_count) {
list_move(&inode->i_list, &inode_unused);
spin_unlock(&inode->i_lock);
continue;
}
if (inode_has_buffers(inode) || inode->i_data.nrpages) {
+ spin_unlock(&wb_inode_list_lock);
__iget(inode);
spin_unlock(&inode->i_lock);
spin_unlock(&inode_lock);
@@ -509,11 +521,16 @@ static void prune_icache(int nr_to_scan)
0, -1);
iput(inode);
spin_lock(&inode_lock);
+again2:
+ spin_lock(&wb_inode_list_lock);
if (inode != list_entry(inode_unused.next,
struct inode, i_list))
continue; /* wrong inode or list_empty */
- spin_lock(&inode->i_lock);
+ if (!spin_trylock(&inode->i_lock)) {
+ spin_unlock(&wb_inode_list_lock);
+ goto again2;
+ }
if (!can_unuse(inode)) {
spin_unlock(&inode->i_lock);
continue;
@@ -531,6 +548,7 @@ static void prune_icache(int nr_to_scan)
else
__count_vm_events(PGINODESTEAL, reap);
spin_unlock(&inode_lock);
+ spin_unlock(&wb_inode_list_lock);
dispose_list(&freeable);
mutex_unlock(&iprune_mutex);
@@ -655,7 +673,9 @@ __inode_add_to_lists(struct super_block
spin_lock(&sb_inode_list_lock);
list_add(&inode->i_sb_list, &sb->s_inodes);
spin_unlock(&sb_inode_list_lock);
+ spin_lock(&wb_inode_list_lock);
list_add(&inode->i_list, &inode_in_use);
+ spin_unlock(&wb_inode_list_lock);
if (head) {
spin_lock(&inode_hash_lock);
hlist_add_head(&inode->i_hash, head);
@@ -1283,14 +1303,16 @@ void generic_delete_inode(struct inode *
{
const struct super_operations *op = inode->i_sb->s_op;
+ spin_lock(&wb_inode_list_lock);
list_del_init(&inode->i_list);
+ spin_unlock(&wb_inode_list_lock);
list_del_init(&inode->i_sb_list);
spin_unlock(&sb_inode_list_lock);
WARN_ON(inode->i_state & I_NEW);
inode->i_state |= I_FREEING;
spin_unlock(&inode->i_lock);
- atomic_dec(&inodes_stat.nr_inodes);
spin_unlock(&inode_lock);
+ atomic_dec(&inodes_stat.nr_inodes);
security_inode_delete(inode);
@@ -1323,8 +1345,11 @@ static void generic_forget_inode(struct
struct super_block *sb = inode->i_sb;
if (!hlist_unhashed(&inode->i_hash)) {
- if (!(inode->i_state & (I_DIRTY|I_SYNC)))
+ if (!(inode->i_state & (I_DIRTY|I_SYNC))) {
+ spin_lock(&wb_inode_list_lock);
list_move(&inode->i_list, &inode_unused);
+ spin_unlock(&wb_inode_list_lock);
+ }
atomic_inc(&inodes_stat.nr_unused);
if (sb->s_flags & MS_ACTIVE) {
spin_unlock(&inode->i_lock);
@@ -1348,14 +1373,16 @@ static void generic_forget_inode(struct
hlist_del_init(&inode->i_hash);
spin_unlock(&inode_hash_lock);
}
+ spin_lock(&wb_inode_list_lock);
list_del_init(&inode->i_list);
+ spin_unlock(&wb_inode_list_lock);
list_del_init(&inode->i_sb_list);
spin_unlock(&sb_inode_list_lock);
WARN_ON(inode->i_state & I_NEW);
inode->i_state |= I_FREEING;
- atomic_dec(&inodes_stat.nr_inodes);
spin_unlock(&inode->i_lock);
spin_unlock(&inode_lock);
+ atomic_dec(&inodes_stat.nr_inodes);
if (inode->i_data.nrpages)
truncate_inode_pages(&inode->i_data, 0);
clear_inode(inode);
@@ -1412,17 +1439,17 @@ void iput(struct inode *inode)
if (inode) {
BUG_ON(inode->i_state == I_CLEAR);
-retry:
+retry1:
spin_lock(&inode->i_lock);
if (inode->i_count == 1) {
if (!spin_trylock(&inode_lock)) {
+retry2:
spin_unlock(&inode->i_lock);
- goto retry;
+ goto retry1;
}
if (!spin_trylock(&sb_inode_list_lock)) {
spin_unlock(&inode_lock);
- spin_unlock(&inode->i_lock);
- goto retry;
+ goto retry2;
}
inode->i_count--;
iput_final(inode);
Index: linux-2.6/include/linux/writeback.h
===================================================================
--- linux-2.6.orig/include/linux/writeback.h
+++ linux-2.6/include/linux/writeback.h
@@ -11,6 +11,7 @@ struct backing_dev_info;
extern spinlock_t inode_lock;
extern spinlock_t sb_inode_list_lock;
+extern spinlock_t wb_inode_list_lock;
extern spinlock_t inode_hash_lock;
extern struct list_head inode_in_use;
extern struct list_head inode_unused;
^ permalink raw reply [flat|nested] 64+ messages in thread
* [patch 25/33] fs: icache protect inode state
2009-09-04 6:51 [patch 00/33] my current vfs scalability patch queue npiggin
` (23 preceding siblings ...)
2009-09-04 6:52 ` [patch 24/33] fs: icache lock lru/writeback lists npiggin
@ 2009-09-04 6:52 ` npiggin
2009-09-04 6:52 ` [patch 26/33] fs: inode atomic last_ino, iunique lock npiggin
` (8 subsequent siblings)
33 siblings, 0 replies; 64+ messages in thread
From: npiggin @ 2009-09-04 6:52 UTC (permalink / raw)
To: linux-fsdevel; +Cc: linux-kernel
[-- Attachment #1: fs-inode_lock-scale-6b.patch --]
[-- Type: text/plain, Size: 6744 bytes --]
Protect i_hash, i_sb_list etc members with i_lock.
---
fs/hugetlbfs/inode.c | 14 +++++++++-----
fs/inode.c | 29 ++++++++++++++++++++++++++---
2 files changed, 35 insertions(+), 8 deletions(-)
Index: linux-2.6/fs/inode.c
===================================================================
--- linux-2.6.orig/fs/inode.c
+++ linux-2.6/fs/inode.c
@@ -363,12 +363,14 @@ static void dispose_list(struct list_hea
clear_inode(inode);
spin_lock(&inode_lock);
+ spin_lock(&sb_inode_list_lock);
+ spin_lock(&inode->i_lock);
spin_lock(&inode_hash_lock);
hlist_del_init(&inode->i_hash);
spin_unlock(&inode_hash_lock);
- spin_lock(&sb_inode_list_lock);
list_del_init(&inode->i_sb_list);
spin_unlock(&sb_inode_list_lock);
+ spin_unlock(&inode->i_lock);
spin_unlock(&inode_lock);
wake_up_inode(inode);
@@ -670,7 +672,6 @@ __inode_add_to_lists(struct super_block
struct inode *inode)
{
atomic_inc(&inodes_stat.nr_inodes);
- spin_lock(&sb_inode_list_lock);
list_add(&inode->i_sb_list, &sb->s_inodes);
spin_unlock(&sb_inode_list_lock);
spin_lock(&wb_inode_list_lock);
@@ -700,7 +701,10 @@ void inode_add_to_lists(struct super_blo
struct hlist_head *head = inode_hashtable + hash(sb, inode->i_ino);
spin_lock(&inode_lock);
+ spin_lock(&sb_inode_list_lock);
+ spin_lock(&inode->i_lock);
__inode_add_to_lists(sb, head, inode);
+ spin_unlock(&inode->i_lock);
spin_unlock(&inode_lock);
}
EXPORT_SYMBOL_GPL(inode_add_to_lists);
@@ -732,9 +736,12 @@ struct inode *new_inode(struct super_blo
inode = alloc_inode(sb);
if (inode) {
spin_lock(&inode_lock);
+ spin_lock(&sb_inode_list_lock);
+ spin_lock(&inode->i_lock);
inode->i_ino = ++last_ino;
inode->i_state = 0;
__inode_add_to_lists(sb, NULL, inode);
+ spin_unlock(&inode->i_lock);
spin_unlock(&inode_lock);
}
return inode;
@@ -796,11 +803,14 @@ static struct inode *get_new_inode(struc
/* We released the lock, so.. */
old = find_inode(sb, head, test, data);
if (!old) {
+ spin_lock(&sb_inode_list_lock);
+ spin_lock(&inode->i_lock);
if (set(inode, data))
goto set_failed;
inode->i_state = I_LOCK|I_NEW;
__inode_add_to_lists(sb, head, inode);
+ spin_unlock(&inode->i_lock);
spin_unlock(&inode_lock);
/* Return the locked inode with I_NEW set, the
@@ -825,6 +835,7 @@ static struct inode *get_new_inode(struc
set_failed:
spin_unlock(&inode->i_lock);
+ spin_unlock(&sb_inode_list_lock);
spin_unlock(&inode_lock);
destroy_inode(inode);
return NULL;
@@ -847,9 +858,12 @@ static struct inode *get_new_inode_fast(
/* We released the lock, so.. */
old = find_inode_fast(sb, head, ino);
if (!old) {
+ spin_lock(&sb_inode_list_lock);
+ spin_lock(&inode->i_lock);
inode->i_ino = ino;
inode->i_state = I_LOCK|I_NEW;
__inode_add_to_lists(sb, head, inode);
+ spin_unlock(&inode->i_lock);
spin_unlock(&inode_lock);
/* Return the locked inode with I_NEW set, the
@@ -1185,6 +1199,7 @@ repeat:
break;
}
if (likely(!node)) {
+ /* XXX: initialize inode->i_lock to locked? */
hlist_add_head(&inode->i_hash, head);
spin_unlock(&inode_hash_lock);
spin_unlock(&inode_lock);
@@ -1233,6 +1248,7 @@ repeat:
break;
}
if (likely(!node)) {
+ /* XXX: initialize inode->i_lock to locked? */
hlist_add_head(&inode->i_hash, head);
spin_unlock(&inode_hash_lock);
spin_unlock(&inode_lock);
@@ -1263,10 +1279,13 @@ EXPORT_SYMBOL(insert_inode_locked4);
void __insert_inode_hash(struct inode *inode, unsigned long hashval)
{
struct hlist_head *head = inode_hashtable + hash(inode->i_sb, hashval);
+
spin_lock(&inode_lock);
+ spin_lock(&inode->i_lock);
spin_lock(&inode_hash_lock);
hlist_add_head(&inode->i_hash, head);
spin_unlock(&inode_hash_lock);
+ spin_unlock(&inode->i_lock);
spin_unlock(&inode_lock);
}
EXPORT_SYMBOL(__insert_inode_hash);
@@ -1280,9 +1299,11 @@ EXPORT_SYMBOL(__insert_inode_hash);
void remove_inode_hash(struct inode *inode)
{
spin_lock(&inode_lock);
+ spin_lock(&inode->i_lock);
spin_lock(&inode_hash_lock);
hlist_del_init(&inode->i_hash);
spin_unlock(&inode_hash_lock);
+ spin_unlock(&inode->i_lock);
spin_unlock(&inode_lock);
}
EXPORT_SYMBOL(remove_inode_hash);
@@ -1330,9 +1351,11 @@ void generic_delete_inode(struct inode *
clear_inode(inode);
}
spin_lock(&inode_lock);
+ spin_lock(&inode->i_lock);
spin_lock(&inode_hash_lock);
hlist_del_init(&inode->i_hash);
spin_unlock(&inode_hash_lock);
+ spin_unlock(&inode->i_lock);
spin_unlock(&inode_lock);
wake_up_inode(inode);
BUG_ON(inode->i_state != I_CLEAR);
@@ -1368,10 +1391,10 @@ static void generic_forget_inode(struct
spin_lock(&inode->i_lock);
WARN_ON(inode->i_state & I_NEW);
inode->i_state &= ~I_WILL_FREE;
- atomic_dec(&inodes_stat.nr_unused);
spin_lock(&inode_hash_lock);
hlist_del_init(&inode->i_hash);
spin_unlock(&inode_hash_lock);
+ atomic_dec(&inodes_stat.nr_unused);
}
spin_lock(&wb_inode_list_lock);
list_del_init(&inode->i_list);
Index: linux-2.6/fs/hugetlbfs/inode.c
===================================================================
--- linux-2.6.orig/fs/hugetlbfs/inode.c
+++ linux-2.6/fs/hugetlbfs/inode.c
@@ -391,12 +391,15 @@ static void hugetlbfs_forget_inode(struc
}
atomic_inc(&inodes_stat.nr_unused);
if (!sb || (sb->s_flags & MS_ACTIVE)) {
+ spin_unlock(&inode->i_lock);
+ spin_unlock(&sb_inode_list_lock);
spin_unlock(&inode_lock);
return;
}
- spin_lock(&inode->i_lock);
+ WARN_ON(inode->i_state & I_NEW);
inode->i_state |= I_WILL_FREE;
spin_unlock(&inode->i_lock);
+ spin_unlock(&sb_inode_list_lock);
spin_unlock(&inode_lock);
/*
* write_inode_now is a noop as we set BDI_CAP_NO_WRITEBACK
@@ -404,27 +407,28 @@ static void hugetlbfs_forget_inode(struc
*/
write_inode_now(inode, 1);
spin_lock(&inode_lock);
+ spin_lock(&sb_inode_list_lock);
spin_lock(&inode->i_lock);
+ WARN_ON(inode->i_state & I_NEW);
inode->i_state &= ~I_WILL_FREE;
spin_lock(&inode_hash_lock);
hlist_del_init(&inode->i_hash);
spin_unlock(&inode_hash_lock);
- spin_unlock(&inode->i_lock);
atomic_dec(&inodes_stat.nr_unused);
}
spin_lock(&wb_inode_list_lock);
list_del_init(&inode->i_list);
spin_unlock(&wb_inode_list_lock);
- spin_lock(&sb_inode_list_lock);
list_del_init(&inode->i_sb_list);
spin_unlock(&sb_inode_list_lock);
- spin_lock(&inode->i_lock);
+ WARN_ON(inode->i_state & I_NEW);
inode->i_state |= I_FREEING;
spin_unlock(&inode->i_lock);
- atomic_dec(&inodes_stat.nr_unused);
spin_unlock(&inode_lock);
+ atomic_dec(&inodes_stat.nr_unused);
truncate_hugepages(inode, 0);
clear_inode(inode);
+ /* XXX: why no wake_up_inode? */
destroy_inode(inode);
}
^ permalink raw reply [flat|nested] 64+ messages in thread
* [patch 26/33] fs: inode atomic last_ino, iunique lock
2009-09-04 6:51 [patch 00/33] my current vfs scalability patch queue npiggin
` (24 preceding siblings ...)
2009-09-04 6:52 ` [patch 25/33] fs: icache protect inode state npiggin
@ 2009-09-04 6:52 ` npiggin
2009-09-04 6:52 ` [patch 27/33] fs: icache remove inode_lock npiggin
` (7 subsequent siblings)
33 siblings, 0 replies; 64+ messages in thread
From: npiggin @ 2009-09-04 6:52 UTC (permalink / raw)
To: linux-fsdevel; +Cc: linux-kernel
[-- Attachment #1: fs-inode_lock-scale-6c.patch --]
[-- Type: text/plain, Size: 2296 bytes --]
Make last_ino atomic in preperation for removing inode_lock.
Make a new lock for iunique counter, for removing inode_lock.
---
fs/inode.c | 28 ++++++++++++++++++++++------
1 file changed, 22 insertions(+), 6 deletions(-)
Index: linux-2.6/fs/inode.c
===================================================================
--- linux-2.6.orig/fs/inode.c
+++ linux-2.6/fs/inode.c
@@ -728,7 +728,7 @@ struct inode *new_inode(struct super_blo
* error if st_ino won't fit in target struct field. Use 32bit counter
* here to attempt to avoid that.
*/
- static unsigned int last_ino;
+ static atomic_t last_ino = ATOMIC_INIT(0);
struct inode *inode;
spin_lock_prefetch(&inode_lock);
@@ -738,7 +738,7 @@ struct inode *new_inode(struct super_blo
spin_lock(&inode_lock);
spin_lock(&sb_inode_list_lock);
spin_lock(&inode->i_lock);
- inode->i_ino = ++last_ino;
+ inode->i_ino = atomic_inc_return(&last_ino);
inode->i_state = 0;
__inode_add_to_lists(sb, NULL, inode);
spin_unlock(&inode->i_lock);
@@ -887,6 +887,22 @@ static struct inode *get_new_inode_fast(
return inode;
}
+static int test_inode_iunique(struct super_block * sb, struct hlist_head *head, unsigned long ino)
+{
+ struct hlist_node *node;
+ struct inode * inode = NULL;
+
+ spin_lock(&inode_hash_lock);
+ hlist_for_each_entry(inode, node, head, i_hash) {
+ if (inode->i_ino == ino && inode->i_sb == sb) {
+ spin_unlock(&inode_hash_lock);
+ return 0;
+ }
+ }
+ spin_unlock(&inode_hash_lock);
+ return 1;
+}
+
/**
* iunique - get a unique inode number
* @sb: superblock
@@ -908,20 +924,20 @@ ino_t iunique(struct super_block *sb, in
* error if st_ino won't fit in target struct field. Use 32bit counter
* here to attempt to avoid that.
*/
+ static DEFINE_SPINLOCK(unique_lock);
static unsigned int counter;
- struct inode *inode;
struct hlist_head *head;
ino_t res;
spin_lock(&inode_lock);
+ spin_lock(&unique_lock);
do {
if (counter <= max_reserved)
counter = max_reserved + 1;
res = counter++;
head = inode_hashtable + hash(sb, res);
- inode = find_inode_fast(sb, head, res);
- spin_unlock(&inode->i_lock);
- } while (inode != NULL);
+ } while (!test_inode_iunique(sb, head, res));
+ spin_unlock(&unique_lock);
spin_unlock(&inode_lock);
return res;
^ permalink raw reply [flat|nested] 64+ messages in thread
* [patch 27/33] fs: icache remove inode_lock
2009-09-04 6:51 [patch 00/33] my current vfs scalability patch queue npiggin
` (25 preceding siblings ...)
2009-09-04 6:52 ` [patch 26/33] fs: inode atomic last_ino, iunique lock npiggin
@ 2009-09-04 6:52 ` npiggin
2009-09-04 6:52 ` [patch 28/33] fs: inode factor hash lock into functions npiggin
` (6 subsequent siblings)
33 siblings, 0 replies; 64+ messages in thread
From: npiggin @ 2009-09-04 6:52 UTC (permalink / raw)
To: linux-fsdevel; +Cc: linux-kernel
[-- Attachment #1: fs-inode_lock-scale-7.patch --]
[-- Type: text/plain, Size: 23995 bytes --]
Remove the global inode_lock
---
fs/buffer.c | 2 -
fs/drop_caches.c | 4 --
fs/fs-writeback.c | 23 +--------------
fs/hugetlbfs/inode.c | 6 ----
fs/inode.c | 65 ++------------------------------------------
fs/notify/inode_mark.c | 14 +++++----
fs/notify/inotify/inotify.c | 16 ++++++----
fs/quota/dquot.c | 6 ----
include/linux/writeback.h | 1
9 files changed, 26 insertions(+), 111 deletions(-)
Index: linux-2.6/fs/buffer.c
===================================================================
--- linux-2.6.orig/fs/buffer.c
+++ linux-2.6/fs/buffer.c
@@ -1145,7 +1145,7 @@ __getblk_slow(struct block_device *bdev,
* inode list.
*
* mark_buffer_dirty() is atomic. It takes bh->b_page->mapping->private_lock,
- * mapping->tree_lock and the global inode_lock.
+ * and mapping->tree_lock.
*/
void mark_buffer_dirty(struct buffer_head *bh)
{
Index: linux-2.6/fs/drop_caches.c
===================================================================
--- linux-2.6.orig/fs/drop_caches.c
+++ linux-2.6/fs/drop_caches.c
@@ -16,7 +16,6 @@ static void drop_pagecache_sb(struct sup
{
struct inode *inode, *toput_inode = NULL;
- spin_lock(&inode_lock);
spin_lock(&sb_inode_list_lock);
list_for_each_entry(inode, &sb->s_inodes, i_sb_list) {
spin_lock(&inode->i_lock);
@@ -28,15 +27,12 @@ static void drop_pagecache_sb(struct sup
__iget(inode);
spin_unlock(&inode->i_lock);
spin_unlock(&sb_inode_list_lock);
- spin_unlock(&inode_lock);
invalidate_mapping_pages(inode->i_mapping, 0, -1);
iput(toput_inode);
toput_inode = inode;
- spin_lock(&inode_lock);
spin_lock(&sb_inode_list_lock);
}
spin_unlock(&sb_inode_list_lock);
- spin_unlock(&inode_lock);
iput(toput_inode);
}
Index: linux-2.6/fs/fs-writeback.c
===================================================================
--- linux-2.6.orig/fs/fs-writeback.c
+++ linux-2.6/fs/fs-writeback.c
@@ -139,7 +139,6 @@ void __mark_inode_dirty(struct inode *in
if (unlikely(block_dump))
block_dump___mark_inode_dirty(inode);
- spin_lock(&inode_lock);
spin_lock(&inode->i_lock);
if ((inode->i_state & flags) != flags) {
const int was_dirty = inode->i_state & I_DIRTY;
@@ -178,7 +177,6 @@ void __mark_inode_dirty(struct inode *in
}
out:
spin_unlock(&inode->i_lock);
- spin_unlock(&inode_lock);
}
EXPORT_SYMBOL(__mark_inode_dirty);
@@ -226,7 +224,7 @@ static void requeue_io(struct inode *ino
static void inode_sync_complete(struct inode *inode)
{
/*
- * Prevent speculative execution through spin_unlock(&inode_lock);
+ * Prevent speculative execution through spin_unlock(&inode->i_lock);
*/
smp_mb();
wake_up_bit(&inode->i_state, __I_SYNC);
@@ -295,9 +293,7 @@ static void inode_wait_for_writeback(str
do {
spin_unlock(&wb_inode_list_lock);
spin_unlock(&inode->i_lock);
- spin_unlock(&inode_lock);
__wait_on_bit(wqh, &wq, inode_wait, TASK_UNINTERRUPTIBLE);
- spin_lock(&inode_lock);
spin_lock(&inode->i_lock);
spin_lock(&wb_inode_list_lock);
} while (inode->i_state & I_SYNC);
@@ -358,7 +354,6 @@ writeback_single_inode(struct inode *ino
spin_unlock(&wb_inode_list_lock);
spin_unlock(&inode->i_lock);
- spin_unlock(&inode_lock);
ret = do_writepages(mapping, wbc);
@@ -375,7 +370,6 @@ writeback_single_inode(struct inode *ino
ret = err;
}
- spin_lock(&inode_lock);
spin_lock(&inode->i_lock);
spin_lock(&wb_inode_list_lock);
inode->i_state &= ~I_SYNC;
@@ -478,7 +472,6 @@ void generic_sync_sb_inodes(struct super
const unsigned long start = jiffies; /* livelock avoidance */
int sync = wbc->sync_mode == WB_SYNC_ALL;
- spin_lock(&inode_lock);
again:
spin_lock(&wb_inode_list_lock);
if (!wbc->for_kupdate || list_empty(&sb->s_io))
@@ -572,10 +565,8 @@ again:
}
spin_unlock(&wb_inode_list_lock);
spin_unlock(&inode->i_lock);
- spin_unlock(&inode_lock);
iput(inode);
cond_resched();
- spin_lock(&inode_lock);
spin_lock(&wb_inode_list_lock);
if (wbc->nr_to_write <= 0) {
wbc->more_io = 1;
@@ -610,7 +601,6 @@ again:
__iget(inode);
spin_unlock(&inode->i_lock);
spin_unlock(&sb_inode_list_lock);
- spin_unlock(&inode_lock);
/*
* We hold a reference to 'inode' so it couldn't have
* been removed from s_inodes list while we dropped the
@@ -626,14 +616,11 @@ again:
cond_resched();
- spin_lock(&inode_lock);
spin_lock(&sb_inode_list_lock);
}
spin_unlock(&sb_inode_list_lock);
- spin_unlock(&inode_lock);
iput(old_inode);
- } else
- spin_unlock(&inode_lock);
+ }
return; /* Leave any unwritten inodes on s_io */
}
@@ -752,13 +739,11 @@ int write_inode_now(struct inode *inode,
wbc.nr_to_write = 0;
might_sleep();
- spin_lock(&inode_lock);
spin_lock(&inode->i_lock);
spin_lock(&wb_inode_list_lock);
ret = writeback_single_inode(inode, &wbc);
spin_unlock(&wb_inode_list_lock);
spin_unlock(&inode->i_lock);
- spin_unlock(&inode_lock);
if (sync)
inode_sync_wait(inode);
return ret;
@@ -780,13 +765,11 @@ int sync_inode(struct inode *inode, stru
{
int ret;
- spin_lock(&inode_lock);
spin_lock(&inode->i_lock);
spin_lock(&wb_inode_list_lock);
ret = writeback_single_inode(inode, wbc);
spin_unlock(&wb_inode_list_lock);
spin_unlock(&inode->i_lock);
- spin_unlock(&inode_lock);
return ret;
}
EXPORT_SYMBOL(sync_inode);
@@ -827,13 +810,11 @@ int generic_osync_inode(struct inode *in
err = err2;
}
- spin_lock(&inode_lock);
spin_lock(&inode->i_lock);
if ((inode->i_state & I_DIRTY) &&
((what & OSYNC_INODE) || (inode->i_state & I_DIRTY_DATASYNC)))
need_write_inode_now = 1;
spin_unlock(&inode->i_lock);
- spin_unlock(&inode_lock);
if (need_write_inode_now) {
err2 = write_inode_now(inode, 1);
Index: linux-2.6/fs/hugetlbfs/inode.c
===================================================================
--- linux-2.6.orig/fs/hugetlbfs/inode.c
+++ linux-2.6/fs/hugetlbfs/inode.c
@@ -379,7 +379,7 @@ static void hugetlbfs_delete_inode(struc
clear_inode(inode);
}
-static void hugetlbfs_forget_inode(struct inode *inode) __releases(inode_lock)
+static void hugetlbfs_forget_inode(struct inode *inode)
{
struct super_block *sb = inode->i_sb;
@@ -393,20 +393,17 @@ static void hugetlbfs_forget_inode(struc
if (!sb || (sb->s_flags & MS_ACTIVE)) {
spin_unlock(&inode->i_lock);
spin_unlock(&sb_inode_list_lock);
- spin_unlock(&inode_lock);
return;
}
WARN_ON(inode->i_state & I_NEW);
inode->i_state |= I_WILL_FREE;
spin_unlock(&inode->i_lock);
spin_unlock(&sb_inode_list_lock);
- spin_unlock(&inode_lock);
/*
* write_inode_now is a noop as we set BDI_CAP_NO_WRITEBACK
* in our backing_dev_info.
*/
write_inode_now(inode, 1);
- spin_lock(&inode_lock);
spin_lock(&sb_inode_list_lock);
spin_lock(&inode->i_lock);
WARN_ON(inode->i_state & I_NEW);
@@ -424,7 +421,6 @@ static void hugetlbfs_forget_inode(struc
WARN_ON(inode->i_state & I_NEW);
inode->i_state |= I_FREEING;
spin_unlock(&inode->i_lock);
- spin_unlock(&inode_lock);
atomic_dec(&inodes_stat.nr_unused);
truncate_hugepages(inode, 0);
clear_inode(inode);
Index: linux-2.6/fs/inode.c
===================================================================
--- linux-2.6.orig/fs/inode.c
+++ linux-2.6/fs/inode.c
@@ -84,7 +84,6 @@ static struct hlist_head *inode_hashtabl
* NOTE! You also have to own the lock if you change
* the i_state of an inode while it is in use..
*/
-DEFINE_SPINLOCK(inode_lock);
DEFINE_SPINLOCK(sb_inode_list_lock);
DEFINE_SPINLOCK(wb_inode_list_lock);
DEFINE_SPINLOCK(inode_hash_lock);
@@ -362,16 +361,14 @@ static void dispose_list(struct list_hea
truncate_inode_pages(&inode->i_data, 0);
clear_inode(inode);
- spin_lock(&inode_lock);
spin_lock(&sb_inode_list_lock);
spin_lock(&inode->i_lock);
spin_lock(&inode_hash_lock);
hlist_del_init(&inode->i_hash);
spin_unlock(&inode_hash_lock);
list_del_init(&inode->i_sb_list);
- spin_unlock(&sb_inode_list_lock);
spin_unlock(&inode->i_lock);
- spin_unlock(&inode_lock);
+ spin_unlock(&sb_inode_list_lock);
wake_up_inode(inode);
destroy_inode(inode);
@@ -399,7 +396,6 @@ static int invalidate_list(struct list_h
* change during umount anymore, and because iprune_mutex keeps
* shrink_icache_memory() away.
*/
- cond_resched_lock(&inode_lock);
cond_resched_lock(&sb_inode_list_lock);
next = next->next;
@@ -444,13 +440,11 @@ int invalidate_inodes(struct super_block
LIST_HEAD(throw_away);
mutex_lock(&iprune_mutex);
- spin_lock(&inode_lock);
spin_lock(&sb_inode_list_lock);
inotify_unmount_inodes(&sb->s_inodes);
fsnotify_unmount_inodes(&sb->s_inodes);
busy = invalidate_list(&sb->s_inodes, &throw_away);
spin_unlock(&sb_inode_list_lock);
- spin_unlock(&inode_lock);
dispose_list(&throw_away);
mutex_unlock(&iprune_mutex);
@@ -493,7 +487,6 @@ static void prune_icache(int nr_to_scan)
unsigned long reap = 0;
mutex_lock(&iprune_mutex);
- spin_lock(&inode_lock);
again:
spin_lock(&wb_inode_list_lock);
for (nr_scanned = 0; nr_scanned < nr_to_scan; nr_scanned++) {
@@ -517,12 +510,10 @@ again:
spin_unlock(&wb_inode_list_lock);
__iget(inode);
spin_unlock(&inode->i_lock);
- spin_unlock(&inode_lock);
if (remove_inode_buffers(inode))
reap += invalidate_mapping_pages(&inode->i_data,
0, -1);
iput(inode);
- spin_lock(&inode_lock);
again2:
spin_lock(&wb_inode_list_lock);
@@ -549,7 +540,6 @@ again2:
__count_vm_events(KSWAPD_INODESTEAL, reap);
else
__count_vm_events(PGINODESTEAL, reap);
- spin_unlock(&inode_lock);
spin_unlock(&wb_inode_list_lock);
dispose_list(&freeable);
@@ -700,12 +690,10 @@ void inode_add_to_lists(struct super_blo
{
struct hlist_head *head = inode_hashtable + hash(sb, inode->i_ino);
- spin_lock(&inode_lock);
spin_lock(&sb_inode_list_lock);
spin_lock(&inode->i_lock);
__inode_add_to_lists(sb, head, inode);
spin_unlock(&inode->i_lock);
- spin_unlock(&inode_lock);
}
EXPORT_SYMBOL_GPL(inode_add_to_lists);
@@ -731,18 +719,14 @@ struct inode *new_inode(struct super_blo
static atomic_t last_ino = ATOMIC_INIT(0);
struct inode *inode;
- spin_lock_prefetch(&inode_lock);
-
inode = alloc_inode(sb);
if (inode) {
- spin_lock(&inode_lock);
spin_lock(&sb_inode_list_lock);
spin_lock(&inode->i_lock);
inode->i_ino = atomic_inc_return(&last_ino);
inode->i_state = 0;
__inode_add_to_lists(sb, NULL, inode);
spin_unlock(&inode->i_lock);
- spin_unlock(&inode_lock);
}
return inode;
}
@@ -799,7 +783,6 @@ static struct inode *get_new_inode(struc
if (inode) {
struct inode *old;
- spin_lock(&inode_lock);
/* We released the lock, so.. */
old = find_inode(sb, head, test, data);
if (!old) {
@@ -811,7 +794,6 @@ static struct inode *get_new_inode(struc
inode->i_state = I_LOCK|I_NEW;
__inode_add_to_lists(sb, head, inode);
spin_unlock(&inode->i_lock);
- spin_unlock(&inode_lock);
/* Return the locked inode with I_NEW set, the
* caller is responsible for filling in the contents
@@ -826,7 +808,6 @@ static struct inode *get_new_inode(struc
*/
__iget(old);
spin_unlock(&old->i_lock);
- spin_unlock(&inode_lock);
destroy_inode(inode);
inode = old;
wait_on_inode(inode);
@@ -836,7 +817,6 @@ static struct inode *get_new_inode(struc
set_failed:
spin_unlock(&inode->i_lock);
spin_unlock(&sb_inode_list_lock);
- spin_unlock(&inode_lock);
destroy_inode(inode);
return NULL;
}
@@ -854,7 +834,6 @@ static struct inode *get_new_inode_fast(
if (inode) {
struct inode *old;
- spin_lock(&inode_lock);
/* We released the lock, so.. */
old = find_inode_fast(sb, head, ino);
if (!old) {
@@ -864,7 +843,6 @@ static struct inode *get_new_inode_fast(
inode->i_state = I_LOCK|I_NEW;
__inode_add_to_lists(sb, head, inode);
spin_unlock(&inode->i_lock);
- spin_unlock(&inode_lock);
/* Return the locked inode with I_NEW set, the
* caller is responsible for filling in the contents
@@ -879,7 +857,6 @@ static struct inode *get_new_inode_fast(
*/
__iget(old);
spin_unlock(&old->i_lock);
- spin_unlock(&inode_lock);
destroy_inode(inode);
inode = old;
wait_on_inode(inode);
@@ -929,7 +906,6 @@ ino_t iunique(struct super_block *sb, in
struct hlist_head *head;
ino_t res;
- spin_lock(&inode_lock);
spin_lock(&unique_lock);
do {
if (counter <= max_reserved)
@@ -938,7 +914,6 @@ ino_t iunique(struct super_block *sb, in
head = inode_hashtable + hash(sb, res);
} while (!test_inode_iunique(sb, head, res));
spin_unlock(&unique_lock);
- spin_unlock(&inode_lock);
return res;
}
@@ -948,7 +923,6 @@ struct inode *igrab(struct inode *inode)
{
struct inode *ret = inode;
- spin_lock(&inode_lock);
spin_lock(&inode->i_lock);
if (!(inode->i_state & (I_FREEING|I_CLEAR|I_WILL_FREE)))
__iget(inode);
@@ -960,7 +934,6 @@ struct inode *igrab(struct inode *inode)
*/
ret = NULL;
spin_unlock(&inode->i_lock);
- spin_unlock(&inode_lock);
return ret;
}
@@ -991,17 +964,14 @@ static struct inode *ifind(struct super_
{
struct inode *inode;
- spin_lock(&inode_lock);
inode = find_inode(sb, head, test, data);
if (inode) {
__iget(inode);
spin_unlock(&inode->i_lock);
- spin_unlock(&inode_lock);
if (likely(wait))
wait_on_inode(inode);
return inode;
}
- spin_unlock(&inode_lock);
return NULL;
}
@@ -1025,16 +995,13 @@ static struct inode *ifind_fast(struct s
{
struct inode *inode;
- spin_lock(&inode_lock);
inode = find_inode_fast(sb, head, ino);
if (inode) {
__iget(inode);
spin_unlock(&inode->i_lock);
- spin_unlock(&inode_lock);
wait_on_inode(inode);
return inode;
}
- spin_unlock(&inode_lock);
return NULL;
}
@@ -1198,7 +1165,6 @@ int insert_inode_locked(struct inode *in
struct hlist_node *node;
struct inode *old = NULL;
- spin_lock(&inode_lock);
repeat:
spin_lock(&inode_hash_lock);
hlist_for_each_entry(old, node, head, i_hash) {
@@ -1218,13 +1184,11 @@ repeat:
/* XXX: initialize inode->i_lock to locked? */
hlist_add_head(&inode->i_hash, head);
spin_unlock(&inode_hash_lock);
- spin_unlock(&inode_lock);
return 0;
}
spin_unlock(&inode_hash_lock);
__iget(old);
spin_unlock(&old->i_lock);
- spin_unlock(&inode_lock);
wait_on_inode(old);
if (unlikely(!hlist_unhashed(&old->i_hash))) {
iput(old);
@@ -1247,7 +1211,6 @@ int insert_inode_locked4(struct inode *i
struct hlist_node *node;
struct inode *old = NULL;
- spin_lock(&inode_lock);
repeat:
spin_lock(&inode_hash_lock);
hlist_for_each_entry(old, node, head, i_hash) {
@@ -1267,13 +1230,11 @@ repeat:
/* XXX: initialize inode->i_lock to locked? */
hlist_add_head(&inode->i_hash, head);
spin_unlock(&inode_hash_lock);
- spin_unlock(&inode_lock);
return 0;
}
spin_unlock(&inode_hash_lock);
__iget(old);
spin_unlock(&old->i_lock);
- spin_unlock(&inode_lock);
wait_on_inode(old);
if (unlikely(!hlist_unhashed(&old->i_hash))) {
iput(old);
@@ -1296,13 +1257,11 @@ void __insert_inode_hash(struct inode *i
{
struct hlist_head *head = inode_hashtable + hash(inode->i_sb, hashval);
- spin_lock(&inode_lock);
spin_lock(&inode->i_lock);
spin_lock(&inode_hash_lock);
hlist_add_head(&inode->i_hash, head);
spin_unlock(&inode_hash_lock);
spin_unlock(&inode->i_lock);
- spin_unlock(&inode_lock);
}
EXPORT_SYMBOL(__insert_inode_hash);
@@ -1314,13 +1273,11 @@ EXPORT_SYMBOL(__insert_inode_hash);
*/
void remove_inode_hash(struct inode *inode)
{
- spin_lock(&inode_lock);
spin_lock(&inode->i_lock);
spin_lock(&inode_hash_lock);
hlist_del_init(&inode->i_hash);
spin_unlock(&inode_hash_lock);
spin_unlock(&inode->i_lock);
- spin_unlock(&inode_lock);
}
EXPORT_SYMBOL(remove_inode_hash);
@@ -1348,7 +1305,6 @@ void generic_delete_inode(struct inode *
WARN_ON(inode->i_state & I_NEW);
inode->i_state |= I_FREEING;
spin_unlock(&inode->i_lock);
- spin_unlock(&inode_lock);
atomic_dec(&inodes_stat.nr_inodes);
security_inode_delete(inode);
@@ -1366,13 +1322,11 @@ void generic_delete_inode(struct inode *
truncate_inode_pages(&inode->i_data, 0);
clear_inode(inode);
}
- spin_lock(&inode_lock);
spin_lock(&inode->i_lock);
spin_lock(&inode_hash_lock);
hlist_del_init(&inode->i_hash);
spin_unlock(&inode_hash_lock);
spin_unlock(&inode->i_lock);
- spin_unlock(&inode_lock);
wake_up_inode(inode);
BUG_ON(inode->i_state != I_CLEAR);
destroy_inode(inode);
@@ -1393,16 +1347,13 @@ static void generic_forget_inode(struct
if (sb->s_flags & MS_ACTIVE) {
spin_unlock(&inode->i_lock);
spin_unlock(&sb_inode_list_lock);
- spin_unlock(&inode_lock);
return;
}
WARN_ON(inode->i_state & I_NEW);
inode->i_state |= I_WILL_FREE;
spin_unlock(&inode->i_lock);
spin_unlock(&sb_inode_list_lock);
- spin_unlock(&inode_lock);
write_inode_now(inode, 1);
- spin_lock(&inode_lock);
spin_lock(&sb_inode_list_lock);
spin_lock(&inode->i_lock);
WARN_ON(inode->i_state & I_NEW);
@@ -1420,7 +1371,6 @@ static void generic_forget_inode(struct
WARN_ON(inode->i_state & I_NEW);
inode->i_state |= I_FREEING;
spin_unlock(&inode->i_lock);
- spin_unlock(&inode_lock);
atomic_dec(&inodes_stat.nr_inodes);
if (inode->i_data.nrpages)
truncate_inode_pages(&inode->i_data, 0);
@@ -1478,17 +1428,12 @@ void iput(struct inode *inode)
if (inode) {
BUG_ON(inode->i_state == I_CLEAR);
-retry1:
+retry:
spin_lock(&inode->i_lock);
if (inode->i_count == 1) {
- if (!spin_trylock(&inode_lock)) {
-retry2:
- spin_unlock(&inode->i_lock);
- goto retry1;
- }
if (!spin_trylock(&sb_inode_list_lock)) {
- spin_unlock(&inode_lock);
- goto retry2;
+ spin_unlock(&inode->i_lock);
+ goto retry;
}
inode->i_count--;
iput_final(inode);
@@ -1683,10 +1628,8 @@ static void __wait_on_freeing_inode(stru
wq = bit_waitqueue(&inode->i_state, __I_LOCK);
prepare_to_wait(wq, &wait.wait, TASK_UNINTERRUPTIBLE);
spin_unlock(&inode->i_lock);
- spin_unlock(&inode_lock);
schedule();
finish_wait(wq, &wait.wait);
- spin_lock(&inode_lock);
}
static __initdata unsigned long ihash_entries;
Index: linux-2.6/fs/notify/inotify/inotify.c
===================================================================
--- linux-2.6.orig/fs/notify/inotify/inotify.c
+++ linux-2.6/fs/notify/inotify/inotify.c
@@ -392,13 +392,16 @@ void inotify_unmount_inodes(struct list_
struct inode *need_iput_tmp;
struct list_head *watches;
+ spin_lock(&inode->i_lock);
/*
* We cannot __iget() an inode in state I_CLEAR, I_FREEING,
* I_WILL_FREE, or I_NEW which is fine because by that point
* the inode cannot have any associated watches.
*/
- if (inode->i_state & (I_CLEAR|I_FREEING|I_WILL_FREE|I_NEW))
+ if (inode->i_state & (I_CLEAR|I_FREEING|I_WILL_FREE|I_NEW)) {
+ spin_unlock(&inode->i_lock);
continue;
+ }
/*
* If i_count is zero, the inode cannot have any watches and
@@ -406,18 +409,21 @@ void inotify_unmount_inodes(struct list_
* evict all inodes with zero i_count from icache which is
* unnecessarily violent and may in fact be illegal to do.
*/
- if (!inode->i_count)
+ if (!inode->i_count) {
+ spin_unlock(&inode->i_lock);
continue;
+ }
need_iput_tmp = need_iput;
need_iput = NULL;
/* In case inotify_remove_watch_locked() drops a reference. */
if (inode != need_iput_tmp) {
- spin_lock(&inode->i_lock);
__iget(inode);
- spin_unlock(&inode->i_lock);
} else
need_iput_tmp = NULL;
+
+ spin_unlock(&inode->i_lock);
+
/* In case the dropping of a reference would nuke next_i. */
if (&next_i->i_sb_list != list) {
spin_lock(&next_i->i_lock);
@@ -437,7 +443,6 @@ void inotify_unmount_inodes(struct list_
* iprune_mutex keeps shrink_icache_memory() away.
*/
spin_unlock(&sb_inode_list_lock);
- spin_unlock(&inode_lock);
if (need_iput_tmp)
iput(need_iput_tmp);
@@ -457,7 +462,6 @@ void inotify_unmount_inodes(struct list_
mutex_unlock(&inode->inotify_mutex);
iput(inode);
- spin_lock(&inode_lock);
spin_lock(&sb_inode_list_lock);
}
}
Index: linux-2.6/include/linux/writeback.h
===================================================================
--- linux-2.6.orig/include/linux/writeback.h
+++ linux-2.6/include/linux/writeback.h
@@ -9,7 +9,6 @@
struct backing_dev_info;
-extern spinlock_t inode_lock;
extern spinlock_t sb_inode_list_lock;
extern spinlock_t wb_inode_list_lock;
extern spinlock_t inode_hash_lock;
Index: linux-2.6/fs/quota/dquot.c
===================================================================
--- linux-2.6.orig/fs/quota/dquot.c
+++ linux-2.6/fs/quota/dquot.c
@@ -821,7 +821,6 @@ static void add_dquot_ref(struct super_b
{
struct inode *inode, *old_inode = NULL;
- spin_lock(&inode_lock);
spin_lock(&sb_inode_list_lock);
list_for_each_entry(inode, &sb->s_inodes, i_sb_list) {
spin_lock(&inode->i_lock);
@@ -841,7 +840,6 @@ static void add_dquot_ref(struct super_b
__iget(inode);
spin_unlock(&inode->i_lock);
spin_unlock(&sb_inode_list_lock);
- spin_unlock(&inode_lock);
iput(old_inode);
sb->dq_op->initialize(inode, type);
@@ -851,11 +849,9 @@ static void add_dquot_ref(struct super_b
* reference and we cannot iput it under inode_lock. So we
* keep the reference and iput it later. */
old_inode = inode;
- spin_lock(&inode_lock);
spin_lock(&sb_inode_list_lock);
}
spin_unlock(&sb_inode_list_lock);
- spin_unlock(&inode_lock);
iput(old_inode);
}
@@ -925,7 +921,6 @@ static void remove_dquot_ref(struct supe
{
struct inode *inode;
- spin_lock(&inode_lock);
spin_lock(&sb_inode_list_lock);
list_for_each_entry(inode, &sb->s_inodes, i_sb_list) {
/*
@@ -938,7 +933,6 @@ static void remove_dquot_ref(struct supe
remove_inode_dquot_ref(inode, type, tofree_head);
}
spin_unlock(&sb_inode_list_lock);
- spin_unlock(&inode_lock);
}
/* Gather all references from inodes and drop them */
Index: linux-2.6/fs/notify/inode_mark.c
===================================================================
--- linux-2.6.orig/fs/notify/inode_mark.c
+++ linux-2.6/fs/notify/inode_mark.c
@@ -369,13 +369,16 @@ void fsnotify_unmount_inodes(struct list
list_for_each_entry_safe(inode, next_i, list, i_sb_list) {
struct inode *need_iput_tmp;
+ spin_lock(&inode->i_lock);
/*
* We cannot __iget() an inode in state I_CLEAR, I_FREEING,
* I_WILL_FREE, or I_NEW which is fine because by that point
* the inode cannot have any associated watches.
*/
- if (inode->i_state & (I_CLEAR|I_FREEING|I_WILL_FREE|I_NEW))
+ if (inode->i_state & (I_CLEAR|I_FREEING|I_WILL_FREE|I_NEW)) {
+ spin_unlock(&inode->i_lock);
continue;
+ }
/*
* If i_count is zero, the inode cannot have any watches and
@@ -383,19 +386,20 @@ void fsnotify_unmount_inodes(struct list
* evict all inodes with zero i_count from icache which is
* unnecessarily violent and may in fact be illegal to do.
*/
- if (!inode->i_count)
+ if (!inode->i_count) {
+ spin_unlock(&inode->i_lock);
continue;
+ }
need_iput_tmp = need_iput;
need_iput = NULL;
/* In case fsnotify_inode_delete() drops a reference. */
if (inode != need_iput_tmp) {
- spin_lock(&inode->i_lock);
__iget(inode);
- spin_unlock(&inode->i_lock);
} else
need_iput_tmp = NULL;
+ spin_unlock(&inode->i_lock);
/* In case the dropping of a reference would nuke next_i. */
if (&next_i->i_sb_list != list) {
@@ -416,7 +420,6 @@ void fsnotify_unmount_inodes(struct list
* iprune_mutex keeps shrink_icache_memory() away.
*/
spin_unlock(&sb_inode_list_lock);
- spin_unlock(&inode_lock);
if (need_iput_tmp)
iput(need_iput_tmp);
@@ -428,7 +431,6 @@ void fsnotify_unmount_inodes(struct list
iput(inode);
- spin_lock(&inode_lock);
spin_lock(&sb_inode_list_lock);
}
}
^ permalink raw reply [flat|nested] 64+ messages in thread
* [patch 28/33] fs: inode factor hash lock into functions
2009-09-04 6:51 [patch 00/33] my current vfs scalability patch queue npiggin
` (26 preceding siblings ...)
2009-09-04 6:52 ` [patch 27/33] fs: icache remove inode_lock npiggin
@ 2009-09-04 6:52 ` npiggin
2009-09-04 6:52 ` [patch 29/33] Remove the global inode_hash_lock and replace it with per-hash-bucket locks. fs: inode per-bucket inode hash locks npiggin
` (5 subsequent siblings)
33 siblings, 0 replies; 64+ messages in thread
From: npiggin @ 2009-09-04 6:52 UTC (permalink / raw)
To: linux-fsdevel; +Cc: linux-kernel
[-- Attachment #1: fs-inode_lock-scale-8.patch --]
[-- Type: text/plain, Size: 4305 bytes --]
Make inode_hash_lock private by adding a function __remove_inode_hash
that can be used by filesystems defining their own drop_inode functions.
---
fs/hugetlbfs/inode.c | 4 +---
fs/inode.c | 32 +++++++++++++++++++-------------
include/linux/fs.h | 1 +
include/linux/writeback.h | 1 -
4 files changed, 21 insertions(+), 17 deletions(-)
Index: linux-2.6/fs/inode.c
===================================================================
--- linux-2.6.orig/fs/inode.c
+++ linux-2.6/fs/inode.c
@@ -86,7 +86,7 @@ static struct hlist_head *inode_hashtabl
*/
DEFINE_SPINLOCK(sb_inode_list_lock);
DEFINE_SPINLOCK(wb_inode_list_lock);
-DEFINE_SPINLOCK(inode_hash_lock);
+static DEFINE_SPINLOCK(inode_hash_lock);
/*
* iprune_mutex provides exclusion between the kswapd or try_to_free_pages
@@ -363,9 +363,7 @@ static void dispose_list(struct list_hea
spin_lock(&sb_inode_list_lock);
spin_lock(&inode->i_lock);
- spin_lock(&inode_hash_lock);
- hlist_del_init(&inode->i_hash);
- spin_unlock(&inode_hash_lock);
+ __remove_inode_hash(inode);
list_del_init(&inode->i_sb_list);
spin_unlock(&inode->i_lock);
spin_unlock(&sb_inode_list_lock);
@@ -1266,6 +1264,20 @@ void __insert_inode_hash(struct inode *i
EXPORT_SYMBOL(__insert_inode_hash);
/**
+ * __remove_inode_hash - remove an inode from the hash
+ * @inode: inode to unhash
+ *
+ * Remove an inode from the superblock. inode->i_lock must be
+ * held.
+ */
+void __remove_inode_hash(struct inode *inode)
+{
+ spin_lock(&inode_hash_lock);
+ hlist_del_init(&inode->i_hash);
+ spin_unlock(&inode_hash_lock);
+}
+
+/**
* remove_inode_hash - remove an inode from the hash
* @inode: inode to unhash
*
@@ -1274,9 +1286,7 @@ EXPORT_SYMBOL(__insert_inode_hash);
void remove_inode_hash(struct inode *inode)
{
spin_lock(&inode->i_lock);
- spin_lock(&inode_hash_lock);
- hlist_del_init(&inode->i_hash);
- spin_unlock(&inode_hash_lock);
+ __remove_inode_hash(inode);
spin_unlock(&inode->i_lock);
}
EXPORT_SYMBOL(remove_inode_hash);
@@ -1323,9 +1333,7 @@ void generic_delete_inode(struct inode *
clear_inode(inode);
}
spin_lock(&inode->i_lock);
- spin_lock(&inode_hash_lock);
- hlist_del_init(&inode->i_hash);
- spin_unlock(&inode_hash_lock);
+ __remove_inode_hash(inode);
spin_unlock(&inode->i_lock);
wake_up_inode(inode);
BUG_ON(inode->i_state != I_CLEAR);
@@ -1358,9 +1366,7 @@ static void generic_forget_inode(struct
spin_lock(&inode->i_lock);
WARN_ON(inode->i_state & I_NEW);
inode->i_state &= ~I_WILL_FREE;
- spin_lock(&inode_hash_lock);
- hlist_del_init(&inode->i_hash);
- spin_unlock(&inode_hash_lock);
+ __remove_inode_hash(inode);
atomic_dec(&inodes_stat.nr_unused);
}
spin_lock(&wb_inode_list_lock);
Index: linux-2.6/fs/hugetlbfs/inode.c
===================================================================
--- linux-2.6.orig/fs/hugetlbfs/inode.c
+++ linux-2.6/fs/hugetlbfs/inode.c
@@ -408,9 +408,7 @@ static void hugetlbfs_forget_inode(struc
spin_lock(&inode->i_lock);
WARN_ON(inode->i_state & I_NEW);
inode->i_state &= ~I_WILL_FREE;
- spin_lock(&inode_hash_lock);
- hlist_del_init(&inode->i_hash);
- spin_unlock(&inode_hash_lock);
+ __remove_inode_hash(inode);
atomic_dec(&inodes_stat.nr_unused);
}
spin_lock(&wb_inode_list_lock);
Index: linux-2.6/include/linux/fs.h
===================================================================
--- linux-2.6.orig/include/linux/fs.h
+++ linux-2.6/include/linux/fs.h
@@ -2176,6 +2176,7 @@ extern int should_remove_suid(struct den
extern int file_remove_suid(struct file *);
extern void __insert_inode_hash(struct inode *, unsigned long hashval);
+extern void __remove_inode_hash(struct inode *);
extern void remove_inode_hash(struct inode *);
static inline void insert_inode_hash(struct inode *inode) {
__insert_inode_hash(inode, inode->i_ino);
Index: linux-2.6/include/linux/writeback.h
===================================================================
--- linux-2.6.orig/include/linux/writeback.h
+++ linux-2.6/include/linux/writeback.h
@@ -11,7 +11,6 @@ struct backing_dev_info;
extern spinlock_t sb_inode_list_lock;
extern spinlock_t wb_inode_list_lock;
-extern spinlock_t inode_hash_lock;
extern struct list_head inode_in_use;
extern struct list_head inode_unused;
^ permalink raw reply [flat|nested] 64+ messages in thread
* [patch 29/33] Remove the global inode_hash_lock and replace it with per-hash-bucket locks. fs: inode per-bucket inode hash locks
2009-09-04 6:51 [patch 00/33] my current vfs scalability patch queue npiggin
` (27 preceding siblings ...)
2009-09-04 6:52 ` [patch 28/33] fs: inode factor hash lock into functions npiggin
@ 2009-09-04 6:52 ` npiggin
2009-09-04 7:05 ` Nick Piggin
2009-09-04 6:52 ` [patch 30/33] fs: inode lazy lru npiggin
` (4 subsequent siblings)
33 siblings, 1 reply; 64+ messages in thread
From: npiggin @ 2009-09-04 6:52 UTC (permalink / raw)
To: linux-fsdevel; +Cc: linux-kernel
[-- Attachment #1: fs-inode_lock-scale-9.patch --]
[-- Type: text/plain, Size: 15292 bytes --]
Todo: should use bit spinlock in hlist_head pointer to save space.
---
fs/inode.c | 167 ++++++++++++++++++++++++++++++++-----------------------------
1 file changed, 90 insertions(+), 77 deletions(-)
Index: linux-2.6/fs/inode.c
===================================================================
--- linux-2.6.orig/fs/inode.c
+++ linux-2.6/fs/inode.c
@@ -76,7 +76,12 @@ static unsigned int i_hash_shift __read_
LIST_HEAD(inode_in_use);
LIST_HEAD(inode_unused);
-static struct hlist_head *inode_hashtable __read_mostly;
+
+struct inode_hash_bucket {
+ spinlock_t lock;
+ struct hlist_head head;
+};
+static struct inode_hash_bucket *inode_hashtable __read_mostly;
/*
* A simple spinlock to protect the list manipulations.
@@ -86,7 +91,6 @@ static struct hlist_head *inode_hashtabl
*/
DEFINE_SPINLOCK(sb_inode_list_lock);
DEFINE_SPINLOCK(wb_inode_list_lock);
-static DEFINE_SPINLOCK(inode_hash_lock);
/*
* iprune_mutex provides exclusion between the kswapd or try_to_free_pages
@@ -582,7 +586,7 @@ static void __wait_on_freeing_inode(stru
* add any additional branch in the common code.
*/
static struct inode *find_inode(struct super_block *sb,
- struct hlist_head *head,
+ struct inode_hash_bucket *b,
int (*test)(struct inode *, void *),
void *data)
{
@@ -590,12 +594,12 @@ static struct inode *find_inode(struct s
struct inode *inode = NULL;
repeat:
- spin_lock(&inode_hash_lock);
- hlist_for_each_entry(inode, node, head, i_hash) {
+ spin_lock(&b->lock);
+ hlist_for_each_entry(inode, node, &b->head, i_hash) {
if (inode->i_sb != sb)
continue;
if (!spin_trylock(&inode->i_lock)) {
- spin_unlock(&inode_hash_lock);
+ spin_unlock(&b->lock);
goto repeat;
}
if (!test(inode, data)) {
@@ -603,13 +607,13 @@ repeat:
continue;
}
if (inode->i_state & (I_FREEING|I_CLEAR|I_WILL_FREE)) {
- spin_unlock(&inode_hash_lock);
+ spin_unlock(&b->lock);
__wait_on_freeing_inode(inode);
goto repeat;
}
break;
}
- spin_unlock(&inode_hash_lock);
+ spin_unlock(&b->lock);
return node ? inode : NULL;
}
@@ -618,30 +622,31 @@ repeat:
* iget_locked for details.
*/
static struct inode *find_inode_fast(struct super_block *sb,
- struct hlist_head *head, unsigned long ino)
+ struct inode_hash_bucket *b,
+ unsigned long ino)
{
struct hlist_node *node;
struct inode *inode = NULL;
repeat:
- spin_lock(&inode_hash_lock);
- hlist_for_each_entry(inode, node, head, i_hash) {
+ spin_lock(&b->lock);
+ hlist_for_each_entry(inode, node, &b->head, i_hash) {
if (inode->i_ino != ino)
continue;
if (inode->i_sb != sb)
continue;
if (!spin_trylock(&inode->i_lock)) {
- spin_unlock(&inode_hash_lock);
+ spin_unlock(&b->lock);
goto repeat;
}
if (inode->i_state & (I_FREEING|I_CLEAR|I_WILL_FREE)) {
- spin_unlock(&inode_hash_lock);
+ spin_unlock(&b->lock);
__wait_on_freeing_inode(inode);
goto repeat;
}
break;
}
- spin_unlock(&inode_hash_lock);
+ spin_unlock(&b->lock);
return node ? inode : NULL;
}
@@ -656,7 +661,7 @@ static unsigned long hash(struct super_b
}
static inline void
-__inode_add_to_lists(struct super_block *sb, struct hlist_head *head,
+__inode_add_to_lists(struct super_block *sb, struct inode_hash_bucket *b,
struct inode *inode)
{
atomic_inc(&inodes_stat.nr_inodes);
@@ -665,10 +670,10 @@ __inode_add_to_lists(struct super_block
spin_lock(&wb_inode_list_lock);
list_add(&inode->i_list, &inode_in_use);
spin_unlock(&wb_inode_list_lock);
- if (head) {
- spin_lock(&inode_hash_lock);
- hlist_add_head(&inode->i_hash, head);
- spin_unlock(&inode_hash_lock);
+ if (b) {
+ spin_lock(&b->lock);
+ hlist_add_head(&inode->i_hash, &b->head);
+ spin_unlock(&b->lock);
}
}
@@ -686,11 +691,11 @@ __inode_add_to_lists(struct super_block
*/
void inode_add_to_lists(struct super_block *sb, struct inode *inode)
{
- struct hlist_head *head = inode_hashtable + hash(sb, inode->i_ino);
+ struct inode_hash_bucket *b = inode_hashtable + hash(sb, inode->i_ino);
spin_lock(&sb_inode_list_lock);
spin_lock(&inode->i_lock);
- __inode_add_to_lists(sb, head, inode);
+ __inode_add_to_lists(sb, b, inode);
spin_unlock(&inode->i_lock);
}
EXPORT_SYMBOL_GPL(inode_add_to_lists);
@@ -770,7 +775,7 @@ EXPORT_SYMBOL(unlock_new_inode);
* -- rmk@arm.uk.linux.org
*/
static struct inode *get_new_inode(struct super_block *sb,
- struct hlist_head *head,
+ struct inode_hash_bucket *b,
int (*test)(struct inode *, void *),
int (*set)(struct inode *, void *),
void *data)
@@ -782,7 +787,7 @@ static struct inode *get_new_inode(struc
struct inode *old;
/* We released the lock, so.. */
- old = find_inode(sb, head, test, data);
+ old = find_inode(sb, b, test, data);
if (!old) {
spin_lock(&sb_inode_list_lock);
spin_lock(&inode->i_lock);
@@ -790,7 +795,7 @@ static struct inode *get_new_inode(struc
goto set_failed;
inode->i_state = I_LOCK|I_NEW;
- __inode_add_to_lists(sb, head, inode);
+ __inode_add_to_lists(sb, b, inode);
spin_unlock(&inode->i_lock);
/* Return the locked inode with I_NEW set, the
@@ -824,7 +829,7 @@ set_failed:
* comment at iget_locked for details.
*/
static struct inode *get_new_inode_fast(struct super_block *sb,
- struct hlist_head *head, unsigned long ino)
+ struct inode_hash_bucket *b, unsigned long ino)
{
struct inode *inode;
@@ -833,13 +838,13 @@ static struct inode *get_new_inode_fast(
struct inode *old;
/* We released the lock, so.. */
- old = find_inode_fast(sb, head, ino);
+ old = find_inode_fast(sb, b, ino);
if (!old) {
spin_lock(&sb_inode_list_lock);
spin_lock(&inode->i_lock);
inode->i_ino = ino;
inode->i_state = I_LOCK|I_NEW;
- __inode_add_to_lists(sb, head, inode);
+ __inode_add_to_lists(sb, b, inode);
spin_unlock(&inode->i_lock);
/* Return the locked inode with I_NEW set, the
@@ -862,19 +867,20 @@ static struct inode *get_new_inode_fast(
return inode;
}
-static int test_inode_iunique(struct super_block * sb, struct hlist_head *head, unsigned long ino)
+static int test_inode_iunique(struct super_block *sb,
+ struct inode_hash_bucket *b, unsigned long ino)
{
struct hlist_node *node;
- struct inode * inode = NULL;
+ struct inode *inode = NULL;
- spin_lock(&inode_hash_lock);
- hlist_for_each_entry(inode, node, head, i_hash) {
+ spin_lock(&b->lock);
+ hlist_for_each_entry(inode, node, &b->head, i_hash) {
if (inode->i_ino == ino && inode->i_sb == sb) {
- spin_unlock(&inode_hash_lock);
+ spin_unlock(&b->lock);
return 0;
}
}
- spin_unlock(&inode_hash_lock);
+ spin_unlock(&b->lock);
return 1;
}
@@ -901,7 +907,7 @@ ino_t iunique(struct super_block *sb, in
*/
static DEFINE_SPINLOCK(unique_lock);
static unsigned int counter;
- struct hlist_head *head;
+ struct inode_hash_bucket *b;
ino_t res;
spin_lock(&unique_lock);
@@ -909,8 +915,8 @@ ino_t iunique(struct super_block *sb, in
if (counter <= max_reserved)
counter = max_reserved + 1;
res = counter++;
- head = inode_hashtable + hash(sb, res);
- } while (!test_inode_iunique(sb, head, res));
+ b = inode_hashtable + hash(sb, res);
+ } while (!test_inode_iunique(sb, b, res));
spin_unlock(&unique_lock);
return res;
@@ -957,12 +963,13 @@ EXPORT_SYMBOL(igrab);
* Note, @test is called with the inode_lock held, so can't sleep.
*/
static struct inode *ifind(struct super_block *sb,
- struct hlist_head *head, int (*test)(struct inode *, void *),
+ struct inode_hash_bucket *b,
+ int (*test)(struct inode *, void *),
void *data, const int wait)
{
struct inode *inode;
- inode = find_inode(sb, head, test, data);
+ inode = find_inode(sb, b, test, data);
if (inode) {
__iget(inode);
spin_unlock(&inode->i_lock);
@@ -989,11 +996,12 @@ static struct inode *ifind(struct super_
* Otherwise NULL is returned.
*/
static struct inode *ifind_fast(struct super_block *sb,
- struct hlist_head *head, unsigned long ino)
+ struct inode_hash_bucket *b,
+ unsigned long ino)
{
struct inode *inode;
- inode = find_inode_fast(sb, head, ino);
+ inode = find_inode_fast(sb, b, ino);
if (inode) {
__iget(inode);
spin_unlock(&inode->i_lock);
@@ -1027,9 +1035,9 @@ static struct inode *ifind_fast(struct s
struct inode *ilookup5_nowait(struct super_block *sb, unsigned long hashval,
int (*test)(struct inode *, void *), void *data)
{
- struct hlist_head *head = inode_hashtable + hash(sb, hashval);
+ struct inode_hash_bucket *b = inode_hashtable + hash(sb, hashval);
- return ifind(sb, head, test, data, 0);
+ return ifind(sb, b, test, data, 0);
}
EXPORT_SYMBOL(ilookup5_nowait);
@@ -1055,9 +1063,9 @@ EXPORT_SYMBOL(ilookup5_nowait);
struct inode *ilookup5(struct super_block *sb, unsigned long hashval,
int (*test)(struct inode *, void *), void *data)
{
- struct hlist_head *head = inode_hashtable + hash(sb, hashval);
+ struct inode_hash_bucket *b = inode_hashtable + hash(sb, hashval);
- return ifind(sb, head, test, data, 1);
+ return ifind(sb, b, test, data, 1);
}
EXPORT_SYMBOL(ilookup5);
@@ -1077,9 +1085,9 @@ EXPORT_SYMBOL(ilookup5);
*/
struct inode *ilookup(struct super_block *sb, unsigned long ino)
{
- struct hlist_head *head = inode_hashtable + hash(sb, ino);
+ struct inode_hash_bucket *b = inode_hashtable + hash(sb, ino);
- return ifind_fast(sb, head, ino);
+ return ifind_fast(sb, b, ino);
}
EXPORT_SYMBOL(ilookup);
@@ -1107,17 +1115,17 @@ struct inode *iget5_locked(struct super_
int (*test)(struct inode *, void *),
int (*set)(struct inode *, void *), void *data)
{
- struct hlist_head *head = inode_hashtable + hash(sb, hashval);
+ struct inode_hash_bucket *b = inode_hashtable + hash(sb, hashval);
struct inode *inode;
- inode = ifind(sb, head, test, data, 1);
+ inode = ifind(sb, b, test, data, 1);
if (inode)
return inode;
/*
* get_new_inode() will do the right thing, re-trying the search
* in case it had to block at any point.
*/
- return get_new_inode(sb, head, test, set, data);
+ return get_new_inode(sb, b, test, set, data);
}
EXPORT_SYMBOL(iget5_locked);
@@ -1138,17 +1146,17 @@ EXPORT_SYMBOL(iget5_locked);
*/
struct inode *iget_locked(struct super_block *sb, unsigned long ino)
{
- struct hlist_head *head = inode_hashtable + hash(sb, ino);
+ struct inode_hash_bucket *b = inode_hashtable + hash(sb, ino);
struct inode *inode;
- inode = ifind_fast(sb, head, ino);
+ inode = ifind_fast(sb, b, ino);
if (inode)
return inode;
/*
* get_new_inode_fast() will do the right thing, re-trying the search
* in case it had to block at any point.
*/
- return get_new_inode_fast(sb, head, ino);
+ return get_new_inode_fast(sb, b, ino);
}
EXPORT_SYMBOL(iget_locked);
@@ -1156,7 +1164,7 @@ int insert_inode_locked(struct inode *in
{
struct super_block *sb = inode->i_sb;
ino_t ino = inode->i_ino;
- struct hlist_head *head = inode_hashtable + hash(sb, ino);
+ struct inode_hash_bucket *b = inode_hashtable + hash(sb, ino);
inode->i_state |= I_LOCK|I_NEW;
while (1) {
@@ -1164,8 +1172,8 @@ int insert_inode_locked(struct inode *in
struct inode *old = NULL;
repeat:
- spin_lock(&inode_hash_lock);
- hlist_for_each_entry(old, node, head, i_hash) {
+ spin_lock(&b->lock);
+ hlist_for_each_entry(old, node, &b->head, i_hash) {
if (old->i_ino != ino)
continue;
if (old->i_sb != sb)
@@ -1173,18 +1181,18 @@ repeat:
if (old->i_state & (I_FREEING|I_CLEAR|I_WILL_FREE))
continue;
if (!spin_trylock(&old->i_lock)) {
- spin_unlock(&inode_hash_lock);
+ spin_unlock(&b->lock);
goto repeat;
}
break;
}
if (likely(!node)) {
/* XXX: initialize inode->i_lock to locked? */
- hlist_add_head(&inode->i_hash, head);
- spin_unlock(&inode_hash_lock);
+ hlist_add_head(&inode->i_hash, &b->head);
+ spin_unlock(&b->lock);
return 0;
}
- spin_unlock(&inode_hash_lock);
+ spin_unlock(&b->lock);
__iget(old);
spin_unlock(&old->i_lock);
wait_on_inode(old);
@@ -1201,7 +1209,7 @@ int insert_inode_locked4(struct inode *i
int (*test)(struct inode *, void *), void *data)
{
struct super_block *sb = inode->i_sb;
- struct hlist_head *head = inode_hashtable + hash(sb, hashval);
+ struct inode_hash_bucket *b = inode_hashtable + hash(sb, hashval);
inode->i_state |= I_LOCK|I_NEW;
@@ -1210,8 +1218,8 @@ int insert_inode_locked4(struct inode *i
struct inode *old = NULL;
repeat:
- spin_lock(&inode_hash_lock);
- hlist_for_each_entry(old, node, head, i_hash) {
+ spin_lock(&b->lock);
+ hlist_for_each_entry(old, node, &b->head, i_hash) {
if (old->i_sb != sb)
continue;
if (!test(old, data))
@@ -1219,18 +1227,18 @@ repeat:
if (old->i_state & (I_FREEING|I_CLEAR|I_WILL_FREE))
continue;
if (!spin_trylock(&old->i_lock)) {
- spin_unlock(&inode_hash_lock);
+ spin_unlock(&b->lock);
goto repeat;
}
break;
}
if (likely(!node)) {
/* XXX: initialize inode->i_lock to locked? */
- hlist_add_head(&inode->i_hash, head);
- spin_unlock(&inode_hash_lock);
+ hlist_add_head(&inode->i_hash, &b->head);
+ spin_unlock(&b->lock);
return 0;
}
- spin_unlock(&inode_hash_lock);
+ spin_unlock(&b->lock);
__iget(old);
spin_unlock(&old->i_lock);
wait_on_inode(old);
@@ -1253,12 +1261,12 @@ EXPORT_SYMBOL(insert_inode_locked4);
*/
void __insert_inode_hash(struct inode *inode, unsigned long hashval)
{
- struct hlist_head *head = inode_hashtable + hash(inode->i_sb, hashval);
+ struct inode_hash_bucket *b = inode_hashtable + hash(inode->i_sb, hashval);
spin_lock(&inode->i_lock);
- spin_lock(&inode_hash_lock);
- hlist_add_head(&inode->i_hash, head);
- spin_unlock(&inode_hash_lock);
+ spin_lock(&b->lock);
+ hlist_add_head(&inode->i_hash, &b->head);
+ spin_unlock(&b->lock);
spin_unlock(&inode->i_lock);
}
EXPORT_SYMBOL(__insert_inode_hash);
@@ -1272,9 +1280,10 @@ EXPORT_SYMBOL(__insert_inode_hash);
*/
void __remove_inode_hash(struct inode *inode)
{
- spin_lock(&inode_hash_lock);
+ struct inode_hash_bucket *b = inode_hashtable + hash(inode->i_sb, inode->i_ino);
+ spin_lock(&b->lock);
hlist_del_init(&inode->i_hash);
- spin_unlock(&inode_hash_lock);
+ spin_unlock(&b->lock);
}
/**
@@ -1663,7 +1672,7 @@ void __init inode_init_early(void)
inode_hashtable =
alloc_large_system_hash("Inode-cache",
- sizeof(struct hlist_head),
+ sizeof(struct inode_hash_bucket),
ihash_entries,
14,
HASH_EARLY,
@@ -1671,8 +1680,10 @@ void __init inode_init_early(void)
&i_hash_mask,
0);
- for (loop = 0; loop < (1 << i_hash_shift); loop++)
- INIT_HLIST_HEAD(&inode_hashtable[loop]);
+ for (loop = 0; loop < (1 << i_hash_shift); loop++) {
+ spin_lock_init(&inode_hashtable[loop].lock);
+ INIT_HLIST_HEAD(&inode_hashtable[loop].head);
+ }
}
void __init inode_init(void)
@@ -1702,8 +1713,10 @@ void __init inode_init(void)
&i_hash_mask,
0);
- for (loop = 0; loop < (1 << i_hash_shift); loop++)
- INIT_HLIST_HEAD(&inode_hashtable[loop]);
+ for (loop = 0; loop < (1 << i_hash_shift); loop++) {
+ spin_lock_init(&inode_hashtable[loop].lock);
+ INIT_HLIST_HEAD(&inode_hashtable[loop].head);
+ }
}
void init_special_inode(struct inode *inode, umode_t mode, dev_t rdev)
^ permalink raw reply [flat|nested] 64+ messages in thread
* [patch 30/33] fs: inode lazy lru
2009-09-04 6:51 [patch 00/33] my current vfs scalability patch queue npiggin
` (28 preceding siblings ...)
2009-09-04 6:52 ` [patch 29/33] Remove the global inode_hash_lock and replace it with per-hash-bucket locks. fs: inode per-bucket inode hash locks npiggin
@ 2009-09-04 6:52 ` npiggin
2009-09-04 6:52 ` [patch 31/33] fs: RCU free inodes npiggin
` (3 subsequent siblings)
33 siblings, 0 replies; 64+ messages in thread
From: npiggin @ 2009-09-04 6:52 UTC (permalink / raw)
To: linux-fsdevel; +Cc: linux-kernel
[-- Attachment #1: fs-inode_lock-scale-10.patch --]
[-- Type: text/plain, Size: 6918 bytes --]
Impelemnt lazy inode lru similarly to dcache. This should reduce inode list
lock acquisition (todo: measure).
---
fs/fs-writeback.c | 2 -
fs/inode.c | 61 +++++++++++++++++++---------------------------
include/linux/fs.h | 7 ++++-
include/linux/writeback.h | 1
4 files changed, 33 insertions(+), 38 deletions(-)
Index: linux-2.6/fs/inode.c
===================================================================
--- linux-2.6.orig/fs/inode.c
+++ linux-2.6/fs/inode.c
@@ -74,7 +74,6 @@ static unsigned int i_hash_shift __read_
* allowing for low-overhead inode sync() operations.
*/
-LIST_HEAD(inode_in_use);
LIST_HEAD(inode_unused);
struct inode_hash_bucket {
@@ -273,6 +272,7 @@ void inode_init_once(struct inode *inode
INIT_HLIST_NODE(&inode->i_hash);
INIT_LIST_HEAD(&inode->i_dentry);
INIT_LIST_HEAD(&inode->i_devices);
+ INIT_LIST_HEAD(&inode->i_list);
INIT_RADIX_TREE(&inode->i_data.page_tree, GFP_ATOMIC);
spin_lock_init(&inode->i_data.tree_lock);
spin_lock_init(&inode->i_data.i_mmap_lock);
@@ -298,24 +298,6 @@ static void init_once(void *foo)
inode_init_once(inode);
}
-/*
- * inode_lock must be held
- */
-void __iget(struct inode *inode)
-{
- assert_spin_locked(&inode->i_lock);
- inode->i_count++;
- if (inode->i_count > 1)
- return;
-
- if (!(inode->i_state & (I_DIRTY|I_SYNC))) {
- spin_lock(&wb_inode_list_lock);
- list_move(&inode->i_list, &inode_in_use);
- spin_unlock(&wb_inode_list_lock);
- }
- atomic_dec(&inodes_stat.nr_unused);
-}
-
/**
* clear_inode - clear an inode
* @inode: inode to clear
@@ -359,7 +341,7 @@ static void dispose_list(struct list_hea
struct inode *inode;
inode = list_first_entry(head, struct inode, i_list);
- list_del(&inode->i_list);
+ list_del_init(&inode->i_list);
if (inode->i_data.nrpages)
truncate_inode_pages(&inode->i_data, 0);
@@ -412,11 +394,12 @@ static int invalidate_list(struct list_h
invalidate_inode_buffers(inode);
if (!inode->i_count) {
spin_lock(&wb_inode_list_lock);
- list_move(&inode->i_list, dispose);
+ list_del(&inode->i_list);
spin_unlock(&wb_inode_list_lock);
WARN_ON(inode->i_state & I_NEW);
inode->i_state |= I_FREEING;
spin_unlock(&inode->i_lock);
+ list_add(&inode->i_list, dispose);
count++;
continue;
}
@@ -503,7 +486,13 @@ again:
spin_unlock(&wb_inode_list_lock);
goto again;
}
- if (inode->i_state || inode->i_count) {
+ if (inode->i_count) {
+ list_del_init(&inode->i_list);
+ spin_unlock(&inode->i_lock);
+ atomic_dec(&inodes_stat.nr_unused);
+ continue;
+ }
+ if (inode->i_state) {
list_move(&inode->i_list, &inode_unused);
spin_unlock(&inode->i_lock);
continue;
@@ -519,6 +508,7 @@ again:
again2:
spin_lock(&wb_inode_list_lock);
+ /* XXX: may no longer work well */
if (inode != list_entry(inode_unused.next,
struct inode, i_list))
continue; /* wrong inode or list_empty */
@@ -667,9 +657,6 @@ __inode_add_to_lists(struct super_block
atomic_inc(&inodes_stat.nr_inodes);
list_add(&inode->i_sb_list, &sb->s_inodes);
spin_unlock(&sb_inode_list_lock);
- spin_lock(&wb_inode_list_lock);
- list_add(&inode->i_list, &inode_in_use);
- spin_unlock(&wb_inode_list_lock);
if (b) {
spin_lock(&b->lock);
hlist_add_head(&inode->i_hash, &b->head);
@@ -1316,9 +1303,11 @@ void generic_delete_inode(struct inode *
{
const struct super_operations *op = inode->i_sb->s_op;
- spin_lock(&wb_inode_list_lock);
- list_del_init(&inode->i_list);
- spin_unlock(&wb_inode_list_lock);
+ if (!list_empty(&inode->i_list)) {
+ spin_lock(&wb_inode_list_lock);
+ list_del_init(&inode->i_list);
+ spin_unlock(&wb_inode_list_lock);
+ }
list_del_init(&inode->i_sb_list);
spin_unlock(&sb_inode_list_lock);
WARN_ON(inode->i_state & I_NEW);
@@ -1355,12 +1344,12 @@ static void generic_forget_inode(struct
struct super_block *sb = inode->i_sb;
if (!hlist_unhashed(&inode->i_hash)) {
- if (!(inode->i_state & (I_DIRTY|I_SYNC))) {
+ if (list_empty(&inode->i_list)) {
spin_lock(&wb_inode_list_lock);
- list_move(&inode->i_list, &inode_unused);
+ list_add(&inode->i_list, &inode_unused);
spin_unlock(&wb_inode_list_lock);
+ atomic_inc(&inodes_stat.nr_unused);
}
- atomic_inc(&inodes_stat.nr_unused);
if (sb->s_flags & MS_ACTIVE) {
spin_unlock(&inode->i_lock);
spin_unlock(&sb_inode_list_lock);
@@ -1376,11 +1365,13 @@ static void generic_forget_inode(struct
WARN_ON(inode->i_state & I_NEW);
inode->i_state &= ~I_WILL_FREE;
__remove_inode_hash(inode);
+ }
+ if (!list_empty(&inode->i_list)) {
+ spin_lock(&wb_inode_list_lock);
+ list_del_init(&inode->i_list);
+ spin_unlock(&wb_inode_list_lock);
atomic_dec(&inodes_stat.nr_unused);
}
- spin_lock(&wb_inode_list_lock);
- list_del_init(&inode->i_list);
- spin_unlock(&wb_inode_list_lock);
list_del_init(&inode->i_sb_list);
spin_unlock(&sb_inode_list_lock);
WARN_ON(inode->i_state & I_NEW);
@@ -1705,7 +1696,7 @@ void __init inode_init(void)
inode_hashtable =
alloc_large_system_hash("Inode-cache",
- sizeof(struct hlist_head),
+ sizeof(struct inode_hash_bucket),
ihash_entries,
14,
0,
Index: linux-2.6/include/linux/fs.h
===================================================================
--- linux-2.6.orig/include/linux/fs.h
+++ linux-2.6/include/linux/fs.h
@@ -2166,7 +2166,6 @@ extern int insert_inode_locked4(struct i
extern int insert_inode_locked(struct inode *);
extern void unlock_new_inode(struct inode *);
-extern void __iget(struct inode * inode);
extern void iget_failed(struct inode *);
extern void clear_inode(struct inode *);
extern void destroy_inode(struct inode *);
@@ -2384,6 +2383,12 @@ extern int generic_show_options(struct s
extern void save_mount_options(struct super_block *sb, char *options);
extern void replace_mount_options(struct super_block *sb, char *options);
+static inline void __iget(struct inode *inode)
+{
+ assert_spin_locked(&inode->i_lock);
+ inode->i_count++;
+}
+
static inline ino_t parent_ino(struct dentry *dentry)
{
ino_t res;
Index: linux-2.6/fs/fs-writeback.c
===================================================================
--- linux-2.6.orig/fs/fs-writeback.c
+++ linux-2.6/fs/fs-writeback.c
@@ -429,7 +429,7 @@ writeback_single_inode(struct inode *ino
/*
* The inode is clean, inuse
*/
- list_move(&inode->i_list, &inode_in_use);
+ list_del_init(&inode->i_list);
} else {
/*
* The inode is clean, unused
Index: linux-2.6/include/linux/writeback.h
===================================================================
--- linux-2.6.orig/include/linux/writeback.h
+++ linux-2.6/include/linux/writeback.h
@@ -11,7 +11,6 @@ struct backing_dev_info;
extern spinlock_t sb_inode_list_lock;
extern spinlock_t wb_inode_list_lock;
-extern struct list_head inode_in_use;
extern struct list_head inode_unused;
/*
^ permalink raw reply [flat|nested] 64+ messages in thread
* [patch 31/33] fs: RCU free inodes
2009-09-04 6:51 [patch 00/33] my current vfs scalability patch queue npiggin
` (29 preceding siblings ...)
2009-09-04 6:52 ` [patch 30/33] fs: inode lazy lru npiggin
@ 2009-09-04 6:52 ` npiggin
2009-09-04 6:52 ` [patch 32/33] fs: rcu walk for i_sb_list npiggin
` (2 subsequent siblings)
33 siblings, 0 replies; 64+ messages in thread
From: npiggin @ 2009-09-04 6:52 UTC (permalink / raw)
To: linux-fsdevel; +Cc: linux-kernel
[-- Attachment #1: fs-inode_rcu.patch --]
[-- Type: text/plain, Size: 7593 bytes --]
RCU free the struct inode. This will allow:
- sb_inode_list_lock to be moved inside i_lock because sb list walkers who want
to take i_lock no longer need to take sb_inode_list_lock to walk the list in
the first place. This will simplify and optimize locking.
- eventually, completely write-free RCU path walking. The inode must be
consulted for permissions when walking, so a write-free reference (ie.
RCU is helpful).
- can potentially simplify things a bit in VM land. May not need to take the
page lock to get back to the page->mapping.
- can remove some nested trylock loops in dcache code
todo: convert all filesystems
---
fs/block_dev.c | 9 ++++++++-
fs/ext2/super.c | 9 ++++++++-
fs/ext3/super.c | 9 ++++++++-
fs/hugetlbfs/inode.c | 9 ++++++++-
fs/inode.c | 9 ++++++++-
fs/proc/inode.c | 9 ++++++++-
include/linux/fs.h | 5 ++++-
ipc/mqueue.c | 9 ++++++++-
net/socket.c | 9 ++++++++-
9 files changed, 68 insertions(+), 9 deletions(-)
Index: linux-2.6/fs/ext2/super.c
===================================================================
--- linux-2.6.orig/fs/ext2/super.c
+++ linux-2.6/fs/ext2/super.c
@@ -157,11 +157,18 @@ static struct inode *ext2_alloc_inode(st
return &ei->vfs_inode;
}
-static void ext2_destroy_inode(struct inode *inode)
+static void ext2_i_callback(struct rcu_head *head)
{
+ struct inode *inode = container_of(head, struct inode, i_rcu);
+ INIT_LIST_HEAD(&inode->i_dentry);
kmem_cache_free(ext2_inode_cachep, EXT2_I(inode));
}
+static void ext2_destroy_inode(struct inode *inode)
+{
+ call_rcu(&inode->i_rcu, ext2_i_callback);
+}
+
static void init_once(void *foo)
{
struct ext2_inode_info *ei = (struct ext2_inode_info *) foo;
Index: linux-2.6/fs/inode.c
===================================================================
--- linux-2.6.orig/fs/inode.c
+++ linux-2.6/fs/inode.c
@@ -252,13 +252,20 @@ void __destroy_inode(struct inode *inode
}
EXPORT_SYMBOL(__destroy_inode);
+static void i_callback(struct rcu_head *head)
+{
+ struct inode *inode = container_of(head, struct inode, i_rcu);
+ INIT_LIST_HEAD(&inode->i_dentry);
+ kmem_cache_free(inode_cachep, inode);
+}
+
void destroy_inode(struct inode *inode)
{
__destroy_inode(inode);
if (inode->i_sb->s_op->destroy_inode)
inode->i_sb->s_op->destroy_inode(inode);
else
- kmem_cache_free(inode_cachep, (inode));
+ call_rcu(&inode->i_rcu, i_callback);
}
/*
Index: linux-2.6/include/linux/fs.h
===================================================================
--- linux-2.6.orig/include/linux/fs.h
+++ linux-2.6/include/linux/fs.h
@@ -718,7 +718,10 @@ struct inode {
struct hlist_node i_hash;
struct list_head i_list;
struct list_head i_sb_list;
- struct list_head i_dentry;
+ union {
+ struct list_head i_dentry;
+ struct rcu_head i_rcu;
+ };
unsigned long i_ino;
unsigned int i_count;
unsigned int i_nlink;
Index: linux-2.6/fs/block_dev.c
===================================================================
--- linux-2.6.orig/fs/block_dev.c
+++ linux-2.6/fs/block_dev.c
@@ -416,14 +416,21 @@ static struct inode *bdev_alloc_inode(st
return &ei->vfs_inode;
}
-static void bdev_destroy_inode(struct inode *inode)
+static void bdev_i_callback(struct rcu_head *head)
{
+ struct inode *inode = container_of(head, struct inode, i_rcu);
struct bdev_inode *bdi = BDEV_I(inode);
+ INIT_LIST_HEAD(&inode->i_dentry);
bdi->bdev.bd_inode_backing_dev_info = NULL;
kmem_cache_free(bdev_cachep, bdi);
}
+static void bdev_destroy_inode(struct inode *inode)
+{
+ call_rcu(&inode->i_rcu, bdev_i_callback);
+}
+
static void init_once(void *foo)
{
struct bdev_inode *ei = (struct bdev_inode *) foo;
Index: linux-2.6/fs/ext3/super.c
===================================================================
--- linux-2.6.orig/fs/ext3/super.c
+++ linux-2.6/fs/ext3/super.c
@@ -469,6 +469,13 @@ static struct inode *ext3_alloc_inode(st
return &ei->vfs_inode;
}
+static void ext3_i_callback(struct rcu_head *head)
+{
+ struct inode *inode = container_of(head, struct inode, i_rcu);
+ INIT_LIST_HEAD(&inode->i_dentry);
+ kmem_cache_free(ext3_inode_cachep, EXT3_I(inode));
+}
+
static void ext3_destroy_inode(struct inode *inode)
{
if (!list_empty(&(EXT3_I(inode)->i_orphan))) {
@@ -479,7 +486,7 @@ static void ext3_destroy_inode(struct in
false);
dump_stack();
}
- kmem_cache_free(ext3_inode_cachep, EXT3_I(inode));
+ call_rcu(&inode->i_rcu, ext3_i_callback);
}
static void init_once(void *foo)
Index: linux-2.6/fs/hugetlbfs/inode.c
===================================================================
--- linux-2.6.orig/fs/hugetlbfs/inode.c
+++ linux-2.6/fs/hugetlbfs/inode.c
@@ -697,11 +697,18 @@ static struct inode *hugetlbfs_alloc_ino
return &p->vfs_inode;
}
+static void hugetlbfs_i_callback(struct rcu_head *head)
+{
+ struct inode *inode = container_of(head, struct inode, i_rcu);
+ INIT_LIST_HEAD(&inode->i_dentry);
+ kmem_cache_free(hugetlbfs_inode_cachep, HUGETLBFS_I(inode));
+}
+
static void hugetlbfs_destroy_inode(struct inode *inode)
{
hugetlbfs_inc_free_inodes(HUGETLBFS_SB(inode->i_sb));
mpol_free_shared_policy(&HUGETLBFS_I(inode)->policy);
- kmem_cache_free(hugetlbfs_inode_cachep, HUGETLBFS_I(inode));
+ call_rcu(&inode->i_rcu, hugetlbfs_i_callback);
}
static const struct address_space_operations hugetlbfs_aops = {
Index: linux-2.6/fs/proc/inode.c
===================================================================
--- linux-2.6.orig/fs/proc/inode.c
+++ linux-2.6/fs/proc/inode.c
@@ -88,11 +88,18 @@ static struct inode *proc_alloc_inode(st
return inode;
}
-static void proc_destroy_inode(struct inode *inode)
+static void proc_i_callback(struct rcu_head *head)
{
+ struct inode *inode = container_of(head, struct inode, i_rcu);
+ INIT_LIST_HEAD(&inode->i_dentry);
kmem_cache_free(proc_inode_cachep, PROC_I(inode));
}
+static void proc_destroy_inode(struct inode *inode)
+{
+ call_rcu(&inode->i_rcu, proc_i_callback);
+}
+
static void init_once(void *foo)
{
struct proc_inode *ei = (struct proc_inode *) foo;
Index: linux-2.6/ipc/mqueue.c
===================================================================
--- linux-2.6.orig/ipc/mqueue.c
+++ linux-2.6/ipc/mqueue.c
@@ -238,11 +238,18 @@ static struct inode *mqueue_alloc_inode(
return &ei->vfs_inode;
}
-static void mqueue_destroy_inode(struct inode *inode)
+static void mqueue_i_callback(struct rcu_head *head)
{
+ struct inode *inode = container_of(head, struct inode, i_rcu);
+ INIT_LIST_HEAD(&inode->i_dentry);
kmem_cache_free(mqueue_inode_cachep, MQUEUE_I(inode));
}
+static void mqueue_destroy_inode(struct inode *inode)
+{
+ call_rcu(&inode->i_rcu, mqueue_i_callback);
+}
+
static void mqueue_delete_inode(struct inode *inode)
{
struct mqueue_inode_info *info;
Index: linux-2.6/net/socket.c
===================================================================
--- linux-2.6.orig/net/socket.c
+++ linux-2.6/net/socket.c
@@ -258,12 +258,19 @@ static struct inode *sock_alloc_inode(st
return &ei->vfs_inode;
}
-static void sock_destroy_inode(struct inode *inode)
+static void sock_i_callback(struct rcu_head *head)
{
+ struct inode *inode = container_of(head, struct inode, i_rcu);
+ INIT_LIST_HEAD(&inode->i_dentry);
kmem_cache_free(sock_inode_cachep,
container_of(inode, struct socket_alloc, vfs_inode));
}
+static void sock_destroy_inode(struct inode *inode)
+{
+ call_rcu(&inode->i_rcu, sock_i_callback);
+}
+
static void init_once(void *foo)
{
struct socket_alloc *ei = (struct socket_alloc *)foo;
^ permalink raw reply [flat|nested] 64+ messages in thread
* [patch 32/33] fs: rcu walk for i_sb_list
2009-09-04 6:51 [patch 00/33] my current vfs scalability patch queue npiggin
` (30 preceding siblings ...)
2009-09-04 6:52 ` [patch 31/33] fs: RCU free inodes npiggin
@ 2009-09-04 6:52 ` npiggin
2009-09-04 6:52 ` [patch 33/33] fs: improve scalability of pseudo filesystems npiggin
2009-09-04 7:05 ` [patch 00/33] my current vfs scalability patch queue Nick Piggin
33 siblings, 0 replies; 64+ messages in thread
From: npiggin @ 2009-09-04 6:52 UTC (permalink / raw)
To: linux-fsdevel; +Cc: linux-kernel
[-- Attachment #1: fs-inode_lock-scale-11.patch --]
[-- Type: text/plain, Size: 11876 bytes --]
This enables locking to be reduced and simplified.
---
fs/drop_caches.c | 10 ++++-----
fs/fs-writeback.c | 10 ++++-----
fs/hugetlbfs/inode.c | 4 ---
fs/inode.c | 45 ++++++++++++++------------------------------
fs/notify/inode_mark.c | 10 ---------
fs/notify/inotify/inotify.c | 10 ---------
fs/quota/dquot.c | 16 +++++++--------
7 files changed, 34 insertions(+), 71 deletions(-)
Index: linux-2.6/fs/drop_caches.c
===================================================================
--- linux-2.6.orig/fs/drop_caches.c
+++ linux-2.6/fs/drop_caches.c
@@ -16,8 +16,8 @@ static void drop_pagecache_sb(struct sup
{
struct inode *inode, *toput_inode = NULL;
- spin_lock(&sb_inode_list_lock);
- list_for_each_entry(inode, &sb->s_inodes, i_sb_list) {
+ rcu_read_lock();
+ list_for_each_entry_rcu(inode, &sb->s_inodes, i_sb_list) {
spin_lock(&inode->i_lock);
if (inode->i_state & (I_FREEING|I_CLEAR|I_WILL_FREE|I_NEW)
|| inode->i_mapping->nrpages == 0) {
@@ -26,13 +26,13 @@ static void drop_pagecache_sb(struct sup
}
__iget(inode);
spin_unlock(&inode->i_lock);
- spin_unlock(&sb_inode_list_lock);
+ rcu_read_unlock();
invalidate_mapping_pages(inode->i_mapping, 0, -1);
iput(toput_inode);
toput_inode = inode;
- spin_lock(&sb_inode_list_lock);
+ rcu_read_lock();
}
- spin_unlock(&sb_inode_list_lock);
+ rcu_read_unlock();
iput(toput_inode);
}
Index: linux-2.6/fs/fs-writeback.c
===================================================================
--- linux-2.6.orig/fs/fs-writeback.c
+++ linux-2.6/fs/fs-writeback.c
@@ -587,8 +587,8 @@ again:
* In which case, the inode may not be on the dirty list, but
* we still have to wait for that writeout.
*/
- spin_lock(&sb_inode_list_lock);
- list_for_each_entry(inode, &sb->s_inodes, i_sb_list) {
+ rcu_read_lock();
+ list_for_each_entry_rcu(inode, &sb->s_inodes, i_sb_list) {
struct address_space *mapping = inode->i_mapping;
spin_lock(&inode->i_lock);
@@ -600,7 +600,7 @@ again:
}
__iget(inode);
spin_unlock(&inode->i_lock);
- spin_unlock(&sb_inode_list_lock);
+ rcu_read_unlock();
/*
* We hold a reference to 'inode' so it couldn't have
* been removed from s_inodes list while we dropped the
@@ -616,9 +616,9 @@ again:
cond_resched();
- spin_lock(&sb_inode_list_lock);
+ rcu_read_lock();
}
- spin_unlock(&sb_inode_list_lock);
+ rcu_read_unlock();
iput(old_inode);
}
Index: linux-2.6/fs/inode.c
===================================================================
--- linux-2.6.orig/fs/inode.c
+++ linux-2.6/fs/inode.c
@@ -354,12 +354,12 @@ static void dispose_list(struct list_hea
truncate_inode_pages(&inode->i_data, 0);
clear_inode(inode);
- spin_lock(&sb_inode_list_lock);
spin_lock(&inode->i_lock);
__remove_inode_hash(inode);
+ spin_lock(&sb_inode_list_lock);
list_del_init(&inode->i_sb_list);
- spin_unlock(&inode->i_lock);
spin_unlock(&sb_inode_list_lock);
+ spin_unlock(&inode->i_lock);
wake_up_inode(inode);
destroy_inode(inode);
@@ -381,14 +381,6 @@ static int invalidate_list(struct list_h
struct list_head *tmp = next;
struct inode *inode;
- /*
- * We can reschedule here without worrying about the list's
- * consistency because the per-sb list of inodes must not
- * change during umount anymore, and because iprune_mutex keeps
- * shrink_icache_memory() away.
- */
- cond_resched_lock(&sb_inode_list_lock);
-
next = next->next;
if (tmp == head)
break;
@@ -431,12 +423,17 @@ int invalidate_inodes(struct super_block
int busy;
LIST_HEAD(throw_away);
+ /*
+ * Don't need to worry about the list's consistency because the per-sb
+ * list of inodes must not change during umount anymore, and because
+ * iprune_mutex keeps shrink_icache_memory() away.
+ */
mutex_lock(&iprune_mutex);
- spin_lock(&sb_inode_list_lock);
+// spin_lock(&sb_inode_list_lock); XXX: is this safe?
inotify_unmount_inodes(&sb->s_inodes);
fsnotify_unmount_inodes(&sb->s_inodes);
busy = invalidate_list(&sb->s_inodes, &throw_away);
- spin_unlock(&sb_inode_list_lock);
+// spin_unlock(&sb_inode_list_lock);
dispose_list(&throw_away);
mutex_unlock(&iprune_mutex);
@@ -662,6 +659,7 @@ __inode_add_to_lists(struct super_block
struct inode *inode)
{
atomic_inc(&inodes_stat.nr_inodes);
+ spin_lock(&sb_inode_list_lock);
list_add(&inode->i_sb_list, &sb->s_inodes);
spin_unlock(&sb_inode_list_lock);
if (b) {
@@ -687,7 +685,6 @@ void inode_add_to_lists(struct super_blo
{
struct inode_hash_bucket *b = inode_hashtable + hash(sb, inode->i_ino);
- spin_lock(&sb_inode_list_lock);
spin_lock(&inode->i_lock);
__inode_add_to_lists(sb, b, inode);
spin_unlock(&inode->i_lock);
@@ -718,7 +715,6 @@ struct inode *new_inode(struct super_blo
inode = alloc_inode(sb);
if (inode) {
- spin_lock(&sb_inode_list_lock);
spin_lock(&inode->i_lock);
inode->i_ino = atomic_inc_return(&last_ino);
inode->i_state = 0;
@@ -783,7 +779,6 @@ static struct inode *get_new_inode(struc
/* We released the lock, so.. */
old = find_inode(sb, b, test, data);
if (!old) {
- spin_lock(&sb_inode_list_lock);
spin_lock(&inode->i_lock);
if (set(inode, data))
goto set_failed;
@@ -813,7 +808,6 @@ static struct inode *get_new_inode(struc
set_failed:
spin_unlock(&inode->i_lock);
- spin_unlock(&sb_inode_list_lock);
destroy_inode(inode);
return NULL;
}
@@ -834,7 +828,6 @@ static struct inode *get_new_inode_fast(
/* We released the lock, so.. */
old = find_inode_fast(sb, b, ino);
if (!old) {
- spin_lock(&sb_inode_list_lock);
spin_lock(&inode->i_lock);
inode->i_ino = ino;
inode->i_state = I_LOCK|I_NEW;
@@ -1315,6 +1308,7 @@ void generic_delete_inode(struct inode *
list_del_init(&inode->i_list);
spin_unlock(&wb_inode_list_lock);
}
+ spin_lock(&sb_inode_list_lock);
list_del_init(&inode->i_sb_list);
spin_unlock(&sb_inode_list_lock);
WARN_ON(inode->i_state & I_NEW);
@@ -1359,15 +1353,12 @@ static void generic_forget_inode(struct
}
if (sb->s_flags & MS_ACTIVE) {
spin_unlock(&inode->i_lock);
- spin_unlock(&sb_inode_list_lock);
return;
}
WARN_ON(inode->i_state & I_NEW);
inode->i_state |= I_WILL_FREE;
spin_unlock(&inode->i_lock);
- spin_unlock(&sb_inode_list_lock);
write_inode_now(inode, 1);
- spin_lock(&sb_inode_list_lock);
spin_lock(&inode->i_lock);
WARN_ON(inode->i_state & I_NEW);
inode->i_state &= ~I_WILL_FREE;
@@ -1379,6 +1370,7 @@ static void generic_forget_inode(struct
spin_unlock(&wb_inode_list_lock);
atomic_dec(&inodes_stat.nr_unused);
}
+ spin_lock(&sb_inode_list_lock);
list_del_init(&inode->i_sb_list);
spin_unlock(&sb_inode_list_lock);
WARN_ON(inode->i_state & I_NEW);
@@ -1441,19 +1433,12 @@ void iput(struct inode *inode)
if (inode) {
BUG_ON(inode->i_state == I_CLEAR);
-retry:
spin_lock(&inode->i_lock);
- if (inode->i_count == 1) {
- if (!spin_trylock(&sb_inode_list_lock)) {
- spin_unlock(&inode->i_lock);
- goto retry;
- }
- inode->i_count--;
+ inode->i_count--;
+ if (inode->i_count == 0)
iput_final(inode);
- } else {
- inode->i_count--;
+ else
spin_unlock(&inode->i_lock);
- }
}
}
EXPORT_SYMBOL(iput);
Index: linux-2.6/fs/notify/inode_mark.c
===================================================================
--- linux-2.6.orig/fs/notify/inode_mark.c
+++ linux-2.6/fs/notify/inode_mark.c
@@ -413,14 +413,6 @@ void fsnotify_unmount_inodes(struct list
spin_unlock(&next_i->i_lock);
}
- /*
- * We can safely drop inode_lock here because we hold
- * references on both inode and next_i. Also no new inodes
- * will be added since the umount has begun. Finally,
- * iprune_mutex keeps shrink_icache_memory() away.
- */
- spin_unlock(&sb_inode_list_lock);
-
if (need_iput_tmp)
iput(need_iput_tmp);
@@ -430,7 +422,5 @@ void fsnotify_unmount_inodes(struct list
fsnotify_inode_delete(inode);
iput(inode);
-
- spin_lock(&sb_inode_list_lock);
}
}
Index: linux-2.6/fs/notify/inotify/inotify.c
===================================================================
--- linux-2.6.orig/fs/notify/inotify/inotify.c
+++ linux-2.6/fs/notify/inotify/inotify.c
@@ -436,14 +436,6 @@ void inotify_unmount_inodes(struct list_
spin_unlock(&next_i->i_lock);
}
- /*
- * We can safely drop inode_lock here because we hold
- * references on both inode and next_i. Also no new inodes
- * will be added since the umount has begun. Finally,
- * iprune_mutex keeps shrink_icache_memory() away.
- */
- spin_unlock(&sb_inode_list_lock);
-
if (need_iput_tmp)
iput(need_iput_tmp);
@@ -461,8 +453,6 @@ void inotify_unmount_inodes(struct list_
}
mutex_unlock(&inode->inotify_mutex);
iput(inode);
-
- spin_lock(&sb_inode_list_lock);
}
}
EXPORT_SYMBOL_GPL(inotify_unmount_inodes);
Index: linux-2.6/fs/quota/dquot.c
===================================================================
--- linux-2.6.orig/fs/quota/dquot.c
+++ linux-2.6/fs/quota/dquot.c
@@ -821,8 +821,8 @@ static void add_dquot_ref(struct super_b
{
struct inode *inode, *old_inode = NULL;
- spin_lock(&sb_inode_list_lock);
- list_for_each_entry(inode, &sb->s_inodes, i_sb_list) {
+ rcu_read_lock();
+ list_for_each_entry_rcu(inode, &sb->s_inodes, i_sb_list) {
spin_lock(&inode->i_lock);
if (inode->i_state & (I_FREEING|I_CLEAR|I_WILL_FREE|I_NEW)) {
spin_unlock(&inode->i_lock);
@@ -839,7 +839,7 @@ static void add_dquot_ref(struct super_b
__iget(inode);
spin_unlock(&inode->i_lock);
- spin_unlock(&sb_inode_list_lock);
+ rcu_read_unlock();
iput(old_inode);
sb->dq_op->initialize(inode, type);
@@ -849,9 +849,9 @@ static void add_dquot_ref(struct super_b
* reference and we cannot iput it under inode_lock. So we
* keep the reference and iput it later. */
old_inode = inode;
- spin_lock(&sb_inode_list_lock);
+ rcu_read_lock();
}
- spin_unlock(&sb_inode_list_lock);
+ rcu_read_unlock();
iput(old_inode);
}
@@ -921,8 +921,8 @@ static void remove_dquot_ref(struct supe
{
struct inode *inode;
- spin_lock(&sb_inode_list_lock);
- list_for_each_entry(inode, &sb->s_inodes, i_sb_list) {
+ rcu_read_lock();
+ list_for_each_entry_rcu(inode, &sb->s_inodes, i_sb_list) {
/*
* We have to scan also I_NEW inodes because they can already
* have quota pointer initialized. Luckily, we need to touch
@@ -932,7 +932,7 @@ static void remove_dquot_ref(struct supe
if (!IS_NOQUOTA(inode))
remove_inode_dquot_ref(inode, type, tofree_head);
}
- spin_unlock(&sb_inode_list_lock);
+ rcu_read_unlock();
}
/* Gather all references from inodes and drop them */
Index: linux-2.6/fs/hugetlbfs/inode.c
===================================================================
--- linux-2.6.orig/fs/hugetlbfs/inode.c
+++ linux-2.6/fs/hugetlbfs/inode.c
@@ -392,19 +392,16 @@ static void hugetlbfs_forget_inode(struc
atomic_inc(&inodes_stat.nr_unused);
if (!sb || (sb->s_flags & MS_ACTIVE)) {
spin_unlock(&inode->i_lock);
- spin_unlock(&sb_inode_list_lock);
return;
}
WARN_ON(inode->i_state & I_NEW);
inode->i_state |= I_WILL_FREE;
spin_unlock(&inode->i_lock);
- spin_unlock(&sb_inode_list_lock);
/*
* write_inode_now is a noop as we set BDI_CAP_NO_WRITEBACK
* in our backing_dev_info.
*/
write_inode_now(inode, 1);
- spin_lock(&sb_inode_list_lock);
spin_lock(&inode->i_lock);
WARN_ON(inode->i_state & I_NEW);
inode->i_state &= ~I_WILL_FREE;
@@ -414,6 +411,7 @@ static void hugetlbfs_forget_inode(struc
spin_lock(&wb_inode_list_lock);
list_del_init(&inode->i_list);
spin_unlock(&wb_inode_list_lock);
+ spin_lock(&sb_inode_list_lock);
list_del_init(&inode->i_sb_list);
spin_unlock(&sb_inode_list_lock);
WARN_ON(inode->i_state & I_NEW);
^ permalink raw reply [flat|nested] 64+ messages in thread
* [patch 33/33] fs: improve scalability of pseudo filesystems
2009-09-04 6:51 [patch 00/33] my current vfs scalability patch queue npiggin
` (31 preceding siblings ...)
2009-09-04 6:52 ` [patch 32/33] fs: rcu walk for i_sb_list npiggin
@ 2009-09-04 6:52 ` npiggin
2009-09-04 7:05 ` [patch 00/33] my current vfs scalability patch queue Nick Piggin
33 siblings, 0 replies; 64+ messages in thread
From: npiggin @ 2009-09-04 6:52 UTC (permalink / raw)
To: linux-fsdevel; +Cc: linux-kernel
[-- Attachment #1: fs-scale-pseudo.patch --]
[-- Type: text/plain, Size: 2808 bytes --]
Regardless of how much we possibly try to scale dcache, there is likely
always going to be some fundamental contention when adding or removing children
under the same parent. Pseudo filesystems do not seem need to have connected
dentries because by definition they are disconnected.
XXX: is this right? I can't see any reason why they need to have a real
parent.
TODO: add a d_instantiate_something() and avoid adding the extra checks
for !d_parent
---
fs/anon_inodes.c | 2 +-
fs/notify/inotify/inotify.c | 2 +-
fs/pipe.c | 2 +-
include/linux/fsnotify_backend.h | 2 +-
net/socket.c | 2 +-
5 files changed, 5 insertions(+), 5 deletions(-)
Index: linux-2.6/fs/notify/inotify/inotify.c
===================================================================
--- linux-2.6.orig/fs/notify/inotify/inotify.c
+++ linux-2.6/fs/notify/inotify/inotify.c
@@ -268,7 +268,7 @@ void inotify_d_instantiate(struct dentry
{
struct dentry *parent;
- if (!inode)
+ if (!inode || !entry->d_parent)
return;
/* XXX: need parent lock in place of dcache_lock? */
Index: linux-2.6/include/linux/fsnotify_backend.h
===================================================================
--- linux-2.6.orig/include/linux/fsnotify_backend.h
+++ linux-2.6/include/linux/fsnotify_backend.h
@@ -291,7 +291,7 @@ static inline void __fsnotify_update_dca
*/
static inline void __fsnotify_d_instantiate(struct dentry *dentry, struct inode *inode)
{
- if (!inode)
+ if (!inode || !dentry->d_parent)
return;
spin_lock(&dentry->d_lock);
Index: linux-2.6/net/socket.c
===================================================================
--- linux-2.6.orig/net/socket.c
+++ linux-2.6/net/socket.c
@@ -367,7 +367,7 @@ static int sock_attach_fd(struct socket
struct dentry *dentry;
struct qstr name = { .name = "" };
- dentry = d_alloc(sock_mnt->mnt_sb->s_root, &name);
+ dentry = d_alloc(NULL, &name);
if (unlikely(!dentry))
return -ENOMEM;
Index: linux-2.6/fs/anon_inodes.c
===================================================================
--- linux-2.6.orig/fs/anon_inodes.c
+++ linux-2.6/fs/anon_inodes.c
@@ -95,7 +95,7 @@ int anon_inode_getfd(const char *name, c
this.name = name;
this.len = strlen(name);
this.hash = 0;
- dentry = d_alloc(anon_inode_mnt->mnt_sb->s_root, &this);
+ dentry = d_alloc(NULL, &this);
if (!dentry)
goto err_put_unused_fd;
Index: linux-2.6/fs/pipe.c
===================================================================
--- linux-2.6.orig/fs/pipe.c
+++ linux-2.6/fs/pipe.c
@@ -952,7 +952,7 @@ struct file *create_write_pipe(int flags
goto err;
err = -ENOMEM;
- dentry = d_alloc(pipe_mnt->mnt_sb->s_root, &name);
+ dentry = d_alloc(NULL, &name);
if (!dentry)
goto err_inode;
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [patch 00/33] my current vfs scalability patch queue
2009-09-04 6:51 [patch 00/33] my current vfs scalability patch queue npiggin
` (32 preceding siblings ...)
2009-09-04 6:52 ` [patch 33/33] fs: improve scalability of pseudo filesystems npiggin
@ 2009-09-04 7:05 ` Nick Piggin
33 siblings, 0 replies; 64+ messages in thread
From: Nick Piggin @ 2009-09-04 7:05 UTC (permalink / raw)
To: linux-fsdevel; +Cc: linux-kernel
On Fri, Sep 04, 2009 at 04:51:41PM +1000, npiggin@suse.de wrote:
> But it is now getting to the point where I will need to get some agreement with
> the approach.
BTW, yes I'm aware numbers and results are needed. I have some things,
but I don't have that big systems to test with, or really interesting
workoads.
I know google is running into lock contention which is why they proposed
the batched iput dput patches. Peter Chubb hit inode lock contention
with reaim (don't konw if that is considered realistic). And SGI have
had various lock contention problems with NFS fileservers.
So aside from reviews and suggestions, what would be most helpful to me
would be people willing to test out their problematic workloads with
me. It's the only way I can try to get them fixed.
Thanks,
Nick
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [patch 29/33] Remove the global inode_hash_lock and replace it with per-hash-bucket locks. fs: inode per-bucket inode hash locks
2009-09-04 6:52 ` [patch 29/33] Remove the global inode_hash_lock and replace it with per-hash-bucket locks. fs: inode per-bucket inode hash locks npiggin
@ 2009-09-04 7:05 ` Nick Piggin
0 siblings, 0 replies; 64+ messages in thread
From: Nick Piggin @ 2009-09-04 7:05 UTC (permalink / raw)
To: linux-fsdevel; +Cc: linux-kernel
And sorry about the messed up subject on this one.
On Fri, Sep 04, 2009 at 04:52:10PM +1000, npiggin@suse.de wrote:
> Todo: should use bit spinlock in hlist_head pointer to save space.
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [patch 08/33] fs: dcache scale nr_dentry
2009-09-04 6:51 ` [patch 08/33] fs: dcache scale nr_dentry npiggin
@ 2009-09-04 14:41 ` Daniel Walker
2009-09-07 7:36 ` Nick Piggin
0 siblings, 1 reply; 64+ messages in thread
From: Daniel Walker @ 2009-09-04 14:41 UTC (permalink / raw)
To: npiggin; +Cc: linux-fsdevel, linux-kernel
On Fri, 2009-09-04 at 16:51 +1000, npiggin@suse.de wrote:
> plain text document attachment (fs-dcache-scale-nr_dentry.patch)
> Make dentry_stat_t.nr_dentry an atomic_t type, and move it from under
> dcache_lock.
> ---
> fs/dcache.c | 20 +++++++++-----------
> include/linux/dcache.h | 4 ++--
> kernel/sysctl.c | 6 ++++++
> 3 files changed, 17 insertions(+), 13 deletions(-)
>
No sign off on this one..
Daniel
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [patch 16/33] fs: dcache per-bucket dcache hash locking
2009-09-04 6:51 ` [patch 16/33] fs: dcache per-bucket dcache hash locking npiggin
@ 2009-09-04 14:51 ` Daniel Walker
2009-09-07 7:38 ` Nick Piggin
0 siblings, 1 reply; 64+ messages in thread
From: Daniel Walker @ 2009-09-04 14:51 UTC (permalink / raw)
To: npiggin; +Cc: linux-fsdevel, linux-kernel
On Fri, 2009-09-04 at 16:51 +1000, npiggin@suse.de wrote:
> +/* This should be called _only_ with a lock pinning the dentry */
> +static inline struct dentry * __dget_locked_dlock(struct dentry
> *dentry)
> +{
> + dentry->d_count++;
> + dentry_lru_del_init(dentry);
> + return dentry;
> +}
> +
> +static inline struct dentry * __dget_locked(struct dentry *dentry)
> +{
Could you run your series through checkpatch, and clean up at least any
errors you find.. Plus add your signed off, most (all?) aren't signed
off..
Daniel
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [patch 04/33] fs: brlock vfsmount_lock
2009-09-04 6:51 ` [patch 04/33] fs: brlock vfsmount_lock npiggin
@ 2009-09-04 15:19 ` Jens Axboe
2009-09-07 7:39 ` Nick Piggin
2009-09-22 15:17 ` Al Viro
1 sibling, 1 reply; 64+ messages in thread
From: Jens Axboe @ 2009-09-04 15:19 UTC (permalink / raw)
To: npiggin; +Cc: linux-fsdevel, linux-kernel
On Fri, Sep 04 2009, npiggin@suse.de wrote:
> Index: linux-2.6/fs/namei.c
> ===================================================================
> --- linux-2.6.orig/fs/namei.c
> +++ linux-2.6/fs/namei.c
> @@ -679,15 +679,16 @@ int follow_up(struct path *path)
> {
> struct vfsmount *parent;
> struct dentry *mountpoint;
> - spin_lock(&vfsmount_lock);
> +
> + vfsmount_read_unlock();
> parent = path->mnt->mnt_parent;
> if (parent == path->mnt) {
> - spin_unlock(&vfsmount_lock);
> + vfsmount_read_unlock();
> return 0;
Hmm, that looks a bit off.
--
Jens Axboe
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [patch 09/33] fs: dcache scale dentry refcount
2009-09-04 6:51 ` [patch 09/33] fs: dcache scale dentry refcount npiggin
@ 2009-09-06 18:01 ` Eric Paris
2009-09-07 7:44 ` Nick Piggin
0 siblings, 1 reply; 64+ messages in thread
From: Eric Paris @ 2009-09-06 18:01 UTC (permalink / raw)
To: npiggin; +Cc: linux-fsdevel, linux-kernel
On Fri, Sep 4, 2009 at 2:51 AM, <npiggin@suse.de> wrote:
> Make d_count non-atomic and protect it with d_lock. This allows us to
> ensure a 0 refcount dentry remains 0 without dcache_lock. It is also
> fairly natural when we start protecting many other dentry members with
> d_lock.
> +struct dentry *dget_parent(struct dentry *dentry)
> +{
> + struct dentry *ret;
> +
> +repeat:
> + spin_lock(&dentry->d_lock);
> + ret = dentry->d_parent;
> + if (!ret)
> + goto out;
> + if (!spin_trylock(&ret->d_lock)) {
> + spin_unlock(&dentry->d_lock);
> + goto repeat;
> + }
> + BUG_ON(!ret->d_count);
> + ret->d_count++;
> + spin_unlock(&ret->d_lock);
> +out:
> + spin_unlock(&dentry->d_lock);
> + return ret;
> +}
> +EXPORT_SYMBOL(dget_parent);
> Index: linux-2.6/fs/notify/inotify/inotify.c
> ===================================================================
> --- linux-2.6.orig/fs/notify/inotify/inotify.c
> +++ linux-2.6/fs/notify/inotify/inotify.c
> @@ -339,18 +339,26 @@ void inotify_dentry_parent_queue_event(s
> if (!(dentry->d_flags & DCACHE_INOTIFY_PARENT_WATCHED))
> return;
>
> +again:
> spin_lock(&dentry->d_lock);
> parent = dentry->d_parent;
> + if (!spin_trylock(&parent->d_lock)) {
> + spin_unlock(&dentry->d_lock);
> + goto again;
> + }
> inode = parent->d_inode;
>
> if (inotify_inode_watched(inode)) {
> - dget(parent);
> + dget_dlock(parent);
> spin_unlock(&dentry->d_lock);
> + spin_unlock(&parent->d_lock);
> inotify_inode_queue_event(inode, mask, cookie, name,
> dentry->d_inode);
> dput(parent);
> - } else
> + } else {
> spin_unlock(&dentry->d_lock);
> + spin_unlock(&parent->d_lock);
> + }
I don't think I understand why in both of these cases you don't need
to check for dentry->d_parent == dentry
> Index: linux-2.6/fs/notify/fsnotify.c
> ===================================================================
> --- linux-2.6.orig/fs/notify/fsnotify.c
> +++ linux-2.6/fs/notify/fsnotify.c
> @@ -87,13 +87,18 @@ void __fsnotify_parent(struct dentry *de
> if (!(dentry->d_flags & DCACHE_FSNOTIFY_PARENT_WATCHED))
> return;
>
> +again:
> spin_lock(&dentry->d_lock);
> parent = dentry->d_parent;
> + if (parent != dentry && !spin_trylock(&parent->d_lock)) {
> + spin_unlock(&dentry->d_lock);
> + goto again;
> + }
> p_inode = parent->d_inode;
>
> if (fsnotify_inode_watches_children(p_inode)) {
> if (p_inode->i_fsnotify_mask & mask) {
> - dget(parent);
> + dget_dlock(parent);
> send = true;
> }
> } else {
And yet in this case we do check for dentry->d_parent == dentry. (my
unknowing self thinks we'd want to check in all places)
-Eric
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [patch 08/33] fs: dcache scale nr_dentry
2009-09-04 14:41 ` Daniel Walker
@ 2009-09-07 7:36 ` Nick Piggin
0 siblings, 0 replies; 64+ messages in thread
From: Nick Piggin @ 2009-09-07 7:36 UTC (permalink / raw)
To: Daniel Walker; +Cc: linux-fsdevel, linux-kernel
On Fri, Sep 04, 2009 at 07:41:02AM -0700, Daniel Walker wrote:
> On Fri, 2009-09-04 at 16:51 +1000, npiggin@suse.de wrote:
> > plain text document attachment (fs-dcache-scale-nr_dentry.patch)
> > Make dentry_stat_t.nr_dentry an atomic_t type, and move it from under
> > dcache_lock.
> > ---
> > fs/dcache.c | 20 +++++++++-----------
> > include/linux/dcache.h | 4 ++--
> > kernel/sysctl.c | 6 ++++++
> > 3 files changed, 17 insertions(+), 13 deletions(-)
> >
>
> No sign off on this one..
Ah, yeah... They're not really signoffable quality. Even a bit too
raw (not enough comments) for detailed review. I just want to get
the high level design out there again for comments because it is a
really big time investment to go further...
Just want to see if I make anyone upset or there are some bright
ideas of how to improve things :)
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [patch 16/33] fs: dcache per-bucket dcache hash locking
2009-09-04 14:51 ` Daniel Walker
@ 2009-09-07 7:38 ` Nick Piggin
0 siblings, 0 replies; 64+ messages in thread
From: Nick Piggin @ 2009-09-07 7:38 UTC (permalink / raw)
To: Daniel Walker; +Cc: linux-fsdevel, linux-kernel
On Fri, Sep 04, 2009 at 07:51:37AM -0700, Daniel Walker wrote:
> On Fri, 2009-09-04 at 16:51 +1000, npiggin@suse.de wrote:
> > +/* This should be called _only_ with a lock pinning the dentry */
> > +static inline struct dentry * __dget_locked_dlock(struct dentry
> > *dentry)
> > +{
> > + dentry->d_count++;
> > + dentry_lru_del_init(dentry);
> > + return dentry;
> > +}
> > +
> > +static inline struct dentry * __dget_locked(struct dentry *dentry)
> > +{
>
> Could you run your series through checkpatch, and clean up at least any
> errors you find..
Sure, I'll try to remember to do that at some point. fs/ code has a lot of
space after *, which I don't always fix up (and arguably is a clearer way
write pointer types). But yeah I'm happy to follow checkpatch style.
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [patch 04/33] fs: brlock vfsmount_lock
2009-09-04 15:19 ` Jens Axboe
@ 2009-09-07 7:39 ` Nick Piggin
0 siblings, 0 replies; 64+ messages in thread
From: Nick Piggin @ 2009-09-07 7:39 UTC (permalink / raw)
To: Jens Axboe; +Cc: linux-fsdevel, linux-kernel
On Fri, Sep 04, 2009 at 05:19:09PM +0200, Jens Axboe wrote:
> On Fri, Sep 04 2009, npiggin@suse.de wrote:
> > Index: linux-2.6/fs/namei.c
> > ===================================================================
> > --- linux-2.6.orig/fs/namei.c
> > +++ linux-2.6/fs/namei.c
> > @@ -679,15 +679,16 @@ int follow_up(struct path *path)
> > {
> > struct vfsmount *parent;
> > struct dentry *mountpoint;
> > - spin_lock(&vfsmount_lock);
> > +
> > + vfsmount_read_unlock();
> > parent = path->mnt->mnt_parent;
> > if (parent == path->mnt) {
> > - spin_unlock(&vfsmount_lock);
> > + vfsmount_read_unlock();
> > return 0;
>
> Hmm, that looks a bit off.
Thanks Jens, good catch. Yes I haven't actually even tested NFS or
NFSD yet, which I should do soon because they do some interesting
things with the dcache.
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [patch 09/33] fs: dcache scale dentry refcount
2009-09-06 18:01 ` Eric Paris
@ 2009-09-07 7:44 ` Nick Piggin
2009-09-07 11:21 ` Eric Paris
0 siblings, 1 reply; 64+ messages in thread
From: Nick Piggin @ 2009-09-07 7:44 UTC (permalink / raw)
To: Eric Paris; +Cc: linux-fsdevel, linux-kernel
On Sun, Sep 06, 2009 at 02:01:05PM -0400, Eric Paris wrote:
> On Fri, Sep 4, 2009 at 2:51 AM, <npiggin@suse.de> wrote:
>
> And yet in this case we do check for dentry->d_parent == dentry. (my
> unknowing self thinks we'd want to check in all places)
I think you're right, I think we need checks there too...
BTW Is there a way to test the fsnotify code? I thought inotify was
supposed to be implemented with fsnotify, but I see quite a lot
of duplicated (or very similar) code... We've also still got
inotify calls in fs/ (inotify_umount_inodes). and CONFIG_INOTIFY and
CONFIG_FSNOTIFY conditionals in there too. Would it be possible to
move that out into fsnotify calls? (fsnotify_inode_init_once or
whatever).
(Sorry to hijack your good review comments :))
Thanks,
Nick
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [patch 05/33] fs: scale mntget/mntput
2009-09-04 6:51 ` [patch 05/33] fs: scale mntget/mntput npiggin
@ 2009-09-07 9:41 ` Nick Piggin
0 siblings, 0 replies; 64+ messages in thread
From: Nick Piggin @ 2009-09-07 9:41 UTC (permalink / raw)
To: linux-fsdevel; +Cc: linux-kernel
On Fri, Sep 04, 2009 at 04:51:46PM +1000, npiggin@suse.de wrote:
> Improve scalability of mntget/mntput by using per-cpu counters protected
> by the reader side of the brlock vfsmount_lock. mnt_mounted keeps track of
> whether the vfsmount is actually attached to the tree so we can shortcut
> expensive checks in mntput.
Ah, I have a problem with count_mnt_count here...
count_mnt_count probably needs write lock otherwise you could count
CPU0, then counter is incremented on CPU0 and decremented on CPUN, then
you count the decrement on CPUN but have missed the increment.
count_mnt_count should be a slowpath anyway I think. may_umount_tree and
do_refcount_check I think need the write lock.
I think I was mis-remembering my mnt writers count per-cpu patches here,
but that's got a much more complex scheme to avoid atomic ops in the
fastpath... potentially we could possibly use the same scheme to speed
up mntget. Or if speed is not a problem then perhaps simplify mnt_want_write
with the use of the brlock vfsmount lock...
Anyway, I'll fix this one up the simple way for now, and I need to look
closely at single-threaded performance of this patchset in some areas anyway
so I'll revisit then.
> ---
> fs/libfs.c | 1
> fs/namespace.c | 122 +++++++++++++++++++++++++++++++++++++++++++-------
> fs/pnode.c | 2
> include/linux/mount.h | 33 ++++---------
> 4 files changed, 121 insertions(+), 37 deletions(-)
>
> Index: linux-2.6/fs/namespace.c
> ===================================================================
> --- linux-2.6.orig/fs/namespace.c
> +++ linux-2.6/fs/namespace.c
> @@ -177,6 +177,49 @@ void mnt_release_group_id(struct vfsmoun
> mnt->mnt_group_id = 0;
> }
>
> +static inline void add_mnt_count(struct vfsmount *mnt, int n)
> +{
> +#ifdef CONFIG_SMP
> + (*per_cpu_ptr(mnt->mnt_count, smp_processor_id())) += n;
> +#else
> + mnt->mnt_count += n;
> +#endif
> +}
> +
> +static inline void inc_mnt_count(struct vfsmount *mnt)
> +{
> +#ifdef CONFIG_SMP
> + (*per_cpu_ptr(mnt->mnt_count, smp_processor_id()))++;
> +#else
> + mnt->mnt_count++;
> +#endif
> +}
> +
> +static inline void dec_mnt_count(struct vfsmount *mnt)
> +{
> +#ifdef CONFIG_SMP
> + (*per_cpu_ptr(mnt->mnt_count, smp_processor_id()))--;
> +#else
> + mnt->mnt_count--;
> +#endif
> +}
> +
> +unsigned int count_mnt_count(struct vfsmount *mnt)
> +{
> +#ifdef CONFIG_SMP
> + unsigned int count = 0;
> + int cpu;
> +
> + for_each_possible_cpu(cpu) {
> + count += *per_cpu_ptr(mnt->mnt_count, cpu);
> + }
> +
> + return count;
> +#else
> + return mnt->mnt_count;
> +#endif
> +}
> +
> struct vfsmount *alloc_vfsmnt(const char *name)
> {
> struct vfsmount *mnt = kmem_cache_zalloc(mnt_cache, GFP_KERNEL);
> @@ -193,7 +236,13 @@ struct vfsmount *alloc_vfsmnt(const char
> goto out_free_id;
> }
>
> - atomic_set(&mnt->mnt_count, 1);
> +#ifdef CONFIG_SMP
> + mnt->mnt_count = alloc_percpu(int);
> + if (!mnt->mnt_count)
> + goto out_free_devname;
> +#else
> + mnt->mnt_count = 0;
> +#endif
> INIT_LIST_HEAD(&mnt->mnt_hash);
> INIT_LIST_HEAD(&mnt->mnt_child);
> INIT_LIST_HEAD(&mnt->mnt_mounts);
> @@ -205,14 +254,19 @@ struct vfsmount *alloc_vfsmnt(const char
> #ifdef CONFIG_SMP
> mnt->mnt_writers = alloc_percpu(int);
> if (!mnt->mnt_writers)
> - goto out_free_devname;
> + goto out_free_mntcount;
> #else
> mnt->mnt_writers = 0;
> #endif
> + preempt_disable();
> + inc_mnt_count(mnt);
> + preempt_enable();
> }
> return mnt;
>
> #ifdef CONFIG_SMP
> +out_free_mntcount:
> + free_percpu(mnt->mnt_count);
> out_free_devname:
> kfree(mnt->mnt_devname);
> #endif
> @@ -526,9 +580,11 @@ static void detach_mnt(struct vfsmount *
> old_path->mnt = mnt->mnt_parent;
> mnt->mnt_parent = mnt;
> mnt->mnt_mountpoint = mnt->mnt_root;
> - list_del_init(&mnt->mnt_child);
> list_del_init(&mnt->mnt_hash);
> + list_del_init(&mnt->mnt_child);
> old_path->dentry->d_mounted--;
> + WARN_ON(mnt->mnt_mounted != 1);
> + mnt->mnt_mounted--;
> }
>
> void mnt_set_mountpoint(struct vfsmount *mnt, struct dentry *dentry,
> @@ -545,6 +601,8 @@ static void attach_mnt(struct vfsmount *
> list_add_tail(&mnt->mnt_hash, mount_hashtable +
> hash(path->mnt, path->dentry));
> list_add_tail(&mnt->mnt_child, &path->mnt->mnt_mounts);
> + WARN_ON(mnt->mnt_mounted != 0);
> + mnt->mnt_mounted++;
> }
>
> /*
> @@ -567,6 +625,8 @@ static void commit_tree(struct vfsmount
> list_add_tail(&mnt->mnt_hash, mount_hashtable +
> hash(parent, mnt->mnt_mountpoint));
> list_add_tail(&mnt->mnt_child, &parent->mnt_mounts);
> + WARN_ON(mnt->mnt_mounted != 0);
> + mnt->mnt_mounted++;
> touch_mnt_namespace(n);
> }
>
> @@ -670,50 +730,80 @@ static inline void __mntput(struct vfsmo
>
> void mntput_no_expire(struct vfsmount *mnt)
> {
> -repeat:
> - /* open-code atomic_dec_and_lock for the vfsmount lock */
> - if (atomic_add_unless(&mnt->mnt_count, -1, 1))
> + if (likely(mnt->mnt_mounted)) {
> + vfsmount_read_lock();
> + if (unlikely(!mnt->mnt_mounted)) {
> + vfsmount_read_unlock();
> + goto repeat;
> + }
> + dec_mnt_count(mnt);
> + BUG_ON(count_mnt_count(mnt) == 0);
> + vfsmount_read_unlock();
> +
> return;
> + }
> +
> +repeat:
> vfsmount_write_lock();
> - if (!atomic_dec_and_test(&mnt->mnt_count)) {
> + BUG_ON(mnt->mnt_mounted);
> + dec_mnt_count(mnt);
> + if (count_mnt_count(mnt)) {
> vfsmount_write_unlock();
> return;
> }
> -
> if (likely(!mnt->mnt_pinned)) {
> vfsmount_write_unlock();
> __mntput(mnt);
> return;
> }
> - atomic_add(mnt->mnt_pinned + 1, &mnt->mnt_count);
> + add_mnt_count(mnt, mnt->mnt_pinned + 1);
> mnt->mnt_pinned = 0;
> vfsmount_write_unlock();
> acct_auto_close_mnt(mnt);
> security_sb_umount_close(mnt);
> goto repeat;
> }
> -
> EXPORT_SYMBOL(mntput_no_expire);
>
> +void mntput(struct vfsmount *mnt)
> +{
> + if (mnt) {
> + /* avoid cacheline pingpong */
> + if (unlikely(mnt->mnt_expiry_mark))
> + mnt->mnt_expiry_mark = 0;
> + mntput_no_expire(mnt);
> + }
> +}
> +EXPORT_SYMBOL(mntput);
> +
> +struct vfsmount *mntget(struct vfsmount *mnt)
> +{
> + if (mnt) {
> + preempt_disable();
> + inc_mnt_count(mnt);
> + preempt_enable();
> + }
> + return mnt;
> +}
> +EXPORT_SYMBOL(mntget);
> +
> void mnt_pin(struct vfsmount *mnt)
> {
> vfsmount_write_lock();
> mnt->mnt_pinned++;
> vfsmount_write_unlock();
> }
> -
> EXPORT_SYMBOL(mnt_pin);
>
> void mnt_unpin(struct vfsmount *mnt)
> {
> vfsmount_write_lock();
> if (mnt->mnt_pinned) {
> - atomic_inc(&mnt->mnt_count);
> + inc_mnt_count(mnt);
> mnt->mnt_pinned--;
> }
> vfsmount_write_unlock();
> }
> -
> EXPORT_SYMBOL(mnt_unpin);
>
> static inline void mangle(struct seq_file *m, const char *s)
> @@ -996,7 +1086,7 @@ int may_umount_tree(struct vfsmount *mnt
>
> vfsmount_read_lock();
> for (p = mnt; p; p = next_mnt(p, mnt)) {
> - actual_refs += atomic_read(&p->mnt_count);
> + actual_refs += count_mnt_count(p);
> minimum_refs += 2;
> }
> vfsmount_read_unlock();
> @@ -1076,6 +1166,8 @@ void umount_tree(struct vfsmount *mnt, i
> __touch_mnt_namespace(p->mnt_ns);
> p->mnt_ns = NULL;
> list_del_init(&p->mnt_child);
> + WARN_ON(p->mnt_mounted != 1);
> + p->mnt_mounted--;
> if (p->mnt_parent != p) {
> p->mnt_parent->mnt_ghosts++;
> p->mnt_mountpoint->d_mounted--;
> @@ -1107,7 +1199,7 @@ static int do_umount(struct vfsmount *mn
> flags & (MNT_FORCE | MNT_DETACH))
> return -EINVAL;
>
> - if (atomic_read(&mnt->mnt_count) != 2)
> + if (count_mnt_count(mnt) != 2)
> return -EBUSY;
>
> if (!xchg(&mnt->mnt_expiry_mark, 1))
> Index: linux-2.6/include/linux/mount.h
> ===================================================================
> --- linux-2.6.orig/include/linux/mount.h
> +++ linux-2.6/include/linux/mount.h
> @@ -56,20 +56,20 @@ struct vfsmount {
> struct mnt_namespace *mnt_ns; /* containing namespace */
> int mnt_id; /* mount identifier */
> int mnt_group_id; /* peer group identifier */
> - /*
> - * We put mnt_count & mnt_expiry_mark at the end of struct vfsmount
> - * to let these frequently modified fields in a separate cache line
> - * (so that reads of mnt_flags wont ping-pong on SMP machines)
> - */
> - atomic_t mnt_count;
> int mnt_expiry_mark; /* true if marked for expiry */
> int mnt_pinned;
> int mnt_ghosts;
> + int mnt_mounted;
> #ifdef CONFIG_SMP
> int *mnt_writers;
> #else
> int mnt_writers;
> #endif
> +#ifdef CONFIG_SMP
> + int *mnt_count;
> +#else
> + int mnt_count;
> +#endif
> };
>
> static inline int *get_mnt_writers_ptr(struct vfsmount *mnt)
> @@ -81,13 +81,6 @@ static inline int *get_mnt_writers_ptr(s
> #endif
> }
>
> -static inline struct vfsmount *mntget(struct vfsmount *mnt)
> -{
> - if (mnt)
> - atomic_inc(&mnt->mnt_count);
> - return mnt;
> -}
> -
> struct file; /* forward dec */
>
> extern void vfsmount_read_lock(void);
> @@ -95,23 +88,21 @@ extern void vfsmount_read_unlock(void);
> extern void vfsmount_write_lock(void);
> extern void vfsmount_write_unlock(void);
>
> +extern unsigned int count_mnt_count(struct vfsmount *mnt);
> +
> extern int mnt_want_write(struct vfsmount *mnt);
> extern int mnt_want_write_file(struct file *file);
> extern int mnt_clone_write(struct vfsmount *mnt);
> extern void mnt_drop_write(struct vfsmount *mnt);
> +
> extern void mntput_no_expire(struct vfsmount *mnt);
> +extern struct vfsmount *mntget(struct vfsmount *mnt);
> +extern void mntput(struct vfsmount *mnt);
> +
> extern void mnt_pin(struct vfsmount *mnt);
> extern void mnt_unpin(struct vfsmount *mnt);
> extern int __mnt_is_readonly(struct vfsmount *mnt);
>
> -static inline void mntput(struct vfsmount *mnt)
> -{
> - if (mnt) {
> - mnt->mnt_expiry_mark = 0;
> - mntput_no_expire(mnt);
> - }
> -}
> -
> extern struct vfsmount *do_kern_mount(const char *fstype, int flags,
> const char *name, void *data);
>
> Index: linux-2.6/fs/pnode.c
> ===================================================================
> --- linux-2.6.orig/fs/pnode.c
> +++ linux-2.6/fs/pnode.c
> @@ -279,7 +279,7 @@ out:
> */
> static inline int do_refcount_check(struct vfsmount *mnt, int count)
> {
> - int mycount = atomic_read(&mnt->mnt_count) - mnt->mnt_ghosts;
> + int mycount = count_mnt_count(mnt) - mnt->mnt_ghosts;
> return (mycount > count);
> }
>
> Index: linux-2.6/fs/libfs.c
> ===================================================================
> --- linux-2.6.orig/fs/libfs.c
> +++ linux-2.6/fs/libfs.c
> @@ -244,6 +244,7 @@ int get_sb_pseudo(struct file_system_typ
> d_instantiate(dentry, root);
> s->s_root = dentry;
> s->s_flags |= MS_ACTIVE;
> + mnt->mnt_mounted++; /* never unmounted, shortcut mntget (XXX: OK?) */
> simple_set_mnt(mnt, s);
> return 0;
>
>
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [patch 09/33] fs: dcache scale dentry refcount
2009-09-07 7:44 ` Nick Piggin
@ 2009-09-07 11:21 ` Eric Paris
2009-09-07 11:35 ` Nick Piggin
0 siblings, 1 reply; 64+ messages in thread
From: Eric Paris @ 2009-09-07 11:21 UTC (permalink / raw)
To: Nick Piggin; +Cc: Eric Paris, linux-fsdevel, linux-kernel
On Mon, 2009-09-07 at 09:44 +0200, Nick Piggin wrote:
> On Sun, Sep 06, 2009 at 02:01:05PM -0400, Eric Paris wrote:
> > On Fri, Sep 4, 2009 at 2:51 AM, <npiggin@suse.de> wrote:
> >
> > And yet in this case we do check for dentry->d_parent == dentry. (my
> > unknowing self thinks we'd want to check in all places)
>
> I think you're right, I think we need checks there too...
>
> BTW Is there a way to test the fsnotify code? I thought inotify was
> supposed to be implemented with fsnotify, but I see quite a lot
> of duplicated (or very similar) code... We've also still got
> inotify calls in fs/ (inotify_umount_inodes). and CONFIG_INOTIFY and
> CONFIG_FSNOTIFY conditionals in there too. Would it be possible to
> move that out into fsnotify calls? (fsnotify_inode_init_once or
> whatever).
>
> (Sorry to hijack your good review comments :))
Actually inotify.c in linux-next is dead code which I plan to remove
in .32. In .31 inotify.c is used by the audit subsystem but inotify as
seen by userspace is implemented on top of fsnotify.
I'll try to pull down the whole patch series and test for any problems.
-Eric
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [patch 09/33] fs: dcache scale dentry refcount
2009-09-07 11:21 ` Eric Paris
@ 2009-09-07 11:35 ` Nick Piggin
0 siblings, 0 replies; 64+ messages in thread
From: Nick Piggin @ 2009-09-07 11:35 UTC (permalink / raw)
To: Eric Paris; +Cc: Eric Paris, linux-fsdevel, linux-kernel
On Mon, Sep 07, 2009 at 07:21:21AM -0400, Eric Paris wrote:
> On Mon, 2009-09-07 at 09:44 +0200, Nick Piggin wrote:
> > On Sun, Sep 06, 2009 at 02:01:05PM -0400, Eric Paris wrote:
> > > On Fri, Sep 4, 2009 at 2:51 AM, <npiggin@suse.de> wrote:
> > >
> > > And yet in this case we do check for dentry->d_parent == dentry. (my
> > > unknowing self thinks we'd want to check in all places)
> >
> > I think you're right, I think we need checks there too...
> >
> > BTW Is there a way to test the fsnotify code? I thought inotify was
> > supposed to be implemented with fsnotify, but I see quite a lot
> > of duplicated (or very similar) code... We've also still got
> > inotify calls in fs/ (inotify_umount_inodes). and CONFIG_INOTIFY and
> > CONFIG_FSNOTIFY conditionals in there too. Would it be possible to
> > move that out into fsnotify calls? (fsnotify_inode_init_once or
> > whatever).
> >
> > (Sorry to hijack your good review comments :))
>
> Actually inotify.c in linux-next is dead code which I plan to remove
> in .32. In .31 inotify.c is used by the audit subsystem but inotify as
> seen by userspace is implemented on top of fsnotify.
Ah, fine. That explains why I had the check in fsnotify but not inotify.
> I'll try to pull down the whole patch series and test for any problems.
I guess I should make it available as a git tree or at least a
patch rollup. People have spotted a few problems and I've found
a few more in testing more filesystems, so maybe hold off testing
and I'll get something newer out soon. It would be very appreciated
though.
Thanks,
Nick
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [patch 04/33] fs: brlock vfsmount_lock
2009-09-04 6:51 ` [patch 04/33] fs: brlock vfsmount_lock npiggin
2009-09-04 15:19 ` Jens Axboe
@ 2009-09-22 15:17 ` Al Viro
2009-09-27 19:56 ` Nick Piggin
1 sibling, 1 reply; 64+ messages in thread
From: Al Viro @ 2009-09-22 15:17 UTC (permalink / raw)
To: npiggin; +Cc: linux-fsdevel, linux-kernel
On Fri, Sep 04, 2009 at 04:51:45PM +1000, npiggin@suse.de wrote:
> Use a brlock for the vfsmount lock.
I like it, but I'd like to see how costly it becomes on heavily SMP boxen.
Creation/removal of bindings as load...
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [patch 04/33] fs: brlock vfsmount_lock
2009-09-22 15:17 ` Al Viro
@ 2009-09-27 19:56 ` Nick Piggin
2009-09-28 13:21 ` Peter Zijlstra
0 siblings, 1 reply; 64+ messages in thread
From: Nick Piggin @ 2009-09-27 19:56 UTC (permalink / raw)
To: Al Viro; +Cc: linux-fsdevel, linux-kernel
On Tue, Sep 22, 2009 at 04:17:51PM +0100, Al Viro wrote:
> On Fri, Sep 04, 2009 at 04:51:45PM +1000, npiggin@suse.de wrote:
> > Use a brlock for the vfsmount lock.
>
> I like it, but I'd like to see how costly it becomes on heavily SMP boxen.
> Creation/removal of bindings as load...
I could test that... Is there some realistic scenario I can try
to implement that exercises this? (failing that, I'll happily
do a microbenchmark).
I was thinking it *might* be possible to do RCU... but especially
coming up with a scheme that avoids synchronize_rcu() in the
umount path is not trivial, so perhaps the simple read/write
annotations with brlock behind the scenes is a more reasonable step.
I do also actually owe you some documentation with this one too,
which I will get around to adding.
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [patch 04/33] fs: brlock vfsmount_lock
2009-09-27 19:56 ` Nick Piggin
@ 2009-09-28 13:21 ` Peter Zijlstra
2009-10-01 2:10 ` Nick Piggin
0 siblings, 1 reply; 64+ messages in thread
From: Peter Zijlstra @ 2009-09-28 13:21 UTC (permalink / raw)
To: Nick Piggin; +Cc: Al Viro, linux-fsdevel, linux-kernel
On Sun, 2009-09-27 at 21:56 +0200, Nick Piggin wrote:
> On Tue, Sep 22, 2009 at 04:17:51PM +0100, Al Viro wrote:
> > On Fri, Sep 04, 2009 at 04:51:45PM +1000, npiggin@suse.de wrote:
> > > Use a brlock for the vfsmount lock.
> >
> > I like it, but I'd like to see how costly it becomes on heavily SMP boxen.
> > Creation/removal of bindings as load...
>
> I could test that... Is there some realistic scenario I can try
> to implement that exercises this? (failing that, I'll happily
> do a microbenchmark).
>
> I was thinking it *might* be possible to do RCU... but especially
> coming up with a scheme that avoids synchronize_rcu() in the
> umount path is not trivial, so perhaps the simple read/write
> annotations with brlock behind the scenes is a more reasonable step.
>
> I do also actually owe you some documentation with this one too,
> which I will get around to adding.
The thing that worries me is that the write-side is very heavy and the
read sides are spinning on it, yielding rather large spin times on large
smp boxen.
It wouldn't nearly be as bad if the read sides could block..
FWIW, spin_lock_nested is limited to 8 subclasses, so your current
implementation will explode on anything larger than an 8-way.
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [patch 03/33] fs: scale files_lock
2009-09-04 6:51 ` [patch 03/33] fs: scale files_lock npiggin
@ 2009-09-28 13:22 ` Peter Zijlstra
2009-09-28 13:24 ` Peter Zijlstra
1 sibling, 0 replies; 64+ messages in thread
From: Peter Zijlstra @ 2009-09-28 13:22 UTC (permalink / raw)
To: npiggin; +Cc: linux-fsdevel, linux-kernel
On Fri, 2009-09-04 at 16:51 +1000, npiggin@suse.de wrote:
> +static void file_list_lock_all(void)
> +{
> + int i;
> + int nr = 0;
> +
> + for_each_possible_cpu(i) {
> + spinlock_t *lock;
> +
> + lock = &per_cpu(files_cpulock, i);
> + spin_lock_nested(lock, nr);
> + nr++;
> + }
> +}
Same here, this'll make lockdep explode on >8-way. And create large spin
times on large smp boxen.
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [patch 03/33] fs: scale files_lock
2009-09-04 6:51 ` [patch 03/33] fs: scale files_lock npiggin
2009-09-28 13:22 ` Peter Zijlstra
@ 2009-09-28 13:24 ` Peter Zijlstra
2009-10-01 2:16 ` Nick Piggin
1 sibling, 1 reply; 64+ messages in thread
From: Peter Zijlstra @ 2009-09-28 13:24 UTC (permalink / raw)
To: npiggin; +Cc: linux-fsdevel, linux-kernel
On Fri, 2009-09-04 at 16:51 +1000, npiggin@suse.de wrote:
> Improve scalability of files_lock by adding per-cpu, per-sb files lists,
> protected with per-cpu locking. Effectively turning it into a big-writer
> lock.
What I did was fine-grain locking the double linked list so that you can
delete items without hitting a global lock.
For addition I added per-cpu list-heads that would be spliced onto the
global list once in a while.
Granted, the code was a tad involved... and hch wanted to get rid of
these lists, which is of course a much better solution.
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [patch 04/33] fs: brlock vfsmount_lock
2009-09-28 13:21 ` Peter Zijlstra
@ 2009-10-01 2:10 ` Nick Piggin
0 siblings, 0 replies; 64+ messages in thread
From: Nick Piggin @ 2009-10-01 2:10 UTC (permalink / raw)
To: Peter Zijlstra; +Cc: Al Viro, linux-fsdevel, linux-kernel
On Mon, Sep 28, 2009 at 03:21:03PM +0200, Peter Zijlstra wrote:
> On Sun, 2009-09-27 at 21:56 +0200, Nick Piggin wrote:
> > On Tue, Sep 22, 2009 at 04:17:51PM +0100, Al Viro wrote:
> > > On Fri, Sep 04, 2009 at 04:51:45PM +1000, npiggin@suse.de wrote:
> > > > Use a brlock for the vfsmount lock.
> > >
> > > I like it, but I'd like to see how costly it becomes on heavily SMP boxen.
> > > Creation/removal of bindings as load...
> >
> > I could test that... Is there some realistic scenario I can try
> > to implement that exercises this? (failing that, I'll happily
> > do a microbenchmark).
> >
> > I was thinking it *might* be possible to do RCU... but especially
> > coming up with a scheme that avoids synchronize_rcu() in the
> > umount path is not trivial, so perhaps the simple read/write
> > annotations with brlock behind the scenes is a more reasonable step.
> >
> > I do also actually owe you some documentation with this one too,
> > which I will get around to adding.
>
> The thing that worries me is that the write-side is very heavy and the
> read sides are spinning on it, yielding rather large spin times on large
> smp boxen.
Well, that might be true. Although I'd say that huge systems that
are doing a reasonable amount of vfs operations will have much
larger spin times today due to pathological queueing on the
global locks (almost to the point of effectively being a livelock).
> It wouldn't nearly be as bad if the read sides could block..
>
> FWIW, spin_lock_nested is limited to 8 subclasses, so your current
> implementation will explode on anything larger than an 8-way.
So OK, a _mergeable_ sequence for this will basicaly look like:
1. document and add read/write lock variations to the call sites
which is just a wrapper around existing lock (no functional
change).
2. add a general brlock implementation, which is nice to lockdep
(basically could just annotate it as an rwlock and be done)
And this is where -rt could also do something more appropriate
if needed.
3. switch vfsmount lock to brlock.
Someone also was interested in using brlocks elsewhere, you'll be
unhappy to know! :) Yes I would really like to see an RCU
implementation _eventually_, but as I say, that is probably a
whole project on its own.
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [patch 03/33] fs: scale files_lock
2009-09-28 13:24 ` Peter Zijlstra
@ 2009-10-01 2:16 ` Nick Piggin
0 siblings, 0 replies; 64+ messages in thread
From: Nick Piggin @ 2009-10-01 2:16 UTC (permalink / raw)
To: Peter Zijlstra; +Cc: linux-fsdevel, linux-kernel
On Mon, Sep 28, 2009 at 03:24:08PM +0200, Peter Zijlstra wrote:
> On Fri, 2009-09-04 at 16:51 +1000, npiggin@suse.de wrote:
> > Improve scalability of files_lock by adding per-cpu, per-sb files lists,
> > protected with per-cpu locking. Effectively turning it into a big-writer
> > lock.
>
> What I did was fine-grain locking the double linked list so that you can
> delete items without hitting a global lock.
>
> For addition I added per-cpu list-heads that would be spliced onto the
> global list once in a while.
>
> Granted, the code was a tad involved... and hch wanted to get rid of
> these lists, which is of course a much better solution.
I did see that of course, and I sent you a critique of it... I
didn't think it was appropriate for reasons I can't remember off
hand (either overly complex for the same task, or had a scalability
problem).
files_lock I would love to see go away completely, and in fact
depending on progress of work to that end, these patches may never
need to be merged. The problem I have is:
1. I don't want to significantly change data structures or cause
avoidable reductions in potential expressiveness of the data
structures we have. (I don't want someone to complain that my
patches suck because they want to be able to traverse files).
2. I need to take out this lock otherwise it become the choke
point and hides the rest of the progress on the rest of the
scalability work.
Again, I think brlock is not such a terrible thing for contention
especially if we're looking at umount slowpath... For this guy
actually though, the read side can probably be turned into RCU
traversal quite easily, I _think_.
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [patch 11/33] fs: dcache scale subdirs
2009-09-04 6:51 ` [patch 11/33] fs: dcache scale subdirs npiggin
@ 2010-06-17 15:13 ` Peter Zijlstra
2010-06-17 16:53 ` Nick Piggin
0 siblings, 1 reply; 64+ messages in thread
From: Peter Zijlstra @ 2010-06-17 15:13 UTC (permalink / raw)
To: npiggin
Cc: linux-fsdevel, linux-kernel, john stultz, John Kacur, Thomas Gleixner
On Fri, 2009-09-04 at 16:51 +1000, npiggin@suse.de wrote:
> @@ -932,6 +984,7 @@ static int select_parent(struct dentry *
> int found = 0;
>
> spin_lock(&dcache_lock);
> + spin_lock(&this_parent->d_lock);
> repeat:
> next = this_parent->d_subdirs.next;
> resume:
> @@ -939,8 +992,9 @@ resume:
> struct list_head *tmp = next;
> struct dentry *dentry = list_entry(tmp, struct dentry, d_u.d_child);
> next = tmp->next;
> + BUG_ON(this_parent == dentry);
>
> - spin_lock(&dentry->d_lock);
> + spin_lock_nested(&dentry->d_lock, DENTRY_D_LOCK_NESTED);
Right, so this isn't going to work well, this dentry recursion is
basically unbounded afaict, so the 2nd subdir will also be locked using
DENRTY_D_LOCKED_NESTED, resulting in the 1st and 2nd subdir both having
the same (sub)class and lockdep doesn't like that much.
Do we really need to keep the whole path locked? One of the comments
seems to suggest we could actually drop some locks and re-acquire.
> dentry_lru_del_init(dentry);
> /*
> * move only zero ref count dentries to the end
> @@ -950,33 +1004,45 @@ resume:
> dentry_lru_add_tail(dentry);
> found++;
> }
> - spin_unlock(&dentry->d_lock);
>
> /*
> * We can return to the caller if we have found some (this
> * ensures forward progress). We'll be coming back to find
> * the rest.
> */
> - if (found && need_resched())
> + if (found && need_resched()) {
> + spin_unlock(&dentry->d_lock);
> goto out;
> + }
>
> /*
> * Descend a level if the d_subdirs list is non-empty.
> */
> if (!list_empty(&dentry->d_subdirs)) {
> + spin_unlock(&this_parent->d_lock);
> + spin_release(&dentry->d_lock.dep_map, 1, _RET_IP_);
> this_parent = dentry;
> + spin_acquire(&this_parent->d_lock.dep_map, 0, 1, _RET_IP_);
> goto repeat;
> }
> +
> + spin_unlock(&dentry->d_lock);
> }
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [patch 11/33] fs: dcache scale subdirs
2010-06-17 15:13 ` Peter Zijlstra
@ 2010-06-17 16:53 ` Nick Piggin
2010-06-21 13:35 ` Peter Zijlstra
0 siblings, 1 reply; 64+ messages in thread
From: Nick Piggin @ 2010-06-17 16:53 UTC (permalink / raw)
To: Peter Zijlstra
Cc: linux-fsdevel, linux-kernel, john stultz, John Kacur, Thomas Gleixner
On Thu, Jun 17, 2010 at 05:13:35PM +0200, Peter Zijlstra wrote:
> On Fri, 2009-09-04 at 16:51 +1000, npiggin@suse.de wrote:
> > @@ -932,6 +984,7 @@ static int select_parent(struct dentry *
> > int found = 0;
> >
> > spin_lock(&dcache_lock);
> > + spin_lock(&this_parent->d_lock);
> > repeat:
> > next = this_parent->d_subdirs.next;
> > resume:
> > @@ -939,8 +992,9 @@ resume:
> > struct list_head *tmp = next;
> > struct dentry *dentry = list_entry(tmp, struct dentry, d_u.d_child);
> > next = tmp->next;
> > + BUG_ON(this_parent == dentry);
> >
> > - spin_lock(&dentry->d_lock);
> > + spin_lock_nested(&dentry->d_lock, DENTRY_D_LOCK_NESTED);
>
> Right, so this isn't going to work well, this dentry recursion is
> basically unbounded afaict, so the 2nd subdir will also be locked using
> DENRTY_D_LOCKED_NESTED, resulting in the 1st and 2nd subdir both having
> the same (sub)class and lockdep doesn't like that much.
No it's a bit of a trucky loop, but it is not unbounded. It takes the
parent, then the child, then it may continue again with the child as
the new parent but in that case it drops the parent lock and tricks
lockdep into not barfing.
> Do we really need to keep the whole path locked? One of the comments
> seems to suggest we could actually drop some locks and re-acquire.
As far as I can tell, RCU should be able to cover it without taking more
than 2 locks at a time. John saw some issues in the -rt tree (I haven't
reproduced yet) so he's locking the full chains there but I hope that
won't be needed.
>
> > dentry_lru_del_init(dentry);
> > /*
> > * move only zero ref count dentries to the end
> > @@ -950,33 +1004,45 @@ resume:
> > dentry_lru_add_tail(dentry);
> > found++;
> > }
> > - spin_unlock(&dentry->d_lock);
> >
> > /*
> > * We can return to the caller if we have found some (this
> > * ensures forward progress). We'll be coming back to find
> > * the rest.
> > */
> > - if (found && need_resched())
> > + if (found && need_resched()) {
> > + spin_unlock(&dentry->d_lock);
> > goto out;
> > + }
> >
> > /*
> > * Descend a level if the d_subdirs list is non-empty.
> > */
> > if (!list_empty(&dentry->d_subdirs)) {
> > + spin_unlock(&this_parent->d_lock);
> > + spin_release(&dentry->d_lock.dep_map, 1, _RET_IP_);
> > this_parent = dentry;
> > + spin_acquire(&this_parent->d_lock.dep_map, 0, 1, _RET_IP_);
> > goto repeat;
^^^ That's what we do when descending.
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [patch 11/33] fs: dcache scale subdirs
2010-06-17 16:53 ` Nick Piggin
@ 2010-06-21 13:35 ` Peter Zijlstra
2010-06-21 14:48 ` Nick Piggin
0 siblings, 1 reply; 64+ messages in thread
From: Peter Zijlstra @ 2010-06-21 13:35 UTC (permalink / raw)
To: Nick Piggin
Cc: linux-fsdevel, linux-kernel, john stultz, John Kacur, Thomas Gleixner
On Fri, 2010-06-18 at 02:53 +1000, Nick Piggin wrote:
> > Right, so this isn't going to work well, this dentry recursion is
> > basically unbounded afaict, so the 2nd subdir will also be locked using
> > DENRTY_D_LOCKED_NESTED, resulting in the 1st and 2nd subdir both having
> > the same (sub)class and lockdep doesn't like that much.
>
> No it's a bit of a trucky loop, but it is not unbounded. It takes the
> parent, then the child, then it may continue again with the child as
> the new parent but in that case it drops the parent lock and tricks
> lockdep into not barfing.
Ah, indeed the thing you pointed out below should work.
> > Do we really need to keep the whole path locked? One of the comments
> > seems to suggest we could actually drop some locks and re-acquire.
>
> As far as I can tell, RCU should be able to cover it without taking more
> than 2 locks at a time. John saw some issues in the -rt tree (I haven't
> reproduced yet) so he's locking the full chains there but I hope that
> won't be needed.
Right, so I was staring at the -rt splat, so its John who created that
wreckage?
static int select_parent(struct dentry * parent)
{
struct dentry *this_parent;
struct list_head *next;
unsigned seq;
int found;
rename_retry:
found = 0;
this_parent = parent;
seq = read_seqbegin(&rename_lock);
spin_lock(&this_parent->d_lock);
repeat:
next = this_parent->d_subdirs.next;
resume:
while (next != &this_parent->d_subdirs) {
struct list_head *tmp = next;
struct dentry *dentry = list_entry(tmp, struct dentry, d_u.d_child);
next = tmp->next;
spin_lock_nested(&dentry->d_lock, DENTRY_D_LOCK_NESTED);
dentry_lru_del_init(dentry);
/*
* move only zero ref count dentries to the end
* of the unused list for prune_dcache
*/
if (!atomic_read(&dentry->d_count)) {
dentry_lru_add_tail(dentry);
found++;
}
/*
* We can return to the caller if we have found some (this
* ensures forward progress). We'll be coming back to find
* the rest.
*/
if (found && need_resched()) {
spin_unlock(&dentry->d_lock);
goto out;
}
/*
* Descend a level if the d_subdirs list is non-empty.
* Note that we keep a hold on the parent lock while
* we descend, so we don't have to reacquire it on
* ascend.
*/
if (!list_empty(&dentry->d_subdirs)) {
this_parent = dentry;
goto repeat;
}
spin_unlock(&dentry->d_lock);
}
/*
* All done at this level ... ascend and resume the search.
*/
if (this_parent != parent) {
struct dentry *tmp;
struct dentry *child;
tmp = this_parent->d_parent;
child = this_parent;
next = child->d_u.d_child.next;
spin_unlock(&this_parent->d_lock);
this_parent = tmp;
goto resume;
}
out:
/* Make sure we unlock all the way back up the tree */
while (this_parent != parent) {
struct dentry *tmp = this_parent->d_parent;
spin_unlock(&this_parent->d_lock);
this_parent = tmp;
}
spin_unlock(&this_parent->d_lock);
if (read_seqretry(&rename_lock, seq))
goto rename_retry;
return found;
}
> > > /*
> > > * Descend a level if the d_subdirs list is non-empty.
> > > */
> > > if (!list_empty(&dentry->d_subdirs)) {
> > > + spin_unlock(&this_parent->d_lock);
> > > + spin_release(&dentry->d_lock.dep_map, 1, _RET_IP_);
> > > this_parent = dentry;
> > > + spin_acquire(&this_parent->d_lock.dep_map, 0, 1, _RET_IP_);
> > > goto repeat;
>
> ^^^ That's what we do when descending.
You can write that as:
lock_set_subclass(&this_parent->d_lock.dep_map, 0, _RET_IP_);
See kernel/sched.c:double_unlock_balance().
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [patch 11/33] fs: dcache scale subdirs
2010-06-21 13:35 ` Peter Zijlstra
@ 2010-06-21 14:48 ` Nick Piggin
2010-06-21 14:55 ` Peter Zijlstra
0 siblings, 1 reply; 64+ messages in thread
From: Nick Piggin @ 2010-06-21 14:48 UTC (permalink / raw)
To: Peter Zijlstra
Cc: linux-fsdevel, linux-kernel, john stultz, John Kacur, Thomas Gleixner
On Mon, Jun 21, 2010 at 03:35:22PM +0200, Peter Zijlstra wrote:
> On Fri, 2010-06-18 at 02:53 +1000, Nick Piggin wrote:
>
> > > Right, so this isn't going to work well, this dentry recursion is
> > > basically unbounded afaict, so the 2nd subdir will also be locked using
> > > DENRTY_D_LOCKED_NESTED, resulting in the 1st and 2nd subdir both having
> > > the same (sub)class and lockdep doesn't like that much.
> >
> > No it's a bit of a trucky loop, but it is not unbounded. It takes the
> > parent, then the child, then it may continue again with the child as
> > the new parent but in that case it drops the parent lock and tricks
> > lockdep into not barfing.
>
> Ah, indeed the thing you pointed out below should work.
>
> > > Do we really need to keep the whole path locked? One of the comments
> > > seems to suggest we could actually drop some locks and re-acquire.
> >
> > As far as I can tell, RCU should be able to cover it without taking more
> > than 2 locks at a time. John saw some issues in the -rt tree (I haven't
> > reproduced yet) so he's locking the full chains there but I hope that
> > won't be needed.
>
> Right, so I was staring at the -rt splat, so its John who created that
> wreckage?
It was, but apparently they saw an RCU bug there somewhere and hit it
with the big hammer. I haven't been able to reproduce it on a non-rt
kernel yet, and I see yet why RCU is not good enough here.
> > > > /*
> > > > * Descend a level if the d_subdirs list is non-empty.
> > > > */
> > > > if (!list_empty(&dentry->d_subdirs)) {
> > > > + spin_unlock(&this_parent->d_lock);
> > > > + spin_release(&dentry->d_lock.dep_map, 1, _RET_IP_);
> > > > this_parent = dentry;
> > > > + spin_acquire(&this_parent->d_lock.dep_map, 0, 1, _RET_IP_);
> > > > goto repeat;
> >
> > ^^^ That's what we do when descending.
>
> You can write that as:
> lock_set_subclass(&this_parent->d_lock.dep_map, 0, _RET_IP_);
>
> See kernel/sched.c:double_unlock_balance().
OK I'll keep that in mind, thanks!
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [patch 11/33] fs: dcache scale subdirs
2010-06-21 14:48 ` Nick Piggin
@ 2010-06-21 14:55 ` Peter Zijlstra
2010-06-22 6:02 ` john stultz
0 siblings, 1 reply; 64+ messages in thread
From: Peter Zijlstra @ 2010-06-21 14:55 UTC (permalink / raw)
To: Nick Piggin
Cc: linux-fsdevel, linux-kernel, john stultz, John Kacur, Thomas Gleixner
On Tue, 2010-06-22 at 00:48 +1000, Nick Piggin wrote:
> > Right, so I was staring at the -rt splat, so its John who created that
> > wreckage?
>
> It was, but apparently they saw an RCU bug there somewhere and hit it
> with the big hammer. I haven't been able to reproduce it on a non-rt
> kernel yet, and I see yet why RCU is not good enough here.
John, could you describe the failure you spotted?
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [patch 11/33] fs: dcache scale subdirs
2010-06-21 14:55 ` Peter Zijlstra
@ 2010-06-22 6:02 ` john stultz
2010-06-22 6:06 ` Nick Piggin
2010-06-22 7:27 ` Peter Zijlstra
0 siblings, 2 replies; 64+ messages in thread
From: john stultz @ 2010-06-22 6:02 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Nick Piggin, linux-fsdevel, linux-kernel, John Kacur, Thomas Gleixner
On Mon, 2010-06-21 at 16:55 +0200, Peter Zijlstra wrote:
> On Tue, 2010-06-22 at 00:48 +1000, Nick Piggin wrote:
> > > Right, so I was staring at the -rt splat, so its John who created that
> > > wreckage?
> >
> > It was, but apparently they saw an RCU bug there somewhere and hit it
> > with the big hammer. I haven't been able to reproduce it on a non-rt
> > kernel yet, and I see yet why RCU is not good enough here.
>
> John, could you describe the failure you spotted?
The problem was that the rcu_read_lock() on the dentry ascending wasn't
preventing d_put/d_kill from removing entries from the parent node. So
the next entry we tried to follow was invalid. So we were getting odd
oopses from select_parent().
I'm not as familiar with the rcu rules there, so the patch I made just
held the locks as it went down the chain. Not ideal of course, but still
an improvement over the dcache_lock that was there prior.
Peter: I'm sorry, I've been out for a few days. Can you give me some
background on what brought this up and what -rt splat you mean?
thanks
-john
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [patch 11/33] fs: dcache scale subdirs
2010-06-22 6:02 ` john stultz
@ 2010-06-22 6:06 ` Nick Piggin
2010-06-22 7:27 ` Peter Zijlstra
1 sibling, 0 replies; 64+ messages in thread
From: Nick Piggin @ 2010-06-22 6:06 UTC (permalink / raw)
To: john stultz
Cc: Peter Zijlstra, linux-fsdevel, linux-kernel, John Kacur, Thomas Gleixner
On Mon, Jun 21, 2010 at 11:02:37PM -0700, John Stultz wrote:
> On Mon, 2010-06-21 at 16:55 +0200, Peter Zijlstra wrote:
> > On Tue, 2010-06-22 at 00:48 +1000, Nick Piggin wrote:
> > > > Right, so I was staring at the -rt splat, so its John who created that
> > > > wreckage?
> > >
> > > It was, but apparently they saw an RCU bug there somewhere and hit it
> > > with the big hammer. I haven't been able to reproduce it on a non-rt
> > > kernel yet, and I see yet why RCU is not good enough here.
> >
> > John, could you describe the failure you spotted?
>
> The problem was that the rcu_read_lock() on the dentry ascending wasn't
> preventing d_put/d_kill from removing entries from the parent node. So
> the next entry we tried to follow was invalid. So we were getting odd
> oopses from select_parent().
Oh, ah OK that makes sense. I was thinking it was a use after grace
period problem. Hmm, I'll think about whether we can fix it better.
>
> I'm not as familiar with the rcu rules there, so the patch I made just
> held the locks as it went down the chain. Not ideal of course, but still
> an improvement over the dcache_lock that was there prior.
>
> Peter: I'm sorry, I've been out for a few days. Can you give me some
> background on what brought this up and what -rt splat you mean?
>
> thanks
> -john
>
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [patch 11/33] fs: dcache scale subdirs
2010-06-22 6:02 ` john stultz
2010-06-22 6:06 ` Nick Piggin
@ 2010-06-22 7:27 ` Peter Zijlstra
2010-06-23 2:03 ` john stultz
1 sibling, 1 reply; 64+ messages in thread
From: Peter Zijlstra @ 2010-06-22 7:27 UTC (permalink / raw)
To: john stultz
Cc: Nick Piggin, linux-fsdevel, linux-kernel, John Kacur, Thomas Gleixner
On Mon, 2010-06-21 at 23:02 -0700, john stultz wrote:
> On Mon, 2010-06-21 at 16:55 +0200, Peter Zijlstra wrote:
> > On Tue, 2010-06-22 at 00:48 +1000, Nick Piggin wrote:
> > > > Right, so I was staring at the -rt splat, so its John who created that
> > > > wreckage?
> > >
> > > It was, but apparently they saw an RCU bug there somewhere and hit it
> > > with the big hammer. I haven't been able to reproduce it on a non-rt
> > > kernel yet, and I see yet why RCU is not good enough here.
> >
> > John, could you describe the failure you spotted?
>
> The problem was that the rcu_read_lock() on the dentry ascending wasn't
> preventing d_put/d_kill from removing entries from the parent node. So
> the next entry we tried to follow was invalid. So we were getting odd
> oopses from select_parent().
>
> I'm not as familiar with the rcu rules there, so the patch I made just
> held the locks as it went down the chain. Not ideal of course, but still
> an improvement over the dcache_lock that was there prior.
>
> Peter: I'm sorry, I've been out for a few days. Can you give me some
> background on what brought this up and what -rt splat you mean?
Well, you make lockdep very unhappy by locking multiple dentries
(unbounded number) all in the same lock class.
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [patch 11/33] fs: dcache scale subdirs
2010-06-22 7:27 ` Peter Zijlstra
@ 2010-06-23 2:03 ` john stultz
2010-06-23 7:23 ` Peter Zijlstra
0 siblings, 1 reply; 64+ messages in thread
From: john stultz @ 2010-06-23 2:03 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Nick Piggin, linux-fsdevel, linux-kernel, John Kacur, Thomas Gleixner
On Tue, 2010-06-22 at 09:27 +0200, Peter Zijlstra wrote:
> On Mon, 2010-06-21 at 23:02 -0700, john stultz wrote:
> > On Mon, 2010-06-21 at 16:55 +0200, Peter Zijlstra wrote:
> > > On Tue, 2010-06-22 at 00:48 +1000, Nick Piggin wrote:
> > > > > Right, so I was staring at the -rt splat, so its John who created that
> > > > > wreckage?
> > > >
> > > > It was, but apparently they saw an RCU bug there somewhere and hit it
> > > > with the big hammer. I haven't been able to reproduce it on a non-rt
> > > > kernel yet, and I see yet why RCU is not good enough here.
> > >
> > > John, could you describe the failure you spotted?
> >
> > The problem was that the rcu_read_lock() on the dentry ascending wasn't
> > preventing d_put/d_kill from removing entries from the parent node. So
> > the next entry we tried to follow was invalid. So we were getting odd
> > oopses from select_parent().
> >
> > I'm not as familiar with the rcu rules there, so the patch I made just
> > held the locks as it went down the chain. Not ideal of course, but still
> > an improvement over the dcache_lock that was there prior.
> >
> > Peter: I'm sorry, I've been out for a few days. Can you give me some
> > background on what brought this up and what -rt splat you mean?
>
> Well, you make lockdep very unhappy by locking multiple dentries
> (unbounded number) all in the same lock class.
So.. Is there a way to tell lockdep that the nesting is ok (I thought
that was what the spin_lock_nested call was doing...)?
Or is locking a (possibly quite long) chain of objects really just a
do-not-do type of operation?
thanks
-john
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [patch 11/33] fs: dcache scale subdirs
2010-06-23 2:03 ` john stultz
@ 2010-06-23 7:23 ` Peter Zijlstra
0 siblings, 0 replies; 64+ messages in thread
From: Peter Zijlstra @ 2010-06-23 7:23 UTC (permalink / raw)
To: john stultz
Cc: Nick Piggin, linux-fsdevel, linux-kernel, John Kacur, Thomas Gleixner
On Tue, 2010-06-22 at 19:03 -0700, john stultz wrote:
> > Well, you make lockdep very unhappy by locking multiple dentries
> > (unbounded number) all in the same lock class.
>
> So.. Is there a way to tell lockdep that the nesting is ok (I thought
> that was what the spin_lock_nested call was doing...)?
spin_lock_nested() allows you to nest a limited number of locks (up to
8, although the usual case is 1).
> Or is locking a (possibly quite long) chain of objects really just a
> do-not-do type of operation?
Usually, yeah. It would be really nice to do this another way (also for
scalability, keeping a large subtree locked is bound to to lead to more
contention).
^ permalink raw reply [flat|nested] 64+ messages in thread
end of thread, other threads:[~2010-06-23 7:23 UTC | newest]
Thread overview: 64+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2009-09-04 6:51 [patch 00/33] my current vfs scalability patch queue npiggin
2009-09-04 6:51 ` [patch 01/33] fs: no games with DCACHE_UNHASHED npiggin
2009-09-04 6:51 ` [patch 02/33] fs: cleanup files_lock npiggin
2009-09-04 6:51 ` [patch 03/33] fs: scale files_lock npiggin
2009-09-28 13:22 ` Peter Zijlstra
2009-09-28 13:24 ` Peter Zijlstra
2009-10-01 2:16 ` Nick Piggin
2009-09-04 6:51 ` [patch 04/33] fs: brlock vfsmount_lock npiggin
2009-09-04 15:19 ` Jens Axboe
2009-09-07 7:39 ` Nick Piggin
2009-09-22 15:17 ` Al Viro
2009-09-27 19:56 ` Nick Piggin
2009-09-28 13:21 ` Peter Zijlstra
2009-10-01 2:10 ` Nick Piggin
2009-09-04 6:51 ` [patch 05/33] fs: scale mntget/mntput npiggin
2009-09-07 9:41 ` Nick Piggin
2009-09-04 6:51 ` [patch 06/33] fs: dcache scale hash npiggin
2009-09-04 6:51 ` [patch 07/33] fs: dcache scale lru npiggin
2009-09-04 6:51 ` [patch 08/33] fs: dcache scale nr_dentry npiggin
2009-09-04 14:41 ` Daniel Walker
2009-09-07 7:36 ` Nick Piggin
2009-09-04 6:51 ` [patch 09/33] fs: dcache scale dentry refcount npiggin
2009-09-06 18:01 ` Eric Paris
2009-09-07 7:44 ` Nick Piggin
2009-09-07 11:21 ` Eric Paris
2009-09-07 11:35 ` Nick Piggin
2009-09-04 6:51 ` [patch 10/33] fs: dcache scale d_unhashed npiggin
2009-09-04 6:51 ` [patch 11/33] fs: dcache scale subdirs npiggin
2010-06-17 15:13 ` Peter Zijlstra
2010-06-17 16:53 ` Nick Piggin
2010-06-21 13:35 ` Peter Zijlstra
2010-06-21 14:48 ` Nick Piggin
2010-06-21 14:55 ` Peter Zijlstra
2010-06-22 6:02 ` john stultz
2010-06-22 6:06 ` Nick Piggin
2010-06-22 7:27 ` Peter Zijlstra
2010-06-23 2:03 ` john stultz
2010-06-23 7:23 ` Peter Zijlstra
2009-09-04 6:51 ` [patch 12/33] fs: scale inode alias list npiggin
2009-09-04 6:51 ` [patch 13/33] fs: use RCU / seqlock logic for reverse and multi-step operaitons npiggin
2009-09-04 6:51 ` [patch 14/33] fs: dcache remove dcache_lock npiggin
2009-09-04 6:51 ` [patch 15/33] fs: dcache reduce dput locking npiggin
2009-09-04 6:51 ` [patch 16/33] fs: dcache per-bucket dcache hash locking npiggin
2009-09-04 14:51 ` Daniel Walker
2009-09-07 7:38 ` Nick Piggin
2009-09-04 6:51 ` [patch 17/33] fs: dcache reduce dcache_inode_lock npiggin
2009-09-04 6:51 ` [patch 18/33] fs: dcache per-inode inode alias locking npiggin
2009-09-04 6:52 ` [patch 19/33] fs: icache lock s_inodes list npiggin
2009-09-04 6:52 ` [patch 20/33] fs: icache lock inode hash npiggin
2009-09-04 6:52 ` [patch 21/33] fs: icache lock i_state npiggin
2009-09-04 6:52 ` [patch 22/33] fs: icache lock i_count npiggin
2009-09-04 6:52 ` [patch 23/33] fs: icache atomic inodes_stat npiggin
2009-09-04 6:52 ` [patch 24/33] fs: icache lock lru/writeback lists npiggin
2009-09-04 6:52 ` [patch 25/33] fs: icache protect inode state npiggin
2009-09-04 6:52 ` [patch 26/33] fs: inode atomic last_ino, iunique lock npiggin
2009-09-04 6:52 ` [patch 27/33] fs: icache remove inode_lock npiggin
2009-09-04 6:52 ` [patch 28/33] fs: inode factor hash lock into functions npiggin
2009-09-04 6:52 ` [patch 29/33] Remove the global inode_hash_lock and replace it with per-hash-bucket locks. fs: inode per-bucket inode hash locks npiggin
2009-09-04 7:05 ` Nick Piggin
2009-09-04 6:52 ` [patch 30/33] fs: inode lazy lru npiggin
2009-09-04 6:52 ` [patch 31/33] fs: RCU free inodes npiggin
2009-09-04 6:52 ` [patch 32/33] fs: rcu walk for i_sb_list npiggin
2009-09-04 6:52 ` [patch 33/33] fs: improve scalability of pseudo filesystems npiggin
2009-09-04 7:05 ` [patch 00/33] my current vfs scalability patch queue Nick Piggin
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).