* [PATCH 0/12] Per-bdi writeback flusher threads v7
@ 2009-05-26 9:33 Jens Axboe
2009-05-26 9:33 ` [PATCH 01/12] ntfs: remove old debug check for dirty data in ntfs_put_super() Jens Axboe
` (13 more replies)
0 siblings, 14 replies; 33+ messages in thread
From: Jens Axboe @ 2009-05-26 9:33 UTC (permalink / raw)
To: linux-kernel, linux-fsdevel
Cc: chris.mason, david, hch, akpm, jack, yanmin_zhang, richard
Hi,
Here's the 7th version of the writeback patches. Changes since
v5/v6:
- Move the sync_supers() to the global bdi_forker_task() thread, so we
don't writeback the supers from all the bdi kupdated() tasks.
- Make bdi_start_writeback() and bdi_writeback_all() be sync when called
with WB_SYNC_ALL only.
- Shuffle some more things around to make a cleaner series. The sync vs
async nature of bdi_writeback_all() and bdi_start_writeback() isn't
consistent through the series, but otherwise things should be sane.
I'd appreciate if Richard and Yanmin could re-run testing with this,
just to make sure that things are sane. For ease of patching, I've
put the full diff here:
http://kernel.dk/writeback-v7.patch
and also stored this in a writeback-v7 branch that will not change,
you can pull that into Linus tree from here:
git://git.kernel.dk/linux-2.6-block.git writeback-v7
--
Jens Axboe
^ permalink raw reply [flat|nested] 33+ messages in thread
* [PATCH 01/12] ntfs: remove old debug check for dirty data in ntfs_put_super()
2009-05-26 9:33 [PATCH 0/12] Per-bdi writeback flusher threads v7 Jens Axboe
@ 2009-05-26 9:33 ` Jens Axboe
2009-05-26 9:33 ` [PATCH 02/12] btrfs: properly register fs backing device Jens Axboe
` (12 subsequent siblings)
13 siblings, 0 replies; 33+ messages in thread
From: Jens Axboe @ 2009-05-26 9:33 UTC (permalink / raw)
To: linux-kernel, linux-fsdevel
Cc: chris.mason, david, hch, akpm, jack, yanmin_zhang, richard, Jens Axboe
This should not trigger anymore, so kill it.
Acked-by: Anton Altaparmakov <aia21@cam.ac.uk>
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
---
fs/ntfs/super.c | 33 +++------------------------------
1 files changed, 3 insertions(+), 30 deletions(-)
diff --git a/fs/ntfs/super.c b/fs/ntfs/super.c
index f76951d..3fc03bd 100644
--- a/fs/ntfs/super.c
+++ b/fs/ntfs/super.c
@@ -2373,39 +2373,12 @@ static void ntfs_put_super(struct super_block *sb)
vol->mftmirr_ino = NULL;
}
/*
- * If any dirty inodes are left, throw away all mft data page cache
- * pages to allow a clean umount. This should never happen any more
- * due to mft.c::ntfs_mft_writepage() cleaning all the dirty pages as
- * the underlying mft records are written out and cleaned. If it does,
- * happen anyway, we want to know...
+ * We should have no dirty inodes left, due to
+ * mft.c::ntfs_mft_writepage() cleaning all the dirty pages as
+ * the underlying mft records are written out and cleaned.
*/
ntfs_commit_inode(vol->mft_ino);
write_inode_now(vol->mft_ino, 1);
- if (sb_has_dirty_inodes(sb)) {
- const char *s1, *s2;
-
- mutex_lock(&vol->mft_ino->i_mutex);
- truncate_inode_pages(vol->mft_ino->i_mapping, 0);
- mutex_unlock(&vol->mft_ino->i_mutex);
- write_inode_now(vol->mft_ino, 1);
- if (sb_has_dirty_inodes(sb)) {
- static const char *_s1 = "inodes";
- static const char *_s2 = "";
- s1 = _s1;
- s2 = _s2;
- } else {
- static const char *_s1 = "mft pages";
- static const char *_s2 = "They have been thrown "
- "away. ";
- s1 = _s1;
- s2 = _s2;
- }
- ntfs_error(sb, "Dirty %s found at umount time. %sYou should "
- "run chkdsk. Please email "
- "linux-ntfs-dev@lists.sourceforge.net and say "
- "that you saw this message. Thank you.", s1,
- s2);
- }
#endif /* NTFS_RW */
iput(vol->mft_ino);
--
1.6.3.rc0.1.gf800
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCH 02/12] btrfs: properly register fs backing device
2009-05-26 9:33 [PATCH 0/12] Per-bdi writeback flusher threads v7 Jens Axboe
2009-05-26 9:33 ` [PATCH 01/12] ntfs: remove old debug check for dirty data in ntfs_put_super() Jens Axboe
@ 2009-05-26 9:33 ` Jens Axboe
2009-05-26 9:33 ` [PATCH 03/12] writeback: move dirty inodes from super_block to backing_dev_info Jens Axboe
` (11 subsequent siblings)
13 siblings, 0 replies; 33+ messages in thread
From: Jens Axboe @ 2009-05-26 9:33 UTC (permalink / raw)
To: linux-kernel, linux-fsdevel
Cc: chris.mason, david, hch, akpm, jack, yanmin_zhang, richard, Jens Axboe
btrfs assigns this bdi to all inodes on that file system, so make
sure it's registered. This isn't really important now, but will be
when we put dirty inodes there. Even now, we miss the stats when the
bdi isn't visible.
Also fixes failure to check bdi_init() return value, and bad inherit of
->capabilities flags from the default bdi.
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
---
fs/btrfs/disk-io.c | 23 ++++++++++++++++++-----
1 files changed, 18 insertions(+), 5 deletions(-)
diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c
index 4b0ea0b..2dc19c9 100644
--- a/fs/btrfs/disk-io.c
+++ b/fs/btrfs/disk-io.c
@@ -1345,12 +1345,24 @@ static void btrfs_unplug_io_fn(struct backing_dev_info *bdi, struct page *page)
free_extent_map(em);
}
+/*
+ * If this fails, caller must call bdi_destroy() to get rid of the
+ * bdi again.
+ */
static int setup_bdi(struct btrfs_fs_info *info, struct backing_dev_info *bdi)
{
- bdi_init(bdi);
+ int err;
+
+ bdi->capabilities = BDI_CAP_MAP_COPY;
+ err = bdi_init(bdi);
+ if (err)
+ return err;
+
+ err = bdi_register(bdi, NULL, "btrfs");
+ if (err)
+ return err;
+
bdi->ra_pages = default_backing_dev_info.ra_pages;
- bdi->state = 0;
- bdi->capabilities = default_backing_dev_info.capabilities;
bdi->unplug_io_fn = btrfs_unplug_io_fn;
bdi->unplug_io_data = info;
bdi->congested_fn = btrfs_congested_fn;
@@ -1574,7 +1586,8 @@ struct btrfs_root *open_ctree(struct super_block *sb,
fs_info->sb = sb;
fs_info->max_extent = (u64)-1;
fs_info->max_inline = 8192 * 1024;
- setup_bdi(fs_info, &fs_info->bdi);
+ if (setup_bdi(fs_info, &fs_info->bdi))
+ goto fail_bdi;
fs_info->btree_inode = new_inode(sb);
fs_info->btree_inode->i_ino = 1;
fs_info->btree_inode->i_nlink = 1;
@@ -1931,8 +1944,8 @@ fail_iput:
btrfs_close_devices(fs_info->fs_devices);
btrfs_mapping_tree_free(&fs_info->mapping_tree);
+fail_bdi:
bdi_destroy(&fs_info->bdi);
-
fail:
kfree(extent_root);
kfree(tree_root);
--
1.6.3.rc0.1.gf800
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCH 03/12] writeback: move dirty inodes from super_block to backing_dev_info
2009-05-26 9:33 [PATCH 0/12] Per-bdi writeback flusher threads v7 Jens Axboe
2009-05-26 9:33 ` [PATCH 01/12] ntfs: remove old debug check for dirty data in ntfs_put_super() Jens Axboe
2009-05-26 9:33 ` [PATCH 02/12] btrfs: properly register fs backing device Jens Axboe
@ 2009-05-26 9:33 ` Jens Axboe
2009-05-26 9:33 ` [PATCH 04/12] writeback: switch to per-bdi threads for flushing data Jens Axboe
` (10 subsequent siblings)
13 siblings, 0 replies; 33+ messages in thread
From: Jens Axboe @ 2009-05-26 9:33 UTC (permalink / raw)
To: linux-kernel, linux-fsdevel
Cc: chris.mason, david, hch, akpm, jack, yanmin_zhang, richard, Jens Axboe
This is a first step at introducing per-bdi flusher threads. We should
have no change in behaviour, although sb_has_dirty_inodes() is now
ridiculously expensive, as there's no easy way to answer that question.
Not a huge problem, since it'll be deleted in subsequent patches.
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
---
fs/fs-writeback.c | 196 +++++++++++++++++++++++++++---------------
fs/super.c | 3 -
include/linux/backing-dev.h | 9 ++
include/linux/fs.h | 5 +-
mm/backing-dev.c | 24 +++++
mm/page-writeback.c | 11 +--
6 files changed, 164 insertions(+), 84 deletions(-)
diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c
index 91013ff..1137408 100644
--- a/fs/fs-writeback.c
+++ b/fs/fs-writeback.c
@@ -25,6 +25,7 @@
#include <linux/buffer_head.h>
#include "internal.h"
+#define inode_to_bdi(inode) ((inode)->i_mapping->backing_dev_info)
/**
* writeback_acquire - attempt to get exclusive writeback access to a device
@@ -158,12 +159,13 @@ void __mark_inode_dirty(struct inode *inode, int flags)
goto out;
/*
- * If the inode was already on s_dirty/s_io/s_more_io, don't
- * reposition it (that would break s_dirty time-ordering).
+ * If the inode was already on b_dirty/b_io/b_more_io, don't
+ * reposition it (that would break b_dirty time-ordering).
*/
if (!was_dirty) {
inode->dirtied_when = jiffies;
- list_move(&inode->i_list, &sb->s_dirty);
+ list_move(&inode->i_list,
+ &inode_to_bdi(inode)->b_dirty);
}
}
out:
@@ -184,31 +186,30 @@ static int write_inode(struct inode *inode, int sync)
* furthest end of its superblock's dirty-inode list.
*
* Before stamping the inode's ->dirtied_when, we check to see whether it is
- * already the most-recently-dirtied inode on the s_dirty list. If that is
+ * already the most-recently-dirtied inode on the b_dirty list. If that is
* the case then the inode must have been redirtied while it was being written
* out and we don't reset its dirtied_when.
*/
static void redirty_tail(struct inode *inode)
{
- struct super_block *sb = inode->i_sb;
+ struct backing_dev_info *bdi = inode_to_bdi(inode);
- if (!list_empty(&sb->s_dirty)) {
- struct inode *tail_inode;
+ if (!list_empty(&bdi->b_dirty)) {
+ struct inode *tail;
- tail_inode = list_entry(sb->s_dirty.next, struct inode, i_list);
- if (time_before(inode->dirtied_when,
- tail_inode->dirtied_when))
+ tail = list_entry(bdi->b_dirty.next, struct inode, i_list);
+ if (time_before(inode->dirtied_when, tail->dirtied_when))
inode->dirtied_when = jiffies;
}
- list_move(&inode->i_list, &sb->s_dirty);
+ list_move(&inode->i_list, &bdi->b_dirty);
}
/*
- * requeue inode for re-scanning after sb->s_io list is exhausted.
+ * requeue inode for re-scanning after bdi->b_io list is exhausted.
*/
static void requeue_io(struct inode *inode)
{
- list_move(&inode->i_list, &inode->i_sb->s_more_io);
+ list_move(&inode->i_list, &inode_to_bdi(inode)->b_more_io);
}
static void inode_sync_complete(struct inode *inode)
@@ -255,18 +256,50 @@ static void move_expired_inodes(struct list_head *delaying_queue,
/*
* Queue all expired dirty inodes for io, eldest first.
*/
-static void queue_io(struct super_block *sb,
- unsigned long *older_than_this)
+static void queue_io(struct backing_dev_info *bdi,
+ unsigned long *older_than_this)
+{
+ list_splice_init(&bdi->b_more_io, bdi->b_io.prev);
+ move_expired_inodes(&bdi->b_dirty, &bdi->b_io, older_than_this);
+}
+
+static int sb_on_inode_list(struct super_block *sb, struct list_head *list)
{
- list_splice_init(&sb->s_more_io, sb->s_io.prev);
- move_expired_inodes(&sb->s_dirty, &sb->s_io, older_than_this);
+ struct inode *inode;
+ int ret = 0;
+
+ spin_lock(&inode_lock);
+ list_for_each_entry(inode, list, i_list) {
+ if (inode->i_sb == sb) {
+ ret = 1;
+ break;
+ }
+ }
+ spin_unlock(&inode_lock);
+ return ret;
}
int sb_has_dirty_inodes(struct super_block *sb)
{
- return !list_empty(&sb->s_dirty) ||
- !list_empty(&sb->s_io) ||
- !list_empty(&sb->s_more_io);
+ struct backing_dev_info *bdi;
+ int ret = 0;
+
+ /*
+ * This is REALLY expensive right now, but it'll go away
+ * when the bdi writeback is introduced
+ */
+ mutex_lock(&bdi_lock);
+ list_for_each_entry(bdi, &bdi_list, bdi_list) {
+ if (sb_on_inode_list(sb, &bdi->b_dirty) ||
+ sb_on_inode_list(sb, &bdi->b_io) ||
+ sb_on_inode_list(sb, &bdi->b_more_io)) {
+ ret = 1;
+ break;
+ }
+ }
+ mutex_unlock(&bdi_lock);
+
+ return ret;
}
EXPORT_SYMBOL(sb_has_dirty_inodes);
@@ -322,11 +355,11 @@ __sync_single_inode(struct inode *inode, struct writeback_control *wbc)
/*
* We didn't write back all the pages. nfs_writepages()
* sometimes bales out without doing anything. Redirty
- * the inode; Move it from s_io onto s_more_io/s_dirty.
+ * the inode; Move it from b_io onto b_more_io/b_dirty.
*/
/*
* akpm: if the caller was the kupdate function we put
- * this inode at the head of s_dirty so it gets first
+ * this inode at the head of b_dirty so it gets first
* consideration. Otherwise, move it to the tail, for
* the reasons described there. I'm not really sure
* how much sense this makes. Presumably I had a good
@@ -336,7 +369,7 @@ __sync_single_inode(struct inode *inode, struct writeback_control *wbc)
if (wbc->for_kupdate) {
/*
* For the kupdate function we move the inode
- * to s_more_io so it will get more writeout as
+ * to b_more_io so it will get more writeout as
* soon as the queue becomes uncongested.
*/
inode->i_state |= I_DIRTY_PAGES;
@@ -402,10 +435,10 @@ __writeback_single_inode(struct inode *inode, struct writeback_control *wbc)
if ((wbc->sync_mode != WB_SYNC_ALL) && (inode->i_state & I_SYNC)) {
/*
* We're skipping this inode because it's locked, and we're not
- * doing writeback-for-data-integrity. Move it to s_more_io so
- * that writeback can proceed with the other inodes on s_io.
+ * doing writeback-for-data-integrity. Move it to b_more_io so
+ * that writeback can proceed with the other inodes on b_io.
* We'll have another go at writing back this inode when we
- * completed a full scan of s_io.
+ * completed a full scan of b_io.
*/
requeue_io(inode);
return 0;
@@ -428,51 +461,34 @@ __writeback_single_inode(struct inode *inode, struct writeback_control *wbc)
return __sync_single_inode(inode, wbc);
}
-/*
- * Write out a superblock's list of dirty inodes. A wait will be performed
- * upon no inodes, all inodes or the final one, depending upon sync_mode.
- *
- * If older_than_this is non-NULL, then only write out inodes which
- * had their first dirtying at a time earlier than *older_than_this.
- *
- * If we're a pdflush thread, then implement pdflush collision avoidance
- * against the entire list.
- *
- * If `bdi' is non-zero then we're being asked to writeback a specific queue.
- * This function assumes that the blockdev superblock's inodes are backed by
- * a variety of queues, so all inodes are searched. For other superblocks,
- * assume that all inodes are backed by the same queue.
- *
- * FIXME: this linear search could get expensive with many fileystems. But
- * how to fix? We need to go from an address_space to all inodes which share
- * a queue with that address_space. (Easy: have a global "dirty superblocks"
- * list).
- *
- * The inodes to be written are parked on sb->s_io. They are moved back onto
- * sb->s_dirty as they are selected for writing. This way, none can be missed
- * on the writer throttling path, and we get decent balancing between many
- * throttled threads: we don't want them all piling up on inode_sync_wait.
- */
-void generic_sync_sb_inodes(struct super_block *sb,
- struct writeback_control *wbc)
+static void generic_sync_bdi_inodes(struct backing_dev_info *bdi,
+ struct writeback_control *wbc,
+ struct super_block *sb,
+ int is_blkdev_sb)
{
const unsigned long start = jiffies; /* livelock avoidance */
- int sync = wbc->sync_mode == WB_SYNC_ALL;
spin_lock(&inode_lock);
- if (!wbc->for_kupdate || list_empty(&sb->s_io))
- queue_io(sb, wbc->older_than_this);
- while (!list_empty(&sb->s_io)) {
- struct inode *inode = list_entry(sb->s_io.prev,
+ if (!wbc->for_kupdate || list_empty(&bdi->b_io))
+ queue_io(bdi, wbc->older_than_this);
+
+ while (!list_empty(&bdi->b_io)) {
+ struct inode *inode = list_entry(bdi->b_io.prev,
struct inode, i_list);
- struct address_space *mapping = inode->i_mapping;
- struct backing_dev_info *bdi = mapping->backing_dev_info;
long pages_skipped;
+ /*
+ * super block given and doesn't match, skip this inode
+ */
+ if (sb && sb != inode->i_sb) {
+ redirty_tail(inode);
+ continue;
+ }
+
if (!bdi_cap_writeback_dirty(bdi)) {
redirty_tail(inode);
- if (sb_is_blkdev_sb(sb)) {
+ if (is_blkdev_sb) {
/*
* Dirty memory-backed blockdev: the ramdisk
* driver does this. Skip just this inode
@@ -494,14 +510,14 @@ void generic_sync_sb_inodes(struct super_block *sb,
if (wbc->nonblocking && bdi_write_congested(bdi)) {
wbc->encountered_congestion = 1;
- if (!sb_is_blkdev_sb(sb))
+ if (!is_blkdev_sb)
break; /* Skip a congested fs */
requeue_io(inode);
continue; /* Skip a congested blockdev */
}
if (wbc->bdi && bdi != wbc->bdi) {
- if (!sb_is_blkdev_sb(sb))
+ if (!is_blkdev_sb)
break; /* fs has the wrong queue */
requeue_io(inode);
continue; /* blockdev has wrong queue */
@@ -539,13 +555,55 @@ void generic_sync_sb_inodes(struct super_block *sb,
wbc->more_io = 1;
break;
}
- if (!list_empty(&sb->s_more_io))
+ if (!list_empty(&bdi->b_more_io))
wbc->more_io = 1;
}
- if (sync) {
+ spin_unlock(&inode_lock);
+ /* Leave any unwritten inodes on b_io */
+}
+
+/*
+ * Write out a superblock's list of dirty inodes. A wait will be performed
+ * upon no inodes, all inodes or the final one, depending upon sync_mode.
+ *
+ * If older_than_this is non-NULL, then only write out inodes which
+ * had their first dirtying at a time earlier than *older_than_this.
+ *
+ * If we're a pdlfush thread, then implement pdflush collision avoidance
+ * against the entire list.
+ *
+ * If `bdi' is non-zero then we're being asked to writeback a specific queue.
+ * This function assumes that the blockdev superblock's inodes are backed by
+ * a variety of queues, so all inodes are searched. For other superblocks,
+ * assume that all inodes are backed by the same queue.
+ *
+ * FIXME: this linear search could get expensive with many fileystems. But
+ * how to fix? We need to go from an address_space to all inodes which share
+ * a queue with that address_space. (Easy: have a global "dirty superblocks"
+ * list).
+ *
+ * The inodes to be written are parked on bdi->b_io. They are moved back onto
+ * bdi->b_dirty as they are selected for writing. This way, none can be missed
+ * on the writer throttling path, and we get decent balancing between many
+ * throttled threads: we don't want them all piling up on inode_sync_wait.
+ */
+void generic_sync_sb_inodes(struct super_block *sb,
+ struct writeback_control *wbc)
+{
+ const int is_blkdev_sb = sb_is_blkdev_sb(sb);
+ struct backing_dev_info *bdi;
+
+ mutex_lock(&bdi_lock);
+ list_for_each_entry(bdi, &bdi_list, bdi_list)
+ generic_sync_bdi_inodes(bdi, wbc, sb, is_blkdev_sb);
+ mutex_unlock(&bdi_lock);
+
+ if (wbc->sync_mode == WB_SYNC_ALL) {
struct inode *inode, *old_inode = NULL;
+ spin_lock(&inode_lock);
+
/*
* Data integrity sync. Must wait for all pages under writeback,
* because there may have been pages dirtied before our sync
@@ -583,10 +641,8 @@ void generic_sync_sb_inodes(struct super_block *sb,
}
spin_unlock(&inode_lock);
iput(old_inode);
- } else
- spin_unlock(&inode_lock);
+ }
- return; /* Leave any unwritten inodes on s_io */
}
EXPORT_SYMBOL_GPL(generic_sync_sb_inodes);
@@ -601,8 +657,8 @@ static void sync_sb_inodes(struct super_block *sb,
*
* Note:
* We don't need to grab a reference to superblock here. If it has non-empty
- * ->s_dirty it's hadn't been killed yet and kill_super() won't proceed
- * past sync_inodes_sb() until the ->s_dirty/s_io/s_more_io lists are all
+ * ->b_dirty it's hadn't been killed yet and kill_super() won't proceed
+ * past sync_inodes_sb() until the ->b_dirty/b_io/b_more_io lists are all
* empty. Since __sync_single_inode() regains inode_lock before it finally moves
* inode from superblock lists we are OK.
*
diff --git a/fs/super.c b/fs/super.c
index 1943fdf..76dd5b2 100644
--- a/fs/super.c
+++ b/fs/super.c
@@ -64,9 +64,6 @@ static struct super_block *alloc_super(struct file_system_type *type)
s = NULL;
goto out;
}
- INIT_LIST_HEAD(&s->s_dirty);
- INIT_LIST_HEAD(&s->s_io);
- INIT_LIST_HEAD(&s->s_more_io);
INIT_LIST_HEAD(&s->s_files);
INIT_LIST_HEAD(&s->s_instances);
INIT_HLIST_HEAD(&s->s_anon);
diff --git a/include/linux/backing-dev.h b/include/linux/backing-dev.h
index 0ec2c59..8719c87 100644
--- a/include/linux/backing-dev.h
+++ b/include/linux/backing-dev.h
@@ -40,6 +40,8 @@ enum bdi_stat_item {
#define BDI_STAT_BATCH (8*(1+ilog2(nr_cpu_ids)))
struct backing_dev_info {
+ struct list_head bdi_list;
+
unsigned long ra_pages; /* max readahead in PAGE_CACHE_SIZE units */
unsigned long state; /* Always use atomic bitops on this */
unsigned int capabilities; /* Device capabilities */
@@ -58,6 +60,10 @@ struct backing_dev_info {
struct device *dev;
+ struct list_head b_dirty; /* dirty inodes */
+ struct list_head b_io; /* parked for writeback */
+ struct list_head b_more_io; /* parked for more writeback */
+
#ifdef CONFIG_DEBUG_FS
struct dentry *debug_dir;
struct dentry *debug_stats;
@@ -72,6 +78,9 @@ int bdi_register(struct backing_dev_info *bdi, struct device *parent,
int bdi_register_dev(struct backing_dev_info *bdi, dev_t dev);
void bdi_unregister(struct backing_dev_info *bdi);
+extern struct mutex bdi_lock;
+extern struct list_head bdi_list;
+
static inline void __add_bdi_stat(struct backing_dev_info *bdi,
enum bdi_stat_item item, s64 amount)
{
diff --git a/include/linux/fs.h b/include/linux/fs.h
index 3b534e5..6b475d4 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -712,7 +712,7 @@ static inline int mapping_writably_mapped(struct address_space *mapping)
struct inode {
struct hlist_node i_hash;
- struct list_head i_list;
+ struct list_head i_list; /* backing dev IO list */
struct list_head i_sb_list;
struct list_head i_dentry;
unsigned long i_ino;
@@ -1329,9 +1329,6 @@ struct super_block {
struct xattr_handler **s_xattr;
struct list_head s_inodes; /* all inodes */
- struct list_head s_dirty; /* dirty inodes */
- struct list_head s_io; /* parked for writeback */
- struct list_head s_more_io; /* parked for more writeback */
struct hlist_head s_anon; /* anonymous dentries for (nfs) exporting */
struct list_head s_files;
/* s_dentry_lru and s_nr_dentry_unused are protected by dcache_lock */
diff --git a/mm/backing-dev.c b/mm/backing-dev.c
index 493b468..de0bbfe 100644
--- a/mm/backing-dev.c
+++ b/mm/backing-dev.c
@@ -22,6 +22,8 @@ struct backing_dev_info default_backing_dev_info = {
EXPORT_SYMBOL_GPL(default_backing_dev_info);
static struct class *bdi_class;
+DEFINE_MUTEX(bdi_lock);
+LIST_HEAD(bdi_list);
#ifdef CONFIG_DEBUG_FS
#include <linux/debugfs.h>
@@ -211,6 +213,10 @@ int bdi_register(struct backing_dev_info *bdi, struct device *parent,
goto exit;
}
+ mutex_lock(&bdi_lock);
+ list_add_tail(&bdi->bdi_list, &bdi_list);
+ mutex_unlock(&bdi_lock);
+
bdi->dev = dev;
bdi_debug_register(bdi, dev_name(dev));
@@ -225,9 +231,17 @@ int bdi_register_dev(struct backing_dev_info *bdi, dev_t dev)
}
EXPORT_SYMBOL(bdi_register_dev);
+static void bdi_remove_from_list(struct backing_dev_info *bdi)
+{
+ mutex_lock(&bdi_lock);
+ list_del(&bdi->bdi_list);
+ mutex_unlock(&bdi_lock);
+}
+
void bdi_unregister(struct backing_dev_info *bdi)
{
if (bdi->dev) {
+ bdi_remove_from_list(bdi);
bdi_debug_unregister(bdi);
device_unregister(bdi->dev);
bdi->dev = NULL;
@@ -245,6 +259,10 @@ int bdi_init(struct backing_dev_info *bdi)
bdi->min_ratio = 0;
bdi->max_ratio = 100;
bdi->max_prop_frac = PROP_FRAC_BASE;
+ INIT_LIST_HEAD(&bdi->bdi_list);
+ INIT_LIST_HEAD(&bdi->b_io);
+ INIT_LIST_HEAD(&bdi->b_dirty);
+ INIT_LIST_HEAD(&bdi->b_more_io);
for (i = 0; i < NR_BDI_STAT_ITEMS; i++) {
err = percpu_counter_init(&bdi->bdi_stat[i], 0);
@@ -259,6 +277,8 @@ int bdi_init(struct backing_dev_info *bdi)
err:
while (i--)
percpu_counter_destroy(&bdi->bdi_stat[i]);
+
+ bdi_remove_from_list(bdi);
}
return err;
@@ -269,6 +289,10 @@ void bdi_destroy(struct backing_dev_info *bdi)
{
int i;
+ WARN_ON(!list_empty(&bdi->b_dirty));
+ WARN_ON(!list_empty(&bdi->b_io));
+ WARN_ON(!list_empty(&bdi->b_more_io));
+
bdi_unregister(bdi);
for (i = 0; i < NR_BDI_STAT_ITEMS; i++)
diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index bb553c3..7c44314 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -319,15 +319,13 @@ static void task_dirty_limit(struct task_struct *tsk, long *pdirty)
/*
*
*/
-static DEFINE_SPINLOCK(bdi_lock);
static unsigned int bdi_min_ratio;
int bdi_set_min_ratio(struct backing_dev_info *bdi, unsigned int min_ratio)
{
int ret = 0;
- unsigned long flags;
- spin_lock_irqsave(&bdi_lock, flags);
+ mutex_lock(&bdi_lock);
if (min_ratio > bdi->max_ratio) {
ret = -EINVAL;
} else {
@@ -339,27 +337,26 @@ int bdi_set_min_ratio(struct backing_dev_info *bdi, unsigned int min_ratio)
ret = -EINVAL;
}
}
- spin_unlock_irqrestore(&bdi_lock, flags);
+ mutex_unlock(&bdi_lock);
return ret;
}
int bdi_set_max_ratio(struct backing_dev_info *bdi, unsigned max_ratio)
{
- unsigned long flags;
int ret = 0;
if (max_ratio > 100)
return -EINVAL;
- spin_lock_irqsave(&bdi_lock, flags);
+ mutex_lock(&bdi_lock);
if (bdi->min_ratio > max_ratio) {
ret = -EINVAL;
} else {
bdi->max_ratio = max_ratio;
bdi->max_prop_frac = (PROP_FRAC_BASE * max_ratio) / 100;
}
- spin_unlock_irqrestore(&bdi_lock, flags);
+ mutex_unlock(&bdi_lock);
return ret;
}
--
1.6.3.rc0.1.gf800
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCH 04/12] writeback: switch to per-bdi threads for flushing data
2009-05-26 9:33 [PATCH 0/12] Per-bdi writeback flusher threads v7 Jens Axboe
` (2 preceding siblings ...)
2009-05-26 9:33 ` [PATCH 03/12] writeback: move dirty inodes from super_block to backing_dev_info Jens Axboe
@ 2009-05-26 9:33 ` Jens Axboe
2009-05-26 9:33 ` [PATCH 05/12] writeback: get rid of pdflush completely Jens Axboe
` (9 subsequent siblings)
13 siblings, 0 replies; 33+ messages in thread
From: Jens Axboe @ 2009-05-26 9:33 UTC (permalink / raw)
To: linux-kernel, linux-fsdevel
Cc: chris.mason, david, hch, akpm, jack, yanmin_zhang, richard, Jens Axboe
This gets rid of pdflush for bdi writeout and kupdated style cleaning.
This is an experiment to see if we get better writeout behaviour with
per-bdi flushing. Some initial tests look pretty encouraging. A sample
ffsb workload that does random writes to files is about 8% faster here
on a simple SATA drive during the benchmark phase. File layout also seems
a LOT more smooth in vmstat:
r b swpd free buff cache si so bi bo in cs us sy id wa
0 1 0 608848 2652 375372 0 0 0 71024 604 24 1 10 48 42
0 1 0 549644 2712 433736 0 0 0 60692 505 27 1 8 48 44
1 0 0 476928 2784 505192 0 0 4 29540 553 24 0 9 53 37
0 1 0 457972 2808 524008 0 0 0 54876 331 16 0 4 38 58
0 1 0 366128 2928 614284 0 0 4 92168 710 58 0 13 53 34
0 1 0 295092 3000 684140 0 0 0 62924 572 23 0 9 53 37
0 1 0 236592 3064 741704 0 0 4 58256 523 17 0 8 48 44
0 1 0 165608 3132 811464 0 0 0 57460 560 21 0 8 54 38
0 1 0 102952 3200 873164 0 0 4 74748 540 29 1 10 48 41
0 1 0 48604 3252 926472 0 0 0 53248 469 29 0 7 47 45
where vanilla tends to fluctuate a lot in the creation phase:
r b swpd free buff cache si so bi bo in cs us sy id wa
1 1 0 678716 5792 303380 0 0 0 74064 565 50 1 11 52 36
1 0 0 662488 5864 319396 0 0 4 352 302 329 0 2 47 51
0 1 0 599312 5924 381468 0 0 0 78164 516 55 0 9 51 40
0 1 0 519952 6008 459516 0 0 4 78156 622 56 1 11 52 37
1 1 0 436640 6092 541632 0 0 0 82244 622 54 0 11 48 41
0 1 0 436640 6092 541660 0 0 0 8 152 39 0 0 51 49
0 1 0 332224 6200 644252 0 0 4 102800 728 46 1 13 49 36
1 0 0 274492 6260 701056 0 0 4 12328 459 49 0 7 50 43
0 1 0 211220 6324 763356 0 0 0 106940 515 37 1 10 51 39
1 0 0 160412 6376 813468 0 0 0 8224 415 43 0 6 49 45
1 1 0 85980 6452 886556 0 0 4 113516 575 39 1 11 54 34
0 2 0 85968 6452 886620 0 0 0 1640 158 211 0 0 46 54
So apart from seemingly behaving better for buffered writeout, this also
allows us to potentially have more than one bdi thread flushing out data.
This may be useful for NUMA type setups.
A 10 disk test with btrfs performs 26% faster with per-bdi flushing. Other
tests pending.
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
---
fs/buffer.c | 2 +-
fs/fs-writeback.c | 314 ++++++++++++++++++++++++++-----------------
fs/sync.c | 2 +-
include/linux/backing-dev.h | 30 ++++
include/linux/fs.h | 3 +-
include/linux/writeback.h | 2 +-
mm/backing-dev.c | 185 ++++++++++++++++++++++++-
mm/page-writeback.c | 141 +------------------
mm/vmscan.c | 2 +-
9 files changed, 411 insertions(+), 270 deletions(-)
diff --git a/fs/buffer.c b/fs/buffer.c
index aed2977..14f0802 100644
--- a/fs/buffer.c
+++ b/fs/buffer.c
@@ -281,7 +281,7 @@ static void free_more_memory(void)
struct zone *zone;
int nid;
- wakeup_pdflush(1024);
+ wakeup_flusher_threads(1024);
yield();
for_each_online_node(nid) {
diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c
index 1137408..a30c164 100644
--- a/fs/fs-writeback.c
+++ b/fs/fs-writeback.c
@@ -19,6 +19,8 @@
#include <linux/sched.h>
#include <linux/fs.h>
#include <linux/mm.h>
+#include <linux/kthread.h>
+#include <linux/freezer.h>
#include <linux/writeback.h>
#include <linux/blkdev.h>
#include <linux/backing-dev.h>
@@ -61,10 +63,191 @@ int writeback_in_progress(struct backing_dev_info *bdi)
*/
static void writeback_release(struct backing_dev_info *bdi)
{
- BUG_ON(!writeback_in_progress(bdi));
+ WARN_ON_ONCE(!writeback_in_progress(bdi));
+ bdi->wb_arg.nr_pages = 0;
+ bdi->wb_arg.sb = NULL;
clear_bit(BDI_pdflush, &bdi->state);
}
+int bdi_start_writeback(struct backing_dev_info *bdi, struct super_block *sb,
+ long nr_pages, enum writeback_sync_modes sync_mode)
+{
+ /*
+ * This only happens the first time someone kicks this bdi, so put
+ * it out-of-line.
+ */
+ if (unlikely(!bdi->task)) {
+ bdi_add_default_flusher_task(bdi);
+ return 1;
+ }
+
+ if (writeback_acquire(bdi)) {
+ bdi->wb_arg.nr_pages = nr_pages;
+ bdi->wb_arg.sb = sb;
+ bdi->wb_arg.sync_mode = sync_mode;
+ /*
+ * make above store seen before the task is woken
+ */
+ smp_mb();
+ wake_up(&bdi->wait);
+ }
+
+ return 0;
+}
+
+/*
+ * The maximum number of pages to writeout in a single bdi flush/kupdate
+ * operation. We do this so we don't hold I_SYNC against an inode for
+ * enormous amounts of time, which would block a userspace task which has
+ * been forced to throttle against that inode. Also, the code reevaluates
+ * the dirty each time it has written this many pages.
+ */
+#define MAX_WRITEBACK_PAGES 1024
+
+/*
+ * Periodic writeback of "old" data.
+ *
+ * Define "old": the first time one of an inode's pages is dirtied, we mark the
+ * dirtying-time in the inode's address_space. So this periodic writeback code
+ * just walks the superblock inode list, writing back any inodes which are
+ * older than a specific point in time.
+ *
+ * Try to run once per dirty_writeback_interval. But if a writeback event
+ * takes longer than a dirty_writeback_interval interval, then leave a
+ * one-second gap.
+ *
+ * older_than_this takes precedence over nr_to_write. So we'll only write back
+ * all dirty pages if they are all attached to "old" mappings.
+ */
+static void bdi_kupdated(struct backing_dev_info *bdi)
+{
+ unsigned long oldest_jif;
+ long nr_to_write;
+ struct writeback_control wbc = {
+ .bdi = bdi,
+ .sync_mode = WB_SYNC_NONE,
+ .older_than_this = &oldest_jif,
+ .nr_to_write = 0,
+ .for_kupdate = 1,
+ .range_cyclic = 1,
+ };
+
+ oldest_jif = jiffies - msecs_to_jiffies(dirty_expire_interval * 10);
+
+ nr_to_write = global_page_state(NR_FILE_DIRTY) +
+ global_page_state(NR_UNSTABLE_NFS) +
+ (inodes_stat.nr_inodes - inodes_stat.nr_unused);
+
+ while (nr_to_write > 0) {
+ wbc.more_io = 0;
+ wbc.encountered_congestion = 0;
+ wbc.nr_to_write = MAX_WRITEBACK_PAGES;
+ generic_sync_bdi_inodes(NULL, &wbc);
+ if (wbc.nr_to_write > 0)
+ break; /* All the old data is written */
+ nr_to_write -= MAX_WRITEBACK_PAGES;
+ }
+}
+
+static inline bool over_bground_thresh(void)
+{
+ unsigned long background_thresh, dirty_thresh;
+
+ get_dirty_limits(&background_thresh, &dirty_thresh, NULL, NULL);
+
+ return (global_page_state(NR_FILE_DIRTY) +
+ global_page_state(NR_UNSTABLE_NFS) >= background_thresh);
+}
+
+static void bdi_pdflush(struct backing_dev_info *bdi)
+{
+ struct writeback_control wbc = {
+ .bdi = bdi,
+ .sync_mode = bdi->wb_arg.sync_mode,
+ .older_than_this = NULL,
+ .range_cyclic = 1,
+ };
+ long nr_pages = bdi->wb_arg.nr_pages;
+
+ for (;;) {
+ if (wbc.sync_mode == WB_SYNC_NONE && nr_pages <= 0 &&
+ !over_bground_thresh())
+ break;
+
+ wbc.more_io = 0;
+ wbc.encountered_congestion = 0;
+ wbc.nr_to_write = MAX_WRITEBACK_PAGES;
+ wbc.pages_skipped = 0;
+ generic_sync_bdi_inodes(bdi->wb_arg.sb, &wbc);
+ nr_pages -= MAX_WRITEBACK_PAGES - wbc.nr_to_write;
+ /*
+ * If we ran out of stuff to write, bail unless more_io got set
+ */
+ if (wbc.nr_to_write > 0 || wbc.pages_skipped > 0) {
+ if (wbc.more_io)
+ continue;
+ break;
+ }
+ }
+}
+
+/*
+ * Handle writeback of dirty data for the device backed by this bdi. Also
+ * wakes up periodically and does kupdated style flushing.
+ */
+int bdi_writeback_task(struct backing_dev_info *bdi)
+{
+ while (!kthread_should_stop()) {
+ unsigned long wait_jiffies;
+ DEFINE_WAIT(wait);
+
+ prepare_to_wait(&bdi->wait, &wait, TASK_INTERRUPTIBLE);
+ wait_jiffies = msecs_to_jiffies(dirty_writeback_interval * 10);
+ schedule_timeout(wait_jiffies);
+ try_to_freeze();
+
+ /*
+ * We get here in two cases:
+ *
+ * schedule_timeout() returned because the dirty writeback
+ * interval has elapsed. If that happens, we will be able
+ * to acquire the writeback lock and will proceed to do
+ * kupdated style writeout.
+ *
+ * Someone called bdi_start_writeback(), which will acquire
+ * the writeback lock. This means our writeback_acquire()
+ * below will fail and we call into bdi_pdflush() for
+ * pdflush style writeout.
+ *
+ */
+ if (writeback_acquire(bdi))
+ bdi_kupdated(bdi);
+ else
+ bdi_pdflush(bdi);
+
+ writeback_release(bdi);
+ finish_wait(&bdi->wait, &wait);
+ }
+
+ return 0;
+}
+
+void bdi_writeback_all(struct super_block *sb, long nr_pages,
+ enum writeback_sync_modes sync_mode)
+{
+ struct backing_dev_info *bdi, *tmp;
+
+ mutex_lock(&bdi_lock);
+
+ list_for_each_entry_safe(bdi, tmp, &bdi_list, bdi_list) {
+ if (!bdi_has_dirty_io(bdi))
+ continue;
+ bdi_start_writeback(bdi, sb, nr_pages, sync_mode);
+ }
+
+ mutex_unlock(&bdi_lock);
+}
+
/**
* __mark_inode_dirty - internal function
* @inode: inode to mark
@@ -263,46 +446,6 @@ static void queue_io(struct backing_dev_info *bdi,
move_expired_inodes(&bdi->b_dirty, &bdi->b_io, older_than_this);
}
-static int sb_on_inode_list(struct super_block *sb, struct list_head *list)
-{
- struct inode *inode;
- int ret = 0;
-
- spin_lock(&inode_lock);
- list_for_each_entry(inode, list, i_list) {
- if (inode->i_sb == sb) {
- ret = 1;
- break;
- }
- }
- spin_unlock(&inode_lock);
- return ret;
-}
-
-int sb_has_dirty_inodes(struct super_block *sb)
-{
- struct backing_dev_info *bdi;
- int ret = 0;
-
- /*
- * This is REALLY expensive right now, but it'll go away
- * when the bdi writeback is introduced
- */
- mutex_lock(&bdi_lock);
- list_for_each_entry(bdi, &bdi_list, bdi_list) {
- if (sb_on_inode_list(sb, &bdi->b_dirty) ||
- sb_on_inode_list(sb, &bdi->b_io) ||
- sb_on_inode_list(sb, &bdi->b_more_io)) {
- ret = 1;
- break;
- }
- }
- mutex_unlock(&bdi_lock);
-
- return ret;
-}
-EXPORT_SYMBOL(sb_has_dirty_inodes);
-
/*
* Write a single inode's dirty pages and inode data out to disk.
* If `wait' is set, wait on the writeout.
@@ -461,11 +604,11 @@ __writeback_single_inode(struct inode *inode, struct writeback_control *wbc)
return __sync_single_inode(inode, wbc);
}
-static void generic_sync_bdi_inodes(struct backing_dev_info *bdi,
- struct writeback_control *wbc,
- struct super_block *sb,
- int is_blkdev_sb)
+void generic_sync_bdi_inodes(struct super_block *sb,
+ struct writeback_control *wbc)
{
+ const int is_blkdev_sb = sb_is_blkdev_sb(sb);
+ struct backing_dev_info *bdi = wbc->bdi;
const unsigned long start = jiffies; /* livelock avoidance */
spin_lock(&inode_lock);
@@ -516,13 +659,6 @@ static void generic_sync_bdi_inodes(struct backing_dev_info *bdi,
continue; /* Skip a congested blockdev */
}
- if (wbc->bdi && bdi != wbc->bdi) {
- if (!is_blkdev_sb)
- break; /* fs has the wrong queue */
- requeue_io(inode);
- continue; /* blockdev has wrong queue */
- }
-
/*
* Was this inode dirtied after sync_sb_inodes was called?
* This keeps sync from extra jobs and livelock.
@@ -530,16 +666,10 @@ static void generic_sync_bdi_inodes(struct backing_dev_info *bdi,
if (inode_dirtied_after(inode, start))
break;
- /* Is another pdflush already flushing this queue? */
- if (current_is_pdflush() && !writeback_acquire(bdi))
- break;
-
BUG_ON(inode->i_state & I_FREEING);
__iget(inode);
pages_skipped = wbc->pages_skipped;
__writeback_single_inode(inode, wbc);
- if (current_is_pdflush())
- writeback_release(bdi);
if (wbc->pages_skipped != pages_skipped) {
/*
* writeback is not making progress due to locked
@@ -578,11 +708,6 @@ static void generic_sync_bdi_inodes(struct backing_dev_info *bdi,
* a variety of queues, so all inodes are searched. For other superblocks,
* assume that all inodes are backed by the same queue.
*
- * FIXME: this linear search could get expensive with many fileystems. But
- * how to fix? We need to go from an address_space to all inodes which share
- * a queue with that address_space. (Easy: have a global "dirty superblocks"
- * list).
- *
* The inodes to be written are parked on bdi->b_io. They are moved back onto
* bdi->b_dirty as they are selected for writing. This way, none can be missed
* on the writer throttling path, and we get decent balancing between many
@@ -591,13 +716,10 @@ static void generic_sync_bdi_inodes(struct backing_dev_info *bdi,
void generic_sync_sb_inodes(struct super_block *sb,
struct writeback_control *wbc)
{
- const int is_blkdev_sb = sb_is_blkdev_sb(sb);
- struct backing_dev_info *bdi;
-
- mutex_lock(&bdi_lock);
- list_for_each_entry(bdi, &bdi_list, bdi_list)
- generic_sync_bdi_inodes(bdi, wbc, sb, is_blkdev_sb);
- mutex_unlock(&bdi_lock);
+ if (wbc->bdi)
+ generic_sync_bdi_inodes(sb, wbc);
+ else
+ bdi_writeback_all(sb, wbc->nr_to_write, wbc->sync_mode);
if (wbc->sync_mode == WB_SYNC_ALL) {
struct inode *inode, *old_inode = NULL;
@@ -653,58 +775,6 @@ static void sync_sb_inodes(struct super_block *sb,
}
/*
- * Start writeback of dirty pagecache data against all unlocked inodes.
- *
- * Note:
- * We don't need to grab a reference to superblock here. If it has non-empty
- * ->b_dirty it's hadn't been killed yet and kill_super() won't proceed
- * past sync_inodes_sb() until the ->b_dirty/b_io/b_more_io lists are all
- * empty. Since __sync_single_inode() regains inode_lock before it finally moves
- * inode from superblock lists we are OK.
- *
- * If `older_than_this' is non-zero then only flush inodes which have a
- * flushtime older than *older_than_this.
- *
- * If `bdi' is non-zero then we will scan the first inode against each
- * superblock until we find the matching ones. One group will be the dirty
- * inodes against a filesystem. Then when we hit the dummy blockdev superblock,
- * sync_sb_inodes will seekout the blockdev which matches `bdi'. Maybe not
- * super-efficient but we're about to do a ton of I/O...
- */
-void
-writeback_inodes(struct writeback_control *wbc)
-{
- struct super_block *sb;
-
- might_sleep();
- spin_lock(&sb_lock);
-restart:
- list_for_each_entry_reverse(sb, &super_blocks, s_list) {
- if (sb_has_dirty_inodes(sb)) {
- /* we're making our own get_super here */
- sb->s_count++;
- spin_unlock(&sb_lock);
- /*
- * If we can't get the readlock, there's no sense in
- * waiting around, most of the time the FS is going to
- * be unmounted by the time it is released.
- */
- if (down_read_trylock(&sb->s_umount)) {
- if (sb->s_root)
- sync_sb_inodes(sb, wbc);
- up_read(&sb->s_umount);
- }
- spin_lock(&sb_lock);
- if (__put_super_and_need_restart(sb))
- goto restart;
- }
- if (wbc->nr_to_write <= 0)
- break;
- }
- spin_unlock(&sb_lock);
-}
-
-/*
* writeback and wait upon the filesystem's dirty inodes. The caller will
* do this in two passes - one to write, and one to wait.
*
diff --git a/fs/sync.c b/fs/sync.c
index 7abc65f..3887f10 100644
--- a/fs/sync.c
+++ b/fs/sync.c
@@ -23,7 +23,7 @@
*/
static void do_sync(unsigned long wait)
{
- wakeup_pdflush(0);
+ wakeup_flusher_threads(0);
sync_inodes(0); /* All mappings, inodes and their blockdevs */
vfs_dq_sync(NULL);
sync_supers(); /* Write the superblocks */
diff --git a/include/linux/backing-dev.h b/include/linux/backing-dev.h
index 8719c87..f164925 100644
--- a/include/linux/backing-dev.h
+++ b/include/linux/backing-dev.h
@@ -13,6 +13,7 @@
#include <linux/proportions.h>
#include <linux/kernel.h>
#include <linux/fs.h>
+#include <linux/writeback.h>
#include <asm/atomic.h>
struct page;
@@ -24,6 +25,7 @@ struct dentry;
*/
enum bdi_state {
BDI_pdflush, /* A pdflush thread is working this device */
+ BDI_pending, /* On its way to being activated */
BDI_async_congested, /* The async (write) queue is getting full */
BDI_sync_congested, /* The sync queue is getting full */
BDI_unused, /* Available bits start here */
@@ -39,6 +41,12 @@ enum bdi_stat_item {
#define BDI_STAT_BATCH (8*(1+ilog2(nr_cpu_ids)))
+struct bdi_writeback_arg {
+ unsigned long nr_pages;
+ struct super_block *sb;
+ enum writeback_sync_modes sync_mode;
+};
+
struct backing_dev_info {
struct list_head bdi_list;
@@ -60,6 +68,9 @@ struct backing_dev_info {
struct device *dev;
+ struct task_struct *task; /* writeback task */
+ wait_queue_head_t wait;
+ struct bdi_writeback_arg wb_arg; /* protected by BDI_pdflush */
struct list_head b_dirty; /* dirty inodes */
struct list_head b_io; /* parked for writeback */
struct list_head b_more_io; /* parked for more writeback */
@@ -77,10 +88,23 @@ int bdi_register(struct backing_dev_info *bdi, struct device *parent,
const char *fmt, ...);
int bdi_register_dev(struct backing_dev_info *bdi, dev_t dev);
void bdi_unregister(struct backing_dev_info *bdi);
+int bdi_start_writeback(struct backing_dev_info *bdi, struct super_block *sb,
+ long nr_pages, enum writeback_sync_modes sync_mode);
+int bdi_writeback_task(struct backing_dev_info *bdi);
+void bdi_writeback_all(struct super_block *sb, long nr_pages,
+ enum writeback_sync_modes sync_mode);
+void bdi_add_default_flusher_task(struct backing_dev_info *bdi);
extern struct mutex bdi_lock;
extern struct list_head bdi_list;
+static inline int bdi_has_dirty_io(struct backing_dev_info *bdi)
+{
+ return !list_empty(&bdi->b_dirty) ||
+ !list_empty(&bdi->b_io) ||
+ !list_empty(&bdi->b_more_io);
+}
+
static inline void __add_bdi_stat(struct backing_dev_info *bdi,
enum bdi_stat_item item, s64 amount)
{
@@ -196,6 +220,7 @@ int bdi_set_max_ratio(struct backing_dev_info *bdi, unsigned int max_ratio);
#define BDI_CAP_EXEC_MAP 0x00000040
#define BDI_CAP_NO_ACCT_WB 0x00000080
#define BDI_CAP_SWAP_BACKED 0x00000100
+#define BDI_CAP_FLUSH_FORKER 0x00000200
#define BDI_CAP_VMFLAGS \
(BDI_CAP_READ_MAP | BDI_CAP_WRITE_MAP | BDI_CAP_EXEC_MAP)
@@ -265,6 +290,11 @@ static inline bool bdi_cap_swap_backed(struct backing_dev_info *bdi)
return bdi->capabilities & BDI_CAP_SWAP_BACKED;
}
+static inline bool bdi_cap_flush_forker(struct backing_dev_info *bdi)
+{
+ return bdi->capabilities & BDI_CAP_FLUSH_FORKER;
+}
+
static inline bool mapping_cap_writeback_dirty(struct address_space *mapping)
{
return bdi_cap_writeback_dirty(mapping->backing_dev_info);
diff --git a/include/linux/fs.h b/include/linux/fs.h
index 6b475d4..ecdc544 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -2063,6 +2063,8 @@ extern int invalidate_inode_pages2_range(struct address_space *mapping,
pgoff_t start, pgoff_t end);
extern void generic_sync_sb_inodes(struct super_block *sb,
struct writeback_control *wbc);
+extern void generic_sync_bdi_inodes(struct super_block *sb,
+ struct writeback_control *);
extern int write_inode_now(struct inode *, int);
extern int filemap_fdatawrite(struct address_space *);
extern int filemap_flush(struct address_space *);
@@ -2180,7 +2182,6 @@ extern int bdev_read_only(struct block_device *);
extern int set_blocksize(struct block_device *, int);
extern int sb_set_blocksize(struct super_block *, int);
extern int sb_min_blocksize(struct super_block *, int);
-extern int sb_has_dirty_inodes(struct super_block *);
extern int generic_file_mmap(struct file *, struct vm_area_struct *);
extern int generic_file_readonly_mmap(struct file *, struct vm_area_struct *);
diff --git a/include/linux/writeback.h b/include/linux/writeback.h
index 9344547..a8e9f78 100644
--- a/include/linux/writeback.h
+++ b/include/linux/writeback.h
@@ -99,7 +99,7 @@ static inline void inode_sync_wait(struct inode *inode)
/*
* mm/page-writeback.c
*/
-int wakeup_pdflush(long nr_pages);
+void wakeup_flusher_threads(long nr_pages);
void laptop_io_completion(void);
void laptop_sync_completion(void);
void throttle_vm_writeout(gfp_t gfp_mask);
diff --git a/mm/backing-dev.c b/mm/backing-dev.c
index de0bbfe..bae3d4f 100644
--- a/mm/backing-dev.c
+++ b/mm/backing-dev.c
@@ -1,8 +1,11 @@
#include <linux/wait.h>
#include <linux/backing-dev.h>
+#include <linux/kthread.h>
+#include <linux/freezer.h>
#include <linux/fs.h>
#include <linux/pagemap.h>
+#include <linux/mm.h>
#include <linux/sched.h>
#include <linux/module.h>
#include <linux/writeback.h>
@@ -16,7 +19,7 @@ EXPORT_SYMBOL(default_unplug_io_fn);
struct backing_dev_info default_backing_dev_info = {
.ra_pages = VM_MAX_READAHEAD * 1024 / PAGE_CACHE_SIZE,
.state = 0,
- .capabilities = BDI_CAP_MAP_COPY,
+ .capabilities = BDI_CAP_MAP_COPY | BDI_CAP_FLUSH_FORKER,
.unplug_io_fn = default_unplug_io_fn,
};
EXPORT_SYMBOL_GPL(default_backing_dev_info);
@@ -24,6 +27,7 @@ EXPORT_SYMBOL_GPL(default_backing_dev_info);
static struct class *bdi_class;
DEFINE_MUTEX(bdi_lock);
LIST_HEAD(bdi_list);
+LIST_HEAD(bdi_pending_list);
#ifdef CONFIG_DEBUG_FS
#include <linux/debugfs.h>
@@ -195,6 +199,139 @@ static int __init default_bdi_init(void)
}
subsys_initcall(default_bdi_init);
+static int bdi_start_fn(void *ptr)
+{
+ struct backing_dev_info *bdi = ptr;
+ struct task_struct *tsk = current;
+
+ /*
+ * Add us to the active bdi_list
+ */
+ mutex_lock(&bdi_lock);
+ list_add(&bdi->bdi_list, &bdi_list);
+ mutex_unlock(&bdi_lock);
+
+ tsk->flags |= PF_FLUSHER | PF_SWAPWRITE;
+ set_freezable();
+
+ /*
+ * Our parent may run at a different priority, just set us to normal
+ */
+ set_user_nice(tsk, 0);
+
+ /*
+ * Clear pending bit and wakeup anybody waiting to tear us down
+ */
+ clear_bit(BDI_pending, &bdi->state);
+ smp_mb__after_clear_bit();
+ wake_up_bit(&bdi->state, BDI_pending);
+
+ return bdi_writeback_task(bdi);
+}
+
+static void bdi_flush_io(struct backing_dev_info *bdi)
+{
+ struct writeback_control wbc = {
+ .bdi = bdi,
+ .sync_mode = WB_SYNC_NONE,
+ .older_than_this = NULL,
+ .range_cyclic = 1,
+ .nr_to_write = 1024,
+ };
+
+ generic_sync_bdi_inodes(NULL, &wbc);
+}
+
+static int bdi_forker_task(void *ptr)
+{
+ struct backing_dev_info *me = ptr;
+ DEFINE_WAIT(wait);
+
+ for (;;) {
+ struct backing_dev_info *bdi, *tmp;
+
+ /*
+ * Do this periodically, like kupdated() did before.
+ */
+ sync_supers();
+
+ /*
+ * Temporary measure, we want to make sure we don't see
+ * dirty data on the default backing_dev_info
+ */
+ if (bdi_has_dirty_io(me))
+ bdi_flush_io(me);
+
+ prepare_to_wait(&me->wait, &wait, TASK_INTERRUPTIBLE);
+
+ mutex_lock(&bdi_lock);
+
+ /*
+ * Check if any existing bdi's have dirty data without
+ * a thread registered. If so, set that up.
+ */
+ list_for_each_entry_safe(bdi, tmp, &bdi_list, bdi_list) {
+ if (bdi->task || !bdi_has_dirty_io(bdi))
+ continue;
+
+ bdi_add_default_flusher_task(bdi);
+ }
+
+ if (list_empty(&bdi_pending_list)) {
+ unsigned long wait;
+
+ mutex_unlock(&bdi_lock);
+ wait = msecs_to_jiffies(dirty_writeback_interval * 10);
+ schedule_timeout(wait);
+ try_to_freeze();
+ continue;
+ }
+
+ bdi = list_entry(bdi_pending_list.next, struct backing_dev_info,
+ bdi_list);
+ list_del_init(&bdi->bdi_list);
+ mutex_unlock(&bdi_lock);
+
+ BUG_ON(bdi->task);
+
+ bdi->task = kthread_run(bdi_start_fn, bdi, "bdi-%s",
+ dev_name(bdi->dev));
+ /*
+ * If task creation fails, then readd the bdi to
+ * the pending list and force writeout of the bdi
+ * from this forker thread. That will free some memory
+ * and we can try again.
+ */
+ if (!bdi->task) {
+ /*
+ * Add this 'bdi' to the back, so we get
+ * a chance to flush other bdi's to free
+ * memory.
+ */
+ mutex_lock(&bdi_lock);
+ list_add_tail(&bdi->bdi_list, &bdi_pending_list);
+ mutex_unlock(&bdi_lock);
+
+ bdi_flush_io(bdi);
+ }
+ }
+
+ finish_wait(&me->wait, &wait);
+ return 0;
+}
+
+void bdi_add_default_flusher_task(struct backing_dev_info *bdi)
+{
+ if (test_and_set_bit(BDI_pending, &bdi->state))
+ return;
+
+ mutex_lock(&bdi_lock);
+ list_move_tail(&bdi->bdi_list, &bdi_pending_list);
+ mutex_unlock(&bdi_lock);
+
+ wake_up(&default_backing_dev_info.wait);
+}
+
int bdi_register(struct backing_dev_info *bdi, struct device *parent,
const char *fmt, ...)
{
@@ -218,8 +355,25 @@ int bdi_register(struct backing_dev_info *bdi, struct device *parent,
mutex_unlock(&bdi_lock);
bdi->dev = dev;
- bdi_debug_register(bdi, dev_name(dev));
+ /*
+ * Just start the forker thread for our default backing_dev_info,
+ * and add other bdi's to the list. They will get a thread created
+ * on-demand when they need it.
+ */
+ if (bdi_cap_flush_forker(bdi)) {
+ bdi->task = kthread_run(bdi_forker_task, bdi, "bdi-%s",
+ dev_name(dev));
+ if (!bdi->task) {
+ mutex_lock(&bdi_lock);
+ list_del(&bdi->bdi_list);
+ mutex_unlock(&bdi_lock);
+ ret = -ENOMEM;
+ goto exit;
+ }
+ }
+
+ bdi_debug_register(bdi, dev_name(dev));
exit:
return ret;
}
@@ -231,8 +385,19 @@ int bdi_register_dev(struct backing_dev_info *bdi, dev_t dev)
}
EXPORT_SYMBOL(bdi_register_dev);
-static void bdi_remove_from_list(struct backing_dev_info *bdi)
+static int sched_wait(void *word)
+{
+ schedule();
+ return 0;
+}
+
+static void bdi_wb_shutdown(struct backing_dev_info *bdi)
{
+ /*
+ * If setup is pending, wait for that to complete first
+ */
+ wait_on_bit(&bdi->state, BDI_pending, sched_wait, TASK_UNINTERRUPTIBLE);
+
mutex_lock(&bdi_lock);
list_del(&bdi->bdi_list);
mutex_unlock(&bdi_lock);
@@ -241,7 +406,13 @@ static void bdi_remove_from_list(struct backing_dev_info *bdi)
void bdi_unregister(struct backing_dev_info *bdi)
{
if (bdi->dev) {
- bdi_remove_from_list(bdi);
+ if (!bdi_cap_flush_forker(bdi)) {
+ bdi_wb_shutdown(bdi);
+ if (bdi->task) {
+ kthread_stop(bdi->task);
+ bdi->task = NULL;
+ }
+ }
bdi_debug_unregister(bdi);
device_unregister(bdi->dev);
bdi->dev = NULL;
@@ -251,14 +422,14 @@ EXPORT_SYMBOL(bdi_unregister);
int bdi_init(struct backing_dev_info *bdi)
{
- int i;
- int err;
+ int i, err;
bdi->dev = NULL;
bdi->min_ratio = 0;
bdi->max_ratio = 100;
bdi->max_prop_frac = PROP_FRAC_BASE;
+ init_waitqueue_head(&bdi->wait);
INIT_LIST_HEAD(&bdi->bdi_list);
INIT_LIST_HEAD(&bdi->b_io);
INIT_LIST_HEAD(&bdi->b_dirty);
@@ -277,8 +448,6 @@ int bdi_init(struct backing_dev_info *bdi)
err:
while (i--)
percpu_counter_destroy(&bdi->bdi_stat[i]);
-
- bdi_remove_from_list(bdi);
}
return err;
diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index 7c44314..54a4a65 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -36,15 +36,6 @@
#include <linux/pagevec.h>
/*
- * The maximum number of pages to writeout in a single bdflush/kupdate
- * operation. We do this so we don't hold I_SYNC against an inode for
- * enormous amounts of time, which would block a userspace task which has
- * been forced to throttle against that inode. Also, the code reevaluates
- * the dirty each time it has written this many pages.
- */
-#define MAX_WRITEBACK_PAGES 1024
-
-/*
* After a CPU has dirtied this many pages, balance_dirty_pages_ratelimited
* will look to see if it needs to force writeback or throttling.
*/
@@ -117,8 +108,6 @@ EXPORT_SYMBOL(laptop_mode);
/* End of sysctl-exported parameters */
-static void background_writeout(unsigned long _min_pages);
-
/*
* Scale the writeback cache size proportional to the relative writeout speeds.
*
@@ -539,7 +528,7 @@ static void balance_dirty_pages(struct address_space *mapping)
* been flushed to permanent storage.
*/
if (bdi_nr_reclaimable) {
- writeback_inodes(&wbc);
+ generic_sync_bdi_inodes(NULL, &wbc);
pages_written += write_chunk - wbc.nr_to_write;
get_dirty_limits(&background_thresh, &dirty_thresh,
&bdi_thresh, bdi);
@@ -590,7 +579,7 @@ static void balance_dirty_pages(struct address_space *mapping)
(!laptop_mode && (global_page_state(NR_FILE_DIRTY)
+ global_page_state(NR_UNSTABLE_NFS)
> background_thresh)))
- pdflush_operation(background_writeout, 0);
+ bdi_start_writeback(bdi, NULL, 0, WB_SYNC_NONE);
}
void set_page_dirty_balance(struct page *page, int page_mkwrite)
@@ -675,152 +664,36 @@ void throttle_vm_writeout(gfp_t gfp_mask)
}
/*
- * writeback at least _min_pages, and keep writing until the amount of dirty
- * memory is less than the background threshold, or until we're all clean.
- */
-static void background_writeout(unsigned long _min_pages)
-{
- long min_pages = _min_pages;
- struct writeback_control wbc = {
- .bdi = NULL,
- .sync_mode = WB_SYNC_NONE,
- .older_than_this = NULL,
- .nr_to_write = 0,
- .nonblocking = 1,
- .range_cyclic = 1,
- };
-
- for ( ; ; ) {
- unsigned long background_thresh;
- unsigned long dirty_thresh;
-
- get_dirty_limits(&background_thresh, &dirty_thresh, NULL, NULL);
- if (global_page_state(NR_FILE_DIRTY) +
- global_page_state(NR_UNSTABLE_NFS) < background_thresh
- && min_pages <= 0)
- break;
- wbc.more_io = 0;
- wbc.encountered_congestion = 0;
- wbc.nr_to_write = MAX_WRITEBACK_PAGES;
- wbc.pages_skipped = 0;
- writeback_inodes(&wbc);
- min_pages -= MAX_WRITEBACK_PAGES - wbc.nr_to_write;
- if (wbc.nr_to_write > 0 || wbc.pages_skipped > 0) {
- /* Wrote less than expected */
- if (wbc.encountered_congestion || wbc.more_io)
- congestion_wait(WRITE, HZ/10);
- else
- break;
- }
- }
-}
-
-/*
* Start writeback of `nr_pages' pages. If `nr_pages' is zero, write back
* the whole world. Returns 0 if a pdflush thread was dispatched. Returns
* -1 if all pdflush threads were busy.
*/
-int wakeup_pdflush(long nr_pages)
+void wakeup_flusher_threads(long nr_pages)
{
if (nr_pages == 0)
nr_pages = global_page_state(NR_FILE_DIRTY) +
global_page_state(NR_UNSTABLE_NFS);
- return pdflush_operation(background_writeout, nr_pages);
+ bdi_writeback_all(NULL, nr_pages, WB_SYNC_NONE);
+ return;
}
-static void wb_timer_fn(unsigned long unused);
static void laptop_timer_fn(unsigned long unused);
-static DEFINE_TIMER(wb_timer, wb_timer_fn, 0, 0);
static DEFINE_TIMER(laptop_mode_wb_timer, laptop_timer_fn, 0, 0);
/*
- * Periodic writeback of "old" data.
- *
- * Define "old": the first time one of an inode's pages is dirtied, we mark the
- * dirtying-time in the inode's address_space. So this periodic writeback code
- * just walks the superblock inode list, writing back any inodes which are
- * older than a specific point in time.
- *
- * Try to run once per dirty_writeback_interval. But if a writeback event
- * takes longer than a dirty_writeback_interval interval, then leave a
- * one-second gap.
- *
- * older_than_this takes precedence over nr_to_write. So we'll only write back
- * all dirty pages if they are all attached to "old" mappings.
- */
-static void wb_kupdate(unsigned long arg)
-{
- unsigned long oldest_jif;
- unsigned long start_jif;
- unsigned long next_jif;
- long nr_to_write;
- struct writeback_control wbc = {
- .bdi = NULL,
- .sync_mode = WB_SYNC_NONE,
- .older_than_this = &oldest_jif,
- .nr_to_write = 0,
- .nonblocking = 1,
- .for_kupdate = 1,
- .range_cyclic = 1,
- };
-
- sync_supers();
-
- oldest_jif = jiffies - msecs_to_jiffies(dirty_expire_interval * 10);
- start_jif = jiffies;
- next_jif = start_jif + msecs_to_jiffies(dirty_writeback_interval * 10);
- nr_to_write = global_page_state(NR_FILE_DIRTY) +
- global_page_state(NR_UNSTABLE_NFS) +
- (inodes_stat.nr_inodes - inodes_stat.nr_unused);
- while (nr_to_write > 0) {
- wbc.more_io = 0;
- wbc.encountered_congestion = 0;
- wbc.nr_to_write = MAX_WRITEBACK_PAGES;
- writeback_inodes(&wbc);
- if (wbc.nr_to_write > 0) {
- if (wbc.encountered_congestion || wbc.more_io)
- congestion_wait(WRITE, HZ/10);
- else
- break; /* All the old data is written */
- }
- nr_to_write -= MAX_WRITEBACK_PAGES - wbc.nr_to_write;
- }
- if (time_before(next_jif, jiffies + HZ))
- next_jif = jiffies + HZ;
- if (dirty_writeback_interval)
- mod_timer(&wb_timer, next_jif);
-}
-
-/*
* sysctl handler for /proc/sys/vm/dirty_writeback_centisecs
*/
int dirty_writeback_centisecs_handler(ctl_table *table, int write,
struct file *file, void __user *buffer, size_t *length, loff_t *ppos)
{
proc_dointvec(table, write, file, buffer, length, ppos);
- if (dirty_writeback_interval)
- mod_timer(&wb_timer, jiffies +
- msecs_to_jiffies(dirty_writeback_interval * 10));
- else
- del_timer(&wb_timer);
return 0;
}
-static void wb_timer_fn(unsigned long unused)
-{
- if (pdflush_operation(wb_kupdate, 0) < 0)
- mod_timer(&wb_timer, jiffies + HZ); /* delay 1 second */
-}
-
-static void laptop_flush(unsigned long unused)
-{
- sys_sync();
-}
-
static void laptop_timer_fn(unsigned long unused)
{
- pdflush_operation(laptop_flush, 0);
+ wakeup_flusher_threads(0);
}
/*
@@ -903,8 +776,6 @@ void __init page_writeback_init(void)
{
int shift;
- mod_timer(&wb_timer,
- jiffies + msecs_to_jiffies(dirty_writeback_interval * 10));
writeback_set_ratelimit();
register_cpu_notifier(&ratelimit_nb);
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 5fa3eda..e37fd38 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -1654,7 +1654,7 @@ static unsigned long do_try_to_free_pages(struct zonelist *zonelist,
*/
if (total_scanned > sc->swap_cluster_max +
sc->swap_cluster_max / 2) {
- wakeup_pdflush(laptop_mode ? 0 : total_scanned);
+ wakeup_flusher_threads(laptop_mode ? 0 : total_scanned);
sc->may_writepage = 1;
}
--
1.6.3.rc0.1.gf800
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCH 05/12] writeback: get rid of pdflush completely
2009-05-26 9:33 [PATCH 0/12] Per-bdi writeback flusher threads v7 Jens Axboe
` (3 preceding siblings ...)
2009-05-26 9:33 ` [PATCH 04/12] writeback: switch to per-bdi threads for flushing data Jens Axboe
@ 2009-05-26 9:33 ` Jens Axboe
2009-05-26 9:33 ` [PATCH 06/12] writeback: separate the flushing state/task from the bdi Jens Axboe
` (8 subsequent siblings)
13 siblings, 0 replies; 33+ messages in thread
From: Jens Axboe @ 2009-05-26 9:33 UTC (permalink / raw)
To: linux-kernel, linux-fsdevel
Cc: chris.mason, david, hch, akpm, jack, yanmin_zhang, richard, Jens Axboe
It is now unused, so kill it off.
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
---
fs/fs-writeback.c | 5 +
include/linux/writeback.h | 12 --
mm/Makefile | 2 +-
mm/pdflush.c | 269 ---------------------------------------------
4 files changed, 6 insertions(+), 282 deletions(-)
delete mode 100644 mm/pdflush.c
diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c
index a30c164..7a558a6 100644
--- a/fs/fs-writeback.c
+++ b/fs/fs-writeback.c
@@ -29,6 +29,11 @@
#define inode_to_bdi(inode) ((inode)->i_mapping->backing_dev_info)
+/*
+ * We don't actually have pdflush, but this one is exported though /proc...
+ */
+int nr_pdflush_threads;
+
/**
* writeback_acquire - attempt to get exclusive writeback access to a device
* @bdi: the device's backing_dev_info structure
diff --git a/include/linux/writeback.h b/include/linux/writeback.h
index a8e9f78..baf04a9 100644
--- a/include/linux/writeback.h
+++ b/include/linux/writeback.h
@@ -14,17 +14,6 @@ extern struct list_head inode_in_use;
extern struct list_head inode_unused;
/*
- * Yes, writeback.h requires sched.h
- * No, sched.h is not included from here.
- */
-static inline int task_is_pdflush(struct task_struct *task)
-{
- return task->flags & PF_FLUSHER;
-}
-
-#define current_is_pdflush() task_is_pdflush(current)
-
-/*
* fs/fs-writeback.c
*/
enum writeback_sync_modes {
@@ -151,7 +140,6 @@ balance_dirty_pages_ratelimited(struct address_space *mapping)
typedef int (*writepage_t)(struct page *page, struct writeback_control *wbc,
void *data);
-int pdflush_operation(void (*fn)(unsigned long), unsigned long arg0);
int generic_writepages(struct address_space *mapping,
struct writeback_control *wbc);
int write_cache_pages(struct address_space *mapping,
diff --git a/mm/Makefile b/mm/Makefile
index ec73c68..2adb811 100644
--- a/mm/Makefile
+++ b/mm/Makefile
@@ -8,7 +8,7 @@ mmu-$(CONFIG_MMU) := fremap.o highmem.o madvise.o memory.o mincore.o \
vmalloc.o
obj-y := bootmem.o filemap.o mempool.o oom_kill.o fadvise.o \
- maccess.o page_alloc.o page-writeback.o pdflush.o \
+ maccess.o page_alloc.o page-writeback.o \
readahead.o swap.o truncate.o vmscan.o shmem.o \
prio_tree.o util.o mmzone.o vmstat.o backing-dev.o \
page_isolation.o mm_init.o $(mmu-y)
diff --git a/mm/pdflush.c b/mm/pdflush.c
deleted file mode 100644
index 235ac44..0000000
--- a/mm/pdflush.c
+++ /dev/null
@@ -1,269 +0,0 @@
-/*
- * mm/pdflush.c - worker threads for writing back filesystem data
- *
- * Copyright (C) 2002, Linus Torvalds.
- *
- * 09Apr2002 Andrew Morton
- * Initial version
- * 29Feb2004 kaos@sgi.com
- * Move worker thread creation to kthread to avoid chewing
- * up stack space with nested calls to kernel_thread.
- */
-
-#include <linux/sched.h>
-#include <linux/list.h>
-#include <linux/signal.h>
-#include <linux/spinlock.h>
-#include <linux/gfp.h>
-#include <linux/init.h>
-#include <linux/module.h>
-#include <linux/fs.h> /* Needed by writeback.h */
-#include <linux/writeback.h> /* Prototypes pdflush_operation() */
-#include <linux/kthread.h>
-#include <linux/cpuset.h>
-#include <linux/freezer.h>
-
-
-/*
- * Minimum and maximum number of pdflush instances
- */
-#define MIN_PDFLUSH_THREADS 2
-#define MAX_PDFLUSH_THREADS 8
-
-static void start_one_pdflush_thread(void);
-
-
-/*
- * The pdflush threads are worker threads for writing back dirty data.
- * Ideally, we'd like one thread per active disk spindle. But the disk
- * topology is very hard to divine at this level. Instead, we take
- * care in various places to prevent more than one pdflush thread from
- * performing writeback against a single filesystem. pdflush threads
- * have the PF_FLUSHER flag set in current->flags to aid in this.
- */
-
-/*
- * All the pdflush threads. Protected by pdflush_lock
- */
-static LIST_HEAD(pdflush_list);
-static DEFINE_SPINLOCK(pdflush_lock);
-
-/*
- * The count of currently-running pdflush threads. Protected
- * by pdflush_lock.
- *
- * Readable by sysctl, but not writable. Published to userspace at
- * /proc/sys/vm/nr_pdflush_threads.
- */
-int nr_pdflush_threads = 0;
-
-/*
- * The time at which the pdflush thread pool last went empty
- */
-static unsigned long last_empty_jifs;
-
-/*
- * The pdflush thread.
- *
- * Thread pool management algorithm:
- *
- * - The minimum and maximum number of pdflush instances are bound
- * by MIN_PDFLUSH_THREADS and MAX_PDFLUSH_THREADS.
- *
- * - If there have been no idle pdflush instances for 1 second, create
- * a new one.
- *
- * - If the least-recently-went-to-sleep pdflush thread has been asleep
- * for more than one second, terminate a thread.
- */
-
-/*
- * A structure for passing work to a pdflush thread. Also for passing
- * state information between pdflush threads. Protected by pdflush_lock.
- */
-struct pdflush_work {
- struct task_struct *who; /* The thread */
- void (*fn)(unsigned long); /* A callback function */
- unsigned long arg0; /* An argument to the callback */
- struct list_head list; /* On pdflush_list, when idle */
- unsigned long when_i_went_to_sleep;
-};
-
-static int __pdflush(struct pdflush_work *my_work)
-{
- current->flags |= PF_FLUSHER | PF_SWAPWRITE;
- set_freezable();
- my_work->fn = NULL;
- my_work->who = current;
- INIT_LIST_HEAD(&my_work->list);
-
- spin_lock_irq(&pdflush_lock);
- for ( ; ; ) {
- struct pdflush_work *pdf;
-
- set_current_state(TASK_INTERRUPTIBLE);
- list_move(&my_work->list, &pdflush_list);
- my_work->when_i_went_to_sleep = jiffies;
- spin_unlock_irq(&pdflush_lock);
- schedule();
- try_to_freeze();
- spin_lock_irq(&pdflush_lock);
- if (!list_empty(&my_work->list)) {
- /*
- * Someone woke us up, but without removing our control
- * structure from the global list. swsusp will do this
- * in try_to_freeze()->refrigerator(). Handle it.
- */
- my_work->fn = NULL;
- continue;
- }
- if (my_work->fn == NULL) {
- printk("pdflush: bogus wakeup\n");
- continue;
- }
- spin_unlock_irq(&pdflush_lock);
-
- (*my_work->fn)(my_work->arg0);
-
- spin_lock_irq(&pdflush_lock);
-
- /*
- * Thread creation: For how long have there been zero
- * available threads?
- *
- * To throttle creation, we reset last_empty_jifs.
- */
- if (time_after(jiffies, last_empty_jifs + 1 * HZ)) {
- if (list_empty(&pdflush_list)) {
- if (nr_pdflush_threads < MAX_PDFLUSH_THREADS) {
- last_empty_jifs = jiffies;
- nr_pdflush_threads++;
- spin_unlock_irq(&pdflush_lock);
- start_one_pdflush_thread();
- spin_lock_irq(&pdflush_lock);
- }
- }
- }
-
- my_work->fn = NULL;
-
- /*
- * Thread destruction: For how long has the sleepiest
- * thread slept?
- */
- if (list_empty(&pdflush_list))
- continue;
- if (nr_pdflush_threads <= MIN_PDFLUSH_THREADS)
- continue;
- pdf = list_entry(pdflush_list.prev, struct pdflush_work, list);
- if (time_after(jiffies, pdf->when_i_went_to_sleep + 1 * HZ)) {
- /* Limit exit rate */
- pdf->when_i_went_to_sleep = jiffies;
- break; /* exeunt */
- }
- }
- nr_pdflush_threads--;
- spin_unlock_irq(&pdflush_lock);
- return 0;
-}
-
-/*
- * Of course, my_work wants to be just a local in __pdflush(). It is
- * separated out in this manner to hopefully prevent the compiler from
- * performing unfortunate optimisations against the auto variables. Because
- * these are visible to other tasks and CPUs. (No problem has actually
- * been observed. This is just paranoia).
- */
-static int pdflush(void *dummy)
-{
- struct pdflush_work my_work;
- cpumask_var_t cpus_allowed;
-
- /*
- * Since the caller doesn't even check kthread_run() worked, let's not
- * freak out too much if this fails.
- */
- if (!alloc_cpumask_var(&cpus_allowed, GFP_KERNEL)) {
- printk(KERN_WARNING "pdflush failed to allocate cpumask\n");
- return 0;
- }
-
- /*
- * pdflush can spend a lot of time doing encryption via dm-crypt. We
- * don't want to do that at keventd's priority.
- */
- set_user_nice(current, 0);
-
- /*
- * Some configs put our parent kthread in a limited cpuset,
- * which kthread() overrides, forcing cpus_allowed == cpu_all_mask.
- * Our needs are more modest - cut back to our cpusets cpus_allowed.
- * This is needed as pdflush's are dynamically created and destroyed.
- * The boottime pdflush's are easily placed w/o these 2 lines.
- */
- cpuset_cpus_allowed(current, cpus_allowed);
- set_cpus_allowed_ptr(current, cpus_allowed);
- free_cpumask_var(cpus_allowed);
-
- return __pdflush(&my_work);
-}
-
-/*
- * Attempt to wake up a pdflush thread, and get it to do some work for you.
- * Returns zero if it indeed managed to find a worker thread, and passed your
- * payload to it.
- */
-int pdflush_operation(void (*fn)(unsigned long), unsigned long arg0)
-{
- unsigned long flags;
- int ret = 0;
-
- BUG_ON(fn == NULL); /* Hard to diagnose if it's deferred */
-
- spin_lock_irqsave(&pdflush_lock, flags);
- if (list_empty(&pdflush_list)) {
- ret = -1;
- } else {
- struct pdflush_work *pdf;
-
- pdf = list_entry(pdflush_list.next, struct pdflush_work, list);
- list_del_init(&pdf->list);
- if (list_empty(&pdflush_list))
- last_empty_jifs = jiffies;
- pdf->fn = fn;
- pdf->arg0 = arg0;
- wake_up_process(pdf->who);
- }
- spin_unlock_irqrestore(&pdflush_lock, flags);
-
- return ret;
-}
-
-static void start_one_pdflush_thread(void)
-{
- struct task_struct *k;
-
- k = kthread_run(pdflush, NULL, "pdflush");
- if (unlikely(IS_ERR(k))) {
- spin_lock_irq(&pdflush_lock);
- nr_pdflush_threads--;
- spin_unlock_irq(&pdflush_lock);
- }
-}
-
-static int __init pdflush_init(void)
-{
- int i;
-
- /*
- * Pre-set nr_pdflush_threads... If we fail to create,
- * the count will be decremented.
- */
- nr_pdflush_threads = MIN_PDFLUSH_THREADS;
-
- for (i = 0; i < MIN_PDFLUSH_THREADS; i++)
- start_one_pdflush_thread();
- return 0;
-}
-
-module_init(pdflush_init);
--
1.6.3.rc0.1.gf800
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCH 06/12] writeback: separate the flushing state/task from the bdi
2009-05-26 9:33 [PATCH 0/12] Per-bdi writeback flusher threads v7 Jens Axboe
` (4 preceding siblings ...)
2009-05-26 9:33 ` [PATCH 05/12] writeback: get rid of pdflush completely Jens Axboe
@ 2009-05-26 9:33 ` Jens Axboe
2009-05-26 9:33 ` [PATCH 07/12] writeback: support > 1 flusher thread per bdi Jens Axboe
` (7 subsequent siblings)
13 siblings, 0 replies; 33+ messages in thread
From: Jens Axboe @ 2009-05-26 9:33 UTC (permalink / raw)
To: linux-kernel, linux-fsdevel
Cc: chris.mason, david, hch, akpm, jack, yanmin_zhang, richard, Jens Axboe
Add a struct bdi_writeback for tracking and handling dirty IO. This
is in preparation for adding > 1 flusher task per bdi.
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
---
fs/fs-writeback.c | 145 ++++++++++++++++++++++++++----------------
include/linux/backing-dev.h | 40 +++++++-----
mm/backing-dev.c | 128 ++++++++++++++++++++++++++++++--------
3 files changed, 215 insertions(+), 98 deletions(-)
diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c
index 7a558a6..e72db8b 100644
--- a/fs/fs-writeback.c
+++ b/fs/fs-writeback.c
@@ -46,9 +46,11 @@ int nr_pdflush_threads;
* unless they implement their own. Which is somewhat inefficient, as this
* may prevent concurrent writeback against multiple devices.
*/
-static int writeback_acquire(struct backing_dev_info *bdi)
+static int writeback_acquire(struct bdi_writeback *wb)
{
- return !test_and_set_bit(BDI_pdflush, &bdi->state);
+ struct backing_dev_info *bdi = wb->bdi;
+
+ return !test_and_set_bit(wb->nr, &bdi->wb_active);
}
/**
@@ -59,19 +61,40 @@ static int writeback_acquire(struct backing_dev_info *bdi)
*/
int writeback_in_progress(struct backing_dev_info *bdi)
{
- return test_bit(BDI_pdflush, &bdi->state);
+ return bdi->wb_active != 0;
}
/**
* writeback_release - relinquish exclusive writeback access against a device.
* @bdi: the device's backing_dev_info structure
*/
-static void writeback_release(struct backing_dev_info *bdi)
+static void writeback_release(struct bdi_writeback *wb)
{
- WARN_ON_ONCE(!writeback_in_progress(bdi));
- bdi->wb_arg.nr_pages = 0;
- bdi->wb_arg.sb = NULL;
- clear_bit(BDI_pdflush, &bdi->state);
+ struct backing_dev_info *bdi = wb->bdi;
+
+ wb->nr_pages = 0;
+ wb->sb = NULL;
+ clear_bit(wb->nr, &bdi->wb_active);
+}
+
+static void wb_start_writeback(struct bdi_writeback *wb, struct super_block *sb,
+ long nr_pages,
+ enum writeback_sync_modes sync_mode)
+{
+ if (!wb_has_dirty_io(wb))
+ return;
+
+ if (writeback_acquire(wb)) {
+ wb->nr_pages = nr_pages;
+ wb->sb = sb;
+ wb->sync_mode = sync_mode;
+
+ /*
+ * make above store seen before the task is woken
+ */
+ smp_mb();
+ wake_up(&wb->wait);
+ }
}
int bdi_start_writeback(struct backing_dev_info *bdi, struct super_block *sb,
@@ -81,22 +104,12 @@ int bdi_start_writeback(struct backing_dev_info *bdi, struct super_block *sb,
* This only happens the first time someone kicks this bdi, so put
* it out-of-line.
*/
- if (unlikely(!bdi->task)) {
+ if (unlikely(!bdi->wb.task)) {
bdi_add_default_flusher_task(bdi);
return 1;
}
- if (writeback_acquire(bdi)) {
- bdi->wb_arg.nr_pages = nr_pages;
- bdi->wb_arg.sb = sb;
- bdi->wb_arg.sync_mode = sync_mode;
- /*
- * make above store seen before the task is woken
- */
- smp_mb();
- wake_up(&bdi->wait);
- }
-
+ wb_start_writeback(&bdi->wb, sb, nr_pages, sync_mode);
return 0;
}
@@ -124,12 +137,12 @@ int bdi_start_writeback(struct backing_dev_info *bdi, struct super_block *sb,
* older_than_this takes precedence over nr_to_write. So we'll only write back
* all dirty pages if they are all attached to "old" mappings.
*/
-static void bdi_kupdated(struct backing_dev_info *bdi)
+static void wb_kupdated(struct bdi_writeback *wb)
{
unsigned long oldest_jif;
long nr_to_write;
struct writeback_control wbc = {
- .bdi = bdi,
+ .bdi = wb->bdi,
.sync_mode = WB_SYNC_NONE,
.older_than_this = &oldest_jif,
.nr_to_write = 0,
@@ -164,15 +177,19 @@ static inline bool over_bground_thresh(void)
global_page_state(NR_UNSTABLE_NFS) >= background_thresh);
}
-static void bdi_pdflush(struct backing_dev_info *bdi)
+static void generic_sync_wb_inodes(struct bdi_writeback *wb,
+ struct super_block *sb,
+ struct writeback_control *wbc);
+
+static void wb_writeback(struct bdi_writeback *wb)
{
struct writeback_control wbc = {
- .bdi = bdi,
- .sync_mode = bdi->wb_arg.sync_mode,
+ .bdi = wb->bdi,
+ .sync_mode = wb->sync_mode,
.older_than_this = NULL,
.range_cyclic = 1,
};
- long nr_pages = bdi->wb_arg.nr_pages;
+ long nr_pages = wb->nr_pages;
for (;;) {
if (wbc.sync_mode == WB_SYNC_NONE && nr_pages <= 0 &&
@@ -183,7 +200,7 @@ static void bdi_pdflush(struct backing_dev_info *bdi)
wbc.encountered_congestion = 0;
wbc.nr_to_write = MAX_WRITEBACK_PAGES;
wbc.pages_skipped = 0;
- generic_sync_bdi_inodes(bdi->wb_arg.sb, &wbc);
+ generic_sync_wb_inodes(wb, wb->sb, &wbc);
nr_pages -= MAX_WRITEBACK_PAGES - wbc.nr_to_write;
/*
* If we ran out of stuff to write, bail unless more_io got set
@@ -200,13 +217,13 @@ static void bdi_pdflush(struct backing_dev_info *bdi)
* Handle writeback of dirty data for the device backed by this bdi. Also
* wakes up periodically and does kupdated style flushing.
*/
-int bdi_writeback_task(struct backing_dev_info *bdi)
+int bdi_writeback_task(struct bdi_writeback *wb)
{
while (!kthread_should_stop()) {
unsigned long wait_jiffies;
DEFINE_WAIT(wait);
- prepare_to_wait(&bdi->wait, &wait, TASK_INTERRUPTIBLE);
+ prepare_to_wait(&wb->wait, &wait, TASK_INTERRUPTIBLE);
wait_jiffies = msecs_to_jiffies(dirty_writeback_interval * 10);
schedule_timeout(wait_jiffies);
try_to_freeze();
@@ -225,13 +242,13 @@ int bdi_writeback_task(struct backing_dev_info *bdi)
* pdflush style writeout.
*
*/
- if (writeback_acquire(bdi))
- bdi_kupdated(bdi);
+ if (writeback_acquire(wb))
+ wb_kupdated(wb);
else
- bdi_pdflush(bdi);
+ wb_writeback(wb);
- writeback_release(bdi);
- finish_wait(&bdi->wait, &wait);
+ writeback_release(wb);
+ finish_wait(&wb->wait, &wait);
}
return 0;
@@ -253,6 +270,14 @@ void bdi_writeback_all(struct super_block *sb, long nr_pages,
mutex_unlock(&bdi_lock);
}
+/*
+ * We have only a single wb per bdi, so just return that.
+ */
+static inline struct bdi_writeback *inode_get_wb(struct inode *inode)
+{
+ return &inode_to_bdi(inode)->wb;
+}
+
/**
* __mark_inode_dirty - internal function
* @inode: inode to mark
@@ -351,9 +376,10 @@ void __mark_inode_dirty(struct inode *inode, int flags)
* reposition it (that would break b_dirty time-ordering).
*/
if (!was_dirty) {
+ struct bdi_writeback *wb = inode_get_wb(inode);
+
inode->dirtied_when = jiffies;
- list_move(&inode->i_list,
- &inode_to_bdi(inode)->b_dirty);
+ list_move(&inode->i_list, &wb->b_dirty);
}
}
out:
@@ -380,16 +406,16 @@ static int write_inode(struct inode *inode, int sync)
*/
static void redirty_tail(struct inode *inode)
{
- struct backing_dev_info *bdi = inode_to_bdi(inode);
+ struct bdi_writeback *wb = inode_get_wb(inode);
- if (!list_empty(&bdi->b_dirty)) {
+ if (!list_empty(&wb->b_dirty)) {
struct inode *tail;
- tail = list_entry(bdi->b_dirty.next, struct inode, i_list);
+ tail = list_entry(wb->b_dirty.next, struct inode, i_list);
if (time_before(inode->dirtied_when, tail->dirtied_when))
inode->dirtied_when = jiffies;
}
- list_move(&inode->i_list, &bdi->b_dirty);
+ list_move(&inode->i_list, &wb->b_dirty);
}
/*
@@ -397,7 +423,9 @@ static void redirty_tail(struct inode *inode)
*/
static void requeue_io(struct inode *inode)
{
- list_move(&inode->i_list, &inode_to_bdi(inode)->b_more_io);
+ struct bdi_writeback *wb = inode_get_wb(inode);
+
+ list_move(&inode->i_list, &wb->b_more_io);
}
static void inode_sync_complete(struct inode *inode)
@@ -444,11 +472,10 @@ static void move_expired_inodes(struct list_head *delaying_queue,
/*
* Queue all expired dirty inodes for io, eldest first.
*/
-static void queue_io(struct backing_dev_info *bdi,
- unsigned long *older_than_this)
+static void queue_io(struct bdi_writeback *wb, unsigned long *older_than_this)
{
- list_splice_init(&bdi->b_more_io, bdi->b_io.prev);
- move_expired_inodes(&bdi->b_dirty, &bdi->b_io, older_than_this);
+ list_splice_init(&wb->b_more_io, wb->b_io.prev);
+ move_expired_inodes(&wb->b_dirty, &wb->b_io, older_than_this);
}
/*
@@ -609,20 +636,20 @@ __writeback_single_inode(struct inode *inode, struct writeback_control *wbc)
return __sync_single_inode(inode, wbc);
}
-void generic_sync_bdi_inodes(struct super_block *sb,
- struct writeback_control *wbc)
+static void generic_sync_wb_inodes(struct bdi_writeback *wb,
+ struct super_block *sb,
+ struct writeback_control *wbc)
{
const int is_blkdev_sb = sb_is_blkdev_sb(sb);
- struct backing_dev_info *bdi = wbc->bdi;
const unsigned long start = jiffies; /* livelock avoidance */
spin_lock(&inode_lock);
- if (!wbc->for_kupdate || list_empty(&bdi->b_io))
- queue_io(bdi, wbc->older_than_this);
+ if (!wbc->for_kupdate || list_empty(&wb->b_io))
+ queue_io(wb, wbc->older_than_this);
- while (!list_empty(&bdi->b_io)) {
- struct inode *inode = list_entry(bdi->b_io.prev,
+ while (!list_empty(&wb->b_io)) {
+ struct inode *inode = list_entry(wb->b_io.prev,
struct inode, i_list);
long pages_skipped;
@@ -634,7 +661,7 @@ void generic_sync_bdi_inodes(struct super_block *sb,
continue;
}
- if (!bdi_cap_writeback_dirty(bdi)) {
+ if (!bdi_cap_writeback_dirty(wb->bdi)) {
redirty_tail(inode);
if (is_blkdev_sb) {
/*
@@ -656,7 +683,7 @@ void generic_sync_bdi_inodes(struct super_block *sb,
continue;
}
- if (wbc->nonblocking && bdi_write_congested(bdi)) {
+ if (wbc->nonblocking && bdi_write_congested(wb->bdi)) {
wbc->encountered_congestion = 1;
if (!is_blkdev_sb)
break; /* Skip a congested fs */
@@ -690,7 +717,7 @@ void generic_sync_bdi_inodes(struct super_block *sb,
wbc->more_io = 1;
break;
}
- if (!list_empty(&bdi->b_more_io))
+ if (!list_empty(&wb->b_more_io))
wbc->more_io = 1;
}
@@ -698,6 +725,14 @@ void generic_sync_bdi_inodes(struct super_block *sb,
/* Leave any unwritten inodes on b_io */
}
+void generic_sync_bdi_inodes(struct super_block *sb,
+ struct writeback_control *wbc)
+{
+ struct backing_dev_info *bdi = wbc->bdi;
+
+ generic_sync_wb_inodes(&bdi->wb, sb, wbc);
+}
+
/*
* Write out a superblock's list of dirty inodes. A wait will be performed
* upon no inodes, all inodes or the final one, depending upon sync_mode.
diff --git a/include/linux/backing-dev.h b/include/linux/backing-dev.h
index f164925..77dc62c 100644
--- a/include/linux/backing-dev.h
+++ b/include/linux/backing-dev.h
@@ -24,8 +24,8 @@ struct dentry;
* Bits in backing_dev_info.state
*/
enum bdi_state {
- BDI_pdflush, /* A pdflush thread is working this device */
BDI_pending, /* On its way to being activated */
+ BDI_wb_alloc, /* Default embedded wb allocated */
BDI_async_congested, /* The async (write) queue is getting full */
BDI_sync_congested, /* The sync queue is getting full */
BDI_unused, /* Available bits start here */
@@ -41,15 +41,23 @@ enum bdi_stat_item {
#define BDI_STAT_BATCH (8*(1+ilog2(nr_cpu_ids)))
-struct bdi_writeback_arg {
- unsigned long nr_pages;
- struct super_block *sb;
+struct bdi_writeback {
+ struct backing_dev_info *bdi; /* our parent bdi */
+ unsigned int nr;
+
+ struct task_struct *task; /* writeback task */
+ wait_queue_head_t wait;
+ struct list_head b_dirty; /* dirty inodes */
+ struct list_head b_io; /* parked for writeback */
+ struct list_head b_more_io; /* parked for more writeback */
+
+ unsigned long nr_pages;
+ struct super_block *sb;
enum writeback_sync_modes sync_mode;
};
struct backing_dev_info {
struct list_head bdi_list;
-
unsigned long ra_pages; /* max readahead in PAGE_CACHE_SIZE units */
unsigned long state; /* Always use atomic bitops on this */
unsigned int capabilities; /* Device capabilities */
@@ -66,14 +74,11 @@ struct backing_dev_info {
unsigned int min_ratio;
unsigned int max_ratio, max_prop_frac;
- struct device *dev;
+ struct bdi_writeback wb; /* default writeback info for this bdi */
+ unsigned long wb_active; /* bitmap of active tasks */
+ unsigned long wb_mask; /* number of registered tasks */
- struct task_struct *task; /* writeback task */
- wait_queue_head_t wait;
- struct bdi_writeback_arg wb_arg; /* protected by BDI_pdflush */
- struct list_head b_dirty; /* dirty inodes */
- struct list_head b_io; /* parked for writeback */
- struct list_head b_more_io; /* parked for more writeback */
+ struct device *dev;
#ifdef CONFIG_DEBUG_FS
struct dentry *debug_dir;
@@ -90,19 +95,20 @@ int bdi_register_dev(struct backing_dev_info *bdi, dev_t dev);
void bdi_unregister(struct backing_dev_info *bdi);
int bdi_start_writeback(struct backing_dev_info *bdi, struct super_block *sb,
long nr_pages, enum writeback_sync_modes sync_mode);
-int bdi_writeback_task(struct backing_dev_info *bdi);
+int bdi_writeback_task(struct bdi_writeback *wb);
void bdi_writeback_all(struct super_block *sb, long nr_pages,
enum writeback_sync_modes sync_mode);
void bdi_add_default_flusher_task(struct backing_dev_info *bdi);
+int bdi_has_dirty_io(struct backing_dev_info *bdi);
extern struct mutex bdi_lock;
extern struct list_head bdi_list;
-static inline int bdi_has_dirty_io(struct backing_dev_info *bdi)
+static inline int wb_has_dirty_io(struct bdi_writeback *wb)
{
- return !list_empty(&bdi->b_dirty) ||
- !list_empty(&bdi->b_io) ||
- !list_empty(&bdi->b_more_io);
+ return !list_empty(&wb->b_dirty) ||
+ !list_empty(&wb->b_io) ||
+ !list_empty(&wb->b_more_io);
}
static inline void __add_bdi_stat(struct backing_dev_info *bdi,
diff --git a/mm/backing-dev.c b/mm/backing-dev.c
index bae3d4f..c8201f0 100644
--- a/mm/backing-dev.c
+++ b/mm/backing-dev.c
@@ -199,10 +199,46 @@ static int __init default_bdi_init(void)
}
subsys_initcall(default_bdi_init);
+static void bdi_wb_init(struct bdi_writeback *wb, struct backing_dev_info *bdi)
+{
+ memset(wb, 0, sizeof(*wb));
+
+ wb->bdi = bdi;
+ init_waitqueue_head(&wb->wait);
+ INIT_LIST_HEAD(&wb->b_dirty);
+ INIT_LIST_HEAD(&wb->b_io);
+ INIT_LIST_HEAD(&wb->b_more_io);
+}
+
+static int wb_assign_nr(struct backing_dev_info *bdi, struct bdi_writeback *wb)
+{
+ set_bit(0, &bdi->wb_mask);
+ wb->nr = 0;
+ return 0;
+}
+
+static void bdi_put_wb(struct backing_dev_info *bdi, struct bdi_writeback *wb)
+{
+ clear_bit(wb->nr, &bdi->wb_mask);
+ clear_bit(BDI_wb_alloc, &bdi->state);
+}
+
+static struct bdi_writeback *bdi_new_wb(struct backing_dev_info *bdi)
+{
+ struct bdi_writeback *wb;
+
+ set_bit(BDI_wb_alloc, &bdi->state);
+ wb = &bdi->wb;
+ wb_assign_nr(bdi, wb);
+ return wb;
+}
+
static int bdi_start_fn(void *ptr)
{
- struct backing_dev_info *bdi = ptr;
+ struct bdi_writeback *wb = ptr;
+ struct backing_dev_info *bdi = wb->bdi;
struct task_struct *tsk = current;
+ int ret;
/*
* Add us to the active bdi_list
@@ -226,7 +262,15 @@ static int bdi_start_fn(void *ptr)
smp_mb__after_clear_bit();
wake_up_bit(&bdi->state, BDI_pending);
- return bdi_writeback_task(bdi);
+ ret = bdi_writeback_task(wb);
+
+ bdi_put_wb(bdi, wb);
+ return ret;
+}
+
+int bdi_has_dirty_io(struct backing_dev_info *bdi)
+{
+ return wb_has_dirty_io(&bdi->wb);
}
static void bdi_flush_io(struct backing_dev_info *bdi)
@@ -244,11 +288,12 @@ static void bdi_flush_io(struct backing_dev_info *bdi)
static int bdi_forker_task(void *ptr)
{
- struct backing_dev_info *me = ptr;
+ struct bdi_writeback *me = ptr;
DEFINE_WAIT(wait);
for (;;) {
struct backing_dev_info *bdi, *tmp;
+ struct bdi_writeback *wb;
/*
* Do this periodically, like kupdated() did before.
@@ -259,8 +304,8 @@ static int bdi_forker_task(void *ptr)
* Temporary measure, we want to make sure we don't see
* dirty data on the default backing_dev_info
*/
- if (bdi_has_dirty_io(me))
- bdi_flush_io(me);
+ if (wb_has_dirty_io(me))
+ bdi_flush_io(me->bdi);
prepare_to_wait(&me->wait, &wait, TASK_INTERRUPTIBLE);
@@ -271,7 +316,7 @@ static int bdi_forker_task(void *ptr)
* a thread registered. If so, set that up.
*/
list_for_each_entry_safe(bdi, tmp, &bdi_list, bdi_list) {
- if (bdi->task || !bdi_has_dirty_io(bdi))
+ if (bdi->wb.task || !bdi_has_dirty_io(bdi))
continue;
bdi_add_default_flusher_task(bdi);
@@ -292,17 +337,22 @@ static int bdi_forker_task(void *ptr)
list_del_init(&bdi->bdi_list);
mutex_unlock(&bdi_lock);
- BUG_ON(bdi->task);
+ wb = bdi_new_wb(bdi);
+ if (!wb)
+ goto readd_flush;
- bdi->task = kthread_run(bdi_start_fn, bdi, "bdi-%s",
+ wb->task = kthread_run(bdi_start_fn, wb, "bdi-%s",
dev_name(bdi->dev));
+
/*
* If task creation fails, then readd the bdi to
* the pending list and force writeout of the bdi
* from this forker thread. That will free some memory
* and we can try again.
*/
- if (!bdi->task) {
+ if (!wb->task) {
+ bdi_put_wb(bdi, wb);
+readd_flush:
/*
* Add this 'bdi' to the back, so we get
* a chance to flush other bdi's to free
@@ -320,8 +370,18 @@ static int bdi_forker_task(void *ptr)
return 0;
}
+/*
+ * Add a new flusher task that gets created for any bdi
+ * that has dirty data pending writeout
+ */
void bdi_add_default_flusher_task(struct backing_dev_info *bdi)
{
+ if (!bdi_cap_writeback_dirty(bdi))
+ return;
+
+ /*
+ * Someone already marked this pending for task creation
+ */
if (test_and_set_bit(BDI_pending, &bdi->state))
return;
@@ -329,7 +389,7 @@ void bdi_add_default_flusher_task(struct backing_dev_info *bdi)
list_move_tail(&bdi->bdi_list, &bdi_pending_list);
mutex_unlock(&bdi_lock);
- wake_up(&default_backing_dev_info.wait);
+ wake_up(&default_backing_dev_info.wb.wait);
}
int bdi_register(struct backing_dev_info *bdi, struct device *parent,
@@ -362,13 +422,23 @@ int bdi_register(struct backing_dev_info *bdi, struct device *parent,
* on-demand when they need it.
*/
if (bdi_cap_flush_forker(bdi)) {
- bdi->task = kthread_run(bdi_forker_task, bdi, "bdi-%s",
+ struct bdi_writeback *wb;
+
+ wb = bdi_new_wb(bdi);
+ if (!wb) {
+ ret = -ENOMEM;
+ goto remove_err;
+ }
+
+ wb->task = kthread_run(bdi_forker_task, wb, "bdi-%s",
dev_name(dev));
- if (!bdi->task) {
+ if (!wb->task) {
+ bdi_put_wb(bdi, wb);
+ ret = -ENOMEM;
+remove_err:
mutex_lock(&bdi_lock);
list_del(&bdi->bdi_list);
mutex_unlock(&bdi_lock);
- ret = -ENOMEM;
goto exit;
}
}
@@ -391,28 +461,37 @@ static int sched_wait(void *word)
return 0;
}
+/*
+ * Remove bdi from global list and shutdown any threads we have running
+ */
static void bdi_wb_shutdown(struct backing_dev_info *bdi)
{
+ if (!bdi_cap_writeback_dirty(bdi))
+ return;
+
/*
* If setup is pending, wait for that to complete first
*/
wait_on_bit(&bdi->state, BDI_pending, sched_wait, TASK_UNINTERRUPTIBLE);
+ /*
+ * Make sure nobody finds us on the bdi_list anymore
+ */
mutex_lock(&bdi_lock);
list_del(&bdi->bdi_list);
mutex_unlock(&bdi_lock);
+
+ /*
+ * Finally, kill the kernel thread
+ */
+ kthread_stop(bdi->wb.task);
}
void bdi_unregister(struct backing_dev_info *bdi)
{
if (bdi->dev) {
- if (!bdi_cap_flush_forker(bdi)) {
+ if (!bdi_cap_flush_forker(bdi))
bdi_wb_shutdown(bdi);
- if (bdi->task) {
- kthread_stop(bdi->task);
- bdi->task = NULL;
- }
- }
bdi_debug_unregister(bdi);
device_unregister(bdi->dev);
bdi->dev = NULL;
@@ -429,11 +508,10 @@ int bdi_init(struct backing_dev_info *bdi)
bdi->min_ratio = 0;
bdi->max_ratio = 100;
bdi->max_prop_frac = PROP_FRAC_BASE;
- init_waitqueue_head(&bdi->wait);
INIT_LIST_HEAD(&bdi->bdi_list);
- INIT_LIST_HEAD(&bdi->b_io);
- INIT_LIST_HEAD(&bdi->b_dirty);
- INIT_LIST_HEAD(&bdi->b_more_io);
+ bdi->wb_mask = bdi->wb_active = 0;
+
+ bdi_wb_init(&bdi->wb, bdi);
for (i = 0; i < NR_BDI_STAT_ITEMS; i++) {
err = percpu_counter_init(&bdi->bdi_stat[i], 0);
@@ -458,9 +536,7 @@ void bdi_destroy(struct backing_dev_info *bdi)
{
int i;
- WARN_ON(!list_empty(&bdi->b_dirty));
- WARN_ON(!list_empty(&bdi->b_io));
- WARN_ON(!list_empty(&bdi->b_more_io));
+ WARN_ON(bdi_has_dirty_io(bdi));
bdi_unregister(bdi);
--
1.6.3.rc0.1.gf800
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCH 07/12] writeback: support > 1 flusher thread per bdi
2009-05-26 9:33 [PATCH 0/12] Per-bdi writeback flusher threads v7 Jens Axboe
` (5 preceding siblings ...)
2009-05-26 9:33 ` [PATCH 06/12] writeback: separate the flushing state/task from the bdi Jens Axboe
@ 2009-05-26 9:33 ` Jens Axboe
2009-06-04 17:44 ` Paul E. McKenney
2009-05-26 9:33 ` [PATCH 08/12] writeback: include default_backing_dev_info in writeback Jens Axboe
` (6 subsequent siblings)
13 siblings, 1 reply; 33+ messages in thread
From: Jens Axboe @ 2009-05-26 9:33 UTC (permalink / raw)
To: linux-kernel, linux-fsdevel
Cc: chris.mason, david, hch, akpm, jack, yanmin_zhang, richard, Jens Axboe
Build on the bdi_writeback support by allowing registration of
more than 1 flusher thread. File systems can call bdi_add_flusher_task(bdi)
to add more flusher threads to the device. If they do so, they must also
provide a super_operations function to return the suitable bdi_writeback
struct from any given inode.
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
---
fs/fs-writeback.c | 448 +++++++++++++++++++++++++++++++++++--------
include/linux/backing-dev.h | 34 +++-
include/linux/fs.h | 3 +
mm/backing-dev.c | 233 ++++++++++++++++++-----
mm/page-writeback.c | 4 +-
5 files changed, 586 insertions(+), 136 deletions(-)
diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c
index e72db8b..8e0902e 100644
--- a/fs/fs-writeback.c
+++ b/fs/fs-writeback.c
@@ -34,83 +34,247 @@
*/
int nr_pdflush_threads;
-/**
- * writeback_acquire - attempt to get exclusive writeback access to a device
- * @bdi: the device's backing_dev_info structure
- *
- * It is a waste of resources to have more than one pdflush thread blocked on
- * a single request queue. Exclusion at the request_queue level is obtained
- * via a flag in the request_queue's backing_dev_info.state.
- *
- * Non-request_queue-backed address_spaces will share default_backing_dev_info,
- * unless they implement their own. Which is somewhat inefficient, as this
- * may prevent concurrent writeback against multiple devices.
+static void generic_sync_wb_inodes(struct bdi_writeback *wb,
+ struct super_block *sb,
+ struct writeback_control *wbc);
+
+/*
+ * Work items for the bdi_writeback threads
*/
-static int writeback_acquire(struct bdi_writeback *wb)
+struct bdi_work {
+ struct list_head list;
+ struct list_head wait_list;
+ struct rcu_head rcu_head;
+
+ unsigned long seen;
+ atomic_t pending;
+
+ unsigned long sb_data;
+ unsigned long nr_pages;
+ enum writeback_sync_modes sync_mode;
+
+ unsigned long state;
+};
+
+static struct super_block *bdi_work_sb(struct bdi_work *work)
{
- struct backing_dev_info *bdi = wb->bdi;
+ return (struct super_block *) (work->sb_data & ~1UL);
+}
+
+static inline bool bdi_work_on_stack(struct bdi_work *work)
+{
+ return work->sb_data & 1UL;
+}
+
+static inline void bdi_work_init(struct bdi_work *work, struct super_block *sb,
+ unsigned long nr_pages,
+ enum writeback_sync_modes sync_mode)
+{
+ INIT_RCU_HEAD(&work->rcu_head);
+ work->sb_data = (unsigned long) sb;
+ work->nr_pages = nr_pages;
+ work->sync_mode = sync_mode;
+ work->state = 1;
+
+ /*
+ * state must not be reordered around the insert
+ */
+ smp_mb();
+}
- return !test_and_set_bit(wb->nr, &bdi->wb_active);
+static inline void bdi_work_init_on_stack(struct bdi_work *work,
+ struct super_block *sb,
+ unsigned long nr_pages,
+ enum writeback_sync_modes sync_mode)
+{
+ bdi_work_init(work, sb, nr_pages, sync_mode);
+ work->sb_data |= 1UL;
}
/**
* writeback_in_progress - determine whether there is writeback in progress
* @bdi: the device's backing_dev_info structure.
*
- * Determine whether there is writeback in progress against a backing device.
+ * Determine whether there is writeback waiting to be handled against a
+ * backing device.
*/
int writeback_in_progress(struct backing_dev_info *bdi)
{
- return bdi->wb_active != 0;
+ return !list_empty(&bdi->work_list);
}
-/**
- * writeback_release - relinquish exclusive writeback access against a device.
- * @bdi: the device's backing_dev_info structure
- */
-static void writeback_release(struct bdi_writeback *wb)
+static void bdi_work_clear(struct bdi_work *work)
{
- struct backing_dev_info *bdi = wb->bdi;
+ clear_bit(0, &work->state);
+ smp_mb__after_clear_bit();
+ wake_up_bit(&work->state, 0);
+}
+
+static void bdi_work_free(struct rcu_head *head)
+{
+ struct bdi_work *work = container_of(head, struct bdi_work, rcu_head);
- wb->nr_pages = 0;
- wb->sb = NULL;
- clear_bit(wb->nr, &bdi->wb_active);
+ if (!bdi_work_on_stack(work))
+ kfree(work);
+ else
+ bdi_work_clear(work);
}
-static void wb_start_writeback(struct bdi_writeback *wb, struct super_block *sb,
- long nr_pages,
- enum writeback_sync_modes sync_mode)
+static void wb_work_complete(struct bdi_work *work)
{
- if (!wb_has_dirty_io(wb))
- return;
+ if (!bdi_work_on_stack(work)) {
+ bdi_work_clear(work);
+
+ if (work->sync_mode == WB_SYNC_NONE)
+ call_rcu(&work->rcu_head, bdi_work_free);
+ } else
+ call_rcu(&work->rcu_head, bdi_work_free);
+}
+
+static void wb_clear_pending(struct bdi_writeback *wb, struct bdi_work *work)
+{
+ /*
+ * The caller has retrieved the work arguments from this work,
+ * drop our reference. If this is the last ref, delete and free it
+ */
+ if (atomic_dec_and_test(&work->pending)) {
+ struct backing_dev_info *bdi = wb->bdi;
- if (writeback_acquire(wb)) {
- wb->nr_pages = nr_pages;
- wb->sb = sb;
- wb->sync_mode = sync_mode;
+ spin_lock(&bdi->wb_lock);
+ list_del_rcu(&work->list);
+ spin_unlock(&bdi->wb_lock);
+
+ wb_work_complete(work);
+ }
+}
+
+static void wb_start_writeback(struct bdi_writeback *wb, struct bdi_work *work)
+{
+ /*
+ * If we failed allocating the bdi work item, wake up the wb thread
+ * always. As a safety precaution, it'll flush out everything
+ */
+ if (!wb_has_dirty_io(wb) && work)
+ wb_clear_pending(wb, work);
+ else
+ wake_up(&wb->wait);
+}
+
+static void bdi_queue_work(struct backing_dev_info *bdi, struct bdi_work *work)
+{
+ if (work) {
+ work->seen = bdi->wb_mask;
+ atomic_set(&work->pending, bdi->wb_cnt);
/*
- * make above store seen before the task is woken
+ * Make sure stores are seen before it appears on the list
*/
smp_mb();
- wake_up(&wb->wait);
+
+ spin_lock(&bdi->wb_lock);
+ list_add_tail_rcu(&work->list, &bdi->work_list);
+ spin_unlock(&bdi->wb_lock);
}
}
-int bdi_start_writeback(struct backing_dev_info *bdi, struct super_block *sb,
- long nr_pages, enum writeback_sync_modes sync_mode)
+static void bdi_sched_work(struct backing_dev_info *bdi, struct bdi_work *work)
+{
+ if (!bdi_wblist_needs_lock(bdi))
+ wb_start_writeback(&bdi->wb, work);
+ else {
+ struct bdi_writeback *wb;
+ int idx;
+
+ idx = srcu_read_lock(&bdi->srcu);
+
+ list_for_each_entry_rcu(wb, &bdi->wb_list, list)
+ wb_start_writeback(wb, work);
+
+ srcu_read_unlock(&bdi->srcu, idx);
+ }
+}
+
+static void __bdi_start_work(struct backing_dev_info *bdi,
+ struct bdi_work *work)
+{
+ /*
+ * If the default thread isn't there, make sure we add it. When
+ * it gets created and wakes up, we'll run this work.
+ */
+ if (unlikely(list_empty_careful(&bdi->wb_list)))
+ bdi_add_default_flusher_task(bdi);
+ else
+ bdi_sched_work(bdi, work);
+}
+
+static void bdi_start_work(struct backing_dev_info *bdi, struct bdi_work *work)
{
/*
- * This only happens the first time someone kicks this bdi, so put
- * it out-of-line.
+ * If the default thread isn't there, make sure we add it. When
+ * it gets created and wakes up, we'll run this work.
*/
- if (unlikely(!bdi->wb.task)) {
+ if (unlikely(list_empty_careful(&bdi->wb_list))) {
+ mutex_lock(&bdi_lock);
bdi_add_default_flusher_task(bdi);
- return 1;
+ mutex_unlock(&bdi_lock);
+ } else
+ bdi_sched_work(bdi, work);
+}
+
+/*
+ * Used for on-stack allocated work items. The caller needs to wait until
+ * the wb threads have acked the work before it's safe to continue.
+ */
+static void bdi_wait_on_work_clear(struct bdi_work *work)
+{
+ wait_on_bit(&work->state, 0, bdi_sched_wait, TASK_UNINTERRUPTIBLE);
+}
+
+static struct bdi_work *bdi_alloc_work(struct super_block *sb, long nr_pages,
+ enum writeback_sync_modes sync_mode)
+{
+ struct bdi_work *work;
+
+ work = kmalloc(sizeof(*work), GFP_ATOMIC);
+ if (work)
+ bdi_work_init(work, sb, nr_pages, sync_mode);
+
+ return work;
+}
+
+void bdi_start_writeback(struct backing_dev_info *bdi, struct super_block *sb,
+ long nr_pages, enum writeback_sync_modes sync_mode)
+{
+ const bool must_wait = sync_mode == WB_SYNC_ALL;
+ struct bdi_work work_stack, *work = NULL;
+
+ if (!must_wait)
+ work = bdi_alloc_work(sb, nr_pages, sync_mode);
+
+ if (!work) {
+ work = &work_stack;
+ bdi_work_init_on_stack(work, sb, nr_pages, sync_mode);
}
- wb_start_writeback(&bdi->wb, sb, nr_pages, sync_mode);
- return 0;
+ bdi_queue_work(bdi, work);
+ bdi_start_work(bdi, work);
+
+ /*
+ * If the sync mode is WB_SYNC_ALL, block waiting for the work to
+ * complete. If not, we only need to wait for the work to be started,
+ * if we allocated it on-stack. We use the same mechanism, if the
+ * wait bit is set in the bdi_work struct, then threads will not
+ * clear pending until after they are done.
+ *
+ * Note that work == &work_stack if must_wait is true, but that
+ * is implementation detail and we make it explicit here for
+ * ease of reading.
+ */
+ if (work == &work_stack || must_wait) {
+ bdi_wait_on_work_clear(work);
+ if (must_wait)
+ call_rcu(&work->rcu_head, bdi_work_free);
+ }
}
/*
@@ -160,7 +324,7 @@ static void wb_kupdated(struct bdi_writeback *wb)
wbc.more_io = 0;
wbc.encountered_congestion = 0;
wbc.nr_to_write = MAX_WRITEBACK_PAGES;
- generic_sync_bdi_inodes(NULL, &wbc);
+ generic_sync_wb_inodes(wb, NULL, &wbc);
if (wbc.nr_to_write > 0)
break; /* All the old data is written */
nr_to_write -= MAX_WRITEBACK_PAGES;
@@ -177,22 +341,19 @@ static inline bool over_bground_thresh(void)
global_page_state(NR_UNSTABLE_NFS) >= background_thresh);
}
-static void generic_sync_wb_inodes(struct bdi_writeback *wb,
- struct super_block *sb,
- struct writeback_control *wbc);
-
-static void wb_writeback(struct bdi_writeback *wb)
+static void __wb_writeback(struct bdi_writeback *wb, long nr_pages,
+ struct super_block *sb,
+ enum writeback_sync_modes sync_mode)
{
struct writeback_control wbc = {
.bdi = wb->bdi,
- .sync_mode = wb->sync_mode,
+ .sync_mode = sync_mode,
.older_than_this = NULL,
.range_cyclic = 1,
};
- long nr_pages = wb->nr_pages;
for (;;) {
- if (wbc.sync_mode == WB_SYNC_NONE && nr_pages <= 0 &&
+ if (sync_mode == WB_SYNC_NONE && nr_pages <= 0 &&
!over_bground_thresh())
break;
@@ -200,7 +361,7 @@ static void wb_writeback(struct bdi_writeback *wb)
wbc.encountered_congestion = 0;
wbc.nr_to_write = MAX_WRITEBACK_PAGES;
wbc.pages_skipped = 0;
- generic_sync_wb_inodes(wb, wb->sb, &wbc);
+ generic_sync_wb_inodes(wb, sb, &wbc);
nr_pages -= MAX_WRITEBACK_PAGES - wbc.nr_to_write;
/*
* If we ran out of stuff to write, bail unless more_io got set
@@ -214,68 +375,175 @@ static void wb_writeback(struct bdi_writeback *wb)
}
/*
+ * Return the next bdi_work struct that hasn't been processed by this
+ * wb thread yet
+ */
+static struct bdi_work *get_next_work_item(struct backing_dev_info *bdi,
+ struct bdi_writeback *wb)
+{
+ struct bdi_work *work, *ret = NULL;
+
+ rcu_read_lock();
+
+ list_for_each_entry_rcu(work, &bdi->work_list, list) {
+ if (!test_and_clear_bit(wb->nr, &work->seen))
+ continue;
+
+ ret = work;
+ break;
+ }
+
+ rcu_read_unlock();
+ return ret;
+}
+
+/*
+ * Retrieve work items and do the writeback they describe
+ */
+static void wb_writeback(struct bdi_writeback *wb)
+{
+ struct backing_dev_info *bdi = wb->bdi;
+ struct bdi_work *work;
+
+ while ((work = get_next_work_item(bdi, wb)) != NULL) {
+ struct super_block *sb = bdi_work_sb(work);
+ long nr_pages = work->nr_pages;
+ enum writeback_sync_modes sync_mode = work->sync_mode;
+
+ /*
+ * If this isn't a data integrity operation, just notify
+ * that we have seen this work and we are now starting it.
+ */
+ if (sync_mode == WB_SYNC_NONE)
+ wb_clear_pending(wb, work);
+
+ __wb_writeback(wb, nr_pages, sb, sync_mode);
+
+ /*
+ * This is a data integrity writeback, so only do the
+ * notification when we have completed the work.
+ */
+ if (sync_mode == WB_SYNC_ALL)
+ wb_clear_pending(wb, work);
+ }
+}
+
+/*
+ * This will be inlined in bdi_writeback_task() once we get rid of any
+ * dirty inodes on the default_backing_dev_info
+ */
+static void wb_do_writeback(struct bdi_writeback *wb)
+{
+ /*
+ * We get here in two cases:
+ *
+ * schedule_timeout() returned because the dirty writeback
+ * interval has elapsed. If that happens, the work item list
+ * will be empty and we will proceed to do kupdated style writeout.
+ *
+ * Someone called bdi_start_writeback(), which put one/more work
+ * items on the work_list. Process those.
+ */
+ if (list_empty(&wb->bdi->work_list))
+ wb_kupdated(wb);
+ else
+ wb_writeback(wb);
+}
+
+/*
* Handle writeback of dirty data for the device backed by this bdi. Also
* wakes up periodically and does kupdated style flushing.
*/
int bdi_writeback_task(struct bdi_writeback *wb)
{
+ DEFINE_WAIT(wait);
+
while (!kthread_should_stop()) {
unsigned long wait_jiffies;
- DEFINE_WAIT(wait);
+
+ wb_do_writeback(wb);
prepare_to_wait(&wb->wait, &wait, TASK_INTERRUPTIBLE);
wait_jiffies = msecs_to_jiffies(dirty_writeback_interval * 10);
schedule_timeout(wait_jiffies);
try_to_freeze();
-
- /*
- * We get here in two cases:
- *
- * schedule_timeout() returned because the dirty writeback
- * interval has elapsed. If that happens, we will be able
- * to acquire the writeback lock and will proceed to do
- * kupdated style writeout.
- *
- * Someone called bdi_start_writeback(), which will acquire
- * the writeback lock. This means our writeback_acquire()
- * below will fail and we call into bdi_pdflush() for
- * pdflush style writeout.
- *
- */
- if (writeback_acquire(wb))
- wb_kupdated(wb);
- else
- wb_writeback(wb);
-
- writeback_release(wb);
- finish_wait(&wb->wait, &wait);
}
+ finish_wait(&wb->wait, &wait);
return 0;
}
+/*
+ * Schedule writeback for all backing devices. Expensive! If this is a data
+ * integrity operation, writeback will be complete when this returns. If
+ * we are simply called for WB_SYNC_NONE, then writeback will merely be
+ * scheduled to run.
+ */
void bdi_writeback_all(struct super_block *sb, long nr_pages,
enum writeback_sync_modes sync_mode)
{
+ const bool must_wait = sync_mode == WB_SYNC_ALL;
struct backing_dev_info *bdi, *tmp;
+ struct bdi_work *work;
+ LIST_HEAD(list);
mutex_lock(&bdi_lock);
list_for_each_entry_safe(bdi, tmp, &bdi_list, bdi_list) {
+ struct bdi_work *work, work_stack;
+
if (!bdi_has_dirty_io(bdi))
continue;
- bdi_start_writeback(bdi, sb, nr_pages, sync_mode);
+
+ work = bdi_alloc_work(sb, nr_pages, sync_mode);
+ if (!work) {
+ work = &work_stack;
+ bdi_work_init_on_stack(work, sb, nr_pages, sync_mode);
+ } else if (must_wait)
+ list_add_tail(&work->wait_list, &list);
+
+ bdi_queue_work(bdi, work);
+ __bdi_start_work(bdi, work);
+
+ /*
+ * Do the wait inline if this came from the stack. This
+ * only happens if we ran out of memory, so should very
+ * rarely trigger.
+ */
+ if (work == &work_stack) {
+ bdi_wait_on_work_clear(work);
+ if (must_wait)
+ call_rcu(&work->rcu_head, bdi_work_free);
+ }
}
mutex_unlock(&bdi_lock);
+
+ /*
+ * If this is for WB_SYNC_ALL, wait for pending work to complete
+ * before returning.
+ */
+ while (!list_empty(&list)) {
+ work = list_entry(list.next, struct bdi_work, wait_list);
+ list_del(&work->wait_list);
+ bdi_wait_on_work_clear(work);
+ call_rcu(&work->rcu_head, bdi_work_free);
+ }
}
/*
- * We have only a single wb per bdi, so just return that.
+ * If the filesystem didn't provide a way to map an inode to a dedicated
+ * flusher thread, it doesn't support more than 1 thread. So we know it's
+ * the default thread, return that.
*/
static inline struct bdi_writeback *inode_get_wb(struct inode *inode)
{
- return &inode_to_bdi(inode)->wb;
+ const struct super_operations *sop = inode->i_sb->s_op;
+
+ if (!sop->inode_get_wb)
+ return &inode_to_bdi(inode)->wb;
+
+ return sop->inode_get_wb(inode);
}
/**
@@ -729,8 +997,24 @@ void generic_sync_bdi_inodes(struct super_block *sb,
struct writeback_control *wbc)
{
struct backing_dev_info *bdi = wbc->bdi;
+ struct bdi_writeback *wb;
- generic_sync_wb_inodes(&bdi->wb, sb, wbc);
+ /*
+ * Common case is just a single wb thread and that is embedded in
+ * the bdi, so it doesn't need locking
+ */
+ if (!bdi_wblist_needs_lock(bdi))
+ generic_sync_wb_inodes(&bdi->wb, sb, wbc);
+ else {
+ int idx;
+
+ idx = srcu_read_lock(&bdi->srcu);
+
+ list_for_each_entry_rcu(wb, &bdi->wb_list, list)
+ generic_sync_wb_inodes(wb, sb, wbc);
+
+ srcu_read_unlock(&bdi->srcu, idx);
+ }
}
/*
@@ -757,7 +1041,7 @@ void generic_sync_sb_inodes(struct super_block *sb,
struct writeback_control *wbc)
{
if (wbc->bdi)
- generic_sync_bdi_inodes(sb, wbc);
+ bdi_start_writeback(wbc->bdi, sb, wbc->nr_to_write, wbc->sync_mode);
else
bdi_writeback_all(sb, wbc->nr_to_write, wbc->sync_mode);
diff --git a/include/linux/backing-dev.h b/include/linux/backing-dev.h
index 77dc62c..0559cf8 100644
--- a/include/linux/backing-dev.h
+++ b/include/linux/backing-dev.h
@@ -13,6 +13,8 @@
#include <linux/proportions.h>
#include <linux/kernel.h>
#include <linux/fs.h>
+#include <linux/sched.h>
+#include <linux/srcu.h>
#include <linux/writeback.h>
#include <asm/atomic.h>
@@ -26,6 +28,7 @@ struct dentry;
enum bdi_state {
BDI_pending, /* On its way to being activated */
BDI_wb_alloc, /* Default embedded wb allocated */
+ BDI_wblist_lock, /* bdi->wb_list now needs locking */
BDI_async_congested, /* The async (write) queue is getting full */
BDI_sync_congested, /* The sync queue is getting full */
BDI_unused, /* Available bits start here */
@@ -42,6 +45,8 @@ enum bdi_stat_item {
#define BDI_STAT_BATCH (8*(1+ilog2(nr_cpu_ids)))
struct bdi_writeback {
+ struct list_head list; /* hangs off the bdi */
+
struct backing_dev_info *bdi; /* our parent bdi */
unsigned int nr;
@@ -50,13 +55,12 @@ struct bdi_writeback {
struct list_head b_dirty; /* dirty inodes */
struct list_head b_io; /* parked for writeback */
struct list_head b_more_io; /* parked for more writeback */
-
- unsigned long nr_pages;
- struct super_block *sb;
- enum writeback_sync_modes sync_mode;
};
+#define BDI_MAX_FLUSHERS 32
+
struct backing_dev_info {
+ struct srcu_struct srcu; /* for wb_list read side protection */
struct list_head bdi_list;
unsigned long ra_pages; /* max readahead in PAGE_CACHE_SIZE units */
unsigned long state; /* Always use atomic bitops on this */
@@ -75,8 +79,12 @@ struct backing_dev_info {
unsigned int max_ratio, max_prop_frac;
struct bdi_writeback wb; /* default writeback info for this bdi */
- unsigned long wb_active; /* bitmap of active tasks */
- unsigned long wb_mask; /* number of registered tasks */
+ spinlock_t wb_lock; /* protects update side of wb_list */
+ struct list_head wb_list; /* the flusher threads hanging off this bdi */
+ unsigned long wb_mask; /* bitmask of registered tasks */
+ unsigned int wb_cnt; /* number of registered tasks */
+
+ struct list_head work_list;
struct device *dev;
@@ -93,17 +101,23 @@ int bdi_register(struct backing_dev_info *bdi, struct device *parent,
const char *fmt, ...);
int bdi_register_dev(struct backing_dev_info *bdi, dev_t dev);
void bdi_unregister(struct backing_dev_info *bdi);
-int bdi_start_writeback(struct backing_dev_info *bdi, struct super_block *sb,
+void bdi_start_writeback(struct backing_dev_info *bdi, struct super_block *sb,
long nr_pages, enum writeback_sync_modes sync_mode);
int bdi_writeback_task(struct bdi_writeback *wb);
void bdi_writeback_all(struct super_block *sb, long nr_pages,
enum writeback_sync_modes sync_mode);
void bdi_add_default_flusher_task(struct backing_dev_info *bdi);
+void bdi_add_flusher_task(struct backing_dev_info *bdi);
int bdi_has_dirty_io(struct backing_dev_info *bdi);
extern struct mutex bdi_lock;
extern struct list_head bdi_list;
+static inline int bdi_wblist_needs_lock(struct backing_dev_info *bdi)
+{
+ return test_bit(BDI_wblist_lock, &bdi->state);
+}
+
static inline int wb_has_dirty_io(struct bdi_writeback *wb)
{
return !list_empty(&wb->b_dirty) ||
@@ -316,4 +330,10 @@ static inline bool mapping_cap_swap_backed(struct address_space *mapping)
return bdi_cap_swap_backed(mapping->backing_dev_info);
}
+static inline int bdi_sched_wait(void *word)
+{
+ schedule();
+ return 0;
+}
+
#endif /* _LINUX_BACKING_DEV_H */
diff --git a/include/linux/fs.h b/include/linux/fs.h
index ecdc544..d3bda5d 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -1550,11 +1550,14 @@ extern ssize_t vfs_readv(struct file *, const struct iovec __user *,
extern ssize_t vfs_writev(struct file *, const struct iovec __user *,
unsigned long, loff_t *);
+struct bdi_writeback;
+
struct super_operations {
struct inode *(*alloc_inode)(struct super_block *sb);
void (*destroy_inode)(struct inode *);
void (*dirty_inode) (struct inode *);
+ struct bdi_writeback *(*inode_get_wb) (struct inode *);
int (*write_inode) (struct inode *, int);
void (*drop_inode) (struct inode *);
void (*delete_inode) (struct inode *);
diff --git a/mm/backing-dev.c b/mm/backing-dev.c
index c8201f0..57e44e3 100644
--- a/mm/backing-dev.c
+++ b/mm/backing-dev.c
@@ -199,53 +199,96 @@ static int __init default_bdi_init(void)
}
subsys_initcall(default_bdi_init);
-static void bdi_wb_init(struct bdi_writeback *wb, struct backing_dev_info *bdi)
+static int wb_assign_nr(struct backing_dev_info *bdi, struct bdi_writeback *wb)
{
- memset(wb, 0, sizeof(*wb));
+ unsigned long mask = BDI_MAX_FLUSHERS - 1;
+ unsigned int nr;
- wb->bdi = bdi;
- init_waitqueue_head(&wb->wait);
- INIT_LIST_HEAD(&wb->b_dirty);
- INIT_LIST_HEAD(&wb->b_io);
- INIT_LIST_HEAD(&wb->b_more_io);
-}
+ do {
+ if ((bdi->wb_mask & mask) == mask)
+ return 1;
+
+ nr = find_first_zero_bit(&bdi->wb_mask, BDI_MAX_FLUSHERS);
+ } while (test_and_set_bit(nr, &bdi->wb_mask));
+
+ wb->nr = nr;
+
+ spin_lock(&bdi->wb_lock);
+ bdi->wb_cnt++;
+ spin_unlock(&bdi->wb_lock);
-static int wb_assign_nr(struct backing_dev_info *bdi, struct bdi_writeback *wb)
-{
- set_bit(0, &bdi->wb_mask);
- wb->nr = 0;
return 0;
}
static void bdi_put_wb(struct backing_dev_info *bdi, struct bdi_writeback *wb)
{
clear_bit(wb->nr, &bdi->wb_mask);
- clear_bit(BDI_wb_alloc, &bdi->state);
+
+ if (wb == &bdi->wb)
+ clear_bit(BDI_wb_alloc, &bdi->state);
+ else
+ kfree(wb);
+
+ spin_lock(&bdi->wb_lock);
+ bdi->wb_cnt--;
+ spin_unlock(&bdi->wb_lock);
+}
+
+static int bdi_wb_init(struct bdi_writeback *wb, struct backing_dev_info *bdi)
+{
+ memset(wb, 0, sizeof(*wb));
+
+ wb->bdi = bdi;
+ init_waitqueue_head(&wb->wait);
+ INIT_LIST_HEAD(&wb->b_dirty);
+ INIT_LIST_HEAD(&wb->b_io);
+ INIT_LIST_HEAD(&wb->b_more_io);
+
+ return wb_assign_nr(bdi, wb);
}
static struct bdi_writeback *bdi_new_wb(struct backing_dev_info *bdi)
{
struct bdi_writeback *wb;
- set_bit(BDI_wb_alloc, &bdi->state);
- wb = &bdi->wb;
- wb_assign_nr(bdi, wb);
+ /*
+ * Default bdi->wb is already assigned, so just return it
+ */
+ if (!test_and_set_bit(BDI_wb_alloc, &bdi->state))
+ wb = &bdi->wb;
+ else {
+ wb = kmalloc(sizeof(struct bdi_writeback), GFP_KERNEL);
+ if (wb) {
+ if (bdi_wb_init(wb, bdi)) {
+ kfree(wb);
+ wb = NULL;
+ }
+ }
+ }
+
return wb;
}
-static int bdi_start_fn(void *ptr)
+static void bdi_task_init(struct backing_dev_info *bdi,
+ struct bdi_writeback *wb)
{
- struct bdi_writeback *wb = ptr;
- struct backing_dev_info *bdi = wb->bdi;
struct task_struct *tsk = current;
- int ret;
+ int was_empty;
/*
- * Add us to the active bdi_list
+ * Add us to the active bdi_list. If we are adding threads beyond
+ * the default embedded bdi_writeback, then we need to start using
+ * proper locking. Check the list for empty first, then set the
+ * BDI_wblist_lock flag if there's > 1 entry on the list now
*/
- mutex_lock(&bdi_lock);
- list_add(&bdi->bdi_list, &bdi_list);
- mutex_unlock(&bdi_lock);
+ spin_lock(&bdi->wb_lock);
+
+ was_empty = list_empty(&bdi->wb_list);
+ list_add_tail_rcu(&wb->list, &bdi->wb_list);
+ if (!was_empty)
+ set_bit(BDI_wblist_lock, &bdi->state);
+
+ spin_unlock(&bdi->wb_lock);
tsk->flags |= PF_FLUSHER | PF_SWAPWRITE;
set_freezable();
@@ -254,6 +297,22 @@ static int bdi_start_fn(void *ptr)
* Our parent may run at a different priority, just set us to normal
*/
set_user_nice(tsk, 0);
+}
+
+static int bdi_start_fn(void *ptr)
+{
+ struct bdi_writeback *wb = ptr;
+ struct backing_dev_info *bdi = wb->bdi;
+ int ret;
+
+ /*
+ * Add us to the active bdi_list
+ */
+ mutex_lock(&bdi_lock);
+ list_add(&bdi->bdi_list, &bdi_list);
+ mutex_unlock(&bdi_lock);
+
+ bdi_task_init(bdi, wb);
/*
* Clear pending bit and wakeup anybody waiting to tear us down
@@ -264,13 +323,44 @@ static int bdi_start_fn(void *ptr)
ret = bdi_writeback_task(wb);
+ /*
+ * Remove us from the list
+ */
+ spin_lock(&bdi->wb_lock);
+ list_del_rcu(&wb->list);
+ spin_unlock(&bdi->wb_lock);
+
+ /*
+ * wait for rcu grace period to end, so we can free wb
+ */
+ synchronize_srcu(&bdi->srcu);
+
bdi_put_wb(bdi, wb);
return ret;
}
int bdi_has_dirty_io(struct backing_dev_info *bdi)
{
- return wb_has_dirty_io(&bdi->wb);
+ struct bdi_writeback *wb;
+ int ret = 0;
+
+ if (!bdi_wblist_needs_lock(bdi))
+ ret = wb_has_dirty_io(&bdi->wb);
+ else {
+ int idx;
+
+ idx = srcu_read_lock(&bdi->srcu);
+
+ list_for_each_entry_rcu(wb, &bdi->wb_list, list) {
+ ret = wb_has_dirty_io(wb);
+ if (ret)
+ break;
+ }
+
+ srcu_read_unlock(&bdi->srcu, idx);
+ }
+
+ return ret;
}
static void bdi_flush_io(struct backing_dev_info *bdi)
@@ -291,6 +381,8 @@ static int bdi_forker_task(void *ptr)
struct bdi_writeback *me = ptr;
DEFINE_WAIT(wait);
+ bdi_task_init(me->bdi, me);
+
for (;;) {
struct backing_dev_info *bdi, *tmp;
struct bdi_writeback *wb;
@@ -371,27 +463,70 @@ readd_flush:
}
/*
- * Add a new flusher task that gets created for any bdi
- * that has dirty data pending writeout
+ * bdi_lock held on entry
*/
-void bdi_add_default_flusher_task(struct backing_dev_info *bdi)
+static void bdi_add_one_flusher_task(struct backing_dev_info *bdi,
+ int(*func)(struct backing_dev_info *))
{
if (!bdi_cap_writeback_dirty(bdi))
return;
/*
- * Someone already marked this pending for task creation
+ * Check with the helper whether to proceed adding a task. Will only
+ * abort if we two or more simultanous calls to
+ * bdi_add_default_flusher_task() occured, further additions will block
+ * waiting for previous additions to finish.
*/
- if (test_and_set_bit(BDI_pending, &bdi->state))
- return;
+ if (!func(bdi)) {
+ list_move_tail(&bdi->bdi_list, &bdi_pending_list);
- mutex_lock(&bdi_lock);
- list_move_tail(&bdi->bdi_list, &bdi_pending_list);
+ /*
+ * We are now on the pending list, wake up bdi_forker_task()
+ * to finish the job and add us back to the active bdi_list
+ */
+ wake_up(&default_backing_dev_info.wb.wait);
+ }
+}
+
+static int flusher_add_helper_block(struct backing_dev_info *bdi)
+{
mutex_unlock(&bdi_lock);
+ wait_on_bit_lock(&bdi->state, BDI_pending, bdi_sched_wait,
+ TASK_UNINTERRUPTIBLE);
+ mutex_lock(&bdi_lock);
+ return 0;
+}
- wake_up(&default_backing_dev_info.wb.wait);
+static int flusher_add_helper_test(struct backing_dev_info *bdi)
+{
+ return test_and_set_bit(BDI_pending, &bdi->state);
+}
+
+/*
+ * Add the default flusher task that gets created for any bdi
+ * that has dirty data pending writeout
+ */
+void bdi_add_default_flusher_task(struct backing_dev_info *bdi)
+{
+ bdi_add_one_flusher_task(bdi, flusher_add_helper_test);
}
+/**
+ * bdi_add_flusher_task - add one more flusher task to this @bdi
+ * @bdi: the bdi
+ *
+ * Add an additional flusher task to this @bdi. Will block waiting on
+ * previous additions, if any.
+ *
+ */
+void bdi_add_flusher_task(struct backing_dev_info *bdi)
+{
+ mutex_lock(&bdi_lock);
+ bdi_add_one_flusher_task(bdi, flusher_add_helper_block);
+ mutex_unlock(&bdi_lock);
+}
+EXPORT_SYMBOL(bdi_add_flusher_task);
+
int bdi_register(struct backing_dev_info *bdi, struct device *parent,
const char *fmt, ...)
{
@@ -455,24 +590,21 @@ int bdi_register_dev(struct backing_dev_info *bdi, dev_t dev)
}
EXPORT_SYMBOL(bdi_register_dev);
-static int sched_wait(void *word)
-{
- schedule();
- return 0;
-}
-
/*
* Remove bdi from global list and shutdown any threads we have running
*/
static void bdi_wb_shutdown(struct backing_dev_info *bdi)
{
+ struct bdi_writeback *wb;
+
if (!bdi_cap_writeback_dirty(bdi))
return;
/*
* If setup is pending, wait for that to complete first
*/
- wait_on_bit(&bdi->state, BDI_pending, sched_wait, TASK_UNINTERRUPTIBLE);
+ wait_on_bit(&bdi->state, BDI_pending, bdi_sched_wait,
+ TASK_UNINTERRUPTIBLE);
/*
* Make sure nobody finds us on the bdi_list anymore
@@ -482,9 +614,11 @@ static void bdi_wb_shutdown(struct backing_dev_info *bdi)
mutex_unlock(&bdi_lock);
/*
- * Finally, kill the kernel thread
+ * Finally, kill the kernel threads. We don't need to be RCU
+ * safe anymore, since the bdi is gone from visibility.
*/
- kthread_stop(bdi->wb.task);
+ list_for_each_entry(wb, &bdi->wb_list, list)
+ kthread_stop(wb->task);
}
void bdi_unregister(struct backing_dev_info *bdi)
@@ -508,8 +642,12 @@ int bdi_init(struct backing_dev_info *bdi)
bdi->min_ratio = 0;
bdi->max_ratio = 100;
bdi->max_prop_frac = PROP_FRAC_BASE;
+ spin_lock_init(&bdi->wb_lock);
+ bdi->wb_mask = 0;
+ bdi->wb_cnt = 0;
INIT_LIST_HEAD(&bdi->bdi_list);
- bdi->wb_mask = bdi->wb_active = 0;
+ INIT_LIST_HEAD(&bdi->wb_list);
+ INIT_LIST_HEAD(&bdi->work_list);
bdi_wb_init(&bdi->wb, bdi);
@@ -519,10 +657,15 @@ int bdi_init(struct backing_dev_info *bdi)
goto err;
}
+ err = init_srcu_struct(&bdi->srcu);
+ if (err)
+ goto err;
+
bdi->dirty_exceeded = 0;
err = prop_local_init_percpu(&bdi->completions);
if (err) {
+ cleanup_srcu_struct(&bdi->srcu);
err:
while (i--)
percpu_counter_destroy(&bdi->bdi_stat[i]);
@@ -540,6 +683,8 @@ void bdi_destroy(struct backing_dev_info *bdi)
bdi_unregister(bdi);
+ cleanup_srcu_struct(&bdi->srcu);
+
for (i = 0; i < NR_BDI_STAT_ITEMS; i++)
percpu_counter_destroy(&bdi->bdi_stat[i]);
diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index 54a4a65..7dd7de7 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -665,8 +665,7 @@ void throttle_vm_writeout(gfp_t gfp_mask)
/*
* Start writeback of `nr_pages' pages. If `nr_pages' is zero, write back
- * the whole world. Returns 0 if a pdflush thread was dispatched. Returns
- * -1 if all pdflush threads were busy.
+ * the whole world.
*/
void wakeup_flusher_threads(long nr_pages)
{
@@ -674,7 +673,6 @@ void wakeup_flusher_threads(long nr_pages)
nr_pages = global_page_state(NR_FILE_DIRTY) +
global_page_state(NR_UNSTABLE_NFS);
bdi_writeback_all(NULL, nr_pages, WB_SYNC_NONE);
- return;
}
static void laptop_timer_fn(unsigned long unused);
--
1.6.3.rc0.1.gf800
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCH 08/12] writeback: include default_backing_dev_info in writeback
2009-05-26 9:33 [PATCH 0/12] Per-bdi writeback flusher threads v7 Jens Axboe
` (6 preceding siblings ...)
2009-05-26 9:33 ` [PATCH 07/12] writeback: support > 1 flusher thread per bdi Jens Axboe
@ 2009-05-26 9:33 ` Jens Axboe
2009-05-26 9:33 ` [PATCH 09/12] writeback: allow sleepy exit of default writeback task Jens Axboe
` (5 subsequent siblings)
13 siblings, 0 replies; 33+ messages in thread
From: Jens Axboe @ 2009-05-26 9:33 UTC (permalink / raw)
To: linux-kernel, linux-fsdevel
Cc: chris.mason, david, hch, akpm, jack, yanmin_zhang, richard, Jens Axboe
We see dirty inodes there occasionally, so better be safe and write them
out.
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
---
fs/fs-writeback.c | 2 +-
include/linux/writeback.h | 1 +
mm/backing-dev.c | 8 ++++++--
3 files changed, 8 insertions(+), 3 deletions(-)
diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c
index 8e0902e..e4f96b9 100644
--- a/fs/fs-writeback.c
+++ b/fs/fs-writeback.c
@@ -432,7 +432,7 @@ static void wb_writeback(struct bdi_writeback *wb)
* This will be inlined in bdi_writeback_task() once we get rid of any
* dirty inodes on the default_backing_dev_info
*/
-static void wb_do_writeback(struct bdi_writeback *wb)
+void wb_do_writeback(struct bdi_writeback *wb)
{
/*
* We get here in two cases:
diff --git a/include/linux/writeback.h b/include/linux/writeback.h
index baf04a9..e414702 100644
--- a/include/linux/writeback.h
+++ b/include/linux/writeback.h
@@ -69,6 +69,7 @@ void writeback_inodes(struct writeback_control *wbc);
int inode_wait(void *);
void sync_inodes_sb(struct super_block *, int wait);
void sync_inodes(int wait);
+void wb_do_writeback(struct bdi_writeback *wb);
/* writeback.h requires fs.h; it, too, is not included from here. */
static inline void wait_on_inode(struct inode *inode)
diff --git a/mm/backing-dev.c b/mm/backing-dev.c
index 57e44e3..977c171 100644
--- a/mm/backing-dev.c
+++ b/mm/backing-dev.c
@@ -396,8 +396,8 @@ static int bdi_forker_task(void *ptr)
* Temporary measure, we want to make sure we don't see
* dirty data on the default backing_dev_info
*/
- if (wb_has_dirty_io(me))
- bdi_flush_io(me->bdi);
+ if (wb_has_dirty_io(me) || !list_empty(&me->bdi->work_list))
+ wb_do_writeback(me);
prepare_to_wait(&me->wait, &wait, TASK_INTERRUPTIBLE);
@@ -424,6 +424,10 @@ static int bdi_forker_task(void *ptr)
continue;
}
+ /*
+ * This is our real job - check for pending entries in
+ * bdi_pending_list, and create the tasks that got added
+ */
bdi = list_entry(bdi_pending_list.next, struct backing_dev_info,
bdi_list);
list_del_init(&bdi->bdi_list);
--
1.6.3.rc0.1.gf800
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCH 09/12] writeback: allow sleepy exit of default writeback task
2009-05-26 9:33 [PATCH 0/12] Per-bdi writeback flusher threads v7 Jens Axboe
` (7 preceding siblings ...)
2009-05-26 9:33 ` [PATCH 08/12] writeback: include default_backing_dev_info in writeback Jens Axboe
@ 2009-05-26 9:33 ` Jens Axboe
2009-05-26 9:33 ` [PATCH 10/12] writeback: add some debug inode list counters to bdi stats Jens Axboe
` (4 subsequent siblings)
13 siblings, 0 replies; 33+ messages in thread
From: Jens Axboe @ 2009-05-26 9:33 UTC (permalink / raw)
To: linux-kernel, linux-fsdevel
Cc: chris.mason, david, hch, akpm, jack, yanmin_zhang, richard, Jens Axboe
Since we do lazy create of default writeback tasks for a bdi, we can
allow sleepy exit if it has been completely idle for 5 minutes.
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
---
fs/fs-writeback.c | 52 ++++++++++++++++++++++++++++++++++--------
include/linux/backing-dev.h | 5 ++++
include/linux/writeback.h | 2 +-
3 files changed, 48 insertions(+), 11 deletions(-)
diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c
index e4f96b9..c61c797 100644
--- a/fs/fs-writeback.c
+++ b/fs/fs-writeback.c
@@ -301,10 +301,10 @@ void bdi_start_writeback(struct backing_dev_info *bdi, struct super_block *sb,
* older_than_this takes precedence over nr_to_write. So we'll only write back
* all dirty pages if they are all attached to "old" mappings.
*/
-static void wb_kupdated(struct bdi_writeback *wb)
+static long wb_kupdated(struct bdi_writeback *wb)
{
unsigned long oldest_jif;
- long nr_to_write;
+ long nr_to_write, wrote = 0;
struct writeback_control wbc = {
.bdi = wb->bdi,
.sync_mode = WB_SYNC_NONE,
@@ -325,10 +325,13 @@ static void wb_kupdated(struct bdi_writeback *wb)
wbc.encountered_congestion = 0;
wbc.nr_to_write = MAX_WRITEBACK_PAGES;
generic_sync_wb_inodes(wb, NULL, &wbc);
+ wrote += MAX_WRITEBACK_PAGES - wbc.nr_to_write;
if (wbc.nr_to_write > 0)
break; /* All the old data is written */
nr_to_write -= MAX_WRITEBACK_PAGES;
}
+
+ return wrote;
}
static inline bool over_bground_thresh(void)
@@ -341,7 +344,7 @@ static inline bool over_bground_thresh(void)
global_page_state(NR_UNSTABLE_NFS) >= background_thresh);
}
-static void __wb_writeback(struct bdi_writeback *wb, long nr_pages,
+static long __wb_writeback(struct bdi_writeback *wb, long nr_pages,
struct super_block *sb,
enum writeback_sync_modes sync_mode)
{
@@ -351,6 +354,7 @@ static void __wb_writeback(struct bdi_writeback *wb, long nr_pages,
.older_than_this = NULL,
.range_cyclic = 1,
};
+ long wrote = 0;
for (;;) {
if (sync_mode == WB_SYNC_NONE && nr_pages <= 0 &&
@@ -363,6 +367,7 @@ static void __wb_writeback(struct bdi_writeback *wb, long nr_pages,
wbc.pages_skipped = 0;
generic_sync_wb_inodes(wb, sb, &wbc);
nr_pages -= MAX_WRITEBACK_PAGES - wbc.nr_to_write;
+ wrote += MAX_WRITEBACK_PAGES - wbc.nr_to_write;
/*
* If we ran out of stuff to write, bail unless more_io got set
*/
@@ -372,6 +377,8 @@ static void __wb_writeback(struct bdi_writeback *wb, long nr_pages,
break;
}
}
+
+ return wrote;
}
/*
@@ -400,10 +407,11 @@ static struct bdi_work *get_next_work_item(struct backing_dev_info *bdi,
/*
* Retrieve work items and do the writeback they describe
*/
-static void wb_writeback(struct bdi_writeback *wb)
+static long wb_writeback(struct bdi_writeback *wb)
{
struct backing_dev_info *bdi = wb->bdi;
struct bdi_work *work;
+ long wrote = 0;
while ((work = get_next_work_item(bdi, wb)) != NULL) {
struct super_block *sb = bdi_work_sb(work);
@@ -417,7 +425,7 @@ static void wb_writeback(struct bdi_writeback *wb)
if (sync_mode == WB_SYNC_NONE)
wb_clear_pending(wb, work);
- __wb_writeback(wb, nr_pages, sb, sync_mode);
+ wrote += __wb_writeback(wb, nr_pages, sb, sync_mode);
/*
* This is a data integrity writeback, so only do the
@@ -426,14 +434,18 @@ static void wb_writeback(struct bdi_writeback *wb)
if (sync_mode == WB_SYNC_ALL)
wb_clear_pending(wb, work);
}
+
+ return wrote;
}
/*
* This will be inlined in bdi_writeback_task() once we get rid of any
* dirty inodes on the default_backing_dev_info
*/
-void wb_do_writeback(struct bdi_writeback *wb)
+long wb_do_writeback(struct bdi_writeback *wb)
{
+ long wrote;
+
/*
* We get here in two cases:
*
@@ -445,9 +457,11 @@ void wb_do_writeback(struct bdi_writeback *wb)
* items on the work_list. Process those.
*/
if (list_empty(&wb->bdi->work_list))
- wb_kupdated(wb);
+ wrote = wb_kupdated(wb);
else
- wb_writeback(wb);
+ wrote = wb_writeback(wb);
+
+ return wrote;
}
/*
@@ -456,12 +470,30 @@ void wb_do_writeback(struct bdi_writeback *wb)
*/
int bdi_writeback_task(struct bdi_writeback *wb)
{
+ unsigned long last_active = jiffies;
+ unsigned long wait_jiffies = -1UL;
+ long pages_written;
DEFINE_WAIT(wait);
while (!kthread_should_stop()) {
- unsigned long wait_jiffies;
- wb_do_writeback(wb);
+ pages_written = wb_do_writeback(wb);
+
+ if (pages_written)
+ last_active = jiffies;
+ else if (wait_jiffies != -1UL) {
+ unsigned long max_idle;
+
+ /*
+ * Longest period of inactivity that we tolerate. If we
+ * see dirty data again later, the task will get
+ * recreated automatically.
+ */
+ max_idle = max(5UL * 60 * HZ, wait_jiffies);
+ if (time_after(jiffies, max_idle + last_active) &&
+ wb_is_default_task(wb))
+ break;
+ }
prepare_to_wait(&wb->wait, &wait, TASK_INTERRUPTIBLE);
wait_jiffies = msecs_to_jiffies(dirty_writeback_interval * 10);
diff --git a/include/linux/backing-dev.h b/include/linux/backing-dev.h
index 0559cf8..9523df3 100644
--- a/include/linux/backing-dev.h
+++ b/include/linux/backing-dev.h
@@ -113,6 +113,11 @@ int bdi_has_dirty_io(struct backing_dev_info *bdi);
extern struct mutex bdi_lock;
extern struct list_head bdi_list;
+static inline int wb_is_default_task(struct bdi_writeback *wb)
+{
+ return wb == &wb->bdi->wb;
+}
+
static inline int bdi_wblist_needs_lock(struct backing_dev_info *bdi)
{
return test_bit(BDI_wblist_lock, &bdi->state);
diff --git a/include/linux/writeback.h b/include/linux/writeback.h
index e414702..30e318b 100644
--- a/include/linux/writeback.h
+++ b/include/linux/writeback.h
@@ -69,7 +69,7 @@ void writeback_inodes(struct writeback_control *wbc);
int inode_wait(void *);
void sync_inodes_sb(struct super_block *, int wait);
void sync_inodes(int wait);
-void wb_do_writeback(struct bdi_writeback *wb);
+long wb_do_writeback(struct bdi_writeback *wb);
/* writeback.h requires fs.h; it, too, is not included from here. */
static inline void wait_on_inode(struct inode *inode)
--
1.6.3.rc0.1.gf800
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCH 10/12] writeback: add some debug inode list counters to bdi stats
2009-05-26 9:33 [PATCH 0/12] Per-bdi writeback flusher threads v7 Jens Axboe
` (8 preceding siblings ...)
2009-05-26 9:33 ` [PATCH 09/12] writeback: allow sleepy exit of default writeback task Jens Axboe
@ 2009-05-26 9:33 ` Jens Axboe
2009-05-26 9:33 ` [PATCH 11/12] writeback: add name to backing_dev_info Jens Axboe
` (3 subsequent siblings)
13 siblings, 0 replies; 33+ messages in thread
From: Jens Axboe @ 2009-05-26 9:33 UTC (permalink / raw)
To: linux-kernel, linux-fsdevel
Cc: chris.mason, david, hch, akpm, jack, yanmin_zhang, richard, Jens Axboe
Add some debug entries to be able to inspect the internal state of
the writeback details.
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
---
mm/backing-dev.c | 38 ++++++++++++++++++++++++++++++++++----
1 files changed, 34 insertions(+), 4 deletions(-)
diff --git a/mm/backing-dev.c b/mm/backing-dev.c
index 977c171..e7b0283 100644
--- a/mm/backing-dev.c
+++ b/mm/backing-dev.c
@@ -43,9 +43,29 @@ static void bdi_debug_init(void)
static int bdi_debug_stats_show(struct seq_file *m, void *v)
{
struct backing_dev_info *bdi = m->private;
+ struct bdi_writeback *wb;
unsigned long background_thresh;
unsigned long dirty_thresh;
unsigned long bdi_thresh;
+ unsigned long nr_dirty, nr_io, nr_more_io, nr_wb;
+ struct inode *inode;
+
+ /*
+ * inode lock is enough here, the bdi->wb_list is protected by
+ * RCU on the reader side
+ */
+ nr_wb = nr_dirty = nr_io = nr_more_io = 0;
+ spin_lock(&inode_lock);
+ list_for_each_entry(wb, &bdi->wb_list, list) {
+ nr_wb++;
+ list_for_each_entry(inode, &wb->b_dirty, i_list)
+ nr_dirty++;
+ list_for_each_entry(inode, &wb->b_io, i_list)
+ nr_io++;
+ list_for_each_entry(inode, &wb->b_more_io, i_list)
+ nr_more_io++;
+ }
+ spin_unlock(&inode_lock);
get_dirty_limits(&background_thresh, &dirty_thresh, &bdi_thresh, bdi);
@@ -55,12 +75,22 @@ static int bdi_debug_stats_show(struct seq_file *m, void *v)
"BdiReclaimable: %8lu kB\n"
"BdiDirtyThresh: %8lu kB\n"
"DirtyThresh: %8lu kB\n"
- "BackgroundThresh: %8lu kB\n",
+ "BackgroundThresh: %8lu kB\n"
+ "WriteBack threads:%8lu\n"
+ "b_dirty: %8lu\n"
+ "b_io: %8lu\n"
+ "b_more_io: %8lu\n"
+ "bdi_list: %8u\n"
+ "state: %8lx\n"
+ "wb_mask: %8lx\n"
+ "wb_list: %8u\n"
+ "wb_cnt: %8u\n",
(unsigned long) K(bdi_stat(bdi, BDI_WRITEBACK)),
(unsigned long) K(bdi_stat(bdi, BDI_RECLAIMABLE)),
- K(bdi_thresh),
- K(dirty_thresh),
- K(background_thresh));
+ K(bdi_thresh), K(dirty_thresh),
+ K(background_thresh), nr_wb, nr_dirty, nr_io, nr_more_io,
+ !list_empty(&bdi->bdi_list), bdi->state, bdi->wb_mask,
+ !list_empty(&bdi->wb_list), bdi->wb_cnt);
#undef K
return 0;
--
1.6.3.rc0.1.gf800
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCH 11/12] writeback: add name to backing_dev_info
2009-05-26 9:33 [PATCH 0/12] Per-bdi writeback flusher threads v7 Jens Axboe
` (9 preceding siblings ...)
2009-05-26 9:33 ` [PATCH 10/12] writeback: add some debug inode list counters to bdi stats Jens Axboe
@ 2009-05-26 9:33 ` Jens Axboe
2009-05-26 9:33 ` [PATCH 12/12] writeback: check for registered bdi in flusher add and inode dirty Jens Axboe
` (2 subsequent siblings)
13 siblings, 0 replies; 33+ messages in thread
From: Jens Axboe @ 2009-05-26 9:33 UTC (permalink / raw)
To: linux-kernel, linux-fsdevel
Cc: chris.mason, david, hch, akpm, jack, yanmin_zhang, richard, Jens Axboe
This enables us to track who does what and print info. Its main use
is catching dirty inodes on the default_backing_dev_info, so we can
fix that up.
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
---
block/blk-core.c | 1 +
drivers/block/aoe/aoeblk.c | 1 +
drivers/char/mem.c | 1 +
fs/btrfs/disk-io.c | 1 +
fs/char_dev.c | 1 +
fs/configfs/inode.c | 1 +
fs/fuse/inode.c | 1 +
fs/hugetlbfs/inode.c | 1 +
fs/nfs/client.c | 1 +
fs/ocfs2/dlm/dlmfs.c | 1 +
fs/ramfs/inode.c | 1 +
fs/sysfs/inode.c | 1 +
fs/ubifs/super.c | 1 +
include/linux/backing-dev.h | 2 ++
kernel/cgroup.c | 1 +
mm/backing-dev.c | 1 +
mm/swap_state.c | 1 +
17 files changed, 18 insertions(+), 0 deletions(-)
diff --git a/block/blk-core.c b/block/blk-core.c
index c89883b..d3f18b5 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -517,6 +517,7 @@ struct request_queue *blk_alloc_queue_node(gfp_t gfp_mask, int node_id)
q->backing_dev_info.unplug_io_fn = blk_backing_dev_unplug;
q->backing_dev_info.unplug_io_data = q;
+ q->backing_dev_info.name = "block";
err = bdi_init(&q->backing_dev_info);
if (err) {
kmem_cache_free(blk_requestq_cachep, q);
diff --git a/drivers/block/aoe/aoeblk.c b/drivers/block/aoe/aoeblk.c
index 2307a27..0efb8fc 100644
--- a/drivers/block/aoe/aoeblk.c
+++ b/drivers/block/aoe/aoeblk.c
@@ -265,6 +265,7 @@ aoeblk_gdalloc(void *vp)
}
blk_queue_make_request(&d->blkq, aoeblk_make_request);
+ d->blkq.backing_dev_info.name = "aoe";
if (bdi_init(&d->blkq.backing_dev_info))
goto err_mempool;
spin_lock_irqsave(&d->lock, flags);
diff --git a/drivers/char/mem.c b/drivers/char/mem.c
index 8f05c38..3b38093 100644
--- a/drivers/char/mem.c
+++ b/drivers/char/mem.c
@@ -820,6 +820,7 @@ static const struct file_operations zero_fops = {
* - permits private mappings, "copies" are taken of the source of zeros
*/
static struct backing_dev_info zero_bdi = {
+ .name = "char/mem",
.capabilities = BDI_CAP_MAP_COPY,
};
diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c
index 2dc19c9..eff2a82 100644
--- a/fs/btrfs/disk-io.c
+++ b/fs/btrfs/disk-io.c
@@ -1353,6 +1353,7 @@ static int setup_bdi(struct btrfs_fs_info *info, struct backing_dev_info *bdi)
{
int err;
+ bdi->name = "btrfs";
bdi->capabilities = BDI_CAP_MAP_COPY;
err = bdi_init(bdi);
if (err)
diff --git a/fs/char_dev.c b/fs/char_dev.c
index 38f7122..350ef9c 100644
--- a/fs/char_dev.c
+++ b/fs/char_dev.c
@@ -32,6 +32,7 @@
* - no readahead or I/O queue unplugging required
*/
struct backing_dev_info directly_mappable_cdev_bdi = {
+ .name = "char",
.capabilities = (
#ifdef CONFIG_MMU
/* permit private copies of the data to be taken */
diff --git a/fs/configfs/inode.c b/fs/configfs/inode.c
index 5d349d3..9a266cd 100644
--- a/fs/configfs/inode.c
+++ b/fs/configfs/inode.c
@@ -46,6 +46,7 @@ static const struct address_space_operations configfs_aops = {
};
static struct backing_dev_info configfs_backing_dev_info = {
+ .name = "configfs",
.ra_pages = 0, /* No readahead */
.capabilities = BDI_CAP_NO_ACCT_AND_WRITEBACK,
};
diff --git a/fs/fuse/inode.c b/fs/fuse/inode.c
index 91f7c85..e5e8b03 100644
--- a/fs/fuse/inode.c
+++ b/fs/fuse/inode.c
@@ -484,6 +484,7 @@ int fuse_conn_init(struct fuse_conn *fc, struct super_block *sb)
INIT_LIST_HEAD(&fc->bg_queue);
INIT_LIST_HEAD(&fc->entry);
atomic_set(&fc->num_waiting, 0);
+ fc->bdi.name = "fuse";
fc->bdi.ra_pages = (VM_MAX_READAHEAD * 1024) / PAGE_CACHE_SIZE;
fc->bdi.unplug_io_fn = default_unplug_io_fn;
/* fuse does it's own writeback accounting */
diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c
index c1462d4..db1e537 100644
--- a/fs/hugetlbfs/inode.c
+++ b/fs/hugetlbfs/inode.c
@@ -43,6 +43,7 @@ static const struct inode_operations hugetlbfs_dir_inode_operations;
static const struct inode_operations hugetlbfs_inode_operations;
static struct backing_dev_info hugetlbfs_backing_dev_info = {
+ .name = "hugetlbfs",
.ra_pages = 0, /* No readahead */
.capabilities = BDI_CAP_NO_ACCT_AND_WRITEBACK,
};
diff --git a/fs/nfs/client.c b/fs/nfs/client.c
index 75c9cd2..3a26d06 100644
--- a/fs/nfs/client.c
+++ b/fs/nfs/client.c
@@ -836,6 +836,7 @@ static void nfs_server_set_fsinfo(struct nfs_server *server, struct nfs_fsinfo *
server->rsize = NFS_MAX_FILE_IO_SIZE;
server->rpages = (server->rsize + PAGE_CACHE_SIZE - 1) >> PAGE_CACHE_SHIFT;
+ server->backing_dev_info.name = "nfs";
server->backing_dev_info.ra_pages = server->rpages * NFS_MAX_READAHEAD;
if (server->wsize > max_rpc_payload)
diff --git a/fs/ocfs2/dlm/dlmfs.c b/fs/ocfs2/dlm/dlmfs.c
index 1c9efb4..02bf178 100644
--- a/fs/ocfs2/dlm/dlmfs.c
+++ b/fs/ocfs2/dlm/dlmfs.c
@@ -325,6 +325,7 @@ clear_fields:
}
static struct backing_dev_info dlmfs_backing_dev_info = {
+ .name = "ocfs2-dlmfs",
.ra_pages = 0, /* No readahead */
.capabilities = BDI_CAP_NO_ACCT_AND_WRITEBACK,
};
diff --git a/fs/ramfs/inode.c b/fs/ramfs/inode.c
index 3a6b193..5a24199 100644
--- a/fs/ramfs/inode.c
+++ b/fs/ramfs/inode.c
@@ -46,6 +46,7 @@ static const struct super_operations ramfs_ops;
static const struct inode_operations ramfs_dir_inode_operations;
static struct backing_dev_info ramfs_backing_dev_info = {
+ .name = "ramfs",
.ra_pages = 0, /* No readahead */
.capabilities = BDI_CAP_NO_ACCT_AND_WRITEBACK |
BDI_CAP_MAP_DIRECT | BDI_CAP_MAP_COPY |
diff --git a/fs/sysfs/inode.c b/fs/sysfs/inode.c
index 555f0ff..e57f98e 100644
--- a/fs/sysfs/inode.c
+++ b/fs/sysfs/inode.c
@@ -29,6 +29,7 @@ static const struct address_space_operations sysfs_aops = {
};
static struct backing_dev_info sysfs_backing_dev_info = {
+ .name = "sysfs",
.ra_pages = 0, /* No readahead */
.capabilities = BDI_CAP_NO_ACCT_AND_WRITEBACK,
};
diff --git a/fs/ubifs/super.c b/fs/ubifs/super.c
index e9f7a75..2349e2c 100644
--- a/fs/ubifs/super.c
+++ b/fs/ubifs/super.c
@@ -1923,6 +1923,7 @@ static int ubifs_fill_super(struct super_block *sb, void *data, int silent)
*
* Read-ahead will be disabled because @c->bdi.ra_pages is 0.
*/
+ c->bdi.name = "ubifs",
c->bdi.capabilities = BDI_CAP_MAP_COPY;
c->bdi.unplug_io_fn = default_unplug_io_fn;
err = bdi_init(&c->bdi);
diff --git a/include/linux/backing-dev.h b/include/linux/backing-dev.h
index 9523df3..f8b9325 100644
--- a/include/linux/backing-dev.h
+++ b/include/linux/backing-dev.h
@@ -70,6 +70,8 @@ struct backing_dev_info {
void (*unplug_io_fn)(struct backing_dev_info *, struct page *);
void *unplug_io_data;
+ char *name;
+
struct percpu_counter bdi_stat[NR_BDI_STAT_ITEMS];
struct prop_local_percpu completions;
diff --git a/kernel/cgroup.c b/kernel/cgroup.c
index a7267bf..0863c5f 100644
--- a/kernel/cgroup.c
+++ b/kernel/cgroup.c
@@ -598,6 +598,7 @@ static struct inode_operations cgroup_dir_inode_operations;
static struct file_operations proc_cgroupstats_operations;
static struct backing_dev_info cgroup_backing_dev_info = {
+ .name = "cgroup",
.capabilities = BDI_CAP_NO_ACCT_AND_WRITEBACK,
};
diff --git a/mm/backing-dev.c b/mm/backing-dev.c
index e7b0283..fb7e00a 100644
--- a/mm/backing-dev.c
+++ b/mm/backing-dev.c
@@ -17,6 +17,7 @@ void default_unplug_io_fn(struct backing_dev_info *bdi, struct page *page)
EXPORT_SYMBOL(default_unplug_io_fn);
struct backing_dev_info default_backing_dev_info = {
+ .name = "default",
.ra_pages = VM_MAX_READAHEAD * 1024 / PAGE_CACHE_SIZE,
.state = 0,
.capabilities = BDI_CAP_MAP_COPY | BDI_CAP_FLUSH_FORKER,
diff --git a/mm/swap_state.c b/mm/swap_state.c
index 3ecea98..323da00 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -34,6 +34,7 @@ static const struct address_space_operations swap_aops = {
};
static struct backing_dev_info swap_backing_dev_info = {
+ .name = "swap",
.capabilities = BDI_CAP_NO_ACCT_AND_WRITEBACK | BDI_CAP_SWAP_BACKED,
.unplug_io_fn = swap_unplug_io_fn,
};
--
1.6.3.rc0.1.gf800
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCH 12/12] writeback: check for registered bdi in flusher add and inode dirty
2009-05-26 9:33 [PATCH 0/12] Per-bdi writeback flusher threads v7 Jens Axboe
` (10 preceding siblings ...)
2009-05-26 9:33 ` [PATCH 11/12] writeback: add name to backing_dev_info Jens Axboe
@ 2009-05-26 9:33 ` Jens Axboe
2009-05-26 15:25 ` [PATCH 0/12] Per-bdi writeback flusher threads v7 Damien Wyart
2009-05-27 5:27 ` Zhang, Yanmin
13 siblings, 0 replies; 33+ messages in thread
From: Jens Axboe @ 2009-05-26 9:33 UTC (permalink / raw)
To: linux-kernel, linux-fsdevel
Cc: chris.mason, david, hch, akpm, jack, yanmin_zhang, richard, Jens Axboe
Also a debugging aid. We want to catch dirty inodes being added to
backing devices that don't do writeback.
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
---
fs/fs-writeback.c | 7 +++++++
include/linux/backing-dev.h | 1 +
mm/backing-dev.c | 6 ++++++
3 files changed, 14 insertions(+), 0 deletions(-)
diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c
index c61c797..a185a16 100644
--- a/fs/fs-writeback.c
+++ b/fs/fs-writeback.c
@@ -677,6 +677,13 @@ void __mark_inode_dirty(struct inode *inode, int flags)
*/
if (!was_dirty) {
struct bdi_writeback *wb = inode_get_wb(inode);
+ struct backing_dev_info *bdi = wb->bdi;
+
+ if (bdi_cap_writeback_dirty(bdi) &&
+ !test_bit(BDI_registered, &bdi->state)) {
+ WARN_ON(1);
+ printk("bdi-%s not registered\n", bdi->name);
+ }
inode->dirtied_when = jiffies;
list_move(&inode->i_list, &wb->b_dirty);
diff --git a/include/linux/backing-dev.h b/include/linux/backing-dev.h
index f8b9325..dee38ec 100644
--- a/include/linux/backing-dev.h
+++ b/include/linux/backing-dev.h
@@ -31,6 +31,7 @@ enum bdi_state {
BDI_wblist_lock, /* bdi->wb_list now needs locking */
BDI_async_congested, /* The async (write) queue is getting full */
BDI_sync_congested, /* The sync queue is getting full */
+ BDI_registered, /* bdi_register() was done */
BDI_unused, /* Available bits start here */
};
diff --git a/mm/backing-dev.c b/mm/backing-dev.c
index fb7e00a..420496c 100644
--- a/mm/backing-dev.c
+++ b/mm/backing-dev.c
@@ -506,6 +506,11 @@ static void bdi_add_one_flusher_task(struct backing_dev_info *bdi,
if (!bdi_cap_writeback_dirty(bdi))
return;
+ if (WARN_ON(!test_bit(BDI_registered, &bdi->state))) {
+ printk("bdi %p/%s is not registered!\n", bdi, bdi->name);
+ return;
+ }
+
/*
* Check with the helper whether to proceed adding a task. Will only
* abort if we two or more simultanous calls to
@@ -614,6 +619,7 @@ remove_err:
}
bdi_debug_register(bdi, dev_name(dev));
+ set_bit(BDI_registered, &bdi->state);
exit:
return ret;
}
--
1.6.3.rc0.1.gf800
^ permalink raw reply related [flat|nested] 33+ messages in thread
* Re: [PATCH 0/12] Per-bdi writeback flusher threads v7
2009-05-26 9:33 [PATCH 0/12] Per-bdi writeback flusher threads v7 Jens Axboe
` (11 preceding siblings ...)
2009-05-26 9:33 ` [PATCH 12/12] writeback: check for registered bdi in flusher add and inode dirty Jens Axboe
@ 2009-05-26 15:25 ` Damien Wyart
2009-05-26 16:41 ` Jens Axboe
2009-05-27 5:27 ` Zhang, Yanmin
13 siblings, 1 reply; 33+ messages in thread
From: Damien Wyart @ 2009-05-26 15:25 UTC (permalink / raw)
To: Jens Axboe
Cc: linux-kernel, linux-fsdevel, chris.mason, david, hch, akpm, jack,
yanmin_zhang, richard
Hello,
I have been playing with v7 since your sending and after a white (short
on laptop, longer on desktop, a few hours), writeback doesn't seem to
work anymore. Manual call to sync hangs (process in D state) and Dirty
value in meminfo gets growing. As previous versions had been heavily
tested, I guess there is some regression in v7.
Best
Damien Wyart
> Here's the 7th version of the writeback patches. Changes since
> v5/v6:
> - Move the sync_supers() to the global bdi_forker_task() thread, so we
> don't writeback the supers from all the bdi kupdated() tasks.
> - Make bdi_start_writeback() and bdi_writeback_all() be sync when called
> with WB_SYNC_ALL only.
> - Shuffle some more things around to make a cleaner series. The sync vs
> async nature of bdi_writeback_all() and bdi_start_writeback() isn't
> consistent through the series, but otherwise things should be sane.
> I'd appreciate if Richard and Yanmin could re-run testing with this,
> just to make sure that things are sane. For ease of patching, I've
> put the full diff here:
> http://kernel.dk/writeback-v7.patch
> and also stored this in a writeback-v7 branch that will not change,
> you can pull that into Linus tree from here:
> git://git.kernel.dk/linux-2.6-block.git writeback-v7
--
Damien Wyart
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCH 0/12] Per-bdi writeback flusher threads v7
2009-05-26 15:25 ` [PATCH 0/12] Per-bdi writeback flusher threads v7 Damien Wyart
@ 2009-05-26 16:41 ` Jens Axboe
2009-05-26 17:08 ` Damien Wyart
0 siblings, 1 reply; 33+ messages in thread
From: Jens Axboe @ 2009-05-26 16:41 UTC (permalink / raw)
To: Damien Wyart
Cc: linux-kernel, linux-fsdevel, chris.mason, david, hch, akpm, jack,
yanmin_zhang, richard
On Tue, May 26 2009, Damien Wyart wrote:
> Hello,
>
> I have been playing with v7 since your sending and after a white (short
> on laptop, longer on desktop, a few hours), writeback doesn't seem to
> work anymore. Manual call to sync hangs (process in D state) and Dirty
> value in meminfo gets growing. As previous versions had been heavily
> tested, I guess there is some regression in v7.
Not good, the prime suspect is the sync notification stuff. I'll take a
look and get that fixed. You didn't happen to catch any sysrq-t back
traces or anything like that? Would be interesting to see where where
bdi-default and the bdi-* threads are stuck.
Can you send me your .config and a dmesg from a booted system? Any
kernel, doesn't have to be the writeback kernel.
--
Jens Axboe
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCH 0/12] Per-bdi writeback flusher threads v7
2009-05-26 16:41 ` Jens Axboe
@ 2009-05-26 17:08 ` Damien Wyart
2009-05-26 17:10 ` Damien Wyart
2009-05-26 20:47 ` Jens Axboe
0 siblings, 2 replies; 33+ messages in thread
From: Damien Wyart @ 2009-05-26 17:08 UTC (permalink / raw)
To: Jens Axboe
Cc: linux-kernel, linux-fsdevel, chris.mason, david, hch, akpm, jack,
yanmin_zhang, richard
[-- Attachment #1: Type: text/plain, Size: 1480 bytes --]
> > I have been playing with v7 since your sending and after a while
> > (short on laptop, longer on desktop, a few hours), writeback doesn't
> > seem to work anymore. Manual call to sync hangs (process in D state)
> > and Dirty value in meminfo gets growing. As previous versions had
> > been heavily tested, I guess there is some regression in v7.
> Not good, the prime suspect is the sync notification stuff. I'll take
> a look and get that fixed. You didn't happen to catch any sysrq-t back
> traces or anything like that? Would be interesting to see where
> bdi-default and the bdi-* threads are stuck.
No, as I was doing many things at the same time and not exclusively
debugging, I just rebooted hard and went back to an upatched kernel when
the problems occured. But I noticed only bdi-default was alive, the
other bdi-* threads had disappeared and the sync commands I had tried
were all in D state. Also I tried to reinstall a kernel .deb (these
systems are Debian) and this got stuck guring installation, when probing
grub config (do not know if there is some sync syscall inthere).
Can try to go further tomorrow but will not have a lot of time...
> Can you send me your .config and a dmesg from a booted system? Any
> kernel, doesn't have to be the writeback kernel.
Ok, I am attaching corresponding config and dmesg from the desktop
system (the laptop which reproduces the problem more easily is at the
office so not reachable until tomorrow).
--
Damien Wyart
[-- Attachment #2: config.2630rc7git1.gz --]
[-- Type: application/octet-stream, Size: 10793 bytes --]
[-- Attachment #3: dmesg.2630rc7git1.gz --]
[-- Type: application/octet-stream, Size: 7257 bytes --]
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCH 0/12] Per-bdi writeback flusher threads v7
2009-05-26 17:08 ` Damien Wyart
@ 2009-05-26 17:10 ` Damien Wyart
2009-05-26 20:47 ` Jens Axboe
1 sibling, 0 replies; 33+ messages in thread
From: Damien Wyart @ 2009-05-26 17:10 UTC (permalink / raw)
To: Jens Axboe
Cc: linux-kernel, linux-fsdevel, chris.mason, david, hch, akpm, jack,
yanmin_zhang, richard
Also note that on both systems, / is ext3, other mountpoints are xfs and
I also have one or two vfat FS mounted. Here is the corresponding fstab:
proc /proc proc defaults 0 0
/dev/sdb8 none swap sw 0 0
/dev/sdb2 / ext3 defaults,data=writeback,errors=remount-ro,relatime 0 1
tmpfs /tmp tmpfs defaults,nosuid,nodev,size=1G 0 0
/tmp /var/tmp none bind 0 0
/dev/sdb5 /usr xfs defaults,noatime,nodiratime 0 2
/dev/sdb6 /home xfs defaults,noatime,nodiratime 0 2
/dev/sdb7 /storage1 xfs defaults,noatime,nodiratime 0 2
/dev/sda5 /storage2 xfs defaults,noatime,nodiratime 0 2
/dev/sdb9 /storage3 xfs defaults,noatime,nodiratime 0 2
#
/dev/sdb1 /storagewin vfat defaults,uid=dw,gid=dw,showexec,quiet 0 0
--
Damien Wyart
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCH 0/12] Per-bdi writeback flusher threads v7
2009-05-26 17:08 ` Damien Wyart
2009-05-26 17:10 ` Damien Wyart
@ 2009-05-26 20:47 ` Jens Axboe
2009-05-26 21:11 ` Jens Axboe
1 sibling, 1 reply; 33+ messages in thread
From: Jens Axboe @ 2009-05-26 20:47 UTC (permalink / raw)
To: Damien Wyart
Cc: linux-kernel, linux-fsdevel, chris.mason, david, hch, akpm, jack,
yanmin_zhang, richard
On Tue, May 26 2009, Damien Wyart wrote:
> > > I have been playing with v7 since your sending and after a while
> > > (short on laptop, longer on desktop, a few hours), writeback doesn't
> > > seem to work anymore. Manual call to sync hangs (process in D state)
> > > and Dirty value in meminfo gets growing. As previous versions had
> > > been heavily tested, I guess there is some regression in v7.
>
> > Not good, the prime suspect is the sync notification stuff. I'll take
> > a look and get that fixed. You didn't happen to catch any sysrq-t back
> > traces or anything like that? Would be interesting to see where
> > bdi-default and the bdi-* threads are stuck.
>
> No, as I was doing many things at the same time and not exclusively
> debugging, I just rebooted hard and went back to an upatched kernel when
> the problems occured. But I noticed only bdi-default was alive, the
> other bdi-* threads had disappeared and the sync commands I had tried
> were all in D state. Also I tried to reinstall a kernel .deb (these
> systems are Debian) and this got stuck guring installation, when probing
> grub config (do not know if there is some sync syscall inthere).
>
> Can try to go further tomorrow but will not have a lot of time...
OK, I spotted the problem. If we fallback to the on-stack allocation in
bdi_writeback_all(), then we do the wait for the work completion with
the bdi_lock mutex held. This can deadlock with bdi_forker_task(), so if
we require that to be invoked to make progress (happens if a thread
needs to be restarted), then we have a deadlock on that mutex.
I'll cook up a fix for this, but probably not before the morning.
--
Jens Axboe
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCH 0/12] Per-bdi writeback flusher threads v7
2009-05-26 20:47 ` Jens Axboe
@ 2009-05-26 21:11 ` Jens Axboe
2009-05-27 5:21 ` Damien Wyart
0 siblings, 1 reply; 33+ messages in thread
From: Jens Axboe @ 2009-05-26 21:11 UTC (permalink / raw)
To: Damien Wyart
Cc: linux-kernel, linux-fsdevel, chris.mason, david, hch, akpm, jack,
yanmin_zhang, richard
On Tue, May 26 2009, Jens Axboe wrote:
> On Tue, May 26 2009, Damien Wyart wrote:
> > > > I have been playing with v7 since your sending and after a while
> > > > (short on laptop, longer on desktop, a few hours), writeback doesn't
> > > > seem to work anymore. Manual call to sync hangs (process in D state)
> > > > and Dirty value in meminfo gets growing. As previous versions had
> > > > been heavily tested, I guess there is some regression in v7.
> >
> > > Not good, the prime suspect is the sync notification stuff. I'll take
> > > a look and get that fixed. You didn't happen to catch any sysrq-t back
> > > traces or anything like that? Would be interesting to see where
> > > bdi-default and the bdi-* threads are stuck.
> >
> > No, as I was doing many things at the same time and not exclusively
> > debugging, I just rebooted hard and went back to an upatched kernel when
> > the problems occured. But I noticed only bdi-default was alive, the
> > other bdi-* threads had disappeared and the sync commands I had tried
> > were all in D state. Also I tried to reinstall a kernel .deb (these
> > systems are Debian) and this got stuck guring installation, when probing
> > grub config (do not know if there is some sync syscall inthere).
> >
> > Can try to go further tomorrow but will not have a lot of time...
>
> OK, I spotted the problem. If we fallback to the on-stack allocation in
> bdi_writeback_all(), then we do the wait for the work completion with
> the bdi_lock mutex held. This can deadlock with bdi_forker_task(), so if
> we require that to be invoked to make progress (happens if a thread
> needs to be restarted), then we have a deadlock on that mutex.
>
> I'll cook up a fix for this, but probably not before the morning.
Untested fix. I think it should work, but I haven't run it here yet.
diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c
index a185a16..1662ede 100644
--- a/fs/fs-writeback.c
+++ b/fs/fs-writeback.c
@@ -122,12 +122,11 @@ static void bdi_work_free(struct rcu_head *head)
static void wb_work_complete(struct bdi_work *work)
{
- if (!bdi_work_on_stack(work)) {
- bdi_work_clear(work);
+ const enum writeback_sync_modes sync_mode = work->sync_mode;
- if (work->sync_mode == WB_SYNC_NONE)
- call_rcu(&work->rcu_head, bdi_work_free);
- } else
+ if (!bdi_work_on_stack(work))
+ bdi_work_clear(work);
+ if (sync_mode == WB_SYNC_NONE || bdi_work_on_stack(work))
call_rcu(&work->rcu_head, bdi_work_free);
}
@@ -272,7 +271,7 @@ void bdi_start_writeback(struct backing_dev_info *bdi, struct super_block *sb,
*/
if (work == &work_stack || must_wait) {
bdi_wait_on_work_clear(work);
- if (must_wait)
+ if (must_wait && work != &work_stack)
call_rcu(&work->rcu_head, bdi_work_free);
}
}
@@ -511,10 +510,9 @@ int bdi_writeback_task(struct bdi_writeback *wb)
* we are simply called for WB_SYNC_NONE, then writeback will merely be
* scheduled to run.
*/
-void bdi_writeback_all(struct super_block *sb, long nr_pages,
- enum writeback_sync_modes sync_mode)
+void bdi_writeback_all(struct super_block *sb, struct writeback_control *wbc)
{
- const bool must_wait = sync_mode == WB_SYNC_ALL;
+ const bool must_wait = wbc->sync_mode == WB_SYNC_ALL;
struct backing_dev_info *bdi, *tmp;
struct bdi_work *work;
LIST_HEAD(list);
@@ -522,31 +520,21 @@ void bdi_writeback_all(struct super_block *sb, long nr_pages,
mutex_lock(&bdi_lock);
list_for_each_entry_safe(bdi, tmp, &bdi_list, bdi_list) {
- struct bdi_work *work, work_stack;
+ struct bdi_work *work;
if (!bdi_has_dirty_io(bdi))
continue;
- work = bdi_alloc_work(sb, nr_pages, sync_mode);
+ work = bdi_alloc_work(sb, wbc->nr_to_write, wbc->sync_mode);
if (!work) {
- work = &work_stack;
- bdi_work_init_on_stack(work, sb, nr_pages, sync_mode);
- } else if (must_wait)
+ generic_sync_bdi_inodes(sb, wbc);
+ continue;
+ }
+ if (must_wait)
list_add_tail(&work->wait_list, &list);
bdi_queue_work(bdi, work);
__bdi_start_work(bdi, work);
-
- /*
- * Do the wait inline if this came from the stack. This
- * only happens if we ran out of memory, so should very
- * rarely trigger.
- */
- if (work == &work_stack) {
- bdi_wait_on_work_clear(work);
- if (must_wait)
- call_rcu(&work->rcu_head, bdi_work_free);
- }
}
mutex_unlock(&bdi_lock);
@@ -1082,7 +1070,7 @@ void generic_sync_sb_inodes(struct super_block *sb,
if (wbc->bdi)
bdi_start_writeback(wbc->bdi, sb, wbc->nr_to_write, wbc->sync_mode);
else
- bdi_writeback_all(sb, wbc->nr_to_write, wbc->sync_mode);
+ bdi_writeback_all(sb, wbc);
if (wbc->sync_mode == WB_SYNC_ALL) {
struct inode *inode, *old_inode = NULL;
diff --git a/include/linux/backing-dev.h b/include/linux/backing-dev.h
index dee38ec..679cfb8 100644
--- a/include/linux/backing-dev.h
+++ b/include/linux/backing-dev.h
@@ -107,8 +107,7 @@ void bdi_unregister(struct backing_dev_info *bdi);
void bdi_start_writeback(struct backing_dev_info *bdi, struct super_block *sb,
long nr_pages, enum writeback_sync_modes sync_mode);
int bdi_writeback_task(struct bdi_writeback *wb);
-void bdi_writeback_all(struct super_block *sb, long nr_pages,
- enum writeback_sync_modes sync_mode);
+void bdi_writeback_all(struct super_block *sb, struct writeback_control *wbc);
void bdi_add_default_flusher_task(struct backing_dev_info *bdi);
void bdi_add_flusher_task(struct backing_dev_info *bdi);
int bdi_has_dirty_io(struct backing_dev_info *bdi);
diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index 7dd7de7..dd403cf 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -669,10 +669,18 @@ void throttle_vm_writeout(gfp_t gfp_mask)
*/
void wakeup_flusher_threads(long nr_pages)
{
+ struct writeback_control wbc = {
+ .sync_mode = WB_SYNC_NONE,
+ .older_than_this = NULL,
+ .range_cyclic = 1,
+ };
+
if (nr_pages == 0)
nr_pages = global_page_state(NR_FILE_DIRTY) +
global_page_state(NR_UNSTABLE_NFS);
- bdi_writeback_all(NULL, nr_pages, WB_SYNC_NONE);
+
+ wbc.nr_to_write = nr_pages;
+ bdi_writeback_all(NULL, &wbc);
}
static void laptop_timer_fn(unsigned long unused);
--
Jens Axboe
^ permalink raw reply related [flat|nested] 33+ messages in thread
* Re: [PATCH 0/12] Per-bdi writeback flusher threads v7
2009-05-26 21:11 ` Jens Axboe
@ 2009-05-27 5:21 ` Damien Wyart
2009-05-27 5:49 ` Damien Wyart
2009-05-27 6:17 ` Jens Axboe
0 siblings, 2 replies; 33+ messages in thread
From: Damien Wyart @ 2009-05-27 5:21 UTC (permalink / raw)
To: Jens Axboe
Cc: linux-kernel, linux-fsdevel, chris.mason, david, hch, akpm, jack,
yanmin_zhang, richard
* Jens Axboe <jens.axboe@oracle.com> [2009-05-26 23:11]:
> > OK, I spotted the problem. If we fallback to the on-stack allocation in
> > bdi_writeback_all(), then we do the wait for the work completion with
> > the bdi_lock mutex held. This can deadlock with bdi_forker_task(), so if
> > we require that to be invoked to make progress (happens if a thread
> > needs to be restarted), then we have a deadlock on that mutex.
> > I'll cook up a fix for this, but probably not before the morning.
> Untested fix. I think it should work, but I haven't run it here yet.
Thanks for your feedback and explanations. Will not be able to test the
patch before the afternoon. Will then give feedback as soon as I can. Of
course, I can also test a new fix or series if you send one in the
morning, after your own testing.
Best
--
Damien Wyart
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCH 0/12] Per-bdi writeback flusher threads v7
2009-05-26 9:33 [PATCH 0/12] Per-bdi writeback flusher threads v7 Jens Axboe
` (12 preceding siblings ...)
2009-05-26 15:25 ` [PATCH 0/12] Per-bdi writeback flusher threads v7 Damien Wyart
@ 2009-05-27 5:27 ` Zhang, Yanmin
2009-05-27 6:17 ` Jens Axboe
13 siblings, 1 reply; 33+ messages in thread
From: Zhang, Yanmin @ 2009-05-27 5:27 UTC (permalink / raw)
To: Jens Axboe
Cc: linux-kernel, linux-fsdevel, chris.mason, david, hch, akpm, jack,
richard
[-- Attachment #1: Type: text/plain, Size: 2263 bytes --]
On Tue, 2009-05-26 at 11:33 +0200, Jens Axboe wrote:
> Hi,
>
> Here's the 7th version of the writeback patches. Changes since
> v5/v6:
>
> - Move the sync_supers() to the global bdi_forker_task() thread, so we
> don't writeback the supers from all the bdi kupdated() tasks.
> - Make bdi_start_writeback() and bdi_writeback_all() be sync when called
> with WB_SYNC_ALL only.
> - Shuffle some more things around to make a cleaner series. The sync vs
> async nature of bdi_writeback_all() and bdi_start_writeback() isn't
> consistent through the series, but otherwise things should be sane.
>
> I'd appreciate if Richard and Yanmin could re-run testing with this,
> just to make sure that things are sane. For ease of patching, I've
> put the full diff here:
>
> http://kernel.dk/writeback-v7.patch
I ported it to 2.6.30-rc6 with some change in file mm/page-write.c, so I
could compare with old data.
See the attachment.
The new testing hits the hang issue again. It seems there is still a race.
INFO: task sync:30013 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
sync D ffffc20000011300 4736 30013 28019
ffffffff8093e350 0000000000000086 0000000000000000 0000000000000000
0000000000021220 0000000000004000 0000000000011300 000000000000c868
ffff880000016c48 ffffe20002934b30 ffff8800720b3780 ffff8800720b3b08
Call Trace:
[<ffffffff802c31d0>] ? bdi_sched_wait+0x0/0xd
[<ffffffff8071e56f>] ? schedule+0x9/0x1e
[<ffffffff802c31d9>] ? bdi_sched_wait+0x9/0xd
[<ffffffff8071eb36>] ? __wait_on_bit+0x41/0x71
[<ffffffff802c31d0>] ? bdi_sched_wait+0x0/0xd
[<ffffffff8071ebd1>] ? out_of_line_wait_on_bit+0x6b/0x77
[<ffffffff8024cc0c>] ? wake_bit_function+0x0/0x23
[<ffffffff8022cfa1>] ? __wake_up+0x30/0x44
[<ffffffff802c2e22>] ? bdi_writeback_all+0x20b/0x24c
[<ffffffff802800ce>] ? pagevec_lookup_tag+0x1a/0x21
[<ffffffff80279248>] ? wait_on_page_writeback_range+0xce/0x11b
[<ffffffff802c2ff3>] ? generic_sync_sb_inodes+0x36/0xe1
[<ffffffff802c3121>] ? sync_inodes_sb+0x83/0x88
[<ffffffff802c316c>] ? __sync_inodes+0x46/0x8f
[<ffffffff802c5d10>] ? do_sync+0x36/0x5a
[<ffffffff802c5d56>] ? sys_sync+0xe/0x14
[<ffffffff8020ba2b>] ? system_call_fastpath+0x16/0x1b
[-- Attachment #2: jens_bdi_fluser_v7_2.6.30-rc6.patch --]
[-- Type: text/x-patch, Size: 81632 bytes --]
diff -Nraup linux-2.6.30-rc6/block/blk-core.c linux-2.6.30-rc6_bdiflusherv7/block/blk-core.c
--- linux-2.6.30-rc6/block/blk-core.c 2009-05-19 11:00:45.000000000 +0800
+++ linux-2.6.30-rc6_bdiflusherv7/block/blk-core.c 2009-05-27 08:59:27.000000000 +0800
@@ -517,6 +517,7 @@ struct request_queue *blk_alloc_queue_no
q->backing_dev_info.unplug_io_fn = blk_backing_dev_unplug;
q->backing_dev_info.unplug_io_data = q;
+ q->backing_dev_info.name = "block";
err = bdi_init(&q->backing_dev_info);
if (err) {
kmem_cache_free(blk_requestq_cachep, q);
diff -Nraup linux-2.6.30-rc6/drivers/block/aoe/aoeblk.c linux-2.6.30-rc6_bdiflusherv7/drivers/block/aoe/aoeblk.c
--- linux-2.6.30-rc6/drivers/block/aoe/aoeblk.c 2009-05-19 11:00:27.000000000 +0800
+++ linux-2.6.30-rc6_bdiflusherv7/drivers/block/aoe/aoeblk.c 2009-05-27 08:59:27.000000000 +0800
@@ -265,6 +265,7 @@ aoeblk_gdalloc(void *vp)
}
blk_queue_make_request(&d->blkq, aoeblk_make_request);
+ d->blkq.backing_dev_info.name = "aoe";
if (bdi_init(&d->blkq.backing_dev_info))
goto err_mempool;
spin_lock_irqsave(&d->lock, flags);
diff -Nraup linux-2.6.30-rc6/drivers/char/mem.c linux-2.6.30-rc6_bdiflusherv7/drivers/char/mem.c
--- linux-2.6.30-rc6/drivers/char/mem.c 2009-05-19 11:00:46.000000000 +0800
+++ linux-2.6.30-rc6_bdiflusherv7/drivers/char/mem.c 2009-05-27 08:59:27.000000000 +0800
@@ -820,6 +820,7 @@ static const struct file_operations zero
* - permits private mappings, "copies" are taken of the source of zeros
*/
static struct backing_dev_info zero_bdi = {
+ .name = "char/mem",
.capabilities = BDI_CAP_MAP_COPY,
};
diff -Nraup linux-2.6.30-rc6/fs/btrfs/disk-io.c linux-2.6.30-rc6_bdiflusherv7/fs/btrfs/disk-io.c
--- linux-2.6.30-rc6/fs/btrfs/disk-io.c 2009-05-19 11:00:56.000000000 +0800
+++ linux-2.6.30-rc6_bdiflusherv7/fs/btrfs/disk-io.c 2009-05-27 08:59:27.000000000 +0800
@@ -1345,12 +1345,25 @@ static void btrfs_unplug_io_fn(struct ba
free_extent_map(em);
}
+/*
+ * If this fails, caller must call bdi_destroy() to get rid of the
+ * bdi again.
+ */
static int setup_bdi(struct btrfs_fs_info *info, struct backing_dev_info *bdi)
{
- bdi_init(bdi);
+ int err;
+
+ bdi->name = "btrfs";
+ bdi->capabilities = BDI_CAP_MAP_COPY;
+ err = bdi_init(bdi);
+ if (err)
+ return err;
+
+ err = bdi_register(bdi, NULL, "btrfs");
+ if (err)
+ return err;
+
bdi->ra_pages = default_backing_dev_info.ra_pages;
- bdi->state = 0;
- bdi->capabilities = default_backing_dev_info.capabilities;
bdi->unplug_io_fn = btrfs_unplug_io_fn;
bdi->unplug_io_data = info;
bdi->congested_fn = btrfs_congested_fn;
@@ -1574,7 +1587,8 @@ struct btrfs_root *open_ctree(struct sup
fs_info->sb = sb;
fs_info->max_extent = (u64)-1;
fs_info->max_inline = 8192 * 1024;
- setup_bdi(fs_info, &fs_info->bdi);
+ if (setup_bdi(fs_info, &fs_info->bdi))
+ goto fail_bdi;
fs_info->btree_inode = new_inode(sb);
fs_info->btree_inode->i_ino = 1;
fs_info->btree_inode->i_nlink = 1;
@@ -1931,8 +1945,8 @@ fail_iput:
btrfs_close_devices(fs_info->fs_devices);
btrfs_mapping_tree_free(&fs_info->mapping_tree);
+fail_bdi:
bdi_destroy(&fs_info->bdi);
-
fail:
kfree(extent_root);
kfree(tree_root);
diff -Nraup linux-2.6.30-rc6/fs/buffer.c linux-2.6.30-rc6_bdiflusherv7/fs/buffer.c
--- linux-2.6.30-rc6/fs/buffer.c 2009-05-19 11:00:56.000000000 +0800
+++ linux-2.6.30-rc6_bdiflusherv7/fs/buffer.c 2009-05-27 08:59:27.000000000 +0800
@@ -281,7 +281,7 @@ static void free_more_memory(void)
struct zone *zone;
int nid;
- wakeup_pdflush(1024);
+ wakeup_flusher_threads(1024);
yield();
for_each_online_node(nid) {
diff -Nraup linux-2.6.30-rc6/fs/char_dev.c linux-2.6.30-rc6_bdiflusherv7/fs/char_dev.c
--- linux-2.6.30-rc6/fs/char_dev.c 2009-05-19 11:00:27.000000000 +0800
+++ linux-2.6.30-rc6_bdiflusherv7/fs/char_dev.c 2009-05-27 08:59:27.000000000 +0800
@@ -32,6 +32,7 @@
* - no readahead or I/O queue unplugging required
*/
struct backing_dev_info directly_mappable_cdev_bdi = {
+ .name = "char",
.capabilities = (
#ifdef CONFIG_MMU
/* permit private copies of the data to be taken */
diff -Nraup linux-2.6.30-rc6/fs/configfs/inode.c linux-2.6.30-rc6_bdiflusherv7/fs/configfs/inode.c
--- linux-2.6.30-rc6/fs/configfs/inode.c 2009-05-19 11:00:27.000000000 +0800
+++ linux-2.6.30-rc6_bdiflusherv7/fs/configfs/inode.c 2009-05-27 08:59:27.000000000 +0800
@@ -46,6 +46,7 @@ static const struct address_space_operat
};
static struct backing_dev_info configfs_backing_dev_info = {
+ .name = "configfs",
.ra_pages = 0, /* No readahead */
.capabilities = BDI_CAP_NO_ACCT_AND_WRITEBACK,
};
diff -Nraup linux-2.6.30-rc6/fs/fs-writeback.c linux-2.6.30-rc6_bdiflusherv7/fs/fs-writeback.c
--- linux-2.6.30-rc6/fs/fs-writeback.c 2009-05-19 11:00:57.000000000 +0800
+++ linux-2.6.30-rc6_bdiflusherv7/fs/fs-writeback.c 2009-05-27 08:59:27.000000000 +0800
@@ -19,49 +19,563 @@
#include <linux/sched.h>
#include <linux/fs.h>
#include <linux/mm.h>
+#include <linux/kthread.h>
+#include <linux/freezer.h>
#include <linux/writeback.h>
#include <linux/blkdev.h>
#include <linux/backing-dev.h>
#include <linux/buffer_head.h>
#include "internal.h"
+#define inode_to_bdi(inode) ((inode)->i_mapping->backing_dev_info)
-/**
- * writeback_acquire - attempt to get exclusive writeback access to a device
- * @bdi: the device's backing_dev_info structure
- *
- * It is a waste of resources to have more than one pdflush thread blocked on
- * a single request queue. Exclusion at the request_queue level is obtained
- * via a flag in the request_queue's backing_dev_info.state.
- *
- * Non-request_queue-backed address_spaces will share default_backing_dev_info,
- * unless they implement their own. Which is somewhat inefficient, as this
- * may prevent concurrent writeback against multiple devices.
+/*
+ * We don't actually have pdflush, but this one is exported though /proc...
+ */
+int nr_pdflush_threads;
+
+static void generic_sync_wb_inodes(struct bdi_writeback *wb,
+ struct super_block *sb,
+ struct writeback_control *wbc);
+
+/*
+ * Work items for the bdi_writeback threads
*/
-static int writeback_acquire(struct backing_dev_info *bdi)
+struct bdi_work {
+ struct list_head list;
+ struct list_head wait_list;
+ struct rcu_head rcu_head;
+
+ unsigned long seen;
+ atomic_t pending;
+
+ unsigned long sb_data;
+ unsigned long nr_pages;
+ enum writeback_sync_modes sync_mode;
+
+ unsigned long state;
+};
+
+static struct super_block *bdi_work_sb(struct bdi_work *work)
+{
+ return (struct super_block *) (work->sb_data & ~1UL);
+}
+
+static inline bool bdi_work_on_stack(struct bdi_work *work)
+{
+ return work->sb_data & 1UL;
+}
+
+static inline void bdi_work_init(struct bdi_work *work, struct super_block *sb,
+ unsigned long nr_pages,
+ enum writeback_sync_modes sync_mode)
+{
+ INIT_RCU_HEAD(&work->rcu_head);
+ work->sb_data = (unsigned long) sb;
+ work->nr_pages = nr_pages;
+ work->sync_mode = sync_mode;
+ work->state = 1;
+
+ /*
+ * state must not be reordered around the insert
+ */
+ smp_mb();
+}
+
+static inline void bdi_work_init_on_stack(struct bdi_work *work,
+ struct super_block *sb,
+ unsigned long nr_pages,
+ enum writeback_sync_modes sync_mode)
{
- return !test_and_set_bit(BDI_pdflush, &bdi->state);
+ bdi_work_init(work, sb, nr_pages, sync_mode);
+ work->sb_data |= 1UL;
}
/**
* writeback_in_progress - determine whether there is writeback in progress
* @bdi: the device's backing_dev_info structure.
*
- * Determine whether there is writeback in progress against a backing device.
+ * Determine whether there is writeback waiting to be handled against a
+ * backing device.
*/
int writeback_in_progress(struct backing_dev_info *bdi)
{
- return test_bit(BDI_pdflush, &bdi->state);
+ return !list_empty(&bdi->work_list);
}
-/**
- * writeback_release - relinquish exclusive writeback access against a device.
- * @bdi: the device's backing_dev_info structure
+static void bdi_work_clear(struct bdi_work *work)
+{
+ clear_bit(0, &work->state);
+ smp_mb__after_clear_bit();
+ wake_up_bit(&work->state, 0);
+}
+
+static void bdi_work_free(struct rcu_head *head)
+{
+ struct bdi_work *work = container_of(head, struct bdi_work, rcu_head);
+
+ if (!bdi_work_on_stack(work))
+ kfree(work);
+ else
+ bdi_work_clear(work);
+}
+
+static void wb_work_complete(struct bdi_work *work)
+{
+ if (!bdi_work_on_stack(work)) {
+ bdi_work_clear(work);
+
+ if (work->sync_mode == WB_SYNC_NONE)
+ call_rcu(&work->rcu_head, bdi_work_free);
+ } else
+ call_rcu(&work->rcu_head, bdi_work_free);
+}
+
+static void wb_clear_pending(struct bdi_writeback *wb, struct bdi_work *work)
+{
+ /*
+ * The caller has retrieved the work arguments from this work,
+ * drop our reference. If this is the last ref, delete and free it
+ */
+ if (atomic_dec_and_test(&work->pending)) {
+ struct backing_dev_info *bdi = wb->bdi;
+
+ spin_lock(&bdi->wb_lock);
+ list_del_rcu(&work->list);
+ spin_unlock(&bdi->wb_lock);
+
+ wb_work_complete(work);
+ }
+}
+
+static void wb_start_writeback(struct bdi_writeback *wb, struct bdi_work *work)
+{
+ /*
+ * If we failed allocating the bdi work item, wake up the wb thread
+ * always. As a safety precaution, it'll flush out everything
+ */
+ if (!wb_has_dirty_io(wb) && work)
+ wb_clear_pending(wb, work);
+ else
+ wake_up(&wb->wait);
+}
+
+static void bdi_queue_work(struct backing_dev_info *bdi, struct bdi_work *work)
+{
+ if (work) {
+ work->seen = bdi->wb_mask;
+ atomic_set(&work->pending, bdi->wb_cnt);
+
+ /*
+ * Make sure stores are seen before it appears on the list
+ */
+ smp_mb();
+
+ spin_lock(&bdi->wb_lock);
+ list_add_tail_rcu(&work->list, &bdi->work_list);
+ spin_unlock(&bdi->wb_lock);
+ }
+}
+
+static void bdi_sched_work(struct backing_dev_info *bdi, struct bdi_work *work)
+{
+ if (!bdi_wblist_needs_lock(bdi))
+ wb_start_writeback(&bdi->wb, work);
+ else {
+ struct bdi_writeback *wb;
+ int idx;
+
+ idx = srcu_read_lock(&bdi->srcu);
+
+ list_for_each_entry_rcu(wb, &bdi->wb_list, list)
+ wb_start_writeback(wb, work);
+
+ srcu_read_unlock(&bdi->srcu, idx);
+ }
+}
+
+static void __bdi_start_work(struct backing_dev_info *bdi,
+ struct bdi_work *work)
+{
+ /*
+ * If the default thread isn't there, make sure we add it. When
+ * it gets created and wakes up, we'll run this work.
+ */
+ if (unlikely(list_empty_careful(&bdi->wb_list)))
+ bdi_add_default_flusher_task(bdi);
+ else
+ bdi_sched_work(bdi, work);
+}
+
+static void bdi_start_work(struct backing_dev_info *bdi, struct bdi_work *work)
+{
+ /*
+ * If the default thread isn't there, make sure we add it. When
+ * it gets created and wakes up, we'll run this work.
+ */
+ if (unlikely(list_empty_careful(&bdi->wb_list))) {
+ mutex_lock(&bdi_lock);
+ bdi_add_default_flusher_task(bdi);
+ mutex_unlock(&bdi_lock);
+ } else
+ bdi_sched_work(bdi, work);
+}
+
+/*
+ * Used for on-stack allocated work items. The caller needs to wait until
+ * the wb threads have acked the work before it's safe to continue.
+ */
+static void bdi_wait_on_work_clear(struct bdi_work *work)
+{
+ wait_on_bit(&work->state, 0, bdi_sched_wait, TASK_UNINTERRUPTIBLE);
+}
+
+static struct bdi_work *bdi_alloc_work(struct super_block *sb, long nr_pages,
+ enum writeback_sync_modes sync_mode)
+{
+ struct bdi_work *work;
+
+ work = kmalloc(sizeof(*work), GFP_ATOMIC);
+ if (work)
+ bdi_work_init(work, sb, nr_pages, sync_mode);
+
+ return work;
+}
+
+void bdi_start_writeback(struct backing_dev_info *bdi, struct super_block *sb,
+ long nr_pages, enum writeback_sync_modes sync_mode)
+{
+ const bool must_wait = sync_mode == WB_SYNC_ALL;
+ struct bdi_work work_stack, *work = NULL;
+
+ if (!must_wait)
+ work = bdi_alloc_work(sb, nr_pages, sync_mode);
+
+ if (!work) {
+ work = &work_stack;
+ bdi_work_init_on_stack(work, sb, nr_pages, sync_mode);
+ }
+
+ bdi_queue_work(bdi, work);
+ bdi_start_work(bdi, work);
+
+ /*
+ * If the sync mode is WB_SYNC_ALL, block waiting for the work to
+ * complete. If not, we only need to wait for the work to be started,
+ * if we allocated it on-stack. We use the same mechanism, if the
+ * wait bit is set in the bdi_work struct, then threads will not
+ * clear pending until after they are done.
+ *
+ * Note that work == &work_stack if must_wait is true, but that
+ * is implementation detail and we make it explicit here for
+ * ease of reading.
+ */
+ if (work == &work_stack || must_wait) {
+ bdi_wait_on_work_clear(work);
+ if (must_wait)
+ call_rcu(&work->rcu_head, bdi_work_free);
+ }
+}
+
+/*
+ * The maximum number of pages to writeout in a single bdi flush/kupdate
+ * operation. We do this so we don't hold I_SYNC against an inode for
+ * enormous amounts of time, which would block a userspace task which has
+ * been forced to throttle against that inode. Also, the code reevaluates
+ * the dirty each time it has written this many pages.
+ */
+#define MAX_WRITEBACK_PAGES 1024
+
+/*
+ * Periodic writeback of "old" data.
+ *
+ * Define "old": the first time one of an inode's pages is dirtied, we mark the
+ * dirtying-time in the inode's address_space. So this periodic writeback code
+ * just walks the superblock inode list, writing back any inodes which are
+ * older than a specific point in time.
+ *
+ * Try to run once per dirty_writeback_interval. But if a writeback event
+ * takes longer than a dirty_writeback_interval interval, then leave a
+ * one-second gap.
+ *
+ * older_than_this takes precedence over nr_to_write. So we'll only write back
+ * all dirty pages if they are all attached to "old" mappings.
+ */
+static long wb_kupdated(struct bdi_writeback *wb)
+{
+ unsigned long oldest_jif;
+ long nr_to_write, wrote = 0;
+ struct writeback_control wbc = {
+ .bdi = wb->bdi,
+ .sync_mode = WB_SYNC_NONE,
+ .older_than_this = &oldest_jif,
+ .nr_to_write = 0,
+ .for_kupdate = 1,
+ .range_cyclic = 1,
+ };
+
+ oldest_jif = jiffies - msecs_to_jiffies(dirty_expire_interval * 10);
+
+ nr_to_write = global_page_state(NR_FILE_DIRTY) +
+ global_page_state(NR_UNSTABLE_NFS) +
+ (inodes_stat.nr_inodes - inodes_stat.nr_unused);
+
+ while (nr_to_write > 0) {
+ wbc.more_io = 0;
+ wbc.encountered_congestion = 0;
+ wbc.nr_to_write = MAX_WRITEBACK_PAGES;
+ generic_sync_wb_inodes(wb, NULL, &wbc);
+ wrote += MAX_WRITEBACK_PAGES - wbc.nr_to_write;
+ if (wbc.nr_to_write > 0)
+ break; /* All the old data is written */
+ nr_to_write -= MAX_WRITEBACK_PAGES;
+ }
+
+ return wrote;
+}
+
+static inline bool over_bground_thresh(void)
+{
+ unsigned long background_thresh, dirty_thresh;
+
+ get_dirty_limits(&background_thresh, &dirty_thresh, NULL, NULL);
+
+ return (global_page_state(NR_FILE_DIRTY) +
+ global_page_state(NR_UNSTABLE_NFS) >= background_thresh);
+}
+
+static long __wb_writeback(struct bdi_writeback *wb, long nr_pages,
+ struct super_block *sb,
+ enum writeback_sync_modes sync_mode)
+{
+ struct writeback_control wbc = {
+ .bdi = wb->bdi,
+ .sync_mode = sync_mode,
+ .older_than_this = NULL,
+ .range_cyclic = 1,
+ };
+ long wrote = 0;
+
+ for (;;) {
+ if (sync_mode == WB_SYNC_NONE && nr_pages <= 0 &&
+ !over_bground_thresh())
+ break;
+
+ wbc.more_io = 0;
+ wbc.encountered_congestion = 0;
+ wbc.nr_to_write = MAX_WRITEBACK_PAGES;
+ wbc.pages_skipped = 0;
+ generic_sync_wb_inodes(wb, sb, &wbc);
+ nr_pages -= MAX_WRITEBACK_PAGES - wbc.nr_to_write;
+ wrote += MAX_WRITEBACK_PAGES - wbc.nr_to_write;
+ /*
+ * If we ran out of stuff to write, bail unless more_io got set
+ */
+ if (wbc.nr_to_write > 0 || wbc.pages_skipped > 0) {
+ if (wbc.more_io)
+ continue;
+ break;
+ }
+ }
+
+ return wrote;
+}
+
+/*
+ * Return the next bdi_work struct that hasn't been processed by this
+ * wb thread yet
+ */
+static struct bdi_work *get_next_work_item(struct backing_dev_info *bdi,
+ struct bdi_writeback *wb)
+{
+ struct bdi_work *work, *ret = NULL;
+
+ rcu_read_lock();
+
+ list_for_each_entry_rcu(work, &bdi->work_list, list) {
+ if (!test_and_clear_bit(wb->nr, &work->seen))
+ continue;
+
+ ret = work;
+ break;
+ }
+
+ rcu_read_unlock();
+ return ret;
+}
+
+/*
+ * Retrieve work items and do the writeback they describe
+ */
+static long wb_writeback(struct bdi_writeback *wb)
+{
+ struct backing_dev_info *bdi = wb->bdi;
+ struct bdi_work *work;
+ long wrote = 0;
+
+ while ((work = get_next_work_item(bdi, wb)) != NULL) {
+ struct super_block *sb = bdi_work_sb(work);
+ long nr_pages = work->nr_pages;
+ enum writeback_sync_modes sync_mode = work->sync_mode;
+
+ /*
+ * If this isn't a data integrity operation, just notify
+ * that we have seen this work and we are now starting it.
+ */
+ if (sync_mode == WB_SYNC_NONE)
+ wb_clear_pending(wb, work);
+
+ wrote += __wb_writeback(wb, nr_pages, sb, sync_mode);
+
+ /*
+ * This is a data integrity writeback, so only do the
+ * notification when we have completed the work.
+ */
+ if (sync_mode == WB_SYNC_ALL)
+ wb_clear_pending(wb, work);
+ }
+
+ return wrote;
+}
+
+/*
+ * This will be inlined in bdi_writeback_task() once we get rid of any
+ * dirty inodes on the default_backing_dev_info
+ */
+long wb_do_writeback(struct bdi_writeback *wb)
+{
+ long wrote;
+
+ /*
+ * We get here in two cases:
+ *
+ * schedule_timeout() returned because the dirty writeback
+ * interval has elapsed. If that happens, the work item list
+ * will be empty and we will proceed to do kupdated style writeout.
+ *
+ * Someone called bdi_start_writeback(), which put one/more work
+ * items on the work_list. Process those.
+ */
+ if (list_empty(&wb->bdi->work_list))
+ wrote = wb_kupdated(wb);
+ else
+ wrote = wb_writeback(wb);
+
+ return wrote;
+}
+
+/*
+ * Handle writeback of dirty data for the device backed by this bdi. Also
+ * wakes up periodically and does kupdated style flushing.
*/
-static void writeback_release(struct backing_dev_info *bdi)
+int bdi_writeback_task(struct bdi_writeback *wb)
{
- BUG_ON(!writeback_in_progress(bdi));
- clear_bit(BDI_pdflush, &bdi->state);
+ unsigned long last_active = jiffies;
+ unsigned long wait_jiffies = -1UL;
+ long pages_written;
+ DEFINE_WAIT(wait);
+
+ while (!kthread_should_stop()) {
+
+ pages_written = wb_do_writeback(wb);
+
+ if (pages_written)
+ last_active = jiffies;
+ else if (wait_jiffies != -1UL) {
+ unsigned long max_idle;
+
+ /*
+ * Longest period of inactivity that we tolerate. If we
+ * see dirty data again later, the task will get
+ * recreated automatically.
+ */
+ max_idle = max(5UL * 60 * HZ, wait_jiffies);
+ if (time_after(jiffies, max_idle + last_active) &&
+ wb_is_default_task(wb))
+ break;
+ }
+
+ prepare_to_wait(&wb->wait, &wait, TASK_INTERRUPTIBLE);
+ wait_jiffies = msecs_to_jiffies(dirty_writeback_interval * 10);
+ schedule_timeout(wait_jiffies);
+ try_to_freeze();
+ }
+
+ finish_wait(&wb->wait, &wait);
+ return 0;
+}
+
+/*
+ * Schedule writeback for all backing devices. Expensive! If this is a data
+ * integrity operation, writeback will be complete when this returns. If
+ * we are simply called for WB_SYNC_NONE, then writeback will merely be
+ * scheduled to run.
+ */
+void bdi_writeback_all(struct super_block *sb, long nr_pages,
+ enum writeback_sync_modes sync_mode)
+{
+ const bool must_wait = sync_mode == WB_SYNC_ALL;
+ struct backing_dev_info *bdi, *tmp;
+ struct bdi_work *work;
+ LIST_HEAD(list);
+
+ mutex_lock(&bdi_lock);
+
+ list_for_each_entry_safe(bdi, tmp, &bdi_list, bdi_list) {
+ struct bdi_work *work, work_stack;
+
+ if (!bdi_has_dirty_io(bdi))
+ continue;
+
+ work = bdi_alloc_work(sb, nr_pages, sync_mode);
+ if (!work) {
+ work = &work_stack;
+ bdi_work_init_on_stack(work, sb, nr_pages, sync_mode);
+ } else if (must_wait)
+ list_add_tail(&work->wait_list, &list);
+
+ bdi_queue_work(bdi, work);
+ __bdi_start_work(bdi, work);
+
+ /*
+ * Do the wait inline if this came from the stack. This
+ * only happens if we ran out of memory, so should very
+ * rarely trigger.
+ */
+ if (work == &work_stack) {
+ bdi_wait_on_work_clear(work);
+ if (must_wait)
+ call_rcu(&work->rcu_head, bdi_work_free);
+ }
+ }
+
+ mutex_unlock(&bdi_lock);
+
+ /*
+ * If this is for WB_SYNC_ALL, wait for pending work to complete
+ * before returning.
+ */
+ while (!list_empty(&list)) {
+ work = list_entry(list.next, struct bdi_work, wait_list);
+ list_del(&work->wait_list);
+ bdi_wait_on_work_clear(work);
+ call_rcu(&work->rcu_head, bdi_work_free);
+ }
+}
+
+/*
+ * If the filesystem didn't provide a way to map an inode to a dedicated
+ * flusher thread, it doesn't support more than 1 thread. So we know it's
+ * the default thread, return that.
+ */
+static inline struct bdi_writeback *inode_get_wb(struct inode *inode)
+{
+ const struct super_operations *sop = inode->i_sb->s_op;
+
+ if (!sop->inode_get_wb)
+ return &inode_to_bdi(inode)->wb;
+
+ return sop->inode_get_wb(inode);
}
/**
@@ -158,12 +672,21 @@ void __mark_inode_dirty(struct inode *in
goto out;
/*
- * If the inode was already on s_dirty/s_io/s_more_io, don't
- * reposition it (that would break s_dirty time-ordering).
+ * If the inode was already on b_dirty/b_io/b_more_io, don't
+ * reposition it (that would break b_dirty time-ordering).
*/
if (!was_dirty) {
+ struct bdi_writeback *wb = inode_get_wb(inode);
+ struct backing_dev_info *bdi = wb->bdi;
+
+ if (bdi_cap_writeback_dirty(bdi) &&
+ !test_bit(BDI_registered, &bdi->state)) {
+ WARN_ON(1);
+ printk("bdi-%s not registered\n", bdi->name);
+ }
+
inode->dirtied_when = jiffies;
- list_move(&inode->i_list, &sb->s_dirty);
+ list_move(&inode->i_list, &wb->b_dirty);
}
}
out:
@@ -184,31 +707,32 @@ static int write_inode(struct inode *ino
* furthest end of its superblock's dirty-inode list.
*
* Before stamping the inode's ->dirtied_when, we check to see whether it is
- * already the most-recently-dirtied inode on the s_dirty list. If that is
+ * already the most-recently-dirtied inode on the b_dirty list. If that is
* the case then the inode must have been redirtied while it was being written
* out and we don't reset its dirtied_when.
*/
static void redirty_tail(struct inode *inode)
{
- struct super_block *sb = inode->i_sb;
+ struct bdi_writeback *wb = inode_get_wb(inode);
- if (!list_empty(&sb->s_dirty)) {
- struct inode *tail_inode;
+ if (!list_empty(&wb->b_dirty)) {
+ struct inode *tail;
- tail_inode = list_entry(sb->s_dirty.next, struct inode, i_list);
- if (time_before(inode->dirtied_when,
- tail_inode->dirtied_when))
+ tail = list_entry(wb->b_dirty.next, struct inode, i_list);
+ if (time_before(inode->dirtied_when, tail->dirtied_when))
inode->dirtied_when = jiffies;
}
- list_move(&inode->i_list, &sb->s_dirty);
+ list_move(&inode->i_list, &wb->b_dirty);
}
/*
- * requeue inode for re-scanning after sb->s_io list is exhausted.
+ * requeue inode for re-scanning after bdi->b_io list is exhausted.
*/
static void requeue_io(struct inode *inode)
{
- list_move(&inode->i_list, &inode->i_sb->s_more_io);
+ struct bdi_writeback *wb = inode_get_wb(inode);
+
+ list_move(&inode->i_list, &wb->b_more_io);
}
static void inode_sync_complete(struct inode *inode)
@@ -255,20 +779,11 @@ static void move_expired_inodes(struct l
/*
* Queue all expired dirty inodes for io, eldest first.
*/
-static void queue_io(struct super_block *sb,
- unsigned long *older_than_this)
-{
- list_splice_init(&sb->s_more_io, sb->s_io.prev);
- move_expired_inodes(&sb->s_dirty, &sb->s_io, older_than_this);
-}
-
-int sb_has_dirty_inodes(struct super_block *sb)
+static void queue_io(struct bdi_writeback *wb, unsigned long *older_than_this)
{
- return !list_empty(&sb->s_dirty) ||
- !list_empty(&sb->s_io) ||
- !list_empty(&sb->s_more_io);
+ list_splice_init(&wb->b_more_io, wb->b_io.prev);
+ move_expired_inodes(&wb->b_dirty, &wb->b_io, older_than_this);
}
-EXPORT_SYMBOL(sb_has_dirty_inodes);
/*
* Write a single inode's dirty pages and inode data out to disk.
@@ -322,11 +837,11 @@ __sync_single_inode(struct inode *inode,
/*
* We didn't write back all the pages. nfs_writepages()
* sometimes bales out without doing anything. Redirty
- * the inode; Move it from s_io onto s_more_io/s_dirty.
+ * the inode; Move it from b_io onto b_more_io/b_dirty.
*/
/*
* akpm: if the caller was the kupdate function we put
- * this inode at the head of s_dirty so it gets first
+ * this inode at the head of b_dirty so it gets first
* consideration. Otherwise, move it to the tail, for
* the reasons described there. I'm not really sure
* how much sense this makes. Presumably I had a good
@@ -336,7 +851,7 @@ __sync_single_inode(struct inode *inode,
if (wbc->for_kupdate) {
/*
* For the kupdate function we move the inode
- * to s_more_io so it will get more writeout as
+ * to b_more_io so it will get more writeout as
* soon as the queue becomes uncongested.
*/
inode->i_state |= I_DIRTY_PAGES;
@@ -402,10 +917,10 @@ __writeback_single_inode(struct inode *i
if ((wbc->sync_mode != WB_SYNC_ALL) && (inode->i_state & I_SYNC)) {
/*
* We're skipping this inode because it's locked, and we're not
- * doing writeback-for-data-integrity. Move it to s_more_io so
- * that writeback can proceed with the other inodes on s_io.
+ * doing writeback-for-data-integrity. Move it to b_more_io so
+ * that writeback can proceed with the other inodes on b_io.
* We'll have another go at writing back this inode when we
- * completed a full scan of s_io.
+ * completed a full scan of b_io.
*/
requeue_io(inode);
return 0;
@@ -428,51 +943,34 @@ __writeback_single_inode(struct inode *i
return __sync_single_inode(inode, wbc);
}
-/*
- * Write out a superblock's list of dirty inodes. A wait will be performed
- * upon no inodes, all inodes or the final one, depending upon sync_mode.
- *
- * If older_than_this is non-NULL, then only write out inodes which
- * had their first dirtying at a time earlier than *older_than_this.
- *
- * If we're a pdflush thread, then implement pdflush collision avoidance
- * against the entire list.
- *
- * If `bdi' is non-zero then we're being asked to writeback a specific queue.
- * This function assumes that the blockdev superblock's inodes are backed by
- * a variety of queues, so all inodes are searched. For other superblocks,
- * assume that all inodes are backed by the same queue.
- *
- * FIXME: this linear search could get expensive with many fileystems. But
- * how to fix? We need to go from an address_space to all inodes which share
- * a queue with that address_space. (Easy: have a global "dirty superblocks"
- * list).
- *
- * The inodes to be written are parked on sb->s_io. They are moved back onto
- * sb->s_dirty as they are selected for writing. This way, none can be missed
- * on the writer throttling path, and we get decent balancing between many
- * throttled threads: we don't want them all piling up on inode_sync_wait.
- */
-void generic_sync_sb_inodes(struct super_block *sb,
- struct writeback_control *wbc)
+static void generic_sync_wb_inodes(struct bdi_writeback *wb,
+ struct super_block *sb,
+ struct writeback_control *wbc)
{
+ const int is_blkdev_sb = sb_is_blkdev_sb(sb);
const unsigned long start = jiffies; /* livelock avoidance */
- int sync = wbc->sync_mode == WB_SYNC_ALL;
spin_lock(&inode_lock);
- if (!wbc->for_kupdate || list_empty(&sb->s_io))
- queue_io(sb, wbc->older_than_this);
- while (!list_empty(&sb->s_io)) {
- struct inode *inode = list_entry(sb->s_io.prev,
+ if (!wbc->for_kupdate || list_empty(&wb->b_io))
+ queue_io(wb, wbc->older_than_this);
+
+ while (!list_empty(&wb->b_io)) {
+ struct inode *inode = list_entry(wb->b_io.prev,
struct inode, i_list);
- struct address_space *mapping = inode->i_mapping;
- struct backing_dev_info *bdi = mapping->backing_dev_info;
long pages_skipped;
- if (!bdi_cap_writeback_dirty(bdi)) {
+ /*
+ * super block given and doesn't match, skip this inode
+ */
+ if (sb && sb != inode->i_sb) {
+ redirty_tail(inode);
+ continue;
+ }
+
+ if (!bdi_cap_writeback_dirty(wb->bdi)) {
redirty_tail(inode);
- if (sb_is_blkdev_sb(sb)) {
+ if (is_blkdev_sb) {
/*
* Dirty memory-backed blockdev: the ramdisk
* driver does this. Skip just this inode
@@ -492,21 +990,14 @@ void generic_sync_sb_inodes(struct super
continue;
}
- if (wbc->nonblocking && bdi_write_congested(bdi)) {
+ if (wbc->nonblocking && bdi_write_congested(wb->bdi)) {
wbc->encountered_congestion = 1;
- if (!sb_is_blkdev_sb(sb))
+ if (!is_blkdev_sb)
break; /* Skip a congested fs */
requeue_io(inode);
continue; /* Skip a congested blockdev */
}
- if (wbc->bdi && bdi != wbc->bdi) {
- if (!sb_is_blkdev_sb(sb))
- break; /* fs has the wrong queue */
- requeue_io(inode);
- continue; /* blockdev has wrong queue */
- }
-
/*
* Was this inode dirtied after sync_sb_inodes was called?
* This keeps sync from extra jobs and livelock.
@@ -514,16 +1005,10 @@ void generic_sync_sb_inodes(struct super
if (inode_dirtied_after(inode, start))
break;
- /* Is another pdflush already flushing this queue? */
- if (current_is_pdflush() && !writeback_acquire(bdi))
- break;
-
BUG_ON(inode->i_state & I_FREEING);
__iget(inode);
pages_skipped = wbc->pages_skipped;
__writeback_single_inode(inode, wbc);
- if (current_is_pdflush())
- writeback_release(bdi);
if (wbc->pages_skipped != pages_skipped) {
/*
* writeback is not making progress due to locked
@@ -539,13 +1024,71 @@ void generic_sync_sb_inodes(struct super
wbc->more_io = 1;
break;
}
- if (!list_empty(&sb->s_more_io))
+ if (!list_empty(&wb->b_more_io))
wbc->more_io = 1;
}
- if (sync) {
+ spin_unlock(&inode_lock);
+ /* Leave any unwritten inodes on b_io */
+}
+
+void generic_sync_bdi_inodes(struct super_block *sb,
+ struct writeback_control *wbc)
+{
+ struct backing_dev_info *bdi = wbc->bdi;
+ struct bdi_writeback *wb;
+
+ /*
+ * Common case is just a single wb thread and that is embedded in
+ * the bdi, so it doesn't need locking
+ */
+ if (!bdi_wblist_needs_lock(bdi))
+ generic_sync_wb_inodes(&bdi->wb, sb, wbc);
+ else {
+ int idx;
+
+ idx = srcu_read_lock(&bdi->srcu);
+
+ list_for_each_entry_rcu(wb, &bdi->wb_list, list)
+ generic_sync_wb_inodes(wb, sb, wbc);
+
+ srcu_read_unlock(&bdi->srcu, idx);
+ }
+}
+
+/*
+ * Write out a superblock's list of dirty inodes. A wait will be performed
+ * upon no inodes, all inodes or the final one, depending upon sync_mode.
+ *
+ * If older_than_this is non-NULL, then only write out inodes which
+ * had their first dirtying at a time earlier than *older_than_this.
+ *
+ * If we're a pdlfush thread, then implement pdflush collision avoidance
+ * against the entire list.
+ *
+ * If `bdi' is non-zero then we're being asked to writeback a specific queue.
+ * This function assumes that the blockdev superblock's inodes are backed by
+ * a variety of queues, so all inodes are searched. For other superblocks,
+ * assume that all inodes are backed by the same queue.
+ *
+ * The inodes to be written are parked on bdi->b_io. They are moved back onto
+ * bdi->b_dirty as they are selected for writing. This way, none can be missed
+ * on the writer throttling path, and we get decent balancing between many
+ * throttled threads: we don't want them all piling up on inode_sync_wait.
+ */
+void generic_sync_sb_inodes(struct super_block *sb,
+ struct writeback_control *wbc)
+{
+ if (wbc->bdi)
+ bdi_start_writeback(wbc->bdi, sb, wbc->nr_to_write, wbc->sync_mode);
+ else
+ bdi_writeback_all(sb, wbc->nr_to_write, wbc->sync_mode);
+
+ if (wbc->sync_mode == WB_SYNC_ALL) {
struct inode *inode, *old_inode = NULL;
+ spin_lock(&inode_lock);
+
/*
* Data integrity sync. Must wait for all pages under writeback,
* because there may have been pages dirtied before our sync
@@ -583,10 +1126,8 @@ void generic_sync_sb_inodes(struct super
}
spin_unlock(&inode_lock);
iput(old_inode);
- } else
- spin_unlock(&inode_lock);
+ }
- return; /* Leave any unwritten inodes on s_io */
}
EXPORT_SYMBOL_GPL(generic_sync_sb_inodes);
@@ -597,58 +1138,6 @@ static void sync_sb_inodes(struct super_
}
/*
- * Start writeback of dirty pagecache data against all unlocked inodes.
- *
- * Note:
- * We don't need to grab a reference to superblock here. If it has non-empty
- * ->s_dirty it's hadn't been killed yet and kill_super() won't proceed
- * past sync_inodes_sb() until the ->s_dirty/s_io/s_more_io lists are all
- * empty. Since __sync_single_inode() regains inode_lock before it finally moves
- * inode from superblock lists we are OK.
- *
- * If `older_than_this' is non-zero then only flush inodes which have a
- * flushtime older than *older_than_this.
- *
- * If `bdi' is non-zero then we will scan the first inode against each
- * superblock until we find the matching ones. One group will be the dirty
- * inodes against a filesystem. Then when we hit the dummy blockdev superblock,
- * sync_sb_inodes will seekout the blockdev which matches `bdi'. Maybe not
- * super-efficient but we're about to do a ton of I/O...
- */
-void
-writeback_inodes(struct writeback_control *wbc)
-{
- struct super_block *sb;
-
- might_sleep();
- spin_lock(&sb_lock);
-restart:
- list_for_each_entry_reverse(sb, &super_blocks, s_list) {
- if (sb_has_dirty_inodes(sb)) {
- /* we're making our own get_super here */
- sb->s_count++;
- spin_unlock(&sb_lock);
- /*
- * If we can't get the readlock, there's no sense in
- * waiting around, most of the time the FS is going to
- * be unmounted by the time it is released.
- */
- if (down_read_trylock(&sb->s_umount)) {
- if (sb->s_root)
- sync_sb_inodes(sb, wbc);
- up_read(&sb->s_umount);
- }
- spin_lock(&sb_lock);
- if (__put_super_and_need_restart(sb))
- goto restart;
- }
- if (wbc->nr_to_write <= 0)
- break;
- }
- spin_unlock(&sb_lock);
-}
-
-/*
* writeback and wait upon the filesystem's dirty inodes. The caller will
* do this in two passes - one to write, and one to wait.
*
diff -Nraup linux-2.6.30-rc6/fs/fuse/inode.c linux-2.6.30-rc6_bdiflusherv7/fs/fuse/inode.c
--- linux-2.6.30-rc6/fs/fuse/inode.c 2009-05-19 11:00:57.000000000 +0800
+++ linux-2.6.30-rc6_bdiflusherv7/fs/fuse/inode.c 2009-05-27 08:59:27.000000000 +0800
@@ -484,6 +484,7 @@ int fuse_conn_init(struct fuse_conn *fc,
INIT_LIST_HEAD(&fc->bg_queue);
INIT_LIST_HEAD(&fc->entry);
atomic_set(&fc->num_waiting, 0);
+ fc->bdi.name = "fuse";
fc->bdi.ra_pages = (VM_MAX_READAHEAD * 1024) / PAGE_CACHE_SIZE;
fc->bdi.unplug_io_fn = default_unplug_io_fn;
/* fuse does it's own writeback accounting */
diff -Nraup linux-2.6.30-rc6/fs/hugetlbfs/inode.c linux-2.6.30-rc6_bdiflusherv7/fs/hugetlbfs/inode.c
--- linux-2.6.30-rc6/fs/hugetlbfs/inode.c 2009-05-19 11:00:57.000000000 +0800
+++ linux-2.6.30-rc6_bdiflusherv7/fs/hugetlbfs/inode.c 2009-05-27 08:59:27.000000000 +0800
@@ -43,6 +43,7 @@ static const struct inode_operations hug
static const struct inode_operations hugetlbfs_inode_operations;
static struct backing_dev_info hugetlbfs_backing_dev_info = {
+ .name = "hugetlbfs",
.ra_pages = 0, /* No readahead */
.capabilities = BDI_CAP_NO_ACCT_AND_WRITEBACK,
};
diff -Nraup linux-2.6.30-rc6/fs/nfs/client.c linux-2.6.30-rc6_bdiflusherv7/fs/nfs/client.c
--- linux-2.6.30-rc6/fs/nfs/client.c 2009-05-19 11:00:57.000000000 +0800
+++ linux-2.6.30-rc6_bdiflusherv7/fs/nfs/client.c 2009-05-27 08:59:27.000000000 +0800
@@ -836,6 +836,7 @@ static void nfs_server_set_fsinfo(struct
server->rsize = NFS_MAX_FILE_IO_SIZE;
server->rpages = (server->rsize + PAGE_CACHE_SIZE - 1) >> PAGE_CACHE_SHIFT;
+ server->backing_dev_info.name = "nfs";
server->backing_dev_info.ra_pages = server->rpages * NFS_MAX_READAHEAD;
if (server->wsize > max_rpc_payload)
diff -Nraup linux-2.6.30-rc6/fs/ntfs/super.c linux-2.6.30-rc6_bdiflusherv7/fs/ntfs/super.c
--- linux-2.6.30-rc6/fs/ntfs/super.c 2009-05-19 11:00:57.000000000 +0800
+++ linux-2.6.30-rc6_bdiflusherv7/fs/ntfs/super.c 2009-05-27 08:59:27.000000000 +0800
@@ -2373,39 +2373,12 @@ static void ntfs_put_super(struct super_
vol->mftmirr_ino = NULL;
}
/*
- * If any dirty inodes are left, throw away all mft data page cache
- * pages to allow a clean umount. This should never happen any more
- * due to mft.c::ntfs_mft_writepage() cleaning all the dirty pages as
- * the underlying mft records are written out and cleaned. If it does,
- * happen anyway, we want to know...
+ * We should have no dirty inodes left, due to
+ * mft.c::ntfs_mft_writepage() cleaning all the dirty pages as
+ * the underlying mft records are written out and cleaned.
*/
ntfs_commit_inode(vol->mft_ino);
write_inode_now(vol->mft_ino, 1);
- if (sb_has_dirty_inodes(sb)) {
- const char *s1, *s2;
-
- mutex_lock(&vol->mft_ino->i_mutex);
- truncate_inode_pages(vol->mft_ino->i_mapping, 0);
- mutex_unlock(&vol->mft_ino->i_mutex);
- write_inode_now(vol->mft_ino, 1);
- if (sb_has_dirty_inodes(sb)) {
- static const char *_s1 = "inodes";
- static const char *_s2 = "";
- s1 = _s1;
- s2 = _s2;
- } else {
- static const char *_s1 = "mft pages";
- static const char *_s2 = "They have been thrown "
- "away. ";
- s1 = _s1;
- s2 = _s2;
- }
- ntfs_error(sb, "Dirty %s found at umount time. %sYou should "
- "run chkdsk. Please email "
- "linux-ntfs-dev@lists.sourceforge.net and say "
- "that you saw this message. Thank you.", s1,
- s2);
- }
#endif /* NTFS_RW */
iput(vol->mft_ino);
diff -Nraup linux-2.6.30-rc6/fs/ocfs2/dlm/dlmfs.c linux-2.6.30-rc6_bdiflusherv7/fs/ocfs2/dlm/dlmfs.c
--- linux-2.6.30-rc6/fs/ocfs2/dlm/dlmfs.c 2009-05-19 11:00:27.000000000 +0800
+++ linux-2.6.30-rc6_bdiflusherv7/fs/ocfs2/dlm/dlmfs.c 2009-05-27 08:59:27.000000000 +0800
@@ -325,6 +325,7 @@ clear_fields:
}
static struct backing_dev_info dlmfs_backing_dev_info = {
+ .name = "ocfs2-dlmfs",
.ra_pages = 0, /* No readahead */
.capabilities = BDI_CAP_NO_ACCT_AND_WRITEBACK,
};
diff -Nraup linux-2.6.30-rc6/fs/ramfs/inode.c linux-2.6.30-rc6_bdiflusherv7/fs/ramfs/inode.c
--- linux-2.6.30-rc6/fs/ramfs/inode.c 2009-05-19 11:00:57.000000000 +0800
+++ linux-2.6.30-rc6_bdiflusherv7/fs/ramfs/inode.c 2009-05-27 08:59:27.000000000 +0800
@@ -46,6 +46,7 @@ static const struct super_operations ram
static const struct inode_operations ramfs_dir_inode_operations;
static struct backing_dev_info ramfs_backing_dev_info = {
+ .name = "ramfs",
.ra_pages = 0, /* No readahead */
.capabilities = BDI_CAP_NO_ACCT_AND_WRITEBACK |
BDI_CAP_MAP_DIRECT | BDI_CAP_MAP_COPY |
diff -Nraup linux-2.6.30-rc6/fs/super.c linux-2.6.30-rc6_bdiflusherv7/fs/super.c
--- linux-2.6.30-rc6/fs/super.c 2009-05-19 11:00:57.000000000 +0800
+++ linux-2.6.30-rc6_bdiflusherv7/fs/super.c 2009-05-27 08:59:27.000000000 +0800
@@ -64,9 +64,6 @@ static struct super_block *alloc_super(s
s = NULL;
goto out;
}
- INIT_LIST_HEAD(&s->s_dirty);
- INIT_LIST_HEAD(&s->s_io);
- INIT_LIST_HEAD(&s->s_more_io);
INIT_LIST_HEAD(&s->s_files);
INIT_LIST_HEAD(&s->s_instances);
INIT_HLIST_HEAD(&s->s_anon);
diff -Nraup linux-2.6.30-rc6/fs/sync.c linux-2.6.30-rc6_bdiflusherv7/fs/sync.c
--- linux-2.6.30-rc6/fs/sync.c 2009-05-19 11:00:57.000000000 +0800
+++ linux-2.6.30-rc6_bdiflusherv7/fs/sync.c 2009-05-27 08:59:27.000000000 +0800
@@ -23,7 +23,7 @@
*/
static void do_sync(unsigned long wait)
{
- wakeup_pdflush(0);
+ wakeup_flusher_threads(0);
sync_inodes(0); /* All mappings, inodes and their blockdevs */
vfs_dq_sync(NULL);
sync_supers(); /* Write the superblocks */
diff -Nraup linux-2.6.30-rc6/fs/sysfs/inode.c linux-2.6.30-rc6_bdiflusherv7/fs/sysfs/inode.c
--- linux-2.6.30-rc6/fs/sysfs/inode.c 2009-05-19 11:00:57.000000000 +0800
+++ linux-2.6.30-rc6_bdiflusherv7/fs/sysfs/inode.c 2009-05-27 08:59:27.000000000 +0800
@@ -29,6 +29,7 @@ static const struct address_space_operat
};
static struct backing_dev_info sysfs_backing_dev_info = {
+ .name = "sysfs",
.ra_pages = 0, /* No readahead */
.capabilities = BDI_CAP_NO_ACCT_AND_WRITEBACK,
};
diff -Nraup linux-2.6.30-rc6/fs/ubifs/super.c linux-2.6.30-rc6_bdiflusherv7/fs/ubifs/super.c
--- linux-2.6.30-rc6/fs/ubifs/super.c 2009-05-19 11:00:57.000000000 +0800
+++ linux-2.6.30-rc6_bdiflusherv7/fs/ubifs/super.c 2009-05-27 08:59:27.000000000 +0800
@@ -1923,6 +1923,7 @@ static int ubifs_fill_super(struct super
*
* Read-ahead will be disabled because @c->bdi.ra_pages is 0.
*/
+ c->bdi.name = "ubifs",
c->bdi.capabilities = BDI_CAP_MAP_COPY;
c->bdi.unplug_io_fn = default_unplug_io_fn;
err = bdi_init(&c->bdi);
diff -Nraup linux-2.6.30-rc6/include/linux/backing-dev.h linux-2.6.30-rc6_bdiflusherv7/include/linux/backing-dev.h
--- linux-2.6.30-rc6/include/linux/backing-dev.h 2009-05-19 11:00:57.000000000 +0800
+++ linux-2.6.30-rc6_bdiflusherv7/include/linux/backing-dev.h 2009-05-27 08:59:27.000000000 +0800
@@ -13,6 +13,9 @@
#include <linux/proportions.h>
#include <linux/kernel.h>
#include <linux/fs.h>
+#include <linux/sched.h>
+#include <linux/srcu.h>
+#include <linux/writeback.h>
#include <asm/atomic.h>
struct page;
@@ -23,9 +26,12 @@ struct dentry;
* Bits in backing_dev_info.state
*/
enum bdi_state {
- BDI_pdflush, /* A pdflush thread is working this device */
+ BDI_pending, /* On its way to being activated */
+ BDI_wb_alloc, /* Default embedded wb allocated */
+ BDI_wblist_lock, /* bdi->wb_list now needs locking */
BDI_async_congested, /* The async (write) queue is getting full */
BDI_sync_congested, /* The sync queue is getting full */
+ BDI_registered, /* bdi_register() was done */
BDI_unused, /* Available bits start here */
};
@@ -39,7 +45,24 @@ enum bdi_stat_item {
#define BDI_STAT_BATCH (8*(1+ilog2(nr_cpu_ids)))
+struct bdi_writeback {
+ struct list_head list; /* hangs off the bdi */
+
+ struct backing_dev_info *bdi; /* our parent bdi */
+ unsigned int nr;
+
+ struct task_struct *task; /* writeback task */
+ wait_queue_head_t wait;
+ struct list_head b_dirty; /* dirty inodes */
+ struct list_head b_io; /* parked for writeback */
+ struct list_head b_more_io; /* parked for more writeback */
+};
+
+#define BDI_MAX_FLUSHERS 32
+
struct backing_dev_info {
+ struct srcu_struct srcu; /* for wb_list read side protection */
+ struct list_head bdi_list;
unsigned long ra_pages; /* max readahead in PAGE_CACHE_SIZE units */
unsigned long state; /* Always use atomic bitops on this */
unsigned int capabilities; /* Device capabilities */
@@ -48,6 +71,8 @@ struct backing_dev_info {
void (*unplug_io_fn)(struct backing_dev_info *, struct page *);
void *unplug_io_data;
+ char *name;
+
struct percpu_counter bdi_stat[NR_BDI_STAT_ITEMS];
struct prop_local_percpu completions;
@@ -56,6 +81,14 @@ struct backing_dev_info {
unsigned int min_ratio;
unsigned int max_ratio, max_prop_frac;
+ struct bdi_writeback wb; /* default writeback info for this bdi */
+ spinlock_t wb_lock; /* protects update side of wb_list */
+ struct list_head wb_list; /* the flusher threads hanging off this bdi */
+ unsigned long wb_mask; /* bitmask of registered tasks */
+ unsigned int wb_cnt; /* number of registered tasks */
+
+ struct list_head work_list;
+
struct device *dev;
#ifdef CONFIG_DEBUG_FS
@@ -71,6 +104,34 @@ int bdi_register(struct backing_dev_info
const char *fmt, ...);
int bdi_register_dev(struct backing_dev_info *bdi, dev_t dev);
void bdi_unregister(struct backing_dev_info *bdi);
+void bdi_start_writeback(struct backing_dev_info *bdi, struct super_block *sb,
+ long nr_pages, enum writeback_sync_modes sync_mode);
+int bdi_writeback_task(struct bdi_writeback *wb);
+void bdi_writeback_all(struct super_block *sb, long nr_pages,
+ enum writeback_sync_modes sync_mode);
+void bdi_add_default_flusher_task(struct backing_dev_info *bdi);
+void bdi_add_flusher_task(struct backing_dev_info *bdi);
+int bdi_has_dirty_io(struct backing_dev_info *bdi);
+
+extern struct mutex bdi_lock;
+extern struct list_head bdi_list;
+
+static inline int wb_is_default_task(struct bdi_writeback *wb)
+{
+ return wb == &wb->bdi->wb;
+}
+
+static inline int bdi_wblist_needs_lock(struct backing_dev_info *bdi)
+{
+ return test_bit(BDI_wblist_lock, &bdi->state);
+}
+
+static inline int wb_has_dirty_io(struct bdi_writeback *wb)
+{
+ return !list_empty(&wb->b_dirty) ||
+ !list_empty(&wb->b_io) ||
+ !list_empty(&wb->b_more_io);
+}
static inline void __add_bdi_stat(struct backing_dev_info *bdi,
enum bdi_stat_item item, s64 amount)
@@ -187,6 +248,7 @@ int bdi_set_max_ratio(struct backing_dev
#define BDI_CAP_EXEC_MAP 0x00000040
#define BDI_CAP_NO_ACCT_WB 0x00000080
#define BDI_CAP_SWAP_BACKED 0x00000100
+#define BDI_CAP_FLUSH_FORKER 0x00000200
#define BDI_CAP_VMFLAGS \
(BDI_CAP_READ_MAP | BDI_CAP_WRITE_MAP | BDI_CAP_EXEC_MAP)
@@ -256,6 +318,11 @@ static inline bool bdi_cap_swap_backed(s
return bdi->capabilities & BDI_CAP_SWAP_BACKED;
}
+static inline bool bdi_cap_flush_forker(struct backing_dev_info *bdi)
+{
+ return bdi->capabilities & BDI_CAP_FLUSH_FORKER;
+}
+
static inline bool mapping_cap_writeback_dirty(struct address_space *mapping)
{
return bdi_cap_writeback_dirty(mapping->backing_dev_info);
@@ -271,4 +338,10 @@ static inline bool mapping_cap_swap_back
return bdi_cap_swap_backed(mapping->backing_dev_info);
}
+static inline int bdi_sched_wait(void *word)
+{
+ schedule();
+ return 0;
+}
+
#endif /* _LINUX_BACKING_DEV_H */
diff -Nraup linux-2.6.30-rc6/include/linux/fs.h linux-2.6.30-rc6_bdiflusherv7/include/linux/fs.h
--- linux-2.6.30-rc6/include/linux/fs.h 2009-05-19 11:00:57.000000000 +0800
+++ linux-2.6.30-rc6_bdiflusherv7/include/linux/fs.h 2009-05-27 08:59:27.000000000 +0800
@@ -712,7 +712,7 @@ static inline int mapping_writably_mappe
struct inode {
struct hlist_node i_hash;
- struct list_head i_list;
+ struct list_head i_list; /* backing dev IO list */
struct list_head i_sb_list;
struct list_head i_dentry;
unsigned long i_ino;
@@ -1329,9 +1329,6 @@ struct super_block {
struct xattr_handler **s_xattr;
struct list_head s_inodes; /* all inodes */
- struct list_head s_dirty; /* dirty inodes */
- struct list_head s_io; /* parked for writeback */
- struct list_head s_more_io; /* parked for more writeback */
struct hlist_head s_anon; /* anonymous dentries for (nfs) exporting */
struct list_head s_files;
/* s_dentry_lru and s_nr_dentry_unused are protected by dcache_lock */
@@ -1553,11 +1550,14 @@ extern ssize_t vfs_readv(struct file *,
extern ssize_t vfs_writev(struct file *, const struct iovec __user *,
unsigned long, loff_t *);
+struct bdi_writeback;
+
struct super_operations {
struct inode *(*alloc_inode)(struct super_block *sb);
void (*destroy_inode)(struct inode *);
void (*dirty_inode) (struct inode *);
+ struct bdi_writeback *(*inode_get_wb) (struct inode *);
int (*write_inode) (struct inode *, int);
void (*drop_inode) (struct inode *);
void (*delete_inode) (struct inode *);
@@ -2066,6 +2066,8 @@ extern int invalidate_inode_pages2_range
pgoff_t start, pgoff_t end);
extern void generic_sync_sb_inodes(struct super_block *sb,
struct writeback_control *wbc);
+extern void generic_sync_bdi_inodes(struct super_block *sb,
+ struct writeback_control *);
extern int write_inode_now(struct inode *, int);
extern int filemap_fdatawrite(struct address_space *);
extern int filemap_flush(struct address_space *);
@@ -2183,7 +2185,6 @@ extern int bdev_read_only(struct block_d
extern int set_blocksize(struct block_device *, int);
extern int sb_set_blocksize(struct super_block *, int);
extern int sb_min_blocksize(struct super_block *, int);
-extern int sb_has_dirty_inodes(struct super_block *);
extern int generic_file_mmap(struct file *, struct vm_area_struct *);
extern int generic_file_readonly_mmap(struct file *, struct vm_area_struct *);
diff -Nraup linux-2.6.30-rc6/include/linux/writeback.h linux-2.6.30-rc6_bdiflusherv7/include/linux/writeback.h
--- linux-2.6.30-rc6/include/linux/writeback.h 2009-05-19 11:00:58.000000000 +0800
+++ linux-2.6.30-rc6_bdiflusherv7/include/linux/writeback.h 2009-05-27 08:59:27.000000000 +0800
@@ -14,17 +14,6 @@ extern struct list_head inode_in_use;
extern struct list_head inode_unused;
/*
- * Yes, writeback.h requires sched.h
- * No, sched.h is not included from here.
- */
-static inline int task_is_pdflush(struct task_struct *task)
-{
- return task->flags & PF_FLUSHER;
-}
-
-#define current_is_pdflush() task_is_pdflush(current)
-
-/*
* fs/fs-writeback.c
*/
enum writeback_sync_modes {
@@ -80,6 +69,7 @@ void writeback_inodes(struct writeback_c
int inode_wait(void *);
void sync_inodes_sb(struct super_block *, int wait);
void sync_inodes(int wait);
+long wb_do_writeback(struct bdi_writeback *wb);
/* writeback.h requires fs.h; it, too, is not included from here. */
static inline void wait_on_inode(struct inode *inode)
@@ -99,7 +89,7 @@ static inline void inode_sync_wait(struc
/*
* mm/page-writeback.c
*/
-int wakeup_pdflush(long nr_pages);
+void wakeup_flusher_threads(long nr_pages);
void laptop_io_completion(void);
void laptop_sync_completion(void);
void throttle_vm_writeout(gfp_t gfp_mask);
@@ -151,7 +141,6 @@ balance_dirty_pages_ratelimited(struct a
typedef int (*writepage_t)(struct page *page, struct writeback_control *wbc,
void *data);
-int pdflush_operation(void (*fn)(unsigned long), unsigned long arg0);
int generic_writepages(struct address_space *mapping,
struct writeback_control *wbc);
int write_cache_pages(struct address_space *mapping,
diff -Nraup linux-2.6.30-rc6/kernel/cgroup.c linux-2.6.30-rc6_bdiflusherv7/kernel/cgroup.c
--- linux-2.6.30-rc6/kernel/cgroup.c 2009-05-19 11:00:58.000000000 +0800
+++ linux-2.6.30-rc6_bdiflusherv7/kernel/cgroup.c 2009-05-27 08:59:27.000000000 +0800
@@ -598,6 +598,7 @@ static struct inode_operations cgroup_di
static struct file_operations proc_cgroupstats_operations;
static struct backing_dev_info cgroup_backing_dev_info = {
+ .name = "cgroup",
.capabilities = BDI_CAP_NO_ACCT_AND_WRITEBACK,
};
diff -Nraup linux-2.6.30-rc6/mm/backing-dev.c linux-2.6.30-rc6_bdiflusherv7/mm/backing-dev.c
--- linux-2.6.30-rc6/mm/backing-dev.c 2009-05-19 11:00:58.000000000 +0800
+++ linux-2.6.30-rc6_bdiflusherv7/mm/backing-dev.c 2009-05-27 08:59:27.000000000 +0800
@@ -1,8 +1,11 @@
#include <linux/wait.h>
#include <linux/backing-dev.h>
+#include <linux/kthread.h>
+#include <linux/freezer.h>
#include <linux/fs.h>
#include <linux/pagemap.h>
+#include <linux/mm.h>
#include <linux/sched.h>
#include <linux/module.h>
#include <linux/writeback.h>
@@ -14,14 +17,18 @@ void default_unplug_io_fn(struct backing
EXPORT_SYMBOL(default_unplug_io_fn);
struct backing_dev_info default_backing_dev_info = {
+ .name = "default",
.ra_pages = VM_MAX_READAHEAD * 1024 / PAGE_CACHE_SIZE,
.state = 0,
- .capabilities = BDI_CAP_MAP_COPY,
+ .capabilities = BDI_CAP_MAP_COPY | BDI_CAP_FLUSH_FORKER,
.unplug_io_fn = default_unplug_io_fn,
};
EXPORT_SYMBOL_GPL(default_backing_dev_info);
static struct class *bdi_class;
+DEFINE_MUTEX(bdi_lock);
+LIST_HEAD(bdi_list);
+LIST_HEAD(bdi_pending_list);
#ifdef CONFIG_DEBUG_FS
#include <linux/debugfs.h>
@@ -37,9 +44,29 @@ static void bdi_debug_init(void)
static int bdi_debug_stats_show(struct seq_file *m, void *v)
{
struct backing_dev_info *bdi = m->private;
+ struct bdi_writeback *wb;
unsigned long background_thresh;
unsigned long dirty_thresh;
unsigned long bdi_thresh;
+ unsigned long nr_dirty, nr_io, nr_more_io, nr_wb;
+ struct inode *inode;
+
+ /*
+ * inode lock is enough here, the bdi->wb_list is protected by
+ * RCU on the reader side
+ */
+ nr_wb = nr_dirty = nr_io = nr_more_io = 0;
+ spin_lock(&inode_lock);
+ list_for_each_entry(wb, &bdi->wb_list, list) {
+ nr_wb++;
+ list_for_each_entry(inode, &wb->b_dirty, i_list)
+ nr_dirty++;
+ list_for_each_entry(inode, &wb->b_io, i_list)
+ nr_io++;
+ list_for_each_entry(inode, &wb->b_more_io, i_list)
+ nr_more_io++;
+ }
+ spin_unlock(&inode_lock);
get_dirty_limits(&background_thresh, &dirty_thresh, &bdi_thresh, bdi);
@@ -49,12 +76,22 @@ static int bdi_debug_stats_show(struct s
"BdiReclaimable: %8lu kB\n"
"BdiDirtyThresh: %8lu kB\n"
"DirtyThresh: %8lu kB\n"
- "BackgroundThresh: %8lu kB\n",
+ "BackgroundThresh: %8lu kB\n"
+ "WriteBack threads:%8lu\n"
+ "b_dirty: %8lu\n"
+ "b_io: %8lu\n"
+ "b_more_io: %8lu\n"
+ "bdi_list: %8u\n"
+ "state: %8lx\n"
+ "wb_mask: %8lx\n"
+ "wb_list: %8u\n"
+ "wb_cnt: %8u\n",
(unsigned long) K(bdi_stat(bdi, BDI_WRITEBACK)),
(unsigned long) K(bdi_stat(bdi, BDI_RECLAIMABLE)),
- K(bdi_thresh),
- K(dirty_thresh),
- K(background_thresh));
+ K(bdi_thresh), K(dirty_thresh),
+ K(background_thresh), nr_wb, nr_dirty, nr_io, nr_more_io,
+ !list_empty(&bdi->bdi_list), bdi->state, bdi->wb_mask,
+ !list_empty(&bdi->wb_list), bdi->wb_cnt);
#undef K
return 0;
@@ -193,6 +230,343 @@ static int __init default_bdi_init(void)
}
subsys_initcall(default_bdi_init);
+static int wb_assign_nr(struct backing_dev_info *bdi, struct bdi_writeback *wb)
+{
+ unsigned long mask = BDI_MAX_FLUSHERS - 1;
+ unsigned int nr;
+
+ do {
+ if ((bdi->wb_mask & mask) == mask)
+ return 1;
+
+ nr = find_first_zero_bit(&bdi->wb_mask, BDI_MAX_FLUSHERS);
+ } while (test_and_set_bit(nr, &bdi->wb_mask));
+
+ wb->nr = nr;
+
+ spin_lock(&bdi->wb_lock);
+ bdi->wb_cnt++;
+ spin_unlock(&bdi->wb_lock);
+
+ return 0;
+}
+
+static void bdi_put_wb(struct backing_dev_info *bdi, struct bdi_writeback *wb)
+{
+ clear_bit(wb->nr, &bdi->wb_mask);
+
+ if (wb == &bdi->wb)
+ clear_bit(BDI_wb_alloc, &bdi->state);
+ else
+ kfree(wb);
+
+ spin_lock(&bdi->wb_lock);
+ bdi->wb_cnt--;
+ spin_unlock(&bdi->wb_lock);
+}
+
+static int bdi_wb_init(struct bdi_writeback *wb, struct backing_dev_info *bdi)
+{
+ memset(wb, 0, sizeof(*wb));
+
+ wb->bdi = bdi;
+ init_waitqueue_head(&wb->wait);
+ INIT_LIST_HEAD(&wb->b_dirty);
+ INIT_LIST_HEAD(&wb->b_io);
+ INIT_LIST_HEAD(&wb->b_more_io);
+
+ return wb_assign_nr(bdi, wb);
+}
+
+static struct bdi_writeback *bdi_new_wb(struct backing_dev_info *bdi)
+{
+ struct bdi_writeback *wb;
+
+ /*
+ * Default bdi->wb is already assigned, so just return it
+ */
+ if (!test_and_set_bit(BDI_wb_alloc, &bdi->state))
+ wb = &bdi->wb;
+ else {
+ wb = kmalloc(sizeof(struct bdi_writeback), GFP_KERNEL);
+ if (wb) {
+ if (bdi_wb_init(wb, bdi)) {
+ kfree(wb);
+ wb = NULL;
+ }
+ }
+ }
+
+ return wb;
+}
+
+static void bdi_task_init(struct backing_dev_info *bdi,
+ struct bdi_writeback *wb)
+{
+ struct task_struct *tsk = current;
+ int was_empty;
+
+ /*
+ * Add us to the active bdi_list. If we are adding threads beyond
+ * the default embedded bdi_writeback, then we need to start using
+ * proper locking. Check the list for empty first, then set the
+ * BDI_wblist_lock flag if there's > 1 entry on the list now
+ */
+ spin_lock(&bdi->wb_lock);
+
+ was_empty = list_empty(&bdi->wb_list);
+ list_add_tail_rcu(&wb->list, &bdi->wb_list);
+ if (!was_empty)
+ set_bit(BDI_wblist_lock, &bdi->state);
+
+ spin_unlock(&bdi->wb_lock);
+
+ tsk->flags |= PF_FLUSHER | PF_SWAPWRITE;
+ set_freezable();
+
+ /*
+ * Our parent may run at a different priority, just set us to normal
+ */
+ set_user_nice(tsk, 0);
+}
+
+static int bdi_start_fn(void *ptr)
+{
+ struct bdi_writeback *wb = ptr;
+ struct backing_dev_info *bdi = wb->bdi;
+ int ret;
+
+ /*
+ * Add us to the active bdi_list
+ */
+ mutex_lock(&bdi_lock);
+ list_add(&bdi->bdi_list, &bdi_list);
+ mutex_unlock(&bdi_lock);
+
+ bdi_task_init(bdi, wb);
+
+ /*
+ * Clear pending bit and wakeup anybody waiting to tear us down
+ */
+ clear_bit(BDI_pending, &bdi->state);
+ smp_mb__after_clear_bit();
+ wake_up_bit(&bdi->state, BDI_pending);
+
+ ret = bdi_writeback_task(wb);
+
+ /*
+ * Remove us from the list
+ */
+ spin_lock(&bdi->wb_lock);
+ list_del_rcu(&wb->list);
+ spin_unlock(&bdi->wb_lock);
+
+ /*
+ * wait for rcu grace period to end, so we can free wb
+ */
+ synchronize_srcu(&bdi->srcu);
+
+ bdi_put_wb(bdi, wb);
+ return ret;
+}
+
+int bdi_has_dirty_io(struct backing_dev_info *bdi)
+{
+ struct bdi_writeback *wb;
+ int ret = 0;
+
+ if (!bdi_wblist_needs_lock(bdi))
+ ret = wb_has_dirty_io(&bdi->wb);
+ else {
+ int idx;
+
+ idx = srcu_read_lock(&bdi->srcu);
+
+ list_for_each_entry_rcu(wb, &bdi->wb_list, list) {
+ ret = wb_has_dirty_io(wb);
+ if (ret)
+ break;
+ }
+
+ srcu_read_unlock(&bdi->srcu, idx);
+ }
+
+ return ret;
+}
+
+static void bdi_flush_io(struct backing_dev_info *bdi)
+{
+ struct writeback_control wbc = {
+ .bdi = bdi,
+ .sync_mode = WB_SYNC_NONE,
+ .older_than_this = NULL,
+ .range_cyclic = 1,
+ .nr_to_write = 1024,
+ };
+
+ generic_sync_bdi_inodes(NULL, &wbc);
+}
+
+static int bdi_forker_task(void *ptr)
+{
+ struct bdi_writeback *me = ptr;
+ DEFINE_WAIT(wait);
+
+ bdi_task_init(me->bdi, me);
+
+ for (;;) {
+ struct backing_dev_info *bdi, *tmp;
+ struct bdi_writeback *wb;
+
+ /*
+ * Do this periodically, like kupdated() did before.
+ */
+ sync_supers();
+
+ /*
+ * Temporary measure, we want to make sure we don't see
+ * dirty data on the default backing_dev_info
+ */
+ if (wb_has_dirty_io(me) || !list_empty(&me->bdi->work_list))
+ wb_do_writeback(me);
+
+ prepare_to_wait(&me->wait, &wait, TASK_INTERRUPTIBLE);
+
+ mutex_lock(&bdi_lock);
+
+ /*
+ * Check if any existing bdi's have dirty data without
+ * a thread registered. If so, set that up.
+ */
+ list_for_each_entry_safe(bdi, tmp, &bdi_list, bdi_list) {
+ if (bdi->wb.task || !bdi_has_dirty_io(bdi))
+ continue;
+
+ bdi_add_default_flusher_task(bdi);
+ }
+
+ if (list_empty(&bdi_pending_list)) {
+ unsigned long wait;
+
+ mutex_unlock(&bdi_lock);
+ wait = msecs_to_jiffies(dirty_writeback_interval * 10);
+ schedule_timeout(wait);
+ try_to_freeze();
+ continue;
+ }
+
+ /*
+ * This is our real job - check for pending entries in
+ * bdi_pending_list, and create the tasks that got added
+ */
+ bdi = list_entry(bdi_pending_list.next, struct backing_dev_info,
+ bdi_list);
+ list_del_init(&bdi->bdi_list);
+ mutex_unlock(&bdi_lock);
+
+ wb = bdi_new_wb(bdi);
+ if (!wb)
+ goto readd_flush;
+
+ wb->task = kthread_run(bdi_start_fn, wb, "bdi-%s",
+ dev_name(bdi->dev));
+
+ /*
+ * If task creation fails, then readd the bdi to
+ * the pending list and force writeout of the bdi
+ * from this forker thread. That will free some memory
+ * and we can try again.
+ */
+ if (!wb->task) {
+ bdi_put_wb(bdi, wb);
+readd_flush:
+ /*
+ * Add this 'bdi' to the back, so we get
+ * a chance to flush other bdi's to free
+ * memory.
+ */
+ mutex_lock(&bdi_lock);
+ list_add_tail(&bdi->bdi_list, &bdi_pending_list);
+ mutex_unlock(&bdi_lock);
+
+ bdi_flush_io(bdi);
+ }
+ }
+
+ finish_wait(&me->wait, &wait);
+ return 0;
+}
+
+/*
+ * bdi_lock held on entry
+ */
+static void bdi_add_one_flusher_task(struct backing_dev_info *bdi,
+ int(*func)(struct backing_dev_info *))
+{
+ if (!bdi_cap_writeback_dirty(bdi))
+ return;
+
+ if (WARN_ON(!test_bit(BDI_registered, &bdi->state))) {
+ printk("bdi %p/%s is not registered!\n", bdi, bdi->name);
+ return;
+ }
+
+ /*
+ * Check with the helper whether to proceed adding a task. Will only
+ * abort if we two or more simultanous calls to
+ * bdi_add_default_flusher_task() occured, further additions will block
+ * waiting for previous additions to finish.
+ */
+ if (!func(bdi)) {
+ list_move_tail(&bdi->bdi_list, &bdi_pending_list);
+
+ /*
+ * We are now on the pending list, wake up bdi_forker_task()
+ * to finish the job and add us back to the active bdi_list
+ */
+ wake_up(&default_backing_dev_info.wb.wait);
+ }
+}
+
+static int flusher_add_helper_block(struct backing_dev_info *bdi)
+{
+ mutex_unlock(&bdi_lock);
+ wait_on_bit_lock(&bdi->state, BDI_pending, bdi_sched_wait,
+ TASK_UNINTERRUPTIBLE);
+ mutex_lock(&bdi_lock);
+ return 0;
+}
+
+static int flusher_add_helper_test(struct backing_dev_info *bdi)
+{
+ return test_and_set_bit(BDI_pending, &bdi->state);
+}
+
+/*
+ * Add the default flusher task that gets created for any bdi
+ * that has dirty data pending writeout
+ */
+void bdi_add_default_flusher_task(struct backing_dev_info *bdi)
+{
+ bdi_add_one_flusher_task(bdi, flusher_add_helper_test);
+}
+
+/**
+ * bdi_add_flusher_task - add one more flusher task to this @bdi
+ * @bdi: the bdi
+ *
+ * Add an additional flusher task to this @bdi. Will block waiting on
+ * previous additions, if any.
+ *
+ */
+void bdi_add_flusher_task(struct backing_dev_info *bdi)
+{
+ mutex_lock(&bdi_lock);
+ bdi_add_one_flusher_task(bdi, flusher_add_helper_block);
+ mutex_unlock(&bdi_lock);
+}
+EXPORT_SYMBOL(bdi_add_flusher_task);
+
int bdi_register(struct backing_dev_info *bdi, struct device *parent,
const char *fmt, ...)
{
@@ -211,9 +585,41 @@ int bdi_register(struct backing_dev_info
goto exit;
}
+ mutex_lock(&bdi_lock);
+ list_add_tail(&bdi->bdi_list, &bdi_list);
+ mutex_unlock(&bdi_lock);
+
bdi->dev = dev;
- bdi_debug_register(bdi, dev_name(dev));
+ /*
+ * Just start the forker thread for our default backing_dev_info,
+ * and add other bdi's to the list. They will get a thread created
+ * on-demand when they need it.
+ */
+ if (bdi_cap_flush_forker(bdi)) {
+ struct bdi_writeback *wb;
+
+ wb = bdi_new_wb(bdi);
+ if (!wb) {
+ ret = -ENOMEM;
+ goto remove_err;
+ }
+
+ wb->task = kthread_run(bdi_forker_task, wb, "bdi-%s",
+ dev_name(dev));
+ if (!wb->task) {
+ bdi_put_wb(bdi, wb);
+ ret = -ENOMEM;
+remove_err:
+ mutex_lock(&bdi_lock);
+ list_del(&bdi->bdi_list);
+ mutex_unlock(&bdi_lock);
+ goto exit;
+ }
+ }
+
+ bdi_debug_register(bdi, dev_name(dev));
+ set_bit(BDI_registered, &bdi->state);
exit:
return ret;
}
@@ -225,9 +631,42 @@ int bdi_register_dev(struct backing_dev_
}
EXPORT_SYMBOL(bdi_register_dev);
+/*
+ * Remove bdi from global list and shutdown any threads we have running
+ */
+static void bdi_wb_shutdown(struct backing_dev_info *bdi)
+{
+ struct bdi_writeback *wb;
+
+ if (!bdi_cap_writeback_dirty(bdi))
+ return;
+
+ /*
+ * If setup is pending, wait for that to complete first
+ */
+ wait_on_bit(&bdi->state, BDI_pending, bdi_sched_wait,
+ TASK_UNINTERRUPTIBLE);
+
+ /*
+ * Make sure nobody finds us on the bdi_list anymore
+ */
+ mutex_lock(&bdi_lock);
+ list_del(&bdi->bdi_list);
+ mutex_unlock(&bdi_lock);
+
+ /*
+ * Finally, kill the kernel threads. We don't need to be RCU
+ * safe anymore, since the bdi is gone from visibility.
+ */
+ list_for_each_entry(wb, &bdi->wb_list, list)
+ kthread_stop(wb->task);
+}
+
void bdi_unregister(struct backing_dev_info *bdi)
{
if (bdi->dev) {
+ if (!bdi_cap_flush_forker(bdi))
+ bdi_wb_shutdown(bdi);
bdi_debug_unregister(bdi);
device_unregister(bdi->dev);
bdi->dev = NULL;
@@ -237,14 +676,21 @@ EXPORT_SYMBOL(bdi_unregister);
int bdi_init(struct backing_dev_info *bdi)
{
- int i;
- int err;
+ int i, err;
bdi->dev = NULL;
bdi->min_ratio = 0;
bdi->max_ratio = 100;
bdi->max_prop_frac = PROP_FRAC_BASE;
+ spin_lock_init(&bdi->wb_lock);
+ bdi->wb_mask = 0;
+ bdi->wb_cnt = 0;
+ INIT_LIST_HEAD(&bdi->bdi_list);
+ INIT_LIST_HEAD(&bdi->wb_list);
+ INIT_LIST_HEAD(&bdi->work_list);
+
+ bdi_wb_init(&bdi->wb, bdi);
for (i = 0; i < NR_BDI_STAT_ITEMS; i++) {
err = percpu_counter_init(&bdi->bdi_stat[i], 0);
@@ -252,10 +698,15 @@ int bdi_init(struct backing_dev_info *bd
goto err;
}
+ err = init_srcu_struct(&bdi->srcu);
+ if (err)
+ goto err;
+
bdi->dirty_exceeded = 0;
err = prop_local_init_percpu(&bdi->completions);
if (err) {
+ cleanup_srcu_struct(&bdi->srcu);
err:
while (i--)
percpu_counter_destroy(&bdi->bdi_stat[i]);
@@ -269,8 +720,12 @@ void bdi_destroy(struct backing_dev_info
{
int i;
+ WARN_ON(bdi_has_dirty_io(bdi));
+
bdi_unregister(bdi);
+ cleanup_srcu_struct(&bdi->srcu);
+
for (i = 0; i < NR_BDI_STAT_ITEMS; i++)
percpu_counter_destroy(&bdi->bdi_stat[i]);
diff -Nraup linux-2.6.30-rc6/mm/Makefile linux-2.6.30-rc6_bdiflusherv7/mm/Makefile
--- linux-2.6.30-rc6/mm/Makefile 2009-05-19 11:00:58.000000000 +0800
+++ linux-2.6.30-rc6_bdiflusherv7/mm/Makefile 2009-05-27 08:59:27.000000000 +0800
@@ -8,7 +8,7 @@ mmu-$(CONFIG_MMU) := fremap.o highmem.o
vmalloc.o
obj-y := bootmem.o filemap.o mempool.o oom_kill.o fadvise.o \
- maccess.o page_alloc.o page-writeback.o pdflush.o \
+ maccess.o page_alloc.o page-writeback.o \
readahead.o swap.o truncate.o vmscan.o shmem.o \
prio_tree.o util.o mmzone.o vmstat.o backing-dev.o \
page_isolation.o mm_init.o $(mmu-y)
diff -Nraup linux-2.6.30-rc6/mm/page-writeback.c linux-2.6.30-rc6_bdiflusherv7/mm/page-writeback.c
--- linux-2.6.30-rc6/mm/page-writeback.c 2009-05-19 11:00:58.000000000 +0800
+++ linux-2.6.30-rc6_bdiflusherv7/mm/page-writeback.c 2009-05-27 09:05:08.000000000 +0800
@@ -36,15 +36,6 @@
#include <linux/pagevec.h>
/*
- * The maximum number of pages to writeout in a single bdflush/kupdate
- * operation. We do this so we don't hold I_SYNC against an inode for
- * enormous amounts of time, which would block a userspace task which has
- * been forced to throttle against that inode. Also, the code reevaluates
- * the dirty each time it has written this many pages.
- */
-#define MAX_WRITEBACK_PAGES 1024
-
-/*
* After a CPU has dirtied this many pages, balance_dirty_pages_ratelimited
* will look to see if it needs to force writeback or throttling.
*/
@@ -117,8 +108,6 @@ EXPORT_SYMBOL(laptop_mode);
/* End of sysctl-exported parameters */
-static void background_writeout(unsigned long _min_pages);
-
/*
* Scale the writeback cache size proportional to the relative writeout speeds.
*
@@ -319,15 +308,13 @@ static void task_dirty_limit(struct task
/*
*
*/
-static DEFINE_SPINLOCK(bdi_lock);
static unsigned int bdi_min_ratio;
int bdi_set_min_ratio(struct backing_dev_info *bdi, unsigned int min_ratio)
{
int ret = 0;
- unsigned long flags;
- spin_lock_irqsave(&bdi_lock, flags);
+ mutex_lock(&bdi_lock);
if (min_ratio > bdi->max_ratio) {
ret = -EINVAL;
} else {
@@ -339,27 +326,26 @@ int bdi_set_min_ratio(struct backing_dev
ret = -EINVAL;
}
}
- spin_unlock_irqrestore(&bdi_lock, flags);
+ mutex_unlock(&bdi_lock);
return ret;
}
int bdi_set_max_ratio(struct backing_dev_info *bdi, unsigned max_ratio)
{
- unsigned long flags;
int ret = 0;
if (max_ratio > 100)
return -EINVAL;
- spin_lock_irqsave(&bdi_lock, flags);
+ mutex_lock(&bdi_lock);
if (bdi->min_ratio > max_ratio) {
ret = -EINVAL;
} else {
bdi->max_ratio = max_ratio;
bdi->max_prop_frac = (PROP_FRAC_BASE * max_ratio) / 100;
}
- spin_unlock_irqrestore(&bdi_lock, flags);
+ mutex_unlock(&bdi_lock);
return ret;
}
@@ -542,7 +528,7 @@ static void balance_dirty_pages(struct a
* been flushed to permanent storage.
*/
if (bdi_nr_reclaimable) {
- writeback_inodes(&wbc);
+ generic_sync_bdi_inodes(NULL, &wbc);
pages_written += write_chunk - wbc.nr_to_write;
get_dirty_limits(&background_thresh, &dirty_thresh,
&bdi_thresh, bdi);
@@ -593,7 +579,7 @@ static void balance_dirty_pages(struct a
(!laptop_mode && (global_page_state(NR_FILE_DIRTY)
+ global_page_state(NR_UNSTABLE_NFS)
> background_thresh)))
- pdflush_operation(background_writeout, 0);
+ bdi_start_writeback(bdi, NULL, 0, WB_SYNC_NONE);
}
void set_page_dirty_balance(struct page *page, int page_mkwrite)
@@ -678,152 +664,34 @@ void throttle_vm_writeout(gfp_t gfp_mask
}
/*
- * writeback at least _min_pages, and keep writing until the amount of dirty
- * memory is less than the background threshold, or until we're all clean.
- */
-static void background_writeout(unsigned long _min_pages)
-{
- long min_pages = _min_pages;
- struct writeback_control wbc = {
- .bdi = NULL,
- .sync_mode = WB_SYNC_NONE,
- .older_than_this = NULL,
- .nr_to_write = 0,
- .nonblocking = 1,
- .range_cyclic = 1,
- };
-
- for ( ; ; ) {
- unsigned long background_thresh;
- unsigned long dirty_thresh;
-
- get_dirty_limits(&background_thresh, &dirty_thresh, NULL, NULL);
- if (global_page_state(NR_FILE_DIRTY) +
- global_page_state(NR_UNSTABLE_NFS) < background_thresh
- && min_pages <= 0)
- break;
- wbc.more_io = 0;
- wbc.encountered_congestion = 0;
- wbc.nr_to_write = MAX_WRITEBACK_PAGES;
- wbc.pages_skipped = 0;
- writeback_inodes(&wbc);
- min_pages -= MAX_WRITEBACK_PAGES - wbc.nr_to_write;
- if (wbc.nr_to_write > 0 || wbc.pages_skipped > 0) {
- /* Wrote less than expected */
- if (wbc.encountered_congestion || wbc.more_io)
- congestion_wait(WRITE, HZ/10);
- else
- break;
- }
- }
-}
-
-/*
* Start writeback of `nr_pages' pages. If `nr_pages' is zero, write back
- * the whole world. Returns 0 if a pdflush thread was dispatched. Returns
- * -1 if all pdflush threads were busy.
+ * the whole world.
*/
-int wakeup_pdflush(long nr_pages)
+void wakeup_flusher_threads(long nr_pages)
{
if (nr_pages == 0)
nr_pages = global_page_state(NR_FILE_DIRTY) +
global_page_state(NR_UNSTABLE_NFS);
- return pdflush_operation(background_writeout, nr_pages);
+ bdi_writeback_all(NULL, nr_pages, WB_SYNC_NONE);
}
-static void wb_timer_fn(unsigned long unused);
static void laptop_timer_fn(unsigned long unused);
-static DEFINE_TIMER(wb_timer, wb_timer_fn, 0, 0);
static DEFINE_TIMER(laptop_mode_wb_timer, laptop_timer_fn, 0, 0);
/*
- * Periodic writeback of "old" data.
- *
- * Define "old": the first time one of an inode's pages is dirtied, we mark the
- * dirtying-time in the inode's address_space. So this periodic writeback code
- * just walks the superblock inode list, writing back any inodes which are
- * older than a specific point in time.
- *
- * Try to run once per dirty_writeback_interval. But if a writeback event
- * takes longer than a dirty_writeback_interval interval, then leave a
- * one-second gap.
- *
- * older_than_this takes precedence over nr_to_write. So we'll only write back
- * all dirty pages if they are all attached to "old" mappings.
- */
-static void wb_kupdate(unsigned long arg)
-{
- unsigned long oldest_jif;
- unsigned long start_jif;
- unsigned long next_jif;
- long nr_to_write;
- struct writeback_control wbc = {
- .bdi = NULL,
- .sync_mode = WB_SYNC_NONE,
- .older_than_this = &oldest_jif,
- .nr_to_write = 0,
- .nonblocking = 1,
- .for_kupdate = 1,
- .range_cyclic = 1,
- };
-
- sync_supers();
-
- oldest_jif = jiffies - msecs_to_jiffies(dirty_expire_interval);
- start_jif = jiffies;
- next_jif = start_jif + msecs_to_jiffies(dirty_writeback_interval * 10);
- nr_to_write = global_page_state(NR_FILE_DIRTY) +
- global_page_state(NR_UNSTABLE_NFS) +
- (inodes_stat.nr_inodes - inodes_stat.nr_unused);
- while (nr_to_write > 0) {
- wbc.more_io = 0;
- wbc.encountered_congestion = 0;
- wbc.nr_to_write = MAX_WRITEBACK_PAGES;
- writeback_inodes(&wbc);
- if (wbc.nr_to_write > 0) {
- if (wbc.encountered_congestion || wbc.more_io)
- congestion_wait(WRITE, HZ/10);
- else
- break; /* All the old data is written */
- }
- nr_to_write -= MAX_WRITEBACK_PAGES - wbc.nr_to_write;
- }
- if (time_before(next_jif, jiffies + HZ))
- next_jif = jiffies + HZ;
- if (dirty_writeback_interval)
- mod_timer(&wb_timer, next_jif);
-}
-
-/*
* sysctl handler for /proc/sys/vm/dirty_writeback_centisecs
*/
int dirty_writeback_centisecs_handler(ctl_table *table, int write,
struct file *file, void __user *buffer, size_t *length, loff_t *ppos)
{
proc_dointvec(table, write, file, buffer, length, ppos);
- if (dirty_writeback_interval)
- mod_timer(&wb_timer, jiffies +
- msecs_to_jiffies(dirty_writeback_interval * 10));
- else
- del_timer(&wb_timer);
return 0;
}
-static void wb_timer_fn(unsigned long unused)
-{
- if (pdflush_operation(wb_kupdate, 0) < 0)
- mod_timer(&wb_timer, jiffies + HZ); /* delay 1 second */
-}
-
-static void laptop_flush(unsigned long unused)
-{
- sys_sync();
-}
-
static void laptop_timer_fn(unsigned long unused)
{
- pdflush_operation(laptop_flush, 0);
+ wakeup_flusher_threads(0);
}
/*
@@ -906,8 +774,6 @@ void __init page_writeback_init(void)
{
int shift;
- mod_timer(&wb_timer,
- jiffies + msecs_to_jiffies(dirty_writeback_interval * 10));
writeback_set_ratelimit();
register_cpu_notifier(&ratelimit_nb);
diff -Nraup linux-2.6.30-rc6/mm/pdflush.c linux-2.6.30-rc6_bdiflusherv7/mm/pdflush.c
--- linux-2.6.30-rc6/mm/pdflush.c 2009-05-19 11:00:58.000000000 +0800
+++ linux-2.6.30-rc6_bdiflusherv7/mm/pdflush.c 1970-01-01 08:00:00.000000000 +0800
@@ -1,269 +0,0 @@
-/*
- * mm/pdflush.c - worker threads for writing back filesystem data
- *
- * Copyright (C) 2002, Linus Torvalds.
- *
- * 09Apr2002 Andrew Morton
- * Initial version
- * 29Feb2004 kaos@sgi.com
- * Move worker thread creation to kthread to avoid chewing
- * up stack space with nested calls to kernel_thread.
- */
-
-#include <linux/sched.h>
-#include <linux/list.h>
-#include <linux/signal.h>
-#include <linux/spinlock.h>
-#include <linux/gfp.h>
-#include <linux/init.h>
-#include <linux/module.h>
-#include <linux/fs.h> /* Needed by writeback.h */
-#include <linux/writeback.h> /* Prototypes pdflush_operation() */
-#include <linux/kthread.h>
-#include <linux/cpuset.h>
-#include <linux/freezer.h>
-
-
-/*
- * Minimum and maximum number of pdflush instances
- */
-#define MIN_PDFLUSH_THREADS 2
-#define MAX_PDFLUSH_THREADS 8
-
-static void start_one_pdflush_thread(void);
-
-
-/*
- * The pdflush threads are worker threads for writing back dirty data.
- * Ideally, we'd like one thread per active disk spindle. But the disk
- * topology is very hard to divine at this level. Instead, we take
- * care in various places to prevent more than one pdflush thread from
- * performing writeback against a single filesystem. pdflush threads
- * have the PF_FLUSHER flag set in current->flags to aid in this.
- */
-
-/*
- * All the pdflush threads. Protected by pdflush_lock
- */
-static LIST_HEAD(pdflush_list);
-static DEFINE_SPINLOCK(pdflush_lock);
-
-/*
- * The count of currently-running pdflush threads. Protected
- * by pdflush_lock.
- *
- * Readable by sysctl, but not writable. Published to userspace at
- * /proc/sys/vm/nr_pdflush_threads.
- */
-int nr_pdflush_threads = 0;
-
-/*
- * The time at which the pdflush thread pool last went empty
- */
-static unsigned long last_empty_jifs;
-
-/*
- * The pdflush thread.
- *
- * Thread pool management algorithm:
- *
- * - The minimum and maximum number of pdflush instances are bound
- * by MIN_PDFLUSH_THREADS and MAX_PDFLUSH_THREADS.
- *
- * - If there have been no idle pdflush instances for 1 second, create
- * a new one.
- *
- * - If the least-recently-went-to-sleep pdflush thread has been asleep
- * for more than one second, terminate a thread.
- */
-
-/*
- * A structure for passing work to a pdflush thread. Also for passing
- * state information between pdflush threads. Protected by pdflush_lock.
- */
-struct pdflush_work {
- struct task_struct *who; /* The thread */
- void (*fn)(unsigned long); /* A callback function */
- unsigned long arg0; /* An argument to the callback */
- struct list_head list; /* On pdflush_list, when idle */
- unsigned long when_i_went_to_sleep;
-};
-
-static int __pdflush(struct pdflush_work *my_work)
-{
- current->flags |= PF_FLUSHER | PF_SWAPWRITE;
- set_freezable();
- my_work->fn = NULL;
- my_work->who = current;
- INIT_LIST_HEAD(&my_work->list);
-
- spin_lock_irq(&pdflush_lock);
- for ( ; ; ) {
- struct pdflush_work *pdf;
-
- set_current_state(TASK_INTERRUPTIBLE);
- list_move(&my_work->list, &pdflush_list);
- my_work->when_i_went_to_sleep = jiffies;
- spin_unlock_irq(&pdflush_lock);
- schedule();
- try_to_freeze();
- spin_lock_irq(&pdflush_lock);
- if (!list_empty(&my_work->list)) {
- /*
- * Someone woke us up, but without removing our control
- * structure from the global list. swsusp will do this
- * in try_to_freeze()->refrigerator(). Handle it.
- */
- my_work->fn = NULL;
- continue;
- }
- if (my_work->fn == NULL) {
- printk("pdflush: bogus wakeup\n");
- continue;
- }
- spin_unlock_irq(&pdflush_lock);
-
- (*my_work->fn)(my_work->arg0);
-
- spin_lock_irq(&pdflush_lock);
-
- /*
- * Thread creation: For how long have there been zero
- * available threads?
- *
- * To throttle creation, we reset last_empty_jifs.
- */
- if (time_after(jiffies, last_empty_jifs + 1 * HZ)) {
- if (list_empty(&pdflush_list)) {
- if (nr_pdflush_threads < MAX_PDFLUSH_THREADS) {
- last_empty_jifs = jiffies;
- nr_pdflush_threads++;
- spin_unlock_irq(&pdflush_lock);
- start_one_pdflush_thread();
- spin_lock_irq(&pdflush_lock);
- }
- }
- }
-
- my_work->fn = NULL;
-
- /*
- * Thread destruction: For how long has the sleepiest
- * thread slept?
- */
- if (list_empty(&pdflush_list))
- continue;
- if (nr_pdflush_threads <= MIN_PDFLUSH_THREADS)
- continue;
- pdf = list_entry(pdflush_list.prev, struct pdflush_work, list);
- if (time_after(jiffies, pdf->when_i_went_to_sleep + 1 * HZ)) {
- /* Limit exit rate */
- pdf->when_i_went_to_sleep = jiffies;
- break; /* exeunt */
- }
- }
- nr_pdflush_threads--;
- spin_unlock_irq(&pdflush_lock);
- return 0;
-}
-
-/*
- * Of course, my_work wants to be just a local in __pdflush(). It is
- * separated out in this manner to hopefully prevent the compiler from
- * performing unfortunate optimisations against the auto variables. Because
- * these are visible to other tasks and CPUs. (No problem has actually
- * been observed. This is just paranoia).
- */
-static int pdflush(void *dummy)
-{
- struct pdflush_work my_work;
- cpumask_var_t cpus_allowed;
-
- /*
- * Since the caller doesn't even check kthread_run() worked, let's not
- * freak out too much if this fails.
- */
- if (!alloc_cpumask_var(&cpus_allowed, GFP_KERNEL)) {
- printk(KERN_WARNING "pdflush failed to allocate cpumask\n");
- return 0;
- }
-
- /*
- * pdflush can spend a lot of time doing encryption via dm-crypt. We
- * don't want to do that at keventd's priority.
- */
- set_user_nice(current, 0);
-
- /*
- * Some configs put our parent kthread in a limited cpuset,
- * which kthread() overrides, forcing cpus_allowed == cpu_all_mask.
- * Our needs are more modest - cut back to our cpusets cpus_allowed.
- * This is needed as pdflush's are dynamically created and destroyed.
- * The boottime pdflush's are easily placed w/o these 2 lines.
- */
- cpuset_cpus_allowed(current, cpus_allowed);
- set_cpus_allowed_ptr(current, cpus_allowed);
- free_cpumask_var(cpus_allowed);
-
- return __pdflush(&my_work);
-}
-
-/*
- * Attempt to wake up a pdflush thread, and get it to do some work for you.
- * Returns zero if it indeed managed to find a worker thread, and passed your
- * payload to it.
- */
-int pdflush_operation(void (*fn)(unsigned long), unsigned long arg0)
-{
- unsigned long flags;
- int ret = 0;
-
- BUG_ON(fn == NULL); /* Hard to diagnose if it's deferred */
-
- spin_lock_irqsave(&pdflush_lock, flags);
- if (list_empty(&pdflush_list)) {
- ret = -1;
- } else {
- struct pdflush_work *pdf;
-
- pdf = list_entry(pdflush_list.next, struct pdflush_work, list);
- list_del_init(&pdf->list);
- if (list_empty(&pdflush_list))
- last_empty_jifs = jiffies;
- pdf->fn = fn;
- pdf->arg0 = arg0;
- wake_up_process(pdf->who);
- }
- spin_unlock_irqrestore(&pdflush_lock, flags);
-
- return ret;
-}
-
-static void start_one_pdflush_thread(void)
-{
- struct task_struct *k;
-
- k = kthread_run(pdflush, NULL, "pdflush");
- if (unlikely(IS_ERR(k))) {
- spin_lock_irq(&pdflush_lock);
- nr_pdflush_threads--;
- spin_unlock_irq(&pdflush_lock);
- }
-}
-
-static int __init pdflush_init(void)
-{
- int i;
-
- /*
- * Pre-set nr_pdflush_threads... If we fail to create,
- * the count will be decremented.
- */
- nr_pdflush_threads = MIN_PDFLUSH_THREADS;
-
- for (i = 0; i < MIN_PDFLUSH_THREADS; i++)
- start_one_pdflush_thread();
- return 0;
-}
-
-module_init(pdflush_init);
diff -Nraup linux-2.6.30-rc6/mm/swap_state.c linux-2.6.30-rc6_bdiflusherv7/mm/swap_state.c
--- linux-2.6.30-rc6/mm/swap_state.c 2009-05-19 11:00:28.000000000 +0800
+++ linux-2.6.30-rc6_bdiflusherv7/mm/swap_state.c 2009-05-27 08:59:27.000000000 +0800
@@ -34,6 +34,7 @@ static const struct address_space_operat
};
static struct backing_dev_info swap_backing_dev_info = {
+ .name = "swap",
.capabilities = BDI_CAP_NO_ACCT_AND_WRITEBACK | BDI_CAP_SWAP_BACKED,
.unplug_io_fn = swap_unplug_io_fn,
};
diff -Nraup linux-2.6.30-rc6/mm/vmscan.c linux-2.6.30-rc6_bdiflusherv7/mm/vmscan.c
--- linux-2.6.30-rc6/mm/vmscan.c 2009-05-19 11:00:58.000000000 +0800
+++ linux-2.6.30-rc6_bdiflusherv7/mm/vmscan.c 2009-05-27 08:59:27.000000000 +0800
@@ -1654,7 +1654,7 @@ static unsigned long do_try_to_free_page
*/
if (total_scanned > sc->swap_cluster_max +
sc->swap_cluster_max / 2) {
- wakeup_pdflush(laptop_mode ? 0 : total_scanned);
+ wakeup_flusher_threads(laptop_mode ? 0 : total_scanned);
sc->may_writepage = 1;
}
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCH 0/12] Per-bdi writeback flusher threads v7
2009-05-27 5:21 ` Damien Wyart
@ 2009-05-27 5:49 ` Damien Wyart
2009-05-27 9:20 ` Jens Axboe
2009-05-27 6:17 ` Jens Axboe
1 sibling, 1 reply; 33+ messages in thread
From: Damien Wyart @ 2009-05-27 5:49 UTC (permalink / raw)
To: Jens Axboe
Cc: linux-kernel, linux-fsdevel, chris.mason, david, hch, akpm, jack,
yanmin_zhang, richard
[-- Attachment #1: Type: text/plain, Size: 1029 bytes --]
> > > OK, I spotted the problem. If we fallback to the on-stack allocation in
> > > bdi_writeback_all(), then we do the wait for the work completion with
> > > the bdi_lock mutex held. This can deadlock with bdi_forker_task(), so if
> > > we require that to be invoked to make progress (happens if a thread
> > > needs to be restarted), then we have a deadlock on that mutex.
> > > I'll cook up a fix for this, but probably not before the morning.
> > Untested fix. I think it should work, but I haven't run it here yet.
> Thanks for your feedback and explanations. Will not be able to test the
> patch before the afternoon. Will then give feedback as soon as I can. Of
> course, I can also test a new fix or series if you send one in the
> morning, after your own testing.
Just ran a quick test with your fix applied and ran into the same
problem: manual sync stays in D state. I could capture SysRq-T, I am
attaching part of the output (which was very big and continuously
displaying).
Hope this helps,
--
Damien Wyart
[-- Attachment #2: sysrq_t.gz --]
[-- Type: text/plain, Size: 5711 bytes --]
\x1f‹\b\b#Ô\x1cJ\0\x03sysrq_t\0í]Is+9r¾ûWð>ÑÑØ\vÕá°\x0föuNãƒ#&&*PX$†¸\rI½Å¿ÞH\0µ‰¤”ÕCö[ºtЫ'\x12‰¥_~™\0\x12\x7f5_W¬Z‘ê7!\x7fcdÕ\x1e÷¯þ|ö«\x17\x7fÜùÍo«ÿ2›Íê\x7fŽÆúßþí¯\x1f}yõ÷\x7f·„óPIõ\x1fÿXýçêdŸ½{Ýø¿/òWò…r¼\b\x1fäDDó|<¯·~ÿznŽf÷\x04"92½Dʤ„\x13››e\x0ekÛ€´c³Þýñøz8GyÌDyœ`å)Ï-Iò\x0eûͦé^[Zš \x02\x15\bôxª4Ðí›Ó×S\x03rAŒ®@\x0e¥hAØ: j’ŒÏf\rÍ!QJíðB¨R}÷>›\x17_DHv[ÄƜΫ?Ì“_\x1dýÁ›³w+±‚19akŽõŠ<°qHíKs8z¿=œ^[hÃë!¶¢u±\x19. åÕçžœ_›ó> j’$ê\x04ÌK‡\x7fGšðvTDýÁ°\´§ôÏù`^7¹gMxÝÙóz¿+ìÑâ¤'®¼íÒ³Æî·Û$‰Cÿd\x1fzÒÚ7²N_w¶yñ_A\x1aÿ\x15z‹”Æ<w¤ÊZ¼ï1ö¶9zãL›\x10!7\r;òœ\x19ÁsÓ^wë/Íé\x1cEm›“ß¹íé ^e\b¿‚®´Ø¾rf9O\x02ŸüywJj^[œ~Š4ÁÐHÀ\x02eù…f:6a\x7fülŽ\x0ešE,Ì0&þøf\x11\x17L†MB¼ÿrnü'¿\x030 ð*©B·J\x04ž\x01Ùnâ»LR"@\x1d÷OÇø\x16:¡m^[…ZôlcÁ¶ídÜÆpL…†&J´"|Ÿ/”Ñšë¬\x05v\x7fHøózòGÀu€€í\x1e«+á|’³Þ\x1d^Ë›\x1c‰\x03ëEqÒ‚&•06\x0f–ÿäü§¤˜iÐa¤hÝ®þžþþ\x0f40òºÎòÂi·?¯Ã×ÆX^[-\x01Œ>Œ˜Ä\x19ÿÔ´ºR—M#¿³a^^[–Ah0©\x02°Gãm¡ˆ¸ßË(íáÐ+…¥4”°ÊêN†‡yÞD3oMj\x0fe€]\bm|6^[÷‹qn¿ûÅØÕßVAYQkAVð\x13\x7f3¢ªøDkiXP\JªÉŠ¤\x1f²¼:þ¥\x17l‰¨*¥ó\x03·^\f\x7féŠckâ$\x16fQ#T_%¡Q\x1c¬’*ÿE\x10\x02ß¡\x0ejêK…\x10Zë5ÆXä
Z\x1d\v+\x1a<ïkz÷¡
ù§”\x12\b=šÏŒs?‹y\x18Õn^`\x1eÁÄf\x0e\vÅ÷¡×µöSz=°Öd\x19j´\x19eŽê\f⑧\x1dÌÑg†•\x18'\x05†.ÐÍb†0waâÞ~Ê&ž±ä¡™\x1fsQw3å\x7f=ï~»ÿ䯒-†…\x02æ)!fà4f½ïðÀ\x03\x1e\x04óM¨V„_gèà;D1¥U6Á&©Ð‚”ªóë´‡È\x18>Ç·`ÎJDI¡J’ðÖýîƒOeU\x13‘Dn·æ\x10»ø”¥P\r£ÅѶ4
2J_\x11Ä|5We¶^[÷$ëð´\x0f\x01¸\x104‰I´\x1eE^[Søí§ÐÛ\x18\rvÊà\x1d¿ïÆN=¹íjøù^[€<—ÖìTÅòg˜Î\x05E\r•dl§d´A\x1alE'¸·Jù;Á¬"iå"º’}qlMJ';EÇvjôÐËíkêKÅÿ¶6šNœ"NÔ Ž‹Ø¥v\x10—>òJw\x0fö\x1dËõ\x10;•F4{Ìëä¸}º úÔ\x03ƒÂÎÌ%&´Ä„¦1!>/&ôÝû²wW™(ÑÕlx!-…;°ÚÃ<\x0fNŒxË&v=EOšÃé¥\x05S\x03Ž\x1f\x13x"T\x19âË„*Ò’\x1c‘üP‚åh÷o\x18Y5ò·Ãq¿í=îLa°ÚǼR%\x0e\x10›ÒüóÕ¿FU6ëMGmgÐÆè9±›¡°ðM#a‘Ó:"‡Æ\r’ZˆÀ84È3#½\x1e\x1aöù¸>ûætˆ–§ô\x11ÍB`ä\x03ïGÞù4ö0»€€`¼³®EŒúw<
ÁÈ,â~ôÿ~„Ý’Be£ °Þ\x1cšýÁ\x03—U)~¦<†ŒdïT\bš§Ávw†\bÓnßø/‡õ\x11\x06ž‚Z{4‹¬ˆeY–;d»ÇR·Ð–ù.a^[¦ÚìO¦q‰h\x7fJs\bpX¢\x03²÷!ÄöõprcB\fÔÎ =\x10âZÍ#Ä\O\b±R4’R=\b~7p“‹ckªõÀd•OÄ5Žµ£ãÀM)\x05^[W£Öjz)Y~¥¯³Þñwb#†ÀMùy\b!Ný,Æ\x0eÔ\fVm`R· ô\fbß[ìFrW"æÏf碤ívè[¢Ò¡õDK)E¯±>\x11à.\x0e$\0ý©Ÿ³dwÏ%²û¨°9»‰O›f \x17¶WaJ«y*ÌøX…\r·ÆKá\aÁ\x1f¨0\x14ÇÖ$nû´L³ô`¹$\x19>:\x15N¥’
k†ywié¤.žq\x12÷V…éð— 5µñ×ðQ÷—‡¨pjØdMçt6Çâ96»S¡(s\x14ñ\x0eº\x1c‚èl÷Îìö§÷‡ŽæTèùÍ‚)\v]ßÆÂŒšÅz£4ΦËrýÊ;Á¯1åfµ¤G„Iÿ\0[$Ú\x15º“ >îw\x17úK)\x1dô—ëYú«¸\x18¯0o¤¤&\f‚ß×ßT\x1c[“ººvrU\x7f{\x13œKÁ'ªj1Fa®þFOúþ¦¿,ú»èï\x03ô7ø³}ÞF÷»×_–'\x7f¯¿j.…n“2±¬¿Y™ˆ\x19\x04\x0fÚ*MÍZ>¢¤]qlMŽ\µ¿2Hp)s\x05þ¢‚TªÌ}\f…îÄ]ªm–"ÛÑw.êνT?†þæî\\x11˜]VGS¸\aë.Þ \x104ù(Fœ\x06|Œúnêü€à´õC\fÃo¼\x05!¼ÍkhøMg5¯eqÓw®i_Ok\x7f:7OñÛÐI&I
\x1d¡\x1d‘{„ºYMBé^[èHœTÞ6v›Öä 0\£·ÿÔTÕä ˆkzhŸ±^ìm±5¾‹Dž^¢»¶†È&OS]£càÄÕ%j´3çõ'ßœ¶‡\x1cuª3Ì~îÓ«D·Q²wG\x16)© E\x10Š’ø¾.;6¿É6‚;ïجMy‹)øêÒòÅ\x10\æ<UØ9{¯0õ#6&TD3ÚŇšÍ~ÿ’&k\rêãÐ\x16My¯hY¹8ú´Ü×£*MªH-vÎÆÑ
Áõä\r\x06ë¼\a^[›w/âÃË-\x11²Ãøƒ9?\x0f½£y9\x05ÅLJ¨º.\x11"XÉÒ\ftN€•U\x7fôŽÊ{îuQÄšÞ\x12®ŸL7TZ\rãhä»ß\x1eÖh\x05\x1e½ÿ2TZ„D#ú}˜ûv\x1d™Ïùüµó¼C\x1cx\x05ô´cîŒÍò¼¹¬Ú±çþ\x12i.‚Gž7å„\x05ÕLJûâØšÜmÏ;Ž\x0ff)u¦ÑR\bB†R¤x\x14¸àw\x11×5sä\b\b§¹\x7f\x7f×¢l‰ \x18Mü=»A`ü2^ž÷[X|‡ý½ë8\x15`\x1e4öÙ¬aª§õ¯\x1f4\x10.*'IG\x19Í“ÿ\x14IZ8zß9\x11l†qa,“Oã\Š\x7f7ÝÚeâRXALZÅs“vMT~7—ž)üúºôÂf;\0R6n}²MdU€–\0(ü^[‘ Ø?¦ÝeÿÈÌÞißV}ïŠ\f\x05¸Tc\x17e@†”oe9"`ã]ñÑF›÷*þƒnÞ»
×j\x02×õL¸\x16·áZ!à\x1a^[(_U×\x03-#¸öFò ;¸V÷„kÿ1\›P-p½Àõ\x02×\v\?\0®…4f\fל̃keÈu¸.‚?€k(Ž©½ºµd®\x13¡O4Û\x16¸N¥î\x06× àš-p½Àõ\x02×\v\?\x04®[?k:“]Ó›p\x04\x7fÄ®)\x1a®+öÎј®Û\x12\fñštìšÝ\x15®1ìš,p½Àõ\x02×\v\?\x06®Å\x04®gdŒp¨oõ@À5vãvüj€k[àºíc×õ\x1dá:þFÀ5]àzë\x05®\x17¸~L0ÄNàšÏ\f†¸^[±ë"øƒ\x04)P\x1c[Sø8v][2Ù¤_JÁ&}O1\x03ô\x1e\—\x7fé»p]Ū\x1f\x04×\v´.к@ë\x0f\x05ÓÀ…˜\aR߆VDà\x02Šck2\x1fïâ¨H‰3«.ÎœJÝ S\x04\x13nÄ\x17¸^àzë\aÀu-(›Àµœ ×òFà¢\bþ\b®±)˜âW\x15bY°Ût§ºÀ…T÷\f\PDœ¹j\x17¸^àzë\x05®ï\x0f×\x10K˜²ëY§^[¹\x14æ:»î\x04\x7f\0×P\x1c[“Elºëã̤Àu*u7¸f\bvÜ\x02×\v\/p½Àõ#ص'ÓMw³’ÁD8$·Øu\x16üAœ\x19Šck¢ï°ë\\x13±-³ÓDÜ©TŽ3£Îâv\b¶êpèz>§wàÚê?Gœù;k΂Ï\v>ÿðø|:˜í8ß^¦™£d!LÀ\x19\x16ZqÌQشΦ.è4\x15£\x14<\x1f\x03*ü: ¾\x1e¬\x1e\x12v´¢…„\x1d\x13|N¥â\x17[oPª~3YÈÕ‡‹ºÓÃ’…,©8¾Q*Žo—%ú;u¯cЬæÊdˆÞì÷‡ÖØ—æË6u* B7‰yë</¦ãSólŽ®ä›)⨃¼ L Ó”0\x1aL†ìf³ðÖ´ÏßAª…f\r3J¦Œ¹Ø“ãÌG£múöå\f!¥i,%\x04æè4æœH%ròÕõ¡ ëÝúôÜÄiyx=³äí¥‹™\fÚ‘%¢um'.÷µð’¤ÒØ\\x04±Y\òNδ‡.ó\0´¡\x12¼u9mFÊ9\x13gYòiÿRòjã³t\vîJÞoÈ%Þ I\x03„V\x18\x1f¸ÏsôK´¼k{j›íÞÁ\x15Jvÿ
\x06\x10’\x19U)\v\x18>ïÉO~‹\x1eóÊ™.‰Ê\vÄ!Jþñ¤53riw-¹ž š·ù
^[ô\¿·@ªâ¬Ï“þeë·‘\aÅ×ÙÅ]$@\x0e:oÆÝÛö½fé¾k"\x15Ê$']îߨ–ù’ˆ(¨æóèõ\x12~³è}\x17ı‡×-ä€2»¯M›YCJ[ƒÏÿË|Éo\x1fÛôùС—€Ë"|ËŒ:?g"”½\x12Ifz%ü"jtÝ+ R·äM
Ã\\x1c[Ó\x14ƒg\x10+ð\x17\x15¤ROaXêν\R\x18.~Óâ7-~Óí7ýì\x14þO“V\x11¤¹úŠ4pë\0"ðDYQ>\x10åé\x05;)â\x0f\x7fÃÎÂÝ\x17îþr÷íù0æî±—‚Óº¿Á'Ú‰´¢PsÔÕËRˆ6\x1f\x04íÓSV\x19á\aÁ#î.\x03QB”\v!kÝ\x17ÇÖtý h_Sú¯Rõú¼/\x15$·:Hä\x06è¹PÝ—\x19gRϺ\bß±žTbòKu\x0fáîwA‚;Ô†úÖJmåçÔJQ®Ÿ=\x1c÷pÛúHTbŽXA?ýå=vãÍîõÐk¯M«k¡\x1d´—²™Ú;¹< ¤ÜŸ ¢à‘öÆXÖ^:h/2\x05)|õz
R*k.">^h¯\x1aJÅ\a%gko.3ˆ›h¯ózúðxíÍ#šæÀ±mÖ»“þƒÝoö½\x06·x_t\x01‚?5\x10$ŸqdƃÐU\bÌŽ€€Ï\x03\x02MÆf\x1cú-\x14WƒàÑÍÔ”µÑ¯\x18>êŠckúhãÖ\x18`\x12\x1aõ¥fÔ”\x026Œökõâ2"ˆÔK]{?È]\x05Õ’ºU¤tN?ÆŒ§¡ÍóüìáÆ‚sclšïacžN=F_„zZà ß\x12£\x02¶
Þ™@ûÄiDò¢bZ\fL¿Âzã;\rR©m5Úßû×A†jCŠ›w£ML…Yz„à {'\x12@'5\x14TšÀr\x14¥èäé\x0f\x02\bi<\x02„š\x05\x10UÉ p\x01\x10à)@¤ãüÝG]qlMác®S€hCå3@äR3jº\x06\x10EÜU€È\x03\x01ÂêÒ¹\x05 \x16€ø9\0âíU¾\x11 æ\x05\x02ªr°²s%àæMÐ\x18ÔU¾¥8¶¦ë\a+ƒ”á\x01*`ž¸šL\‰\*ÚuQá] å„ìË\w%\x1cÜü4yÈ¥ºf=Ä•HÝËs mØÉ7K§õ¬Å…X\ˆßëB¸ C`ób U9ªwÍ…p—\f¡¢fø¨+ŽÉ~È\x10ú\b#£¢0„TjFMW]ˆ,î:CH\x03\x18‚#¤tna\b\vCø1\x19ÂéyÿùŸS†@§\fÍt!J¦œÑRh[?\bþˆ! 3åÀWÍ{gyë\x12lœ$!ëJÅ\x0fZ©(f%¿—’åVúmMÝCùŽu¶ÍK\x05Z\x19òP†°ØåŸÆ.+©§¡½t\x11Â\x1c»ì¯‡ö:ÁS»\{2|Ô\x15ÇÖ\x14С½Ö)ÛÙå Åœš®zîYÜ\r»ì»Ðž…ºSç\x16»¼Øå\x1fÓ.oöOëÝØ.S+Ô(—3×lN\x06;°{t|(0ÅÉ[)\x06Áƒ\x15\x0eÁDÓ(VŠñ¨hªê‹ckâ׉»&2¢·\x06Îl´U%Xo\b\x19J¥\x10\x1cn£o‘+Þv.u…2¢xZ½04T¶|g°ÝbFŸ~Ï\x12>ÔÞïxÉSܪ4^[±óè\x1ejÇHÅÌ›fÐVÌjǽí'#Úò^Ë Qp‹i:¼;ãNè»èØ“Û®Vc\x1d{Ë}µ‚g\x18E쪗'S\x1d«R\x06u$÷Mű5]7–\x06]»ŽvŠ7Ü7—\x02îë˜ÂàF/å&÷¥Ã\x03T@\x15™ÄêŒñ¾B°ìoÂ}©ª9ËÛ0\x0fëCo‹ÒÖP…Þøõ\0K¤jÃGí*;ø˜¥éä þvùºªØ¨{°\a3ísôiŸ#~+ûö\x14>b¨¨tª^[ªÈ)vþ˹»É¦K»Ñ;¢yP¬Û“;šQBCÀU¶èùÀI«Ûa^66‘\x1d{xÂ4L-6c³>\x1fïù\x1füÌÿ¹\x12&üïþø4Æí„6££I\WÀ“\0¾±ŒÅ^\x1eM¢êÚÑ$¡ë0=šTŠckr×&q¯d¤*¹\x02\x7fQ›y4©\x13w ×ÃѤþ;\x17uç^.G“–£IX?øѤ9WÛèÀ›'¿:úƒ7gïVr\x05#rÂ.×qÂ\>gÒ®Ï}=žÒ¦?ÁÓQe&·ËYšå,Í\x04L˜‘^ß:K\x13–£4â«à[µõ½¼’ÎÛË{Ç\b\x1f\vUaª!(ŠmÍÎz R\x12ð[¢1“Q"È€ßçu×0“b\x055Úãø9Ó<¹íSÄàXºã›ì\rßLÏ3øfõæ8\r03RÏ ¼»F†½W\r¾zó^5ËEȜυJL×Ⱥ{Õ¬q\x15¦O½¸·|óÆE=oêN\x0fClGY\x12tý\x11lNY5Àç©ÛxÄtâs\x14ï1~g|ŽÍâs”KWçðÉyhØþé\x18œÿ\x14ä/%û\x12C{)\v«û9X\x1dóÜ‘jØ•\x17{›è\x05œêï›6#©“(\v\x05c"\x15M•ËDŠ¦sýLc—à¿W‚Ç ÷‰\x19Fͬ÷\x1d)ó)Ó8z9â;\x0eFB\b1ÈË\x10¢”bf\bñ~4ÊkÃz\x1aU€\@s4\x1aïC¢þïô<YlI«Œ¼\x1a-hÖÕ*¯k"©Í4ŸPÚµ—6^[t‚? QØ|BðUqc+²qDé²\x01Ñv\v=\x1d‰J¥€ìHâ0ƒÝ\x13¤´“¦Öæ½,§mpœË\x10+ÐÂVZå\aC\x1f´ã!µ'{B>¾yÛœÖO‡ˆRë\x1d AJ‹2#Ðþ¯/lJ+Ô\x10‹>CsN¯'h\x11x-à$Xô®¥;m¤‹P4Ìïÿ\x1eÞT7¿k•„\x1a•/\vH¸\x18/Øg\x0eßêñ4ésöZ\x164²\x12Úµ-ç¼/ŽI]ÝH—årØ;DEm5ï>òC©î˨š\x14o\x03ä\x02Žât+ê‹iíóGJÚqø;–òÞ–‡ðÅÄܽ̀Ê}\x11›ýþ%ò„³If8Íp4\x17J½KÒr¬&gB\x04É)\x1aï ¨ÑT/HÇ_@`I Aæ4ê\x1ejWi'2+hÝ:[¸nÑTÎ w±)Q=![Ÿ›ý®i“˜ÄÅ$š%Bƒêë\ršÃ6cƒlqðÀ£Û‡f³Þù7Mk¡‹-ÚjÞ©i@SDž™‰›@hù\r?áèä0•6”ömJS)¥Ë*PW¥u\x17ƒ¦b•nK6ø'¿óǵÍ|ì\x04±²½KsÁ ™\x19ƒfK*¯$)‹‰\x02»ÈO…v¯£$߯æ\x0e²`Š_Sá;Ù²ÖŒù&ô
à@ \rL\x14\x11\x06kUd˜ÄVÿX^[\x15ßú/ú72]8Ú¨rã4rÃK¥LÞi?¦ 5çöŠºÁÁ\fžƒµW\x03Yɽã>tA4:\rd¥RÀÁjzåI\b×i 4¾µÊ~\x14Èj#Ž\x0f¥º‡‡Ø¨ÔÏì“\x1dœ\x01³òz„ J ŽÎJ÷Ón\x1a¯jBä\x15t+ÛA\fh‰ý–ɦ$‰\x13j0 i‘:€\x18™–ƒÐQö(‡ÉkrÈ,1Ì:•1ååü\œè\x14&Px\x1fÚ:¦Þˆ ³$\x10ÎHžÔùïM–Ó<ûÍ!-AT\x19'ÿ\x1fWºÂ²þÍ\0\0
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCH 0/12] Per-bdi writeback flusher threads v7
2009-05-27 5:27 ` Zhang, Yanmin
@ 2009-05-27 6:17 ` Jens Axboe
2009-06-02 2:07 ` Zhang, Yanmin
0 siblings, 1 reply; 33+ messages in thread
From: Jens Axboe @ 2009-05-27 6:17 UTC (permalink / raw)
To: Zhang, Yanmin
Cc: linux-kernel, linux-fsdevel, chris.mason, david, hch, akpm, jack,
richard
On Wed, May 27 2009, Zhang, Yanmin wrote:
> On Tue, 2009-05-26 at 11:33 +0200, Jens Axboe wrote:
> > Hi,
> >
> > Here's the 7th version of the writeback patches. Changes since
> > v5/v6:
> >
> > - Move the sync_supers() to the global bdi_forker_task() thread, so we
> > don't writeback the supers from all the bdi kupdated() tasks.
> > - Make bdi_start_writeback() and bdi_writeback_all() be sync when called
> > with WB_SYNC_ALL only.
> > - Shuffle some more things around to make a cleaner series. The sync vs
> > async nature of bdi_writeback_all() and bdi_start_writeback() isn't
> > consistent through the series, but otherwise things should be sane.
> >
> > I'd appreciate if Richard and Yanmin could re-run testing with this,
> > just to make sure that things are sane. For ease of patching, I've
> > put the full diff here:
> >
> > http://kernel.dk/writeback-v7.patch
> I ported it to 2.6.30-rc6 with some change in file mm/page-write.c, so I
> could compare with old data.
>
> See the attachment.
>
> The new testing hits the hang issue again. It seems there is still a race.
It's actually not a race, it's a deadlock on the bdi_lock. If you find
the bdi-default task, it should be stuck in the mutex slow path. I
posted this quickie [1] yesterday but didn't test it, I'll test it today and
post a v8.
[1] http://lkml.org/lkml/2009/5/26/401
> INFO: task sync:30013 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> sync D ffffc20000011300 4736 30013 28019
> ffffffff8093e350 0000000000000086 0000000000000000 0000000000000000
> 0000000000021220 0000000000004000 0000000000011300 000000000000c868
> ffff880000016c48 ffffe20002934b30 ffff8800720b3780 ffff8800720b3b08
> Call Trace:
> [<ffffffff802c31d0>] ? bdi_sched_wait+0x0/0xd
> [<ffffffff8071e56f>] ? schedule+0x9/0x1e
> [<ffffffff802c31d9>] ? bdi_sched_wait+0x9/0xd
> [<ffffffff8071eb36>] ? __wait_on_bit+0x41/0x71
> [<ffffffff802c31d0>] ? bdi_sched_wait+0x0/0xd
> [<ffffffff8071ebd1>] ? out_of_line_wait_on_bit+0x6b/0x77
> [<ffffffff8024cc0c>] ? wake_bit_function+0x0/0x23
> [<ffffffff8022cfa1>] ? __wake_up+0x30/0x44
> [<ffffffff802c2e22>] ? bdi_writeback_all+0x20b/0x24c
> [<ffffffff802800ce>] ? pagevec_lookup_tag+0x1a/0x21
> [<ffffffff80279248>] ? wait_on_page_writeback_range+0xce/0x11b
> [<ffffffff802c2ff3>] ? generic_sync_sb_inodes+0x36/0xe1
> [<ffffffff802c3121>] ? sync_inodes_sb+0x83/0x88
> [<ffffffff802c316c>] ? __sync_inodes+0x46/0x8f
> [<ffffffff802c5d10>] ? do_sync+0x36/0x5a
> [<ffffffff802c5d56>] ? sys_sync+0xe/0x14
> [<ffffffff8020ba2b>] ? system_call_fastpath+0x16/0x1b
>
>
> diff -Nraup linux-2.6.30-rc6/block/blk-core.c linux-2.6.30-rc6_bdiflusherv7/block/blk-core.c
> --- linux-2.6.30-rc6/block/blk-core.c 2009-05-19 11:00:45.000000000 +0800
> +++ linux-2.6.30-rc6_bdiflusherv7/block/blk-core.c 2009-05-27 08:59:27.000000000 +0800
> @@ -517,6 +517,7 @@ struct request_queue *blk_alloc_queue_no
>
> q->backing_dev_info.unplug_io_fn = blk_backing_dev_unplug;
> q->backing_dev_info.unplug_io_data = q;
> + q->backing_dev_info.name = "block";
> err = bdi_init(&q->backing_dev_info);
> if (err) {
> kmem_cache_free(blk_requestq_cachep, q);
> diff -Nraup linux-2.6.30-rc6/drivers/block/aoe/aoeblk.c linux-2.6.30-rc6_bdiflusherv7/drivers/block/aoe/aoeblk.c
> --- linux-2.6.30-rc6/drivers/block/aoe/aoeblk.c 2009-05-19 11:00:27.000000000 +0800
> +++ linux-2.6.30-rc6_bdiflusherv7/drivers/block/aoe/aoeblk.c 2009-05-27 08:59:27.000000000 +0800
> @@ -265,6 +265,7 @@ aoeblk_gdalloc(void *vp)
> }
>
> blk_queue_make_request(&d->blkq, aoeblk_make_request);
> + d->blkq.backing_dev_info.name = "aoe";
> if (bdi_init(&d->blkq.backing_dev_info))
> goto err_mempool;
> spin_lock_irqsave(&d->lock, flags);
> diff -Nraup linux-2.6.30-rc6/drivers/char/mem.c linux-2.6.30-rc6_bdiflusherv7/drivers/char/mem.c
> --- linux-2.6.30-rc6/drivers/char/mem.c 2009-05-19 11:00:46.000000000 +0800
> +++ linux-2.6.30-rc6_bdiflusherv7/drivers/char/mem.c 2009-05-27 08:59:27.000000000 +0800
> @@ -820,6 +820,7 @@ static const struct file_operations zero
> * - permits private mappings, "copies" are taken of the source of zeros
> */
> static struct backing_dev_info zero_bdi = {
> + .name = "char/mem",
> .capabilities = BDI_CAP_MAP_COPY,
> };
>
> diff -Nraup linux-2.6.30-rc6/fs/btrfs/disk-io.c linux-2.6.30-rc6_bdiflusherv7/fs/btrfs/disk-io.c
> --- linux-2.6.30-rc6/fs/btrfs/disk-io.c 2009-05-19 11:00:56.000000000 +0800
> +++ linux-2.6.30-rc6_bdiflusherv7/fs/btrfs/disk-io.c 2009-05-27 08:59:27.000000000 +0800
> @@ -1345,12 +1345,25 @@ static void btrfs_unplug_io_fn(struct ba
> free_extent_map(em);
> }
>
> +/*
> + * If this fails, caller must call bdi_destroy() to get rid of the
> + * bdi again.
> + */
> static int setup_bdi(struct btrfs_fs_info *info, struct backing_dev_info *bdi)
> {
> - bdi_init(bdi);
> + int err;
> +
> + bdi->name = "btrfs";
> + bdi->capabilities = BDI_CAP_MAP_COPY;
> + err = bdi_init(bdi);
> + if (err)
> + return err;
> +
> + err = bdi_register(bdi, NULL, "btrfs");
> + if (err)
> + return err;
> +
> bdi->ra_pages = default_backing_dev_info.ra_pages;
> - bdi->state = 0;
> - bdi->capabilities = default_backing_dev_info.capabilities;
> bdi->unplug_io_fn = btrfs_unplug_io_fn;
> bdi->unplug_io_data = info;
> bdi->congested_fn = btrfs_congested_fn;
> @@ -1574,7 +1587,8 @@ struct btrfs_root *open_ctree(struct sup
> fs_info->sb = sb;
> fs_info->max_extent = (u64)-1;
> fs_info->max_inline = 8192 * 1024;
> - setup_bdi(fs_info, &fs_info->bdi);
> + if (setup_bdi(fs_info, &fs_info->bdi))
> + goto fail_bdi;
> fs_info->btree_inode = new_inode(sb);
> fs_info->btree_inode->i_ino = 1;
> fs_info->btree_inode->i_nlink = 1;
> @@ -1931,8 +1945,8 @@ fail_iput:
>
> btrfs_close_devices(fs_info->fs_devices);
> btrfs_mapping_tree_free(&fs_info->mapping_tree);
> +fail_bdi:
> bdi_destroy(&fs_info->bdi);
> -
> fail:
> kfree(extent_root);
> kfree(tree_root);
> diff -Nraup linux-2.6.30-rc6/fs/buffer.c linux-2.6.30-rc6_bdiflusherv7/fs/buffer.c
> --- linux-2.6.30-rc6/fs/buffer.c 2009-05-19 11:00:56.000000000 +0800
> +++ linux-2.6.30-rc6_bdiflusherv7/fs/buffer.c 2009-05-27 08:59:27.000000000 +0800
> @@ -281,7 +281,7 @@ static void free_more_memory(void)
> struct zone *zone;
> int nid;
>
> - wakeup_pdflush(1024);
> + wakeup_flusher_threads(1024);
> yield();
>
> for_each_online_node(nid) {
> diff -Nraup linux-2.6.30-rc6/fs/char_dev.c linux-2.6.30-rc6_bdiflusherv7/fs/char_dev.c
> --- linux-2.6.30-rc6/fs/char_dev.c 2009-05-19 11:00:27.000000000 +0800
> +++ linux-2.6.30-rc6_bdiflusherv7/fs/char_dev.c 2009-05-27 08:59:27.000000000 +0800
> @@ -32,6 +32,7 @@
> * - no readahead or I/O queue unplugging required
> */
> struct backing_dev_info directly_mappable_cdev_bdi = {
> + .name = "char",
> .capabilities = (
> #ifdef CONFIG_MMU
> /* permit private copies of the data to be taken */
> diff -Nraup linux-2.6.30-rc6/fs/configfs/inode.c linux-2.6.30-rc6_bdiflusherv7/fs/configfs/inode.c
> --- linux-2.6.30-rc6/fs/configfs/inode.c 2009-05-19 11:00:27.000000000 +0800
> +++ linux-2.6.30-rc6_bdiflusherv7/fs/configfs/inode.c 2009-05-27 08:59:27.000000000 +0800
> @@ -46,6 +46,7 @@ static const struct address_space_operat
> };
>
> static struct backing_dev_info configfs_backing_dev_info = {
> + .name = "configfs",
> .ra_pages = 0, /* No readahead */
> .capabilities = BDI_CAP_NO_ACCT_AND_WRITEBACK,
> };
> diff -Nraup linux-2.6.30-rc6/fs/fs-writeback.c linux-2.6.30-rc6_bdiflusherv7/fs/fs-writeback.c
> --- linux-2.6.30-rc6/fs/fs-writeback.c 2009-05-19 11:00:57.000000000 +0800
> +++ linux-2.6.30-rc6_bdiflusherv7/fs/fs-writeback.c 2009-05-27 08:59:27.000000000 +0800
> @@ -19,49 +19,563 @@
> #include <linux/sched.h>
> #include <linux/fs.h>
> #include <linux/mm.h>
> +#include <linux/kthread.h>
> +#include <linux/freezer.h>
> #include <linux/writeback.h>
> #include <linux/blkdev.h>
> #include <linux/backing-dev.h>
> #include <linux/buffer_head.h>
> #include "internal.h"
>
> +#define inode_to_bdi(inode) ((inode)->i_mapping->backing_dev_info)
>
> -/**
> - * writeback_acquire - attempt to get exclusive writeback access to a device
> - * @bdi: the device's backing_dev_info structure
> - *
> - * It is a waste of resources to have more than one pdflush thread blocked on
> - * a single request queue. Exclusion at the request_queue level is obtained
> - * via a flag in the request_queue's backing_dev_info.state.
> - *
> - * Non-request_queue-backed address_spaces will share default_backing_dev_info,
> - * unless they implement their own. Which is somewhat inefficient, as this
> - * may prevent concurrent writeback against multiple devices.
> +/*
> + * We don't actually have pdflush, but this one is exported though /proc...
> + */
> +int nr_pdflush_threads;
> +
> +static void generic_sync_wb_inodes(struct bdi_writeback *wb,
> + struct super_block *sb,
> + struct writeback_control *wbc);
> +
> +/*
> + * Work items for the bdi_writeback threads
> */
> -static int writeback_acquire(struct backing_dev_info *bdi)
> +struct bdi_work {
> + struct list_head list;
> + struct list_head wait_list;
> + struct rcu_head rcu_head;
> +
> + unsigned long seen;
> + atomic_t pending;
> +
> + unsigned long sb_data;
> + unsigned long nr_pages;
> + enum writeback_sync_modes sync_mode;
> +
> + unsigned long state;
> +};
> +
> +static struct super_block *bdi_work_sb(struct bdi_work *work)
> +{
> + return (struct super_block *) (work->sb_data & ~1UL);
> +}
> +
> +static inline bool bdi_work_on_stack(struct bdi_work *work)
> +{
> + return work->sb_data & 1UL;
> +}
> +
> +static inline void bdi_work_init(struct bdi_work *work, struct super_block *sb,
> + unsigned long nr_pages,
> + enum writeback_sync_modes sync_mode)
> +{
> + INIT_RCU_HEAD(&work->rcu_head);
> + work->sb_data = (unsigned long) sb;
> + work->nr_pages = nr_pages;
> + work->sync_mode = sync_mode;
> + work->state = 1;
> +
> + /*
> + * state must not be reordered around the insert
> + */
> + smp_mb();
> +}
> +
> +static inline void bdi_work_init_on_stack(struct bdi_work *work,
> + struct super_block *sb,
> + unsigned long nr_pages,
> + enum writeback_sync_modes sync_mode)
> {
> - return !test_and_set_bit(BDI_pdflush, &bdi->state);
> + bdi_work_init(work, sb, nr_pages, sync_mode);
> + work->sb_data |= 1UL;
> }
>
> /**
> * writeback_in_progress - determine whether there is writeback in progress
> * @bdi: the device's backing_dev_info structure.
> *
> - * Determine whether there is writeback in progress against a backing device.
> + * Determine whether there is writeback waiting to be handled against a
> + * backing device.
> */
> int writeback_in_progress(struct backing_dev_info *bdi)
> {
> - return test_bit(BDI_pdflush, &bdi->state);
> + return !list_empty(&bdi->work_list);
> }
>
> -/**
> - * writeback_release - relinquish exclusive writeback access against a device.
> - * @bdi: the device's backing_dev_info structure
> +static void bdi_work_clear(struct bdi_work *work)
> +{
> + clear_bit(0, &work->state);
> + smp_mb__after_clear_bit();
> + wake_up_bit(&work->state, 0);
> +}
> +
> +static void bdi_work_free(struct rcu_head *head)
> +{
> + struct bdi_work *work = container_of(head, struct bdi_work, rcu_head);
> +
> + if (!bdi_work_on_stack(work))
> + kfree(work);
> + else
> + bdi_work_clear(work);
> +}
> +
> +static void wb_work_complete(struct bdi_work *work)
> +{
> + if (!bdi_work_on_stack(work)) {
> + bdi_work_clear(work);
> +
> + if (work->sync_mode == WB_SYNC_NONE)
> + call_rcu(&work->rcu_head, bdi_work_free);
> + } else
> + call_rcu(&work->rcu_head, bdi_work_free);
> +}
> +
> +static void wb_clear_pending(struct bdi_writeback *wb, struct bdi_work *work)
> +{
> + /*
> + * The caller has retrieved the work arguments from this work,
> + * drop our reference. If this is the last ref, delete and free it
> + */
> + if (atomic_dec_and_test(&work->pending)) {
> + struct backing_dev_info *bdi = wb->bdi;
> +
> + spin_lock(&bdi->wb_lock);
> + list_del_rcu(&work->list);
> + spin_unlock(&bdi->wb_lock);
> +
> + wb_work_complete(work);
> + }
> +}
> +
> +static void wb_start_writeback(struct bdi_writeback *wb, struct bdi_work *work)
> +{
> + /*
> + * If we failed allocating the bdi work item, wake up the wb thread
> + * always. As a safety precaution, it'll flush out everything
> + */
> + if (!wb_has_dirty_io(wb) && work)
> + wb_clear_pending(wb, work);
> + else
> + wake_up(&wb->wait);
> +}
> +
> +static void bdi_queue_work(struct backing_dev_info *bdi, struct bdi_work *work)
> +{
> + if (work) {
> + work->seen = bdi->wb_mask;
> + atomic_set(&work->pending, bdi->wb_cnt);
> +
> + /*
> + * Make sure stores are seen before it appears on the list
> + */
> + smp_mb();
> +
> + spin_lock(&bdi->wb_lock);
> + list_add_tail_rcu(&work->list, &bdi->work_list);
> + spin_unlock(&bdi->wb_lock);
> + }
> +}
> +
> +static void bdi_sched_work(struct backing_dev_info *bdi, struct bdi_work *work)
> +{
> + if (!bdi_wblist_needs_lock(bdi))
> + wb_start_writeback(&bdi->wb, work);
> + else {
> + struct bdi_writeback *wb;
> + int idx;
> +
> + idx = srcu_read_lock(&bdi->srcu);
> +
> + list_for_each_entry_rcu(wb, &bdi->wb_list, list)
> + wb_start_writeback(wb, work);
> +
> + srcu_read_unlock(&bdi->srcu, idx);
> + }
> +}
> +
> +static void __bdi_start_work(struct backing_dev_info *bdi,
> + struct bdi_work *work)
> +{
> + /*
> + * If the default thread isn't there, make sure we add it. When
> + * it gets created and wakes up, we'll run this work.
> + */
> + if (unlikely(list_empty_careful(&bdi->wb_list)))
> + bdi_add_default_flusher_task(bdi);
> + else
> + bdi_sched_work(bdi, work);
> +}
> +
> +static void bdi_start_work(struct backing_dev_info *bdi, struct bdi_work *work)
> +{
> + /*
> + * If the default thread isn't there, make sure we add it. When
> + * it gets created and wakes up, we'll run this work.
> + */
> + if (unlikely(list_empty_careful(&bdi->wb_list))) {
> + mutex_lock(&bdi_lock);
> + bdi_add_default_flusher_task(bdi);
> + mutex_unlock(&bdi_lock);
> + } else
> + bdi_sched_work(bdi, work);
> +}
> +
> +/*
> + * Used for on-stack allocated work items. The caller needs to wait until
> + * the wb threads have acked the work before it's safe to continue.
> + */
> +static void bdi_wait_on_work_clear(struct bdi_work *work)
> +{
> + wait_on_bit(&work->state, 0, bdi_sched_wait, TASK_UNINTERRUPTIBLE);
> +}
> +
> +static struct bdi_work *bdi_alloc_work(struct super_block *sb, long nr_pages,
> + enum writeback_sync_modes sync_mode)
> +{
> + struct bdi_work *work;
> +
> + work = kmalloc(sizeof(*work), GFP_ATOMIC);
> + if (work)
> + bdi_work_init(work, sb, nr_pages, sync_mode);
> +
> + return work;
> +}
> +
> +void bdi_start_writeback(struct backing_dev_info *bdi, struct super_block *sb,
> + long nr_pages, enum writeback_sync_modes sync_mode)
> +{
> + const bool must_wait = sync_mode == WB_SYNC_ALL;
> + struct bdi_work work_stack, *work = NULL;
> +
> + if (!must_wait)
> + work = bdi_alloc_work(sb, nr_pages, sync_mode);
> +
> + if (!work) {
> + work = &work_stack;
> + bdi_work_init_on_stack(work, sb, nr_pages, sync_mode);
> + }
> +
> + bdi_queue_work(bdi, work);
> + bdi_start_work(bdi, work);
> +
> + /*
> + * If the sync mode is WB_SYNC_ALL, block waiting for the work to
> + * complete. If not, we only need to wait for the work to be started,
> + * if we allocated it on-stack. We use the same mechanism, if the
> + * wait bit is set in the bdi_work struct, then threads will not
> + * clear pending until after they are done.
> + *
> + * Note that work == &work_stack if must_wait is true, but that
> + * is implementation detail and we make it explicit here for
> + * ease of reading.
> + */
> + if (work == &work_stack || must_wait) {
> + bdi_wait_on_work_clear(work);
> + if (must_wait)
> + call_rcu(&work->rcu_head, bdi_work_free);
> + }
> +}
> +
> +/*
> + * The maximum number of pages to writeout in a single bdi flush/kupdate
> + * operation. We do this so we don't hold I_SYNC against an inode for
> + * enormous amounts of time, which would block a userspace task which has
> + * been forced to throttle against that inode. Also, the code reevaluates
> + * the dirty each time it has written this many pages.
> + */
> +#define MAX_WRITEBACK_PAGES 1024
> +
> +/*
> + * Periodic writeback of "old" data.
> + *
> + * Define "old": the first time one of an inode's pages is dirtied, we mark the
> + * dirtying-time in the inode's address_space. So this periodic writeback code
> + * just walks the superblock inode list, writing back any inodes which are
> + * older than a specific point in time.
> + *
> + * Try to run once per dirty_writeback_interval. But if a writeback event
> + * takes longer than a dirty_writeback_interval interval, then leave a
> + * one-second gap.
> + *
> + * older_than_this takes precedence over nr_to_write. So we'll only write back
> + * all dirty pages if they are all attached to "old" mappings.
> + */
> +static long wb_kupdated(struct bdi_writeback *wb)
> +{
> + unsigned long oldest_jif;
> + long nr_to_write, wrote = 0;
> + struct writeback_control wbc = {
> + .bdi = wb->bdi,
> + .sync_mode = WB_SYNC_NONE,
> + .older_than_this = &oldest_jif,
> + .nr_to_write = 0,
> + .for_kupdate = 1,
> + .range_cyclic = 1,
> + };
> +
> + oldest_jif = jiffies - msecs_to_jiffies(dirty_expire_interval * 10);
> +
> + nr_to_write = global_page_state(NR_FILE_DIRTY) +
> + global_page_state(NR_UNSTABLE_NFS) +
> + (inodes_stat.nr_inodes - inodes_stat.nr_unused);
> +
> + while (nr_to_write > 0) {
> + wbc.more_io = 0;
> + wbc.encountered_congestion = 0;
> + wbc.nr_to_write = MAX_WRITEBACK_PAGES;
> + generic_sync_wb_inodes(wb, NULL, &wbc);
> + wrote += MAX_WRITEBACK_PAGES - wbc.nr_to_write;
> + if (wbc.nr_to_write > 0)
> + break; /* All the old data is written */
> + nr_to_write -= MAX_WRITEBACK_PAGES;
> + }
> +
> + return wrote;
> +}
> +
> +static inline bool over_bground_thresh(void)
> +{
> + unsigned long background_thresh, dirty_thresh;
> +
> + get_dirty_limits(&background_thresh, &dirty_thresh, NULL, NULL);
> +
> + return (global_page_state(NR_FILE_DIRTY) +
> + global_page_state(NR_UNSTABLE_NFS) >= background_thresh);
> +}
> +
> +static long __wb_writeback(struct bdi_writeback *wb, long nr_pages,
> + struct super_block *sb,
> + enum writeback_sync_modes sync_mode)
> +{
> + struct writeback_control wbc = {
> + .bdi = wb->bdi,
> + .sync_mode = sync_mode,
> + .older_than_this = NULL,
> + .range_cyclic = 1,
> + };
> + long wrote = 0;
> +
> + for (;;) {
> + if (sync_mode == WB_SYNC_NONE && nr_pages <= 0 &&
> + !over_bground_thresh())
> + break;
> +
> + wbc.more_io = 0;
> + wbc.encountered_congestion = 0;
> + wbc.nr_to_write = MAX_WRITEBACK_PAGES;
> + wbc.pages_skipped = 0;
> + generic_sync_wb_inodes(wb, sb, &wbc);
> + nr_pages -= MAX_WRITEBACK_PAGES - wbc.nr_to_write;
> + wrote += MAX_WRITEBACK_PAGES - wbc.nr_to_write;
> + /*
> + * If we ran out of stuff to write, bail unless more_io got set
> + */
> + if (wbc.nr_to_write > 0 || wbc.pages_skipped > 0) {
> + if (wbc.more_io)
> + continue;
> + break;
> + }
> + }
> +
> + return wrote;
> +}
> +
> +/*
> + * Return the next bdi_work struct that hasn't been processed by this
> + * wb thread yet
> + */
> +static struct bdi_work *get_next_work_item(struct backing_dev_info *bdi,
> + struct bdi_writeback *wb)
> +{
> + struct bdi_work *work, *ret = NULL;
> +
> + rcu_read_lock();
> +
> + list_for_each_entry_rcu(work, &bdi->work_list, list) {
> + if (!test_and_clear_bit(wb->nr, &work->seen))
> + continue;
> +
> + ret = work;
> + break;
> + }
> +
> + rcu_read_unlock();
> + return ret;
> +}
> +
> +/*
> + * Retrieve work items and do the writeback they describe
> + */
> +static long wb_writeback(struct bdi_writeback *wb)
> +{
> + struct backing_dev_info *bdi = wb->bdi;
> + struct bdi_work *work;
> + long wrote = 0;
> +
> + while ((work = get_next_work_item(bdi, wb)) != NULL) {
> + struct super_block *sb = bdi_work_sb(work);
> + long nr_pages = work->nr_pages;
> + enum writeback_sync_modes sync_mode = work->sync_mode;
> +
> + /*
> + * If this isn't a data integrity operation, just notify
> + * that we have seen this work and we are now starting it.
> + */
> + if (sync_mode == WB_SYNC_NONE)
> + wb_clear_pending(wb, work);
> +
> + wrote += __wb_writeback(wb, nr_pages, sb, sync_mode);
> +
> + /*
> + * This is a data integrity writeback, so only do the
> + * notification when we have completed the work.
> + */
> + if (sync_mode == WB_SYNC_ALL)
> + wb_clear_pending(wb, work);
> + }
> +
> + return wrote;
> +}
> +
> +/*
> + * This will be inlined in bdi_writeback_task() once we get rid of any
> + * dirty inodes on the default_backing_dev_info
> + */
> +long wb_do_writeback(struct bdi_writeback *wb)
> +{
> + long wrote;
> +
> + /*
> + * We get here in two cases:
> + *
> + * schedule_timeout() returned because the dirty writeback
> + * interval has elapsed. If that happens, the work item list
> + * will be empty and we will proceed to do kupdated style writeout.
> + *
> + * Someone called bdi_start_writeback(), which put one/more work
> + * items on the work_list. Process those.
> + */
> + if (list_empty(&wb->bdi->work_list))
> + wrote = wb_kupdated(wb);
> + else
> + wrote = wb_writeback(wb);
> +
> + return wrote;
> +}
> +
> +/*
> + * Handle writeback of dirty data for the device backed by this bdi. Also
> + * wakes up periodically and does kupdated style flushing.
> */
> -static void writeback_release(struct backing_dev_info *bdi)
> +int bdi_writeback_task(struct bdi_writeback *wb)
> {
> - BUG_ON(!writeback_in_progress(bdi));
> - clear_bit(BDI_pdflush, &bdi->state);
> + unsigned long last_active = jiffies;
> + unsigned long wait_jiffies = -1UL;
> + long pages_written;
> + DEFINE_WAIT(wait);
> +
> + while (!kthread_should_stop()) {
> +
> + pages_written = wb_do_writeback(wb);
> +
> + if (pages_written)
> + last_active = jiffies;
> + else if (wait_jiffies != -1UL) {
> + unsigned long max_idle;
> +
> + /*
> + * Longest period of inactivity that we tolerate. If we
> + * see dirty data again later, the task will get
> + * recreated automatically.
> + */
> + max_idle = max(5UL * 60 * HZ, wait_jiffies);
> + if (time_after(jiffies, max_idle + last_active) &&
> + wb_is_default_task(wb))
> + break;
> + }
> +
> + prepare_to_wait(&wb->wait, &wait, TASK_INTERRUPTIBLE);
> + wait_jiffies = msecs_to_jiffies(dirty_writeback_interval * 10);
> + schedule_timeout(wait_jiffies);
> + try_to_freeze();
> + }
> +
> + finish_wait(&wb->wait, &wait);
> + return 0;
> +}
> +
> +/*
> + * Schedule writeback for all backing devices. Expensive! If this is a data
> + * integrity operation, writeback will be complete when this returns. If
> + * we are simply called for WB_SYNC_NONE, then writeback will merely be
> + * scheduled to run.
> + */
> +void bdi_writeback_all(struct super_block *sb, long nr_pages,
> + enum writeback_sync_modes sync_mode)
> +{
> + const bool must_wait = sync_mode == WB_SYNC_ALL;
> + struct backing_dev_info *bdi, *tmp;
> + struct bdi_work *work;
> + LIST_HEAD(list);
> +
> + mutex_lock(&bdi_lock);
> +
> + list_for_each_entry_safe(bdi, tmp, &bdi_list, bdi_list) {
> + struct bdi_work *work, work_stack;
> +
> + if (!bdi_has_dirty_io(bdi))
> + continue;
> +
> + work = bdi_alloc_work(sb, nr_pages, sync_mode);
> + if (!work) {
> + work = &work_stack;
> + bdi_work_init_on_stack(work, sb, nr_pages, sync_mode);
> + } else if (must_wait)
> + list_add_tail(&work->wait_list, &list);
> +
> + bdi_queue_work(bdi, work);
> + __bdi_start_work(bdi, work);
> +
> + /*
> + * Do the wait inline if this came from the stack. This
> + * only happens if we ran out of memory, so should very
> + * rarely trigger.
> + */
> + if (work == &work_stack) {
> + bdi_wait_on_work_clear(work);
> + if (must_wait)
> + call_rcu(&work->rcu_head, bdi_work_free);
> + }
> + }
> +
> + mutex_unlock(&bdi_lock);
> +
> + /*
> + * If this is for WB_SYNC_ALL, wait for pending work to complete
> + * before returning.
> + */
> + while (!list_empty(&list)) {
> + work = list_entry(list.next, struct bdi_work, wait_list);
> + list_del(&work->wait_list);
> + bdi_wait_on_work_clear(work);
> + call_rcu(&work->rcu_head, bdi_work_free);
> + }
> +}
> +
> +/*
> + * If the filesystem didn't provide a way to map an inode to a dedicated
> + * flusher thread, it doesn't support more than 1 thread. So we know it's
> + * the default thread, return that.
> + */
> +static inline struct bdi_writeback *inode_get_wb(struct inode *inode)
> +{
> + const struct super_operations *sop = inode->i_sb->s_op;
> +
> + if (!sop->inode_get_wb)
> + return &inode_to_bdi(inode)->wb;
> +
> + return sop->inode_get_wb(inode);
> }
>
> /**
> @@ -158,12 +672,21 @@ void __mark_inode_dirty(struct inode *in
> goto out;
>
> /*
> - * If the inode was already on s_dirty/s_io/s_more_io, don't
> - * reposition it (that would break s_dirty time-ordering).
> + * If the inode was already on b_dirty/b_io/b_more_io, don't
> + * reposition it (that would break b_dirty time-ordering).
> */
> if (!was_dirty) {
> + struct bdi_writeback *wb = inode_get_wb(inode);
> + struct backing_dev_info *bdi = wb->bdi;
> +
> + if (bdi_cap_writeback_dirty(bdi) &&
> + !test_bit(BDI_registered, &bdi->state)) {
> + WARN_ON(1);
> + printk("bdi-%s not registered\n", bdi->name);
> + }
> +
> inode->dirtied_when = jiffies;
> - list_move(&inode->i_list, &sb->s_dirty);
> + list_move(&inode->i_list, &wb->b_dirty);
> }
> }
> out:
> @@ -184,31 +707,32 @@ static int write_inode(struct inode *ino
> * furthest end of its superblock's dirty-inode list.
> *
> * Before stamping the inode's ->dirtied_when, we check to see whether it is
> - * already the most-recently-dirtied inode on the s_dirty list. If that is
> + * already the most-recently-dirtied inode on the b_dirty list. If that is
> * the case then the inode must have been redirtied while it was being written
> * out and we don't reset its dirtied_when.
> */
> static void redirty_tail(struct inode *inode)
> {
> - struct super_block *sb = inode->i_sb;
> + struct bdi_writeback *wb = inode_get_wb(inode);
>
> - if (!list_empty(&sb->s_dirty)) {
> - struct inode *tail_inode;
> + if (!list_empty(&wb->b_dirty)) {
> + struct inode *tail;
>
> - tail_inode = list_entry(sb->s_dirty.next, struct inode, i_list);
> - if (time_before(inode->dirtied_when,
> - tail_inode->dirtied_when))
> + tail = list_entry(wb->b_dirty.next, struct inode, i_list);
> + if (time_before(inode->dirtied_when, tail->dirtied_when))
> inode->dirtied_when = jiffies;
> }
> - list_move(&inode->i_list, &sb->s_dirty);
> + list_move(&inode->i_list, &wb->b_dirty);
> }
>
> /*
> - * requeue inode for re-scanning after sb->s_io list is exhausted.
> + * requeue inode for re-scanning after bdi->b_io list is exhausted.
> */
> static void requeue_io(struct inode *inode)
> {
> - list_move(&inode->i_list, &inode->i_sb->s_more_io);
> + struct bdi_writeback *wb = inode_get_wb(inode);
> +
> + list_move(&inode->i_list, &wb->b_more_io);
> }
>
> static void inode_sync_complete(struct inode *inode)
> @@ -255,20 +779,11 @@ static void move_expired_inodes(struct l
> /*
> * Queue all expired dirty inodes for io, eldest first.
> */
> -static void queue_io(struct super_block *sb,
> - unsigned long *older_than_this)
> -{
> - list_splice_init(&sb->s_more_io, sb->s_io.prev);
> - move_expired_inodes(&sb->s_dirty, &sb->s_io, older_than_this);
> -}
> -
> -int sb_has_dirty_inodes(struct super_block *sb)
> +static void queue_io(struct bdi_writeback *wb, unsigned long *older_than_this)
> {
> - return !list_empty(&sb->s_dirty) ||
> - !list_empty(&sb->s_io) ||
> - !list_empty(&sb->s_more_io);
> + list_splice_init(&wb->b_more_io, wb->b_io.prev);
> + move_expired_inodes(&wb->b_dirty, &wb->b_io, older_than_this);
> }
> -EXPORT_SYMBOL(sb_has_dirty_inodes);
>
> /*
> * Write a single inode's dirty pages and inode data out to disk.
> @@ -322,11 +837,11 @@ __sync_single_inode(struct inode *inode,
> /*
> * We didn't write back all the pages. nfs_writepages()
> * sometimes bales out without doing anything. Redirty
> - * the inode; Move it from s_io onto s_more_io/s_dirty.
> + * the inode; Move it from b_io onto b_more_io/b_dirty.
> */
> /*
> * akpm: if the caller was the kupdate function we put
> - * this inode at the head of s_dirty so it gets first
> + * this inode at the head of b_dirty so it gets first
> * consideration. Otherwise, move it to the tail, for
> * the reasons described there. I'm not really sure
> * how much sense this makes. Presumably I had a good
> @@ -336,7 +851,7 @@ __sync_single_inode(struct inode *inode,
> if (wbc->for_kupdate) {
> /*
> * For the kupdate function we move the inode
> - * to s_more_io so it will get more writeout as
> + * to b_more_io so it will get more writeout as
> * soon as the queue becomes uncongested.
> */
> inode->i_state |= I_DIRTY_PAGES;
> @@ -402,10 +917,10 @@ __writeback_single_inode(struct inode *i
> if ((wbc->sync_mode != WB_SYNC_ALL) && (inode->i_state & I_SYNC)) {
> /*
> * We're skipping this inode because it's locked, and we're not
> - * doing writeback-for-data-integrity. Move it to s_more_io so
> - * that writeback can proceed with the other inodes on s_io.
> + * doing writeback-for-data-integrity. Move it to b_more_io so
> + * that writeback can proceed with the other inodes on b_io.
> * We'll have another go at writing back this inode when we
> - * completed a full scan of s_io.
> + * completed a full scan of b_io.
> */
> requeue_io(inode);
> return 0;
> @@ -428,51 +943,34 @@ __writeback_single_inode(struct inode *i
> return __sync_single_inode(inode, wbc);
> }
>
> -/*
> - * Write out a superblock's list of dirty inodes. A wait will be performed
> - * upon no inodes, all inodes or the final one, depending upon sync_mode.
> - *
> - * If older_than_this is non-NULL, then only write out inodes which
> - * had their first dirtying at a time earlier than *older_than_this.
> - *
> - * If we're a pdflush thread, then implement pdflush collision avoidance
> - * against the entire list.
> - *
> - * If `bdi' is non-zero then we're being asked to writeback a specific queue.
> - * This function assumes that the blockdev superblock's inodes are backed by
> - * a variety of queues, so all inodes are searched. For other superblocks,
> - * assume that all inodes are backed by the same queue.
> - *
> - * FIXME: this linear search could get expensive with many fileystems. But
> - * how to fix? We need to go from an address_space to all inodes which share
> - * a queue with that address_space. (Easy: have a global "dirty superblocks"
> - * list).
> - *
> - * The inodes to be written are parked on sb->s_io. They are moved back onto
> - * sb->s_dirty as they are selected for writing. This way, none can be missed
> - * on the writer throttling path, and we get decent balancing between many
> - * throttled threads: we don't want them all piling up on inode_sync_wait.
> - */
> -void generic_sync_sb_inodes(struct super_block *sb,
> - struct writeback_control *wbc)
> +static void generic_sync_wb_inodes(struct bdi_writeback *wb,
> + struct super_block *sb,
> + struct writeback_control *wbc)
> {
> + const int is_blkdev_sb = sb_is_blkdev_sb(sb);
> const unsigned long start = jiffies; /* livelock avoidance */
> - int sync = wbc->sync_mode == WB_SYNC_ALL;
>
> spin_lock(&inode_lock);
> - if (!wbc->for_kupdate || list_empty(&sb->s_io))
> - queue_io(sb, wbc->older_than_this);
>
> - while (!list_empty(&sb->s_io)) {
> - struct inode *inode = list_entry(sb->s_io.prev,
> + if (!wbc->for_kupdate || list_empty(&wb->b_io))
> + queue_io(wb, wbc->older_than_this);
> +
> + while (!list_empty(&wb->b_io)) {
> + struct inode *inode = list_entry(wb->b_io.prev,
> struct inode, i_list);
> - struct address_space *mapping = inode->i_mapping;
> - struct backing_dev_info *bdi = mapping->backing_dev_info;
> long pages_skipped;
>
> - if (!bdi_cap_writeback_dirty(bdi)) {
> + /*
> + * super block given and doesn't match, skip this inode
> + */
> + if (sb && sb != inode->i_sb) {
> + redirty_tail(inode);
> + continue;
> + }
> +
> + if (!bdi_cap_writeback_dirty(wb->bdi)) {
> redirty_tail(inode);
> - if (sb_is_blkdev_sb(sb)) {
> + if (is_blkdev_sb) {
> /*
> * Dirty memory-backed blockdev: the ramdisk
> * driver does this. Skip just this inode
> @@ -492,21 +990,14 @@ void generic_sync_sb_inodes(struct super
> continue;
> }
>
> - if (wbc->nonblocking && bdi_write_congested(bdi)) {
> + if (wbc->nonblocking && bdi_write_congested(wb->bdi)) {
> wbc->encountered_congestion = 1;
> - if (!sb_is_blkdev_sb(sb))
> + if (!is_blkdev_sb)
> break; /* Skip a congested fs */
> requeue_io(inode);
> continue; /* Skip a congested blockdev */
> }
>
> - if (wbc->bdi && bdi != wbc->bdi) {
> - if (!sb_is_blkdev_sb(sb))
> - break; /* fs has the wrong queue */
> - requeue_io(inode);
> - continue; /* blockdev has wrong queue */
> - }
> -
> /*
> * Was this inode dirtied after sync_sb_inodes was called?
> * This keeps sync from extra jobs and livelock.
> @@ -514,16 +1005,10 @@ void generic_sync_sb_inodes(struct super
> if (inode_dirtied_after(inode, start))
> break;
>
> - /* Is another pdflush already flushing this queue? */
> - if (current_is_pdflush() && !writeback_acquire(bdi))
> - break;
> -
> BUG_ON(inode->i_state & I_FREEING);
> __iget(inode);
> pages_skipped = wbc->pages_skipped;
> __writeback_single_inode(inode, wbc);
> - if (current_is_pdflush())
> - writeback_release(bdi);
> if (wbc->pages_skipped != pages_skipped) {
> /*
> * writeback is not making progress due to locked
> @@ -539,13 +1024,71 @@ void generic_sync_sb_inodes(struct super
> wbc->more_io = 1;
> break;
> }
> - if (!list_empty(&sb->s_more_io))
> + if (!list_empty(&wb->b_more_io))
> wbc->more_io = 1;
> }
>
> - if (sync) {
> + spin_unlock(&inode_lock);
> + /* Leave any unwritten inodes on b_io */
> +}
> +
> +void generic_sync_bdi_inodes(struct super_block *sb,
> + struct writeback_control *wbc)
> +{
> + struct backing_dev_info *bdi = wbc->bdi;
> + struct bdi_writeback *wb;
> +
> + /*
> + * Common case is just a single wb thread and that is embedded in
> + * the bdi, so it doesn't need locking
> + */
> + if (!bdi_wblist_needs_lock(bdi))
> + generic_sync_wb_inodes(&bdi->wb, sb, wbc);
> + else {
> + int idx;
> +
> + idx = srcu_read_lock(&bdi->srcu);
> +
> + list_for_each_entry_rcu(wb, &bdi->wb_list, list)
> + generic_sync_wb_inodes(wb, sb, wbc);
> +
> + srcu_read_unlock(&bdi->srcu, idx);
> + }
> +}
> +
> +/*
> + * Write out a superblock's list of dirty inodes. A wait will be performed
> + * upon no inodes, all inodes or the final one, depending upon sync_mode.
> + *
> + * If older_than_this is non-NULL, then only write out inodes which
> + * had their first dirtying at a time earlier than *older_than_this.
> + *
> + * If we're a pdlfush thread, then implement pdflush collision avoidance
> + * against the entire list.
> + *
> + * If `bdi' is non-zero then we're being asked to writeback a specific queue.
> + * This function assumes that the blockdev superblock's inodes are backed by
> + * a variety of queues, so all inodes are searched. For other superblocks,
> + * assume that all inodes are backed by the same queue.
> + *
> + * The inodes to be written are parked on bdi->b_io. They are moved back onto
> + * bdi->b_dirty as they are selected for writing. This way, none can be missed
> + * on the writer throttling path, and we get decent balancing between many
> + * throttled threads: we don't want them all piling up on inode_sync_wait.
> + */
> +void generic_sync_sb_inodes(struct super_block *sb,
> + struct writeback_control *wbc)
> +{
> + if (wbc->bdi)
> + bdi_start_writeback(wbc->bdi, sb, wbc->nr_to_write, wbc->sync_mode);
> + else
> + bdi_writeback_all(sb, wbc->nr_to_write, wbc->sync_mode);
> +
> + if (wbc->sync_mode == WB_SYNC_ALL) {
> struct inode *inode, *old_inode = NULL;
>
> + spin_lock(&inode_lock);
> +
> /*
> * Data integrity sync. Must wait for all pages under writeback,
> * because there may have been pages dirtied before our sync
> @@ -583,10 +1126,8 @@ void generic_sync_sb_inodes(struct super
> }
> spin_unlock(&inode_lock);
> iput(old_inode);
> - } else
> - spin_unlock(&inode_lock);
> + }
>
> - return; /* Leave any unwritten inodes on s_io */
> }
> EXPORT_SYMBOL_GPL(generic_sync_sb_inodes);
>
> @@ -597,58 +1138,6 @@ static void sync_sb_inodes(struct super_
> }
>
> /*
> - * Start writeback of dirty pagecache data against all unlocked inodes.
> - *
> - * Note:
> - * We don't need to grab a reference to superblock here. If it has non-empty
> - * ->s_dirty it's hadn't been killed yet and kill_super() won't proceed
> - * past sync_inodes_sb() until the ->s_dirty/s_io/s_more_io lists are all
> - * empty. Since __sync_single_inode() regains inode_lock before it finally moves
> - * inode from superblock lists we are OK.
> - *
> - * If `older_than_this' is non-zero then only flush inodes which have a
> - * flushtime older than *older_than_this.
> - *
> - * If `bdi' is non-zero then we will scan the first inode against each
> - * superblock until we find the matching ones. One group will be the dirty
> - * inodes against a filesystem. Then when we hit the dummy blockdev superblock,
> - * sync_sb_inodes will seekout the blockdev which matches `bdi'. Maybe not
> - * super-efficient but we're about to do a ton of I/O...
> - */
> -void
> -writeback_inodes(struct writeback_control *wbc)
> -{
> - struct super_block *sb;
> -
> - might_sleep();
> - spin_lock(&sb_lock);
> -restart:
> - list_for_each_entry_reverse(sb, &super_blocks, s_list) {
> - if (sb_has_dirty_inodes(sb)) {
> - /* we're making our own get_super here */
> - sb->s_count++;
> - spin_unlock(&sb_lock);
> - /*
> - * If we can't get the readlock, there's no sense in
> - * waiting around, most of the time the FS is going to
> - * be unmounted by the time it is released.
> - */
> - if (down_read_trylock(&sb->s_umount)) {
> - if (sb->s_root)
> - sync_sb_inodes(sb, wbc);
> - up_read(&sb->s_umount);
> - }
> - spin_lock(&sb_lock);
> - if (__put_super_and_need_restart(sb))
> - goto restart;
> - }
> - if (wbc->nr_to_write <= 0)
> - break;
> - }
> - spin_unlock(&sb_lock);
> -}
> -
> -/*
> * writeback and wait upon the filesystem's dirty inodes. The caller will
> * do this in two passes - one to write, and one to wait.
> *
> diff -Nraup linux-2.6.30-rc6/fs/fuse/inode.c linux-2.6.30-rc6_bdiflusherv7/fs/fuse/inode.c
> --- linux-2.6.30-rc6/fs/fuse/inode.c 2009-05-19 11:00:57.000000000 +0800
> +++ linux-2.6.30-rc6_bdiflusherv7/fs/fuse/inode.c 2009-05-27 08:59:27.000000000 +0800
> @@ -484,6 +484,7 @@ int fuse_conn_init(struct fuse_conn *fc,
> INIT_LIST_HEAD(&fc->bg_queue);
> INIT_LIST_HEAD(&fc->entry);
> atomic_set(&fc->num_waiting, 0);
> + fc->bdi.name = "fuse";
> fc->bdi.ra_pages = (VM_MAX_READAHEAD * 1024) / PAGE_CACHE_SIZE;
> fc->bdi.unplug_io_fn = default_unplug_io_fn;
> /* fuse does it's own writeback accounting */
> diff -Nraup linux-2.6.30-rc6/fs/hugetlbfs/inode.c linux-2.6.30-rc6_bdiflusherv7/fs/hugetlbfs/inode.c
> --- linux-2.6.30-rc6/fs/hugetlbfs/inode.c 2009-05-19 11:00:57.000000000 +0800
> +++ linux-2.6.30-rc6_bdiflusherv7/fs/hugetlbfs/inode.c 2009-05-27 08:59:27.000000000 +0800
> @@ -43,6 +43,7 @@ static const struct inode_operations hug
> static const struct inode_operations hugetlbfs_inode_operations;
>
> static struct backing_dev_info hugetlbfs_backing_dev_info = {
> + .name = "hugetlbfs",
> .ra_pages = 0, /* No readahead */
> .capabilities = BDI_CAP_NO_ACCT_AND_WRITEBACK,
> };
> diff -Nraup linux-2.6.30-rc6/fs/nfs/client.c linux-2.6.30-rc6_bdiflusherv7/fs/nfs/client.c
> --- linux-2.6.30-rc6/fs/nfs/client.c 2009-05-19 11:00:57.000000000 +0800
> +++ linux-2.6.30-rc6_bdiflusherv7/fs/nfs/client.c 2009-05-27 08:59:27.000000000 +0800
> @@ -836,6 +836,7 @@ static void nfs_server_set_fsinfo(struct
> server->rsize = NFS_MAX_FILE_IO_SIZE;
> server->rpages = (server->rsize + PAGE_CACHE_SIZE - 1) >> PAGE_CACHE_SHIFT;
>
> + server->backing_dev_info.name = "nfs";
> server->backing_dev_info.ra_pages = server->rpages * NFS_MAX_READAHEAD;
>
> if (server->wsize > max_rpc_payload)
> diff -Nraup linux-2.6.30-rc6/fs/ntfs/super.c linux-2.6.30-rc6_bdiflusherv7/fs/ntfs/super.c
> --- linux-2.6.30-rc6/fs/ntfs/super.c 2009-05-19 11:00:57.000000000 +0800
> +++ linux-2.6.30-rc6_bdiflusherv7/fs/ntfs/super.c 2009-05-27 08:59:27.000000000 +0800
> @@ -2373,39 +2373,12 @@ static void ntfs_put_super(struct super_
> vol->mftmirr_ino = NULL;
> }
> /*
> - * If any dirty inodes are left, throw away all mft data page cache
> - * pages to allow a clean umount. This should never happen any more
> - * due to mft.c::ntfs_mft_writepage() cleaning all the dirty pages as
> - * the underlying mft records are written out and cleaned. If it does,
> - * happen anyway, we want to know...
> + * We should have no dirty inodes left, due to
> + * mft.c::ntfs_mft_writepage() cleaning all the dirty pages as
> + * the underlying mft records are written out and cleaned.
> */
> ntfs_commit_inode(vol->mft_ino);
> write_inode_now(vol->mft_ino, 1);
> - if (sb_has_dirty_inodes(sb)) {
> - const char *s1, *s2;
> -
> - mutex_lock(&vol->mft_ino->i_mutex);
> - truncate_inode_pages(vol->mft_ino->i_mapping, 0);
> - mutex_unlock(&vol->mft_ino->i_mutex);
> - write_inode_now(vol->mft_ino, 1);
> - if (sb_has_dirty_inodes(sb)) {
> - static const char *_s1 = "inodes";
> - static const char *_s2 = "";
> - s1 = _s1;
> - s2 = _s2;
> - } else {
> - static const char *_s1 = "mft pages";
> - static const char *_s2 = "They have been thrown "
> - "away. ";
> - s1 = _s1;
> - s2 = _s2;
> - }
> - ntfs_error(sb, "Dirty %s found at umount time. %sYou should "
> - "run chkdsk. Please email "
> - "linux-ntfs-dev@lists.sourceforge.net and say "
> - "that you saw this message. Thank you.", s1,
> - s2);
> - }
> #endif /* NTFS_RW */
>
> iput(vol->mft_ino);
> diff -Nraup linux-2.6.30-rc6/fs/ocfs2/dlm/dlmfs.c linux-2.6.30-rc6_bdiflusherv7/fs/ocfs2/dlm/dlmfs.c
> --- linux-2.6.30-rc6/fs/ocfs2/dlm/dlmfs.c 2009-05-19 11:00:27.000000000 +0800
> +++ linux-2.6.30-rc6_bdiflusherv7/fs/ocfs2/dlm/dlmfs.c 2009-05-27 08:59:27.000000000 +0800
> @@ -325,6 +325,7 @@ clear_fields:
> }
>
> static struct backing_dev_info dlmfs_backing_dev_info = {
> + .name = "ocfs2-dlmfs",
> .ra_pages = 0, /* No readahead */
> .capabilities = BDI_CAP_NO_ACCT_AND_WRITEBACK,
> };
> diff -Nraup linux-2.6.30-rc6/fs/ramfs/inode.c linux-2.6.30-rc6_bdiflusherv7/fs/ramfs/inode.c
> --- linux-2.6.30-rc6/fs/ramfs/inode.c 2009-05-19 11:00:57.000000000 +0800
> +++ linux-2.6.30-rc6_bdiflusherv7/fs/ramfs/inode.c 2009-05-27 08:59:27.000000000 +0800
> @@ -46,6 +46,7 @@ static const struct super_operations ram
> static const struct inode_operations ramfs_dir_inode_operations;
>
> static struct backing_dev_info ramfs_backing_dev_info = {
> + .name = "ramfs",
> .ra_pages = 0, /* No readahead */
> .capabilities = BDI_CAP_NO_ACCT_AND_WRITEBACK |
> BDI_CAP_MAP_DIRECT | BDI_CAP_MAP_COPY |
> diff -Nraup linux-2.6.30-rc6/fs/super.c linux-2.6.30-rc6_bdiflusherv7/fs/super.c
> --- linux-2.6.30-rc6/fs/super.c 2009-05-19 11:00:57.000000000 +0800
> +++ linux-2.6.30-rc6_bdiflusherv7/fs/super.c 2009-05-27 08:59:27.000000000 +0800
> @@ -64,9 +64,6 @@ static struct super_block *alloc_super(s
> s = NULL;
> goto out;
> }
> - INIT_LIST_HEAD(&s->s_dirty);
> - INIT_LIST_HEAD(&s->s_io);
> - INIT_LIST_HEAD(&s->s_more_io);
> INIT_LIST_HEAD(&s->s_files);
> INIT_LIST_HEAD(&s->s_instances);
> INIT_HLIST_HEAD(&s->s_anon);
> diff -Nraup linux-2.6.30-rc6/fs/sync.c linux-2.6.30-rc6_bdiflusherv7/fs/sync.c
> --- linux-2.6.30-rc6/fs/sync.c 2009-05-19 11:00:57.000000000 +0800
> +++ linux-2.6.30-rc6_bdiflusherv7/fs/sync.c 2009-05-27 08:59:27.000000000 +0800
> @@ -23,7 +23,7 @@
> */
> static void do_sync(unsigned long wait)
> {
> - wakeup_pdflush(0);
> + wakeup_flusher_threads(0);
> sync_inodes(0); /* All mappings, inodes and their blockdevs */
> vfs_dq_sync(NULL);
> sync_supers(); /* Write the superblocks */
> diff -Nraup linux-2.6.30-rc6/fs/sysfs/inode.c linux-2.6.30-rc6_bdiflusherv7/fs/sysfs/inode.c
> --- linux-2.6.30-rc6/fs/sysfs/inode.c 2009-05-19 11:00:57.000000000 +0800
> +++ linux-2.6.30-rc6_bdiflusherv7/fs/sysfs/inode.c 2009-05-27 08:59:27.000000000 +0800
> @@ -29,6 +29,7 @@ static const struct address_space_operat
> };
>
> static struct backing_dev_info sysfs_backing_dev_info = {
> + .name = "sysfs",
> .ra_pages = 0, /* No readahead */
> .capabilities = BDI_CAP_NO_ACCT_AND_WRITEBACK,
> };
> diff -Nraup linux-2.6.30-rc6/fs/ubifs/super.c linux-2.6.30-rc6_bdiflusherv7/fs/ubifs/super.c
> --- linux-2.6.30-rc6/fs/ubifs/super.c 2009-05-19 11:00:57.000000000 +0800
> +++ linux-2.6.30-rc6_bdiflusherv7/fs/ubifs/super.c 2009-05-27 08:59:27.000000000 +0800
> @@ -1923,6 +1923,7 @@ static int ubifs_fill_super(struct super
> *
> * Read-ahead will be disabled because @c->bdi.ra_pages is 0.
> */
> + c->bdi.name = "ubifs",
> c->bdi.capabilities = BDI_CAP_MAP_COPY;
> c->bdi.unplug_io_fn = default_unplug_io_fn;
> err = bdi_init(&c->bdi);
> diff -Nraup linux-2.6.30-rc6/include/linux/backing-dev.h linux-2.6.30-rc6_bdiflusherv7/include/linux/backing-dev.h
> --- linux-2.6.30-rc6/include/linux/backing-dev.h 2009-05-19 11:00:57.000000000 +0800
> +++ linux-2.6.30-rc6_bdiflusherv7/include/linux/backing-dev.h 2009-05-27 08:59:27.000000000 +0800
> @@ -13,6 +13,9 @@
> #include <linux/proportions.h>
> #include <linux/kernel.h>
> #include <linux/fs.h>
> +#include <linux/sched.h>
> +#include <linux/srcu.h>
> +#include <linux/writeback.h>
> #include <asm/atomic.h>
>
> struct page;
> @@ -23,9 +26,12 @@ struct dentry;
> * Bits in backing_dev_info.state
> */
> enum bdi_state {
> - BDI_pdflush, /* A pdflush thread is working this device */
> + BDI_pending, /* On its way to being activated */
> + BDI_wb_alloc, /* Default embedded wb allocated */
> + BDI_wblist_lock, /* bdi->wb_list now needs locking */
> BDI_async_congested, /* The async (write) queue is getting full */
> BDI_sync_congested, /* The sync queue is getting full */
> + BDI_registered, /* bdi_register() was done */
> BDI_unused, /* Available bits start here */
> };
>
> @@ -39,7 +45,24 @@ enum bdi_stat_item {
>
> #define BDI_STAT_BATCH (8*(1+ilog2(nr_cpu_ids)))
>
> +struct bdi_writeback {
> + struct list_head list; /* hangs off the bdi */
> +
> + struct backing_dev_info *bdi; /* our parent bdi */
> + unsigned int nr;
> +
> + struct task_struct *task; /* writeback task */
> + wait_queue_head_t wait;
> + struct list_head b_dirty; /* dirty inodes */
> + struct list_head b_io; /* parked for writeback */
> + struct list_head b_more_io; /* parked for more writeback */
> +};
> +
> +#define BDI_MAX_FLUSHERS 32
> +
> struct backing_dev_info {
> + struct srcu_struct srcu; /* for wb_list read side protection */
> + struct list_head bdi_list;
> unsigned long ra_pages; /* max readahead in PAGE_CACHE_SIZE units */
> unsigned long state; /* Always use atomic bitops on this */
> unsigned int capabilities; /* Device capabilities */
> @@ -48,6 +71,8 @@ struct backing_dev_info {
> void (*unplug_io_fn)(struct backing_dev_info *, struct page *);
> void *unplug_io_data;
>
> + char *name;
> +
> struct percpu_counter bdi_stat[NR_BDI_STAT_ITEMS];
>
> struct prop_local_percpu completions;
> @@ -56,6 +81,14 @@ struct backing_dev_info {
> unsigned int min_ratio;
> unsigned int max_ratio, max_prop_frac;
>
> + struct bdi_writeback wb; /* default writeback info for this bdi */
> + spinlock_t wb_lock; /* protects update side of wb_list */
> + struct list_head wb_list; /* the flusher threads hanging off this bdi */
> + unsigned long wb_mask; /* bitmask of registered tasks */
> + unsigned int wb_cnt; /* number of registered tasks */
> +
> + struct list_head work_list;
> +
> struct device *dev;
>
> #ifdef CONFIG_DEBUG_FS
> @@ -71,6 +104,34 @@ int bdi_register(struct backing_dev_info
> const char *fmt, ...);
> int bdi_register_dev(struct backing_dev_info *bdi, dev_t dev);
> void bdi_unregister(struct backing_dev_info *bdi);
> +void bdi_start_writeback(struct backing_dev_info *bdi, struct super_block *sb,
> + long nr_pages, enum writeback_sync_modes sync_mode);
> +int bdi_writeback_task(struct bdi_writeback *wb);
> +void bdi_writeback_all(struct super_block *sb, long nr_pages,
> + enum writeback_sync_modes sync_mode);
> +void bdi_add_default_flusher_task(struct backing_dev_info *bdi);
> +void bdi_add_flusher_task(struct backing_dev_info *bdi);
> +int bdi_has_dirty_io(struct backing_dev_info *bdi);
> +
> +extern struct mutex bdi_lock;
> +extern struct list_head bdi_list;
> +
> +static inline int wb_is_default_task(struct bdi_writeback *wb)
> +{
> + return wb == &wb->bdi->wb;
> +}
> +
> +static inline int bdi_wblist_needs_lock(struct backing_dev_info *bdi)
> +{
> + return test_bit(BDI_wblist_lock, &bdi->state);
> +}
> +
> +static inline int wb_has_dirty_io(struct bdi_writeback *wb)
> +{
> + return !list_empty(&wb->b_dirty) ||
> + !list_empty(&wb->b_io) ||
> + !list_empty(&wb->b_more_io);
> +}
>
> static inline void __add_bdi_stat(struct backing_dev_info *bdi,
> enum bdi_stat_item item, s64 amount)
> @@ -187,6 +248,7 @@ int bdi_set_max_ratio(struct backing_dev
> #define BDI_CAP_EXEC_MAP 0x00000040
> #define BDI_CAP_NO_ACCT_WB 0x00000080
> #define BDI_CAP_SWAP_BACKED 0x00000100
> +#define BDI_CAP_FLUSH_FORKER 0x00000200
>
> #define BDI_CAP_VMFLAGS \
> (BDI_CAP_READ_MAP | BDI_CAP_WRITE_MAP | BDI_CAP_EXEC_MAP)
> @@ -256,6 +318,11 @@ static inline bool bdi_cap_swap_backed(s
> return bdi->capabilities & BDI_CAP_SWAP_BACKED;
> }
>
> +static inline bool bdi_cap_flush_forker(struct backing_dev_info *bdi)
> +{
> + return bdi->capabilities & BDI_CAP_FLUSH_FORKER;
> +}
> +
> static inline bool mapping_cap_writeback_dirty(struct address_space *mapping)
> {
> return bdi_cap_writeback_dirty(mapping->backing_dev_info);
> @@ -271,4 +338,10 @@ static inline bool mapping_cap_swap_back
> return bdi_cap_swap_backed(mapping->backing_dev_info);
> }
>
> +static inline int bdi_sched_wait(void *word)
> +{
> + schedule();
> + return 0;
> +}
> +
> #endif /* _LINUX_BACKING_DEV_H */
> diff -Nraup linux-2.6.30-rc6/include/linux/fs.h linux-2.6.30-rc6_bdiflusherv7/include/linux/fs.h
> --- linux-2.6.30-rc6/include/linux/fs.h 2009-05-19 11:00:57.000000000 +0800
> +++ linux-2.6.30-rc6_bdiflusherv7/include/linux/fs.h 2009-05-27 08:59:27.000000000 +0800
> @@ -712,7 +712,7 @@ static inline int mapping_writably_mappe
>
> struct inode {
> struct hlist_node i_hash;
> - struct list_head i_list;
> + struct list_head i_list; /* backing dev IO list */
> struct list_head i_sb_list;
> struct list_head i_dentry;
> unsigned long i_ino;
> @@ -1329,9 +1329,6 @@ struct super_block {
> struct xattr_handler **s_xattr;
>
> struct list_head s_inodes; /* all inodes */
> - struct list_head s_dirty; /* dirty inodes */
> - struct list_head s_io; /* parked for writeback */
> - struct list_head s_more_io; /* parked for more writeback */
> struct hlist_head s_anon; /* anonymous dentries for (nfs) exporting */
> struct list_head s_files;
> /* s_dentry_lru and s_nr_dentry_unused are protected by dcache_lock */
> @@ -1553,11 +1550,14 @@ extern ssize_t vfs_readv(struct file *,
> extern ssize_t vfs_writev(struct file *, const struct iovec __user *,
> unsigned long, loff_t *);
>
> +struct bdi_writeback;
> +
> struct super_operations {
> struct inode *(*alloc_inode)(struct super_block *sb);
> void (*destroy_inode)(struct inode *);
>
> void (*dirty_inode) (struct inode *);
> + struct bdi_writeback *(*inode_get_wb) (struct inode *);
> int (*write_inode) (struct inode *, int);
> void (*drop_inode) (struct inode *);
> void (*delete_inode) (struct inode *);
> @@ -2066,6 +2066,8 @@ extern int invalidate_inode_pages2_range
> pgoff_t start, pgoff_t end);
> extern void generic_sync_sb_inodes(struct super_block *sb,
> struct writeback_control *wbc);
> +extern void generic_sync_bdi_inodes(struct super_block *sb,
> + struct writeback_control *);
> extern int write_inode_now(struct inode *, int);
> extern int filemap_fdatawrite(struct address_space *);
> extern int filemap_flush(struct address_space *);
> @@ -2183,7 +2185,6 @@ extern int bdev_read_only(struct block_d
> extern int set_blocksize(struct block_device *, int);
> extern int sb_set_blocksize(struct super_block *, int);
> extern int sb_min_blocksize(struct super_block *, int);
> -extern int sb_has_dirty_inodes(struct super_block *);
>
> extern int generic_file_mmap(struct file *, struct vm_area_struct *);
> extern int generic_file_readonly_mmap(struct file *, struct vm_area_struct *);
> diff -Nraup linux-2.6.30-rc6/include/linux/writeback.h linux-2.6.30-rc6_bdiflusherv7/include/linux/writeback.h
> --- linux-2.6.30-rc6/include/linux/writeback.h 2009-05-19 11:00:58.000000000 +0800
> +++ linux-2.6.30-rc6_bdiflusherv7/include/linux/writeback.h 2009-05-27 08:59:27.000000000 +0800
> @@ -14,17 +14,6 @@ extern struct list_head inode_in_use;
> extern struct list_head inode_unused;
>
> /*
> - * Yes, writeback.h requires sched.h
> - * No, sched.h is not included from here.
> - */
> -static inline int task_is_pdflush(struct task_struct *task)
> -{
> - return task->flags & PF_FLUSHER;
> -}
> -
> -#define current_is_pdflush() task_is_pdflush(current)
> -
> -/*
> * fs/fs-writeback.c
> */
> enum writeback_sync_modes {
> @@ -80,6 +69,7 @@ void writeback_inodes(struct writeback_c
> int inode_wait(void *);
> void sync_inodes_sb(struct super_block *, int wait);
> void sync_inodes(int wait);
> +long wb_do_writeback(struct bdi_writeback *wb);
>
> /* writeback.h requires fs.h; it, too, is not included from here. */
> static inline void wait_on_inode(struct inode *inode)
> @@ -99,7 +89,7 @@ static inline void inode_sync_wait(struc
> /*
> * mm/page-writeback.c
> */
> -int wakeup_pdflush(long nr_pages);
> +void wakeup_flusher_threads(long nr_pages);
> void laptop_io_completion(void);
> void laptop_sync_completion(void);
> void throttle_vm_writeout(gfp_t gfp_mask);
> @@ -151,7 +141,6 @@ balance_dirty_pages_ratelimited(struct a
> typedef int (*writepage_t)(struct page *page, struct writeback_control *wbc,
> void *data);
>
> -int pdflush_operation(void (*fn)(unsigned long), unsigned long arg0);
> int generic_writepages(struct address_space *mapping,
> struct writeback_control *wbc);
> int write_cache_pages(struct address_space *mapping,
> diff -Nraup linux-2.6.30-rc6/kernel/cgroup.c linux-2.6.30-rc6_bdiflusherv7/kernel/cgroup.c
> --- linux-2.6.30-rc6/kernel/cgroup.c 2009-05-19 11:00:58.000000000 +0800
> +++ linux-2.6.30-rc6_bdiflusherv7/kernel/cgroup.c 2009-05-27 08:59:27.000000000 +0800
> @@ -598,6 +598,7 @@ static struct inode_operations cgroup_di
> static struct file_operations proc_cgroupstats_operations;
>
> static struct backing_dev_info cgroup_backing_dev_info = {
> + .name = "cgroup",
> .capabilities = BDI_CAP_NO_ACCT_AND_WRITEBACK,
> };
>
> diff -Nraup linux-2.6.30-rc6/mm/backing-dev.c linux-2.6.30-rc6_bdiflusherv7/mm/backing-dev.c
> --- linux-2.6.30-rc6/mm/backing-dev.c 2009-05-19 11:00:58.000000000 +0800
> +++ linux-2.6.30-rc6_bdiflusherv7/mm/backing-dev.c 2009-05-27 08:59:27.000000000 +0800
> @@ -1,8 +1,11 @@
>
> #include <linux/wait.h>
> #include <linux/backing-dev.h>
> +#include <linux/kthread.h>
> +#include <linux/freezer.h>
> #include <linux/fs.h>
> #include <linux/pagemap.h>
> +#include <linux/mm.h>
> #include <linux/sched.h>
> #include <linux/module.h>
> #include <linux/writeback.h>
> @@ -14,14 +17,18 @@ void default_unplug_io_fn(struct backing
> EXPORT_SYMBOL(default_unplug_io_fn);
>
> struct backing_dev_info default_backing_dev_info = {
> + .name = "default",
> .ra_pages = VM_MAX_READAHEAD * 1024 / PAGE_CACHE_SIZE,
> .state = 0,
> - .capabilities = BDI_CAP_MAP_COPY,
> + .capabilities = BDI_CAP_MAP_COPY | BDI_CAP_FLUSH_FORKER,
> .unplug_io_fn = default_unplug_io_fn,
> };
> EXPORT_SYMBOL_GPL(default_backing_dev_info);
>
> static struct class *bdi_class;
> +DEFINE_MUTEX(bdi_lock);
> +LIST_HEAD(bdi_list);
> +LIST_HEAD(bdi_pending_list);
>
> #ifdef CONFIG_DEBUG_FS
> #include <linux/debugfs.h>
> @@ -37,9 +44,29 @@ static void bdi_debug_init(void)
> static int bdi_debug_stats_show(struct seq_file *m, void *v)
> {
> struct backing_dev_info *bdi = m->private;
> + struct bdi_writeback *wb;
> unsigned long background_thresh;
> unsigned long dirty_thresh;
> unsigned long bdi_thresh;
> + unsigned long nr_dirty, nr_io, nr_more_io, nr_wb;
> + struct inode *inode;
> +
> + /*
> + * inode lock is enough here, the bdi->wb_list is protected by
> + * RCU on the reader side
> + */
> + nr_wb = nr_dirty = nr_io = nr_more_io = 0;
> + spin_lock(&inode_lock);
> + list_for_each_entry(wb, &bdi->wb_list, list) {
> + nr_wb++;
> + list_for_each_entry(inode, &wb->b_dirty, i_list)
> + nr_dirty++;
> + list_for_each_entry(inode, &wb->b_io, i_list)
> + nr_io++;
> + list_for_each_entry(inode, &wb->b_more_io, i_list)
> + nr_more_io++;
> + }
> + spin_unlock(&inode_lock);
>
> get_dirty_limits(&background_thresh, &dirty_thresh, &bdi_thresh, bdi);
>
> @@ -49,12 +76,22 @@ static int bdi_debug_stats_show(struct s
> "BdiReclaimable: %8lu kB\n"
> "BdiDirtyThresh: %8lu kB\n"
> "DirtyThresh: %8lu kB\n"
> - "BackgroundThresh: %8lu kB\n",
> + "BackgroundThresh: %8lu kB\n"
> + "WriteBack threads:%8lu\n"
> + "b_dirty: %8lu\n"
> + "b_io: %8lu\n"
> + "b_more_io: %8lu\n"
> + "bdi_list: %8u\n"
> + "state: %8lx\n"
> + "wb_mask: %8lx\n"
> + "wb_list: %8u\n"
> + "wb_cnt: %8u\n",
> (unsigned long) K(bdi_stat(bdi, BDI_WRITEBACK)),
> (unsigned long) K(bdi_stat(bdi, BDI_RECLAIMABLE)),
> - K(bdi_thresh),
> - K(dirty_thresh),
> - K(background_thresh));
> + K(bdi_thresh), K(dirty_thresh),
> + K(background_thresh), nr_wb, nr_dirty, nr_io, nr_more_io,
> + !list_empty(&bdi->bdi_list), bdi->state, bdi->wb_mask,
> + !list_empty(&bdi->wb_list), bdi->wb_cnt);
> #undef K
>
> return 0;
> @@ -193,6 +230,343 @@ static int __init default_bdi_init(void)
> }
> subsys_initcall(default_bdi_init);
>
> +static int wb_assign_nr(struct backing_dev_info *bdi, struct bdi_writeback *wb)
> +{
> + unsigned long mask = BDI_MAX_FLUSHERS - 1;
> + unsigned int nr;
> +
> + do {
> + if ((bdi->wb_mask & mask) == mask)
> + return 1;
> +
> + nr = find_first_zero_bit(&bdi->wb_mask, BDI_MAX_FLUSHERS);
> + } while (test_and_set_bit(nr, &bdi->wb_mask));
> +
> + wb->nr = nr;
> +
> + spin_lock(&bdi->wb_lock);
> + bdi->wb_cnt++;
> + spin_unlock(&bdi->wb_lock);
> +
> + return 0;
> +}
> +
> +static void bdi_put_wb(struct backing_dev_info *bdi, struct bdi_writeback *wb)
> +{
> + clear_bit(wb->nr, &bdi->wb_mask);
> +
> + if (wb == &bdi->wb)
> + clear_bit(BDI_wb_alloc, &bdi->state);
> + else
> + kfree(wb);
> +
> + spin_lock(&bdi->wb_lock);
> + bdi->wb_cnt--;
> + spin_unlock(&bdi->wb_lock);
> +}
> +
> +static int bdi_wb_init(struct bdi_writeback *wb, struct backing_dev_info *bdi)
> +{
> + memset(wb, 0, sizeof(*wb));
> +
> + wb->bdi = bdi;
> + init_waitqueue_head(&wb->wait);
> + INIT_LIST_HEAD(&wb->b_dirty);
> + INIT_LIST_HEAD(&wb->b_io);
> + INIT_LIST_HEAD(&wb->b_more_io);
> +
> + return wb_assign_nr(bdi, wb);
> +}
> +
> +static struct bdi_writeback *bdi_new_wb(struct backing_dev_info *bdi)
> +{
> + struct bdi_writeback *wb;
> +
> + /*
> + * Default bdi->wb is already assigned, so just return it
> + */
> + if (!test_and_set_bit(BDI_wb_alloc, &bdi->state))
> + wb = &bdi->wb;
> + else {
> + wb = kmalloc(sizeof(struct bdi_writeback), GFP_KERNEL);
> + if (wb) {
> + if (bdi_wb_init(wb, bdi)) {
> + kfree(wb);
> + wb = NULL;
> + }
> + }
> + }
> +
> + return wb;
> +}
> +
> +static void bdi_task_init(struct backing_dev_info *bdi,
> + struct bdi_writeback *wb)
> +{
> + struct task_struct *tsk = current;
> + int was_empty;
> +
> + /*
> + * Add us to the active bdi_list. If we are adding threads beyond
> + * the default embedded bdi_writeback, then we need to start using
> + * proper locking. Check the list for empty first, then set the
> + * BDI_wblist_lock flag if there's > 1 entry on the list now
> + */
> + spin_lock(&bdi->wb_lock);
> +
> + was_empty = list_empty(&bdi->wb_list);
> + list_add_tail_rcu(&wb->list, &bdi->wb_list);
> + if (!was_empty)
> + set_bit(BDI_wblist_lock, &bdi->state);
> +
> + spin_unlock(&bdi->wb_lock);
> +
> + tsk->flags |= PF_FLUSHER | PF_SWAPWRITE;
> + set_freezable();
> +
> + /*
> + * Our parent may run at a different priority, just set us to normal
> + */
> + set_user_nice(tsk, 0);
> +}
> +
> +static int bdi_start_fn(void *ptr)
> +{
> + struct bdi_writeback *wb = ptr;
> + struct backing_dev_info *bdi = wb->bdi;
> + int ret;
> +
> + /*
> + * Add us to the active bdi_list
> + */
> + mutex_lock(&bdi_lock);
> + list_add(&bdi->bdi_list, &bdi_list);
> + mutex_unlock(&bdi_lock);
> +
> + bdi_task_init(bdi, wb);
> +
> + /*
> + * Clear pending bit and wakeup anybody waiting to tear us down
> + */
> + clear_bit(BDI_pending, &bdi->state);
> + smp_mb__after_clear_bit();
> + wake_up_bit(&bdi->state, BDI_pending);
> +
> + ret = bdi_writeback_task(wb);
> +
> + /*
> + * Remove us from the list
> + */
> + spin_lock(&bdi->wb_lock);
> + list_del_rcu(&wb->list);
> + spin_unlock(&bdi->wb_lock);
> +
> + /*
> + * wait for rcu grace period to end, so we can free wb
> + */
> + synchronize_srcu(&bdi->srcu);
> +
> + bdi_put_wb(bdi, wb);
> + return ret;
> +}
> +
> +int bdi_has_dirty_io(struct backing_dev_info *bdi)
> +{
> + struct bdi_writeback *wb;
> + int ret = 0;
> +
> + if (!bdi_wblist_needs_lock(bdi))
> + ret = wb_has_dirty_io(&bdi->wb);
> + else {
> + int idx;
> +
> + idx = srcu_read_lock(&bdi->srcu);
> +
> + list_for_each_entry_rcu(wb, &bdi->wb_list, list) {
> + ret = wb_has_dirty_io(wb);
> + if (ret)
> + break;
> + }
> +
> + srcu_read_unlock(&bdi->srcu, idx);
> + }
> +
> + return ret;
> +}
> +
> +static void bdi_flush_io(struct backing_dev_info *bdi)
> +{
> + struct writeback_control wbc = {
> + .bdi = bdi,
> + .sync_mode = WB_SYNC_NONE,
> + .older_than_this = NULL,
> + .range_cyclic = 1,
> + .nr_to_write = 1024,
> + };
> +
> + generic_sync_bdi_inodes(NULL, &wbc);
> +}
> +
> +static int bdi_forker_task(void *ptr)
> +{
> + struct bdi_writeback *me = ptr;
> + DEFINE_WAIT(wait);
> +
> + bdi_task_init(me->bdi, me);
> +
> + for (;;) {
> + struct backing_dev_info *bdi, *tmp;
> + struct bdi_writeback *wb;
> +
> + /*
> + * Do this periodically, like kupdated() did before.
> + */
> + sync_supers();
> +
> + /*
> + * Temporary measure, we want to make sure we don't see
> + * dirty data on the default backing_dev_info
> + */
> + if (wb_has_dirty_io(me) || !list_empty(&me->bdi->work_list))
> + wb_do_writeback(me);
> +
> + prepare_to_wait(&me->wait, &wait, TASK_INTERRUPTIBLE);
> +
> + mutex_lock(&bdi_lock);
> +
> + /*
> + * Check if any existing bdi's have dirty data without
> + * a thread registered. If so, set that up.
> + */
> + list_for_each_entry_safe(bdi, tmp, &bdi_list, bdi_list) {
> + if (bdi->wb.task || !bdi_has_dirty_io(bdi))
> + continue;
> +
> + bdi_add_default_flusher_task(bdi);
> + }
> +
> + if (list_empty(&bdi_pending_list)) {
> + unsigned long wait;
> +
> + mutex_unlock(&bdi_lock);
> + wait = msecs_to_jiffies(dirty_writeback_interval * 10);
> + schedule_timeout(wait);
> + try_to_freeze();
> + continue;
> + }
> +
> + /*
> + * This is our real job - check for pending entries in
> + * bdi_pending_list, and create the tasks that got added
> + */
> + bdi = list_entry(bdi_pending_list.next, struct backing_dev_info,
> + bdi_list);
> + list_del_init(&bdi->bdi_list);
> + mutex_unlock(&bdi_lock);
> +
> + wb = bdi_new_wb(bdi);
> + if (!wb)
> + goto readd_flush;
> +
> + wb->task = kthread_run(bdi_start_fn, wb, "bdi-%s",
> + dev_name(bdi->dev));
> +
> + /*
> + * If task creation fails, then readd the bdi to
> + * the pending list and force writeout of the bdi
> + * from this forker thread. That will free some memory
> + * and we can try again.
> + */
> + if (!wb->task) {
> + bdi_put_wb(bdi, wb);
> +readd_flush:
> + /*
> + * Add this 'bdi' to the back, so we get
> + * a chance to flush other bdi's to free
> + * memory.
> + */
> + mutex_lock(&bdi_lock);
> + list_add_tail(&bdi->bdi_list, &bdi_pending_list);
> + mutex_unlock(&bdi_lock);
> +
> + bdi_flush_io(bdi);
> + }
> + }
> +
> + finish_wait(&me->wait, &wait);
> + return 0;
> +}
> +
> +/*
> + * bdi_lock held on entry
> + */
> +static void bdi_add_one_flusher_task(struct backing_dev_info *bdi,
> + int(*func)(struct backing_dev_info *))
> +{
> + if (!bdi_cap_writeback_dirty(bdi))
> + return;
> +
> + if (WARN_ON(!test_bit(BDI_registered, &bdi->state))) {
> + printk("bdi %p/%s is not registered!\n", bdi, bdi->name);
> + return;
> + }
> +
> + /*
> + * Check with the helper whether to proceed adding a task. Will only
> + * abort if we two or more simultanous calls to
> + * bdi_add_default_flusher_task() occured, further additions will block
> + * waiting for previous additions to finish.
> + */
> + if (!func(bdi)) {
> + list_move_tail(&bdi->bdi_list, &bdi_pending_list);
> +
> + /*
> + * We are now on the pending list, wake up bdi_forker_task()
> + * to finish the job and add us back to the active bdi_list
> + */
> + wake_up(&default_backing_dev_info.wb.wait);
> + }
> +}
> +
> +static int flusher_add_helper_block(struct backing_dev_info *bdi)
> +{
> + mutex_unlock(&bdi_lock);
> + wait_on_bit_lock(&bdi->state, BDI_pending, bdi_sched_wait,
> + TASK_UNINTERRUPTIBLE);
> + mutex_lock(&bdi_lock);
> + return 0;
> +}
> +
> +static int flusher_add_helper_test(struct backing_dev_info *bdi)
> +{
> + return test_and_set_bit(BDI_pending, &bdi->state);
> +}
> +
> +/*
> + * Add the default flusher task that gets created for any bdi
> + * that has dirty data pending writeout
> + */
> +void bdi_add_default_flusher_task(struct backing_dev_info *bdi)
> +{
> + bdi_add_one_flusher_task(bdi, flusher_add_helper_test);
> +}
> +
> +/**
> + * bdi_add_flusher_task - add one more flusher task to this @bdi
> + * @bdi: the bdi
> + *
> + * Add an additional flusher task to this @bdi. Will block waiting on
> + * previous additions, if any.
> + *
> + */
> +void bdi_add_flusher_task(struct backing_dev_info *bdi)
> +{
> + mutex_lock(&bdi_lock);
> + bdi_add_one_flusher_task(bdi, flusher_add_helper_block);
> + mutex_unlock(&bdi_lock);
> +}
> +EXPORT_SYMBOL(bdi_add_flusher_task);
> +
> int bdi_register(struct backing_dev_info *bdi, struct device *parent,
> const char *fmt, ...)
> {
> @@ -211,9 +585,41 @@ int bdi_register(struct backing_dev_info
> goto exit;
> }
>
> + mutex_lock(&bdi_lock);
> + list_add_tail(&bdi->bdi_list, &bdi_list);
> + mutex_unlock(&bdi_lock);
> +
> bdi->dev = dev;
> - bdi_debug_register(bdi, dev_name(dev));
>
> + /*
> + * Just start the forker thread for our default backing_dev_info,
> + * and add other bdi's to the list. They will get a thread created
> + * on-demand when they need it.
> + */
> + if (bdi_cap_flush_forker(bdi)) {
> + struct bdi_writeback *wb;
> +
> + wb = bdi_new_wb(bdi);
> + if (!wb) {
> + ret = -ENOMEM;
> + goto remove_err;
> + }
> +
> + wb->task = kthread_run(bdi_forker_task, wb, "bdi-%s",
> + dev_name(dev));
> + if (!wb->task) {
> + bdi_put_wb(bdi, wb);
> + ret = -ENOMEM;
> +remove_err:
> + mutex_lock(&bdi_lock);
> + list_del(&bdi->bdi_list);
> + mutex_unlock(&bdi_lock);
> + goto exit;
> + }
> + }
> +
> + bdi_debug_register(bdi, dev_name(dev));
> + set_bit(BDI_registered, &bdi->state);
> exit:
> return ret;
> }
> @@ -225,9 +631,42 @@ int bdi_register_dev(struct backing_dev_
> }
> EXPORT_SYMBOL(bdi_register_dev);
>
> +/*
> + * Remove bdi from global list and shutdown any threads we have running
> + */
> +static void bdi_wb_shutdown(struct backing_dev_info *bdi)
> +{
> + struct bdi_writeback *wb;
> +
> + if (!bdi_cap_writeback_dirty(bdi))
> + return;
> +
> + /*
> + * If setup is pending, wait for that to complete first
> + */
> + wait_on_bit(&bdi->state, BDI_pending, bdi_sched_wait,
> + TASK_UNINTERRUPTIBLE);
> +
> + /*
> + * Make sure nobody finds us on the bdi_list anymore
> + */
> + mutex_lock(&bdi_lock);
> + list_del(&bdi->bdi_list);
> + mutex_unlock(&bdi_lock);
> +
> + /*
> + * Finally, kill the kernel threads. We don't need to be RCU
> + * safe anymore, since the bdi is gone from visibility.
> + */
> + list_for_each_entry(wb, &bdi->wb_list, list)
> + kthread_stop(wb->task);
> +}
> +
> void bdi_unregister(struct backing_dev_info *bdi)
> {
> if (bdi->dev) {
> + if (!bdi_cap_flush_forker(bdi))
> + bdi_wb_shutdown(bdi);
> bdi_debug_unregister(bdi);
> device_unregister(bdi->dev);
> bdi->dev = NULL;
> @@ -237,14 +676,21 @@ EXPORT_SYMBOL(bdi_unregister);
>
> int bdi_init(struct backing_dev_info *bdi)
> {
> - int i;
> - int err;
> + int i, err;
>
> bdi->dev = NULL;
>
> bdi->min_ratio = 0;
> bdi->max_ratio = 100;
> bdi->max_prop_frac = PROP_FRAC_BASE;
> + spin_lock_init(&bdi->wb_lock);
> + bdi->wb_mask = 0;
> + bdi->wb_cnt = 0;
> + INIT_LIST_HEAD(&bdi->bdi_list);
> + INIT_LIST_HEAD(&bdi->wb_list);
> + INIT_LIST_HEAD(&bdi->work_list);
> +
> + bdi_wb_init(&bdi->wb, bdi);
>
> for (i = 0; i < NR_BDI_STAT_ITEMS; i++) {
> err = percpu_counter_init(&bdi->bdi_stat[i], 0);
> @@ -252,10 +698,15 @@ int bdi_init(struct backing_dev_info *bd
> goto err;
> }
>
> + err = init_srcu_struct(&bdi->srcu);
> + if (err)
> + goto err;
> +
> bdi->dirty_exceeded = 0;
> err = prop_local_init_percpu(&bdi->completions);
>
> if (err) {
> + cleanup_srcu_struct(&bdi->srcu);
> err:
> while (i--)
> percpu_counter_destroy(&bdi->bdi_stat[i]);
> @@ -269,8 +720,12 @@ void bdi_destroy(struct backing_dev_info
> {
> int i;
>
> + WARN_ON(bdi_has_dirty_io(bdi));
> +
> bdi_unregister(bdi);
>
> + cleanup_srcu_struct(&bdi->srcu);
> +
> for (i = 0; i < NR_BDI_STAT_ITEMS; i++)
> percpu_counter_destroy(&bdi->bdi_stat[i]);
>
> diff -Nraup linux-2.6.30-rc6/mm/Makefile linux-2.6.30-rc6_bdiflusherv7/mm/Makefile
> --- linux-2.6.30-rc6/mm/Makefile 2009-05-19 11:00:58.000000000 +0800
> +++ linux-2.6.30-rc6_bdiflusherv7/mm/Makefile 2009-05-27 08:59:27.000000000 +0800
> @@ -8,7 +8,7 @@ mmu-$(CONFIG_MMU) := fremap.o highmem.o
> vmalloc.o
>
> obj-y := bootmem.o filemap.o mempool.o oom_kill.o fadvise.o \
> - maccess.o page_alloc.o page-writeback.o pdflush.o \
> + maccess.o page_alloc.o page-writeback.o \
> readahead.o swap.o truncate.o vmscan.o shmem.o \
> prio_tree.o util.o mmzone.o vmstat.o backing-dev.o \
> page_isolation.o mm_init.o $(mmu-y)
> diff -Nraup linux-2.6.30-rc6/mm/page-writeback.c linux-2.6.30-rc6_bdiflusherv7/mm/page-writeback.c
> --- linux-2.6.30-rc6/mm/page-writeback.c 2009-05-19 11:00:58.000000000 +0800
> +++ linux-2.6.30-rc6_bdiflusherv7/mm/page-writeback.c 2009-05-27 09:05:08.000000000 +0800
> @@ -36,15 +36,6 @@
> #include <linux/pagevec.h>
>
> /*
> - * The maximum number of pages to writeout in a single bdflush/kupdate
> - * operation. We do this so we don't hold I_SYNC against an inode for
> - * enormous amounts of time, which would block a userspace task which has
> - * been forced to throttle against that inode. Also, the code reevaluates
> - * the dirty each time it has written this many pages.
> - */
> -#define MAX_WRITEBACK_PAGES 1024
> -
> -/*
> * After a CPU has dirtied this many pages, balance_dirty_pages_ratelimited
> * will look to see if it needs to force writeback or throttling.
> */
> @@ -117,8 +108,6 @@ EXPORT_SYMBOL(laptop_mode);
> /* End of sysctl-exported parameters */
>
>
> -static void background_writeout(unsigned long _min_pages);
> -
> /*
> * Scale the writeback cache size proportional to the relative writeout speeds.
> *
> @@ -319,15 +308,13 @@ static void task_dirty_limit(struct task
> /*
> *
> */
> -static DEFINE_SPINLOCK(bdi_lock);
> static unsigned int bdi_min_ratio;
>
> int bdi_set_min_ratio(struct backing_dev_info *bdi, unsigned int min_ratio)
> {
> int ret = 0;
> - unsigned long flags;
>
> - spin_lock_irqsave(&bdi_lock, flags);
> + mutex_lock(&bdi_lock);
> if (min_ratio > bdi->max_ratio) {
> ret = -EINVAL;
> } else {
> @@ -339,27 +326,26 @@ int bdi_set_min_ratio(struct backing_dev
> ret = -EINVAL;
> }
> }
> - spin_unlock_irqrestore(&bdi_lock, flags);
> + mutex_unlock(&bdi_lock);
>
> return ret;
> }
>
> int bdi_set_max_ratio(struct backing_dev_info *bdi, unsigned max_ratio)
> {
> - unsigned long flags;
> int ret = 0;
>
> if (max_ratio > 100)
> return -EINVAL;
>
> - spin_lock_irqsave(&bdi_lock, flags);
> + mutex_lock(&bdi_lock);
> if (bdi->min_ratio > max_ratio) {
> ret = -EINVAL;
> } else {
> bdi->max_ratio = max_ratio;
> bdi->max_prop_frac = (PROP_FRAC_BASE * max_ratio) / 100;
> }
> - spin_unlock_irqrestore(&bdi_lock, flags);
> + mutex_unlock(&bdi_lock);
>
> return ret;
> }
> @@ -542,7 +528,7 @@ static void balance_dirty_pages(struct a
> * been flushed to permanent storage.
> */
> if (bdi_nr_reclaimable) {
> - writeback_inodes(&wbc);
> + generic_sync_bdi_inodes(NULL, &wbc);
> pages_written += write_chunk - wbc.nr_to_write;
> get_dirty_limits(&background_thresh, &dirty_thresh,
> &bdi_thresh, bdi);
> @@ -593,7 +579,7 @@ static void balance_dirty_pages(struct a
> (!laptop_mode && (global_page_state(NR_FILE_DIRTY)
> + global_page_state(NR_UNSTABLE_NFS)
> > background_thresh)))
> - pdflush_operation(background_writeout, 0);
> + bdi_start_writeback(bdi, NULL, 0, WB_SYNC_NONE);
> }
>
> void set_page_dirty_balance(struct page *page, int page_mkwrite)
> @@ -678,152 +664,34 @@ void throttle_vm_writeout(gfp_t gfp_mask
> }
>
> /*
> - * writeback at least _min_pages, and keep writing until the amount of dirty
> - * memory is less than the background threshold, or until we're all clean.
> - */
> -static void background_writeout(unsigned long _min_pages)
> -{
> - long min_pages = _min_pages;
> - struct writeback_control wbc = {
> - .bdi = NULL,
> - .sync_mode = WB_SYNC_NONE,
> - .older_than_this = NULL,
> - .nr_to_write = 0,
> - .nonblocking = 1,
> - .range_cyclic = 1,
> - };
> -
> - for ( ; ; ) {
> - unsigned long background_thresh;
> - unsigned long dirty_thresh;
> -
> - get_dirty_limits(&background_thresh, &dirty_thresh, NULL, NULL);
> - if (global_page_state(NR_FILE_DIRTY) +
> - global_page_state(NR_UNSTABLE_NFS) < background_thresh
> - && min_pages <= 0)
> - break;
> - wbc.more_io = 0;
> - wbc.encountered_congestion = 0;
> - wbc.nr_to_write = MAX_WRITEBACK_PAGES;
> - wbc.pages_skipped = 0;
> - writeback_inodes(&wbc);
> - min_pages -= MAX_WRITEBACK_PAGES - wbc.nr_to_write;
> - if (wbc.nr_to_write > 0 || wbc.pages_skipped > 0) {
> - /* Wrote less than expected */
> - if (wbc.encountered_congestion || wbc.more_io)
> - congestion_wait(WRITE, HZ/10);
> - else
> - break;
> - }
> - }
> -}
> -
> -/*
> * Start writeback of `nr_pages' pages. If `nr_pages' is zero, write back
> - * the whole world. Returns 0 if a pdflush thread was dispatched. Returns
> - * -1 if all pdflush threads were busy.
> + * the whole world.
> */
> -int wakeup_pdflush(long nr_pages)
> +void wakeup_flusher_threads(long nr_pages)
> {
> if (nr_pages == 0)
> nr_pages = global_page_state(NR_FILE_DIRTY) +
> global_page_state(NR_UNSTABLE_NFS);
> - return pdflush_operation(background_writeout, nr_pages);
> + bdi_writeback_all(NULL, nr_pages, WB_SYNC_NONE);
> }
>
> -static void wb_timer_fn(unsigned long unused);
> static void laptop_timer_fn(unsigned long unused);
>
> -static DEFINE_TIMER(wb_timer, wb_timer_fn, 0, 0);
> static DEFINE_TIMER(laptop_mode_wb_timer, laptop_timer_fn, 0, 0);
>
> /*
> - * Periodic writeback of "old" data.
> - *
> - * Define "old": the first time one of an inode's pages is dirtied, we mark the
> - * dirtying-time in the inode's address_space. So this periodic writeback code
> - * just walks the superblock inode list, writing back any inodes which are
> - * older than a specific point in time.
> - *
> - * Try to run once per dirty_writeback_interval. But if a writeback event
> - * takes longer than a dirty_writeback_interval interval, then leave a
> - * one-second gap.
> - *
> - * older_than_this takes precedence over nr_to_write. So we'll only write back
> - * all dirty pages if they are all attached to "old" mappings.
> - */
> -static void wb_kupdate(unsigned long arg)
> -{
> - unsigned long oldest_jif;
> - unsigned long start_jif;
> - unsigned long next_jif;
> - long nr_to_write;
> - struct writeback_control wbc = {
> - .bdi = NULL,
> - .sync_mode = WB_SYNC_NONE,
> - .older_than_this = &oldest_jif,
> - .nr_to_write = 0,
> - .nonblocking = 1,
> - .for_kupdate = 1,
> - .range_cyclic = 1,
> - };
> -
> - sync_supers();
> -
> - oldest_jif = jiffies - msecs_to_jiffies(dirty_expire_interval);
> - start_jif = jiffies;
> - next_jif = start_jif + msecs_to_jiffies(dirty_writeback_interval * 10);
> - nr_to_write = global_page_state(NR_FILE_DIRTY) +
> - global_page_state(NR_UNSTABLE_NFS) +
> - (inodes_stat.nr_inodes - inodes_stat.nr_unused);
> - while (nr_to_write > 0) {
> - wbc.more_io = 0;
> - wbc.encountered_congestion = 0;
> - wbc.nr_to_write = MAX_WRITEBACK_PAGES;
> - writeback_inodes(&wbc);
> - if (wbc.nr_to_write > 0) {
> - if (wbc.encountered_congestion || wbc.more_io)
> - congestion_wait(WRITE, HZ/10);
> - else
> - break; /* All the old data is written */
> - }
> - nr_to_write -= MAX_WRITEBACK_PAGES - wbc.nr_to_write;
> - }
> - if (time_before(next_jif, jiffies + HZ))
> - next_jif = jiffies + HZ;
> - if (dirty_writeback_interval)
> - mod_timer(&wb_timer, next_jif);
> -}
> -
> -/*
> * sysctl handler for /proc/sys/vm/dirty_writeback_centisecs
> */
> int dirty_writeback_centisecs_handler(ctl_table *table, int write,
> struct file *file, void __user *buffer, size_t *length, loff_t *ppos)
> {
> proc_dointvec(table, write, file, buffer, length, ppos);
> - if (dirty_writeback_interval)
> - mod_timer(&wb_timer, jiffies +
> - msecs_to_jiffies(dirty_writeback_interval * 10));
> - else
> - del_timer(&wb_timer);
> return 0;
> }
>
> -static void wb_timer_fn(unsigned long unused)
> -{
> - if (pdflush_operation(wb_kupdate, 0) < 0)
> - mod_timer(&wb_timer, jiffies + HZ); /* delay 1 second */
> -}
> -
> -static void laptop_flush(unsigned long unused)
> -{
> - sys_sync();
> -}
> -
> static void laptop_timer_fn(unsigned long unused)
> {
> - pdflush_operation(laptop_flush, 0);
> + wakeup_flusher_threads(0);
> }
>
> /*
> @@ -906,8 +774,6 @@ void __init page_writeback_init(void)
> {
> int shift;
>
> - mod_timer(&wb_timer,
> - jiffies + msecs_to_jiffies(dirty_writeback_interval * 10));
> writeback_set_ratelimit();
> register_cpu_notifier(&ratelimit_nb);
>
> diff -Nraup linux-2.6.30-rc6/mm/pdflush.c linux-2.6.30-rc6_bdiflusherv7/mm/pdflush.c
> --- linux-2.6.30-rc6/mm/pdflush.c 2009-05-19 11:00:58.000000000 +0800
> +++ linux-2.6.30-rc6_bdiflusherv7/mm/pdflush.c 1970-01-01 08:00:00.000000000 +0800
> @@ -1,269 +0,0 @@
> -/*
> - * mm/pdflush.c - worker threads for writing back filesystem data
> - *
> - * Copyright (C) 2002, Linus Torvalds.
> - *
> - * 09Apr2002 Andrew Morton
> - * Initial version
> - * 29Feb2004 kaos@sgi.com
> - * Move worker thread creation to kthread to avoid chewing
> - * up stack space with nested calls to kernel_thread.
> - */
> -
> -#include <linux/sched.h>
> -#include <linux/list.h>
> -#include <linux/signal.h>
> -#include <linux/spinlock.h>
> -#include <linux/gfp.h>
> -#include <linux/init.h>
> -#include <linux/module.h>
> -#include <linux/fs.h> /* Needed by writeback.h */
> -#include <linux/writeback.h> /* Prototypes pdflush_operation() */
> -#include <linux/kthread.h>
> -#include <linux/cpuset.h>
> -#include <linux/freezer.h>
> -
> -
> -/*
> - * Minimum and maximum number of pdflush instances
> - */
> -#define MIN_PDFLUSH_THREADS 2
> -#define MAX_PDFLUSH_THREADS 8
> -
> -static void start_one_pdflush_thread(void);
> -
> -
> -/*
> - * The pdflush threads are worker threads for writing back dirty data.
> - * Ideally, we'd like one thread per active disk spindle. But the disk
> - * topology is very hard to divine at this level. Instead, we take
> - * care in various places to prevent more than one pdflush thread from
> - * performing writeback against a single filesystem. pdflush threads
> - * have the PF_FLUSHER flag set in current->flags to aid in this.
> - */
> -
> -/*
> - * All the pdflush threads. Protected by pdflush_lock
> - */
> -static LIST_HEAD(pdflush_list);
> -static DEFINE_SPINLOCK(pdflush_lock);
> -
> -/*
> - * The count of currently-running pdflush threads. Protected
> - * by pdflush_lock.
> - *
> - * Readable by sysctl, but not writable. Published to userspace at
> - * /proc/sys/vm/nr_pdflush_threads.
> - */
> -int nr_pdflush_threads = 0;
> -
> -/*
> - * The time at which the pdflush thread pool last went empty
> - */
> -static unsigned long last_empty_jifs;
> -
> -/*
> - * The pdflush thread.
> - *
> - * Thread pool management algorithm:
> - *
> - * - The minimum and maximum number of pdflush instances are bound
> - * by MIN_PDFLUSH_THREADS and MAX_PDFLUSH_THREADS.
> - *
> - * - If there have been no idle pdflush instances for 1 second, create
> - * a new one.
> - *
> - * - If the least-recently-went-to-sleep pdflush thread has been asleep
> - * for more than one second, terminate a thread.
> - */
> -
> -/*
> - * A structure for passing work to a pdflush thread. Also for passing
> - * state information between pdflush threads. Protected by pdflush_lock.
> - */
> -struct pdflush_work {
> - struct task_struct *who; /* The thread */
> - void (*fn)(unsigned long); /* A callback function */
> - unsigned long arg0; /* An argument to the callback */
> - struct list_head list; /* On pdflush_list, when idle */
> - unsigned long when_i_went_to_sleep;
> -};
> -
> -static int __pdflush(struct pdflush_work *my_work)
> -{
> - current->flags |= PF_FLUSHER | PF_SWAPWRITE;
> - set_freezable();
> - my_work->fn = NULL;
> - my_work->who = current;
> - INIT_LIST_HEAD(&my_work->list);
> -
> - spin_lock_irq(&pdflush_lock);
> - for ( ; ; ) {
> - struct pdflush_work *pdf;
> -
> - set_current_state(TASK_INTERRUPTIBLE);
> - list_move(&my_work->list, &pdflush_list);
> - my_work->when_i_went_to_sleep = jiffies;
> - spin_unlock_irq(&pdflush_lock);
> - schedule();
> - try_to_freeze();
> - spin_lock_irq(&pdflush_lock);
> - if (!list_empty(&my_work->list)) {
> - /*
> - * Someone woke us up, but without removing our control
> - * structure from the global list. swsusp will do this
> - * in try_to_freeze()->refrigerator(). Handle it.
> - */
> - my_work->fn = NULL;
> - continue;
> - }
> - if (my_work->fn == NULL) {
> - printk("pdflush: bogus wakeup\n");
> - continue;
> - }
> - spin_unlock_irq(&pdflush_lock);
> -
> - (*my_work->fn)(my_work->arg0);
> -
> - spin_lock_irq(&pdflush_lock);
> -
> - /*
> - * Thread creation: For how long have there been zero
> - * available threads?
> - *
> - * To throttle creation, we reset last_empty_jifs.
> - */
> - if (time_after(jiffies, last_empty_jifs + 1 * HZ)) {
> - if (list_empty(&pdflush_list)) {
> - if (nr_pdflush_threads < MAX_PDFLUSH_THREADS) {
> - last_empty_jifs = jiffies;
> - nr_pdflush_threads++;
> - spin_unlock_irq(&pdflush_lock);
> - start_one_pdflush_thread();
> - spin_lock_irq(&pdflush_lock);
> - }
> - }
> - }
> -
> - my_work->fn = NULL;
> -
> - /*
> - * Thread destruction: For how long has the sleepiest
> - * thread slept?
> - */
> - if (list_empty(&pdflush_list))
> - continue;
> - if (nr_pdflush_threads <= MIN_PDFLUSH_THREADS)
> - continue;
> - pdf = list_entry(pdflush_list.prev, struct pdflush_work, list);
> - if (time_after(jiffies, pdf->when_i_went_to_sleep + 1 * HZ)) {
> - /* Limit exit rate */
> - pdf->when_i_went_to_sleep = jiffies;
> - break; /* exeunt */
> - }
> - }
> - nr_pdflush_threads--;
> - spin_unlock_irq(&pdflush_lock);
> - return 0;
> -}
> -
> -/*
> - * Of course, my_work wants to be just a local in __pdflush(). It is
> - * separated out in this manner to hopefully prevent the compiler from
> - * performing unfortunate optimisations against the auto variables. Because
> - * these are visible to other tasks and CPUs. (No problem has actually
> - * been observed. This is just paranoia).
> - */
> -static int pdflush(void *dummy)
> -{
> - struct pdflush_work my_work;
> - cpumask_var_t cpus_allowed;
> -
> - /*
> - * Since the caller doesn't even check kthread_run() worked, let's not
> - * freak out too much if this fails.
> - */
> - if (!alloc_cpumask_var(&cpus_allowed, GFP_KERNEL)) {
> - printk(KERN_WARNING "pdflush failed to allocate cpumask\n");
> - return 0;
> - }
> -
> - /*
> - * pdflush can spend a lot of time doing encryption via dm-crypt. We
> - * don't want to do that at keventd's priority.
> - */
> - set_user_nice(current, 0);
> -
> - /*
> - * Some configs put our parent kthread in a limited cpuset,
> - * which kthread() overrides, forcing cpus_allowed == cpu_all_mask.
> - * Our needs are more modest - cut back to our cpusets cpus_allowed.
> - * This is needed as pdflush's are dynamically created and destroyed.
> - * The boottime pdflush's are easily placed w/o these 2 lines.
> - */
> - cpuset_cpus_allowed(current, cpus_allowed);
> - set_cpus_allowed_ptr(current, cpus_allowed);
> - free_cpumask_var(cpus_allowed);
> -
> - return __pdflush(&my_work);
> -}
> -
> -/*
> - * Attempt to wake up a pdflush thread, and get it to do some work for you.
> - * Returns zero if it indeed managed to find a worker thread, and passed your
> - * payload to it.
> - */
> -int pdflush_operation(void (*fn)(unsigned long), unsigned long arg0)
> -{
> - unsigned long flags;
> - int ret = 0;
> -
> - BUG_ON(fn == NULL); /* Hard to diagnose if it's deferred */
> -
> - spin_lock_irqsave(&pdflush_lock, flags);
> - if (list_empty(&pdflush_list)) {
> - ret = -1;
> - } else {
> - struct pdflush_work *pdf;
> -
> - pdf = list_entry(pdflush_list.next, struct pdflush_work, list);
> - list_del_init(&pdf->list);
> - if (list_empty(&pdflush_list))
> - last_empty_jifs = jiffies;
> - pdf->fn = fn;
> - pdf->arg0 = arg0;
> - wake_up_process(pdf->who);
> - }
> - spin_unlock_irqrestore(&pdflush_lock, flags);
> -
> - return ret;
> -}
> -
> -static void start_one_pdflush_thread(void)
> -{
> - struct task_struct *k;
> -
> - k = kthread_run(pdflush, NULL, "pdflush");
> - if (unlikely(IS_ERR(k))) {
> - spin_lock_irq(&pdflush_lock);
> - nr_pdflush_threads--;
> - spin_unlock_irq(&pdflush_lock);
> - }
> -}
> -
> -static int __init pdflush_init(void)
> -{
> - int i;
> -
> - /*
> - * Pre-set nr_pdflush_threads... If we fail to create,
> - * the count will be decremented.
> - */
> - nr_pdflush_threads = MIN_PDFLUSH_THREADS;
> -
> - for (i = 0; i < MIN_PDFLUSH_THREADS; i++)
> - start_one_pdflush_thread();
> - return 0;
> -}
> -
> -module_init(pdflush_init);
> diff -Nraup linux-2.6.30-rc6/mm/swap_state.c linux-2.6.30-rc6_bdiflusherv7/mm/swap_state.c
> --- linux-2.6.30-rc6/mm/swap_state.c 2009-05-19 11:00:28.000000000 +0800
> +++ linux-2.6.30-rc6_bdiflusherv7/mm/swap_state.c 2009-05-27 08:59:27.000000000 +0800
> @@ -34,6 +34,7 @@ static const struct address_space_operat
> };
>
> static struct backing_dev_info swap_backing_dev_info = {
> + .name = "swap",
> .capabilities = BDI_CAP_NO_ACCT_AND_WRITEBACK | BDI_CAP_SWAP_BACKED,
> .unplug_io_fn = swap_unplug_io_fn,
> };
> diff -Nraup linux-2.6.30-rc6/mm/vmscan.c linux-2.6.30-rc6_bdiflusherv7/mm/vmscan.c
> --- linux-2.6.30-rc6/mm/vmscan.c 2009-05-19 11:00:58.000000000 +0800
> +++ linux-2.6.30-rc6_bdiflusherv7/mm/vmscan.c 2009-05-27 08:59:27.000000000 +0800
> @@ -1654,7 +1654,7 @@ static unsigned long do_try_to_free_page
> */
> if (total_scanned > sc->swap_cluster_max +
> sc->swap_cluster_max / 2) {
> - wakeup_pdflush(laptop_mode ? 0 : total_scanned);
> + wakeup_flusher_threads(laptop_mode ? 0 : total_scanned);
> sc->may_writepage = 1;
> }
>
--
Jens Axboe
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCH 0/12] Per-bdi writeback flusher threads v7
2009-05-27 5:21 ` Damien Wyart
2009-05-27 5:49 ` Damien Wyart
@ 2009-05-27 6:17 ` Jens Axboe
1 sibling, 0 replies; 33+ messages in thread
From: Jens Axboe @ 2009-05-27 6:17 UTC (permalink / raw)
To: Damien Wyart
Cc: linux-kernel, linux-fsdevel, chris.mason, david, hch, akpm, jack,
yanmin_zhang, richard
On Wed, May 27 2009, Damien Wyart wrote:
> * Jens Axboe <jens.axboe@oracle.com> [2009-05-26 23:11]:
> > > OK, I spotted the problem. If we fallback to the on-stack allocation in
> > > bdi_writeback_all(), then we do the wait for the work completion with
> > > the bdi_lock mutex held. This can deadlock with bdi_forker_task(), so if
> > > we require that to be invoked to make progress (happens if a thread
> > > needs to be restarted), then we have a deadlock on that mutex.
>
> > > I'll cook up a fix for this, but probably not before the morning.
>
> > Untested fix. I think it should work, but I haven't run it here yet.
>
> Thanks for your feedback and explanations. Will not be able to test the
> patch before the afternoon. Will then give feedback as soon as I can. Of
> course, I can also test a new fix or series if you send one in the
> morning, after your own testing.
Just wait for the new series, I'll be posting it this morning. And
thanks for your testing and report!
--
Jens Axboe
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCH 0/12] Per-bdi writeback flusher threads v7
2009-05-27 5:49 ` Damien Wyart
@ 2009-05-27 9:20 ` Jens Axboe
2009-05-27 13:15 ` Damien Wyart
0 siblings, 1 reply; 33+ messages in thread
From: Jens Axboe @ 2009-05-27 9:20 UTC (permalink / raw)
To: Damien Wyart
Cc: linux-kernel, linux-fsdevel, chris.mason, david, hch, akpm, jack,
yanmin_zhang, richard
On Wed, May 27 2009, Damien Wyart wrote:
> > > > OK, I spotted the problem. If we fallback to the on-stack allocation in
> > > > bdi_writeback_all(), then we do the wait for the work completion with
> > > > the bdi_lock mutex held. This can deadlock with bdi_forker_task(), so if
> > > > we require that to be invoked to make progress (happens if a thread
> > > > needs to be restarted), then we have a deadlock on that mutex.
>
> > > > I'll cook up a fix for this, but probably not before the morning.
>
> > > Untested fix. I think it should work, but I haven't run it here yet.
>
> > Thanks for your feedback and explanations. Will not be able to test the
> > patch before the afternoon. Will then give feedback as soon as I can. Of
> > course, I can also test a new fix or series if you send one in the
> > morning, after your own testing.
>
> Just ran a quick test with your fix applied and ran into the same
> problem: manual sync stays in D state. I could capture SysRq-T, I am
> attaching part of the output (which was very big and continuously
> displaying).
Yeah, that patch was still buggy. I have it fixed now locally, I'll post
a v8 within the hour!
--
Jens Axboe
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCH 0/12] Per-bdi writeback flusher threads v7
2009-05-27 9:20 ` Jens Axboe
@ 2009-05-27 13:15 ` Damien Wyart
2009-05-27 15:05 ` Jens Axboe
0 siblings, 1 reply; 33+ messages in thread
From: Damien Wyart @ 2009-05-27 13:15 UTC (permalink / raw)
To: Jens Axboe
Cc: linux-kernel, linux-fsdevel, chris.mason, david, hch, akpm, jack,
yanmin_zhang, richard
> Yeah, that patch was still buggy. I have it fixed now locally, I'll post
> a v8 within the hour!
The two systems having problems with v7 now run v8 and everything seems
smooth for now. Thanks for the bugfixes.
--
Damien Wyart
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCH 0/12] Per-bdi writeback flusher threads v7
2009-05-27 13:15 ` Damien Wyart
@ 2009-05-27 15:05 ` Jens Axboe
2009-05-27 21:06 ` Andrew Morton
0 siblings, 1 reply; 33+ messages in thread
From: Jens Axboe @ 2009-05-27 15:05 UTC (permalink / raw)
To: Damien Wyart
Cc: linux-kernel, linux-fsdevel, chris.mason, david, hch, akpm, jack,
yanmin_zhang, richard
On Wed, May 27 2009, Damien Wyart wrote:
> > Yeah, that patch was still buggy. I have it fixed now locally, I'll post
> > a v8 within the hour!
>
> The two systems having problems with v7 now run v8 and everything seems
> smooth for now. Thanks for the bugfixes.
Good, so at least that one got fixed. I ran testing with artificial
bdi_work allocation failures to verify the oom path, so I pretty much
expected your case to be fixed now. But it's also nice to have
reassurance!
--
Jens Axboe
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCH 0/12] Per-bdi writeback flusher threads v7
2009-05-27 15:05 ` Jens Axboe
@ 2009-05-27 21:06 ` Andrew Morton
2009-05-28 10:20 ` Jens Axboe
0 siblings, 1 reply; 33+ messages in thread
From: Andrew Morton @ 2009-05-27 21:06 UTC (permalink / raw)
To: Jens Axboe
Cc: damien.wyart, linux-kernel, linux-fsdevel, chris.mason, david,
hch, jack, yanmin_zhang, richard
On Wed, 27 May 2009 17:05:47 +0200
Jens Axboe <jens.axboe@oracle.com> wrote:
> I ran testing with artificial
> bdi_work allocation failures to verify the oom path
I bet you didn't use lib/fault-inject.c :(
For tricky pieces of core kernel like this, there's a good case for
including and permanently maintaining fault injection testing
capabilities.
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCH 0/12] Per-bdi writeback flusher threads v7
2009-05-27 21:06 ` Andrew Morton
@ 2009-05-28 10:20 ` Jens Axboe
0 siblings, 0 replies; 33+ messages in thread
From: Jens Axboe @ 2009-05-28 10:20 UTC (permalink / raw)
To: Andrew Morton
Cc: damien.wyart, linux-kernel, linux-fsdevel, chris.mason, david,
hch, jack, yanmin_zhang, richard
On Wed, May 27 2009, Andrew Morton wrote:
> On Wed, 27 May 2009 17:05:47 +0200
> Jens Axboe <jens.axboe@oracle.com> wrote:
>
> > I ran testing with artificial
> > bdi_work allocation failures to verify the oom path
>
> I bet you didn't use lib/fault-inject.c :(
I didn't, it was a 2-second hack to add a
static int i;
if (!(i++ % 4))
return NULL;
so I didn't look further...
> For tricky pieces of core kernel like this, there's a good case for
> including and permanently maintaining fault injection testing
> capabilities.
I agree.
--
Jens Axboe
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCH 0/12] Per-bdi writeback flusher threads v7
2009-05-27 6:17 ` Jens Axboe
@ 2009-06-02 2:07 ` Zhang, Yanmin
2009-06-02 11:53 ` Jens Axboe
0 siblings, 1 reply; 33+ messages in thread
From: Zhang, Yanmin @ 2009-06-02 2:07 UTC (permalink / raw)
To: Jens Axboe
Cc: linux-kernel, linux-fsdevel, chris.mason, david, hch, akpm, jack,
richard
On Wed, 2009-05-27 at 08:17 +0200, Jens Axboe wrote:
> On Wed, May 27 2009, Zhang, Yanmin wrote:
> > On Tue, 2009-05-26 at 11:33 +0200, Jens Axboe wrote:
> > > Hi,
> > >
> > > Here's the 7th version of the writeback patches. Changes since
> > > v5/v6:
> > >
> > > - Move the sync_supers() to the global bdi_forker_task() thread, so we
> > > don't writeback the supers from all the bdi kupdated() tasks.
> > > - Make bdi_start_writeback() and bdi_writeback_all() be sync when called
> > > with WB_SYNC_ALL only.
> > > - Shuffle some more things around to make a cleaner series. The sync vs
> > > async nature of bdi_writeback_all() and bdi_start_writeback() isn't
> > > consistent through the series, but otherwise things should be sane.
> > >
> > > I'd appreciate if Richard and Yanmin could re-run testing with this,
> > > just to make sure that things are sane. For ease of patching, I've
> > > put the full diff here:
> > >
> > > http://kernel.dk/writeback-v7.patch
> > I ported it to 2.6.30-rc6 with some change in file mm/page-write.c, so I
> > could compare with old data.
> >
> > See the attachment.
> >
> > The new testing hits the hang issue again. It seems there is still a race.
>
> It's actually not a race, it's a deadlock on the bdi_lock. If you find
> the bdi-default task, it should be stuck in the mutex slow path. I
> posted this quickie [1] yesterday but didn't test it, I'll test it today and
> post a v8.
>
> [1] http://lkml.org/lkml/2009/5/26/401
Jens,
I checked V9 and found it includes the fix. However, the issue appears again on another
machine when doing 'umount -f /dev/sda3'.
INFO: task umount:17110 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
umount D ffffc20000011300 6128 17110 16225
ffffffff8093e350 0000000000000086 0000000000000000 0000000000000000
ffffe200006482e8 0000000000004000 0000000000011300 000000000000c868
ffff8801be07dd48 0000000000000000 ffff8801be5f8000 ffff8801be5f8388
Call Trace:
[<ffffffff802c316c>] ? bdi_sched_wait+0x0/0xd
[<ffffffff8071e53f>] ? schedule+0x9/0x1e
[<ffffffff802c3175>] ? bdi_sched_wait+0x9/0xd
[<ffffffff8071eb06>] ? __wait_on_bit+0x41/0x71
[<ffffffff802c316c>] ? bdi_sched_wait+0x0/0xd
[<ffffffff8071eba1>] ? out_of_line_wait_on_bit+0x6b/0x77
[<ffffffff8024cbf0>] ? wake_bit_function+0x0/0x23
[<ffffffff802c2e1f>] ? bdi_writeback_all+0x117/0x14e
[<ffffffff802c2f8f>] ? generic_sync_sb_inodes+0x31/0xdc
[<ffffffff802c30bd>] ? sync_inodes_sb+0x83/0x88
[<ffffffff802ab4f3>] ? fsync_super+0x9/0x16
[<ffffffff802ab7a8>] ? generic_shutdown_super+0x25/0xde
[<ffffffff802ab883>] ? kill_block_super+0x22/0x3a
[<ffffffff802abec9>] ? deactivate_super+0x5f/0x76
[<ffffffff802bdf5f>] ? sys_umount+0x2cd/0x2fc
[<ffffffff8020ba2b>] ? system_call_fastpath+0x16/0x1b
Then, I run command sync and hit a panic:
[root@lkp-ne02 ~]# BUG: unable to handle kernel paging request at 0000000561887e50
IP: [<ffffffff8022d121>] task_rq_lock+0x28/0x6b
PGD bc551067 PUD 0
Oops: 0000 [#1] SMP
last sysfs file: /sys/class/net/eth1/ifindex
CPU 9
Modules linked in:
Pid: 27470, comm: sync Not tainted 2.6.30-rc7-bdiflusherv9 #1 X8DTN
RIP: 0010:[<ffffffff8022d121>] [<ffffffff8022d121>] task_rq_lock+0x28/0x6b
RSP: 0018:ffff8800bccd9e48 EFLAGS: 00010096
RAX: 00000000bc1d0420 RBX: 0000000000011300 RCX: 0000000000000000
RDX: 00000000000110fd RSI: ffff8800bccd9e88 RDI: ffff8800bd703090
RBP: 0000000000011300 R08: fefefefefefefeff R09: 0000000000000000
R10: 00000034cbd6da70 R11: 00000034cbacab70 R12: ffff8800bccd9e88
R13: ffff8800bd703090 R14: 000000000000000f R15: 0000000000000000
FS: 00007f04696dd6f0(0000) GS:ffffc20001200000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000561887e50 CR3: 00000000bc0d6000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process sync (pid: 27470, threadinfo ffff8800bccd8000, task ffff8800bc4ee810)
Stack:
00000034cbaca000 000000000000000f ffff8800bd703090 ffff8800bccd9f18
ffff8800bd780138 ffffffff80233889 0000000000000000 ffffffff802c2be6
0000000000000296 ffff8800bcd6f900 ffff8800bd5a9d10 ffff8800bccd9f18
Call Trace:
[<ffffffff80233889>] ? try_to_wake_up+0x8f/0x2d0
[<ffffffff802c2be6>] ? bdi_alloc_work+0x28/0xa6
[<ffffffff802c2e45>] ? bdi_writeback_all+0x13d/0x14e
[<ffffffff8027f151>] ? wakeup_flusher_threads+0x4c/0x51
[<ffffffff802c5c39>] ? do_sync+0xb/0x5a
[<ffffffff802c5caa>] ? sys_sync+0xe/0x14
[<ffffffff8020ba2b>] ? system_call_fastpath+0x16/0x1b
Code: 41 59 c3 41 55 49 89 fd 41 54 49 89 f4 55 48 c7 c5 00 13 01 00 53 48 83 ec 08 9c 58 fa 49 89 04 24 49 8b 45 08 48 89 e
RIP [<ffffffff8022d121>] task_rq_lock+0x28/0x6b
RSP <ffff8800bccd9e48>
CR2: 0000000561887e50
Another machine hangs twice with 2.6.30-rc7+bdiflusherV9
after it's idle for hours without testing. Serial console
didn't capture any useful info.
As for perforamnce testing result, there is no big variation from previous vesions.
>
> > INFO: task sync:30013 blocked for more than 120 seconds.
> > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > sync D ffffc20000011300 4736 30013 28019
> > ffffffff8093e350 0000000000000086 0000000000000000 0000000000000000
> > 0000000000021220 0000000000004000 0000000000011300 000000000000c868
> > ffff880000016c48 ffffe20002934b30 ffff8800720b3780 ffff8800720b3b08
> > Call Trace:
> > [<ffffffff802c31d0>] ? bdi_sched_wait+0x0/0xd
> > [<ffffffff8071e56f>] ? schedule+0x9/0x1e
> > [<ffffffff802c31d9>] ? bdi_sched_wait+0x9/0xd
> > [<ffffffff8071eb36>] ? __wait_on_bit+0x41/0x71
> > [<ffffffff802c31d0>] ? bdi_sched_wait+0x0/0xd
> > [<ffffffff8071ebd1>] ? out_of_line_wait_on_bit+0x6b/0x77
> > [<ffffffff8024cc0c>] ? wake_bit_function+0x0/0x23
> > [<ffffffff8022cfa1>] ? __wake_up+0x30/0x44
> > [<ffffffff802c2e22>] ? bdi_writeback_all+0x20b/0x24c
> > [<ffffffff802800ce>] ? pagevec_lookup_tag+0x1a/0x21
> > [<ffffffff80279248>] ? wait_on_page_writeback_range+0xce/0x11b
> > [<ffffffff802c2ff3>] ? generic_sync_sb_inodes+0x36/0xe1
> > [<ffffffff802c3121>] ? sync_inodes_sb+0x83/0x88
> > [<ffffffff802c316c>] ? __sync_inodes+0x46/0x8f
> > [<ffffffff802c5d10>] ? do_sync+0x36/0x5a
> > [<ffffffff802c5d56>] ? sys_sync+0xe/0x14
> > [<ffffffff8020ba2b>] ? system_call_fastpath+0x16/0x1b
> >
> >
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCH 0/12] Per-bdi writeback flusher threads v7
2009-06-02 2:07 ` Zhang, Yanmin
@ 2009-06-02 11:53 ` Jens Axboe
0 siblings, 0 replies; 33+ messages in thread
From: Jens Axboe @ 2009-06-02 11:53 UTC (permalink / raw)
To: Zhang, Yanmin
Cc: linux-kernel, linux-fsdevel, chris.mason, david, hch, akpm, jack,
richard
On Tue, Jun 02 2009, Zhang, Yanmin wrote:
> On Wed, 2009-05-27 at 08:17 +0200, Jens Axboe wrote:
> > On Wed, May 27 2009, Zhang, Yanmin wrote:
> > > On Tue, 2009-05-26 at 11:33 +0200, Jens Axboe wrote:
> > > > Hi,
> > > >
> > > > Here's the 7th version of the writeback patches. Changes since
> > > > v5/v6:
> > > >
> > > > - Move the sync_supers() to the global bdi_forker_task() thread, so we
> > > > don't writeback the supers from all the bdi kupdated() tasks.
> > > > - Make bdi_start_writeback() and bdi_writeback_all() be sync when called
> > > > with WB_SYNC_ALL only.
> > > > - Shuffle some more things around to make a cleaner series. The sync vs
> > > > async nature of bdi_writeback_all() and bdi_start_writeback() isn't
> > > > consistent through the series, but otherwise things should be sane.
> > > >
> > > > I'd appreciate if Richard and Yanmin could re-run testing with this,
> > > > just to make sure that things are sane. For ease of patching, I've
> > > > put the full diff here:
> > > >
> > > > http://kernel.dk/writeback-v7.patch
> > > I ported it to 2.6.30-rc6 with some change in file mm/page-write.c, so I
> > > could compare with old data.
> > >
> > > See the attachment.
> > >
> > > The new testing hits the hang issue again. It seems there is still a race.
> >
> > It's actually not a race, it's a deadlock on the bdi_lock. If you find
> > the bdi-default task, it should be stuck in the mutex slow path. I
> > posted this quickie [1] yesterday but didn't test it, I'll test it today and
> > post a v8.
> >
>
>
> > [1] http://lkml.org/lkml/2009/5/26/401
> Jens,
>
> I checked V9 and found it includes the fix. However, the issue appears again on another
> machine when doing 'umount -f /dev/sda3'.
>
> INFO: task umount:17110 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> umount D ffffc20000011300 6128 17110 16225
> ffffffff8093e350 0000000000000086 0000000000000000 0000000000000000
> ffffe200006482e8 0000000000004000 0000000000011300 000000000000c868
> ffff8801be07dd48 0000000000000000 ffff8801be5f8000 ffff8801be5f8388
> Call Trace:
> [<ffffffff802c316c>] ? bdi_sched_wait+0x0/0xd
> [<ffffffff8071e53f>] ? schedule+0x9/0x1e
> [<ffffffff802c3175>] ? bdi_sched_wait+0x9/0xd
> [<ffffffff8071eb06>] ? __wait_on_bit+0x41/0x71
> [<ffffffff802c316c>] ? bdi_sched_wait+0x0/0xd
> [<ffffffff8071eba1>] ? out_of_line_wait_on_bit+0x6b/0x77
> [<ffffffff8024cbf0>] ? wake_bit_function+0x0/0x23
> [<ffffffff802c2e1f>] ? bdi_writeback_all+0x117/0x14e
> [<ffffffff802c2f8f>] ? generic_sync_sb_inodes+0x31/0xdc
> [<ffffffff802c30bd>] ? sync_inodes_sb+0x83/0x88
> [<ffffffff802ab4f3>] ? fsync_super+0x9/0x16
> [<ffffffff802ab7a8>] ? generic_shutdown_super+0x25/0xde
> [<ffffffff802ab883>] ? kill_block_super+0x22/0x3a
> [<ffffffff802abec9>] ? deactivate_super+0x5f/0x76
> [<ffffffff802bdf5f>] ? sys_umount+0x2cd/0x2fc
> [<ffffffff8020ba2b>] ? system_call_fastpath+0x16/0x1b
>
>
>
> Then, I run command sync and hit a panic:
>
> [root@lkp-ne02 ~]# BUG: unable to handle kernel paging request at 0000000561887e50
> IP: [<ffffffff8022d121>] task_rq_lock+0x28/0x6b
> PGD bc551067 PUD 0
> Oops: 0000 [#1] SMP
> last sysfs file: /sys/class/net/eth1/ifindex
> CPU 9
> Modules linked in:
> Pid: 27470, comm: sync Not tainted 2.6.30-rc7-bdiflusherv9 #1 X8DTN
> RIP: 0010:[<ffffffff8022d121>] [<ffffffff8022d121>] task_rq_lock+0x28/0x6b
> RSP: 0018:ffff8800bccd9e48 EFLAGS: 00010096
> RAX: 00000000bc1d0420 RBX: 0000000000011300 RCX: 0000000000000000
> RDX: 00000000000110fd RSI: ffff8800bccd9e88 RDI: ffff8800bd703090
> RBP: 0000000000011300 R08: fefefefefefefeff R09: 0000000000000000
> R10: 00000034cbd6da70 R11: 00000034cbacab70 R12: ffff8800bccd9e88
> R13: ffff8800bd703090 R14: 000000000000000f R15: 0000000000000000
> FS: 00007f04696dd6f0(0000) GS:ffffc20001200000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 0000000561887e50 CR3: 00000000bc0d6000 CR4: 00000000000006e0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Process sync (pid: 27470, threadinfo ffff8800bccd8000, task ffff8800bc4ee810)
> Stack:
> 00000034cbaca000 000000000000000f ffff8800bd703090 ffff8800bccd9f18
> ffff8800bd780138 ffffffff80233889 0000000000000000 ffffffff802c2be6
> 0000000000000296 ffff8800bcd6f900 ffff8800bd5a9d10 ffff8800bccd9f18
> Call Trace:
> [<ffffffff80233889>] ? try_to_wake_up+0x8f/0x2d0
> [<ffffffff802c2be6>] ? bdi_alloc_work+0x28/0xa6
> [<ffffffff802c2e45>] ? bdi_writeback_all+0x13d/0x14e
> [<ffffffff8027f151>] ? wakeup_flusher_threads+0x4c/0x51
> [<ffffffff802c5c39>] ? do_sync+0xb/0x5a
> [<ffffffff802c5caa>] ? sys_sync+0xe/0x14
> [<ffffffff8020ba2b>] ? system_call_fastpath+0x16/0x1b
> Code: 41 59 c3 41 55 49 89 fd 41 54 49 89 f4 55 48 c7 c5 00 13 01 00 53 48 83 ec 08 9c 58 fa 49 89 04 24 49 8b 45 08 48 89 e
> RIP [<ffffffff8022d121>] task_rq_lock+0x28/0x6b
> RSP <ffff8800bccd9e48>
> CR2: 0000000561887e50
>
>
>
> Another machine hangs twice with 2.6.30-rc7+bdiflusherV9
> after it's idle for hours without testing. Serial console
> didn't capture any useful info.
I'm looking into the issue that Jan Kara pointed out, we do appear still
to have a race with umount and super block referencing.
> As for perforamnce testing result, there is no big variation from
> previous vesions.
That's good, at least :-)
--
Jens Axboe
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCH 07/12] writeback: support > 1 flusher thread per bdi
2009-05-26 9:33 ` [PATCH 07/12] writeback: support > 1 flusher thread per bdi Jens Axboe
@ 2009-06-04 17:44 ` Paul E. McKenney
2009-06-04 19:48 ` Jens Axboe
0 siblings, 1 reply; 33+ messages in thread
From: Paul E. McKenney @ 2009-06-04 17:44 UTC (permalink / raw)
To: Jens Axboe
Cc: linux-kernel, linux-fsdevel, chris.mason, david, hch, akpm, jack,
yanmin_zhang, richard
On Tue, May 26, 2009 at 11:33:45AM +0200, Jens Axboe wrote:
> Build on the bdi_writeback support by allowing registration of
> more than 1 flusher thread. File systems can call bdi_add_flusher_task(bdi)
> to add more flusher threads to the device. If they do so, they must also
> provide a super_operations function to return the suitable bdi_writeback
> struct from any given inode.
Looks good from an RCU perspective. SRCU used for wb_list, RCU for the
other RCU-protected data. ;-)
Thanx, Paul
> Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
> ---
> fs/fs-writeback.c | 448 +++++++++++++++++++++++++++++++++++--------
> include/linux/backing-dev.h | 34 +++-
> include/linux/fs.h | 3 +
> mm/backing-dev.c | 233 ++++++++++++++++++-----
> mm/page-writeback.c | 4 +-
> 5 files changed, 586 insertions(+), 136 deletions(-)
>
> diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c
> index e72db8b..8e0902e 100644
> --- a/fs/fs-writeback.c
> +++ b/fs/fs-writeback.c
> @@ -34,83 +34,247 @@
> */
> int nr_pdflush_threads;
>
> -/**
> - * writeback_acquire - attempt to get exclusive writeback access to a device
> - * @bdi: the device's backing_dev_info structure
> - *
> - * It is a waste of resources to have more than one pdflush thread blocked on
> - * a single request queue. Exclusion at the request_queue level is obtained
> - * via a flag in the request_queue's backing_dev_info.state.
> - *
> - * Non-request_queue-backed address_spaces will share default_backing_dev_info,
> - * unless they implement their own. Which is somewhat inefficient, as this
> - * may prevent concurrent writeback against multiple devices.
> +static void generic_sync_wb_inodes(struct bdi_writeback *wb,
> + struct super_block *sb,
> + struct writeback_control *wbc);
> +
> +/*
> + * Work items for the bdi_writeback threads
> */
> -static int writeback_acquire(struct bdi_writeback *wb)
> +struct bdi_work {
> + struct list_head list;
> + struct list_head wait_list;
> + struct rcu_head rcu_head;
> +
> + unsigned long seen;
> + atomic_t pending;
> +
> + unsigned long sb_data;
> + unsigned long nr_pages;
> + enum writeback_sync_modes sync_mode;
> +
> + unsigned long state;
> +};
> +
> +static struct super_block *bdi_work_sb(struct bdi_work *work)
> {
> - struct backing_dev_info *bdi = wb->bdi;
> + return (struct super_block *) (work->sb_data & ~1UL);
> +}
> +
> +static inline bool bdi_work_on_stack(struct bdi_work *work)
> +{
> + return work->sb_data & 1UL;
> +}
> +
> +static inline void bdi_work_init(struct bdi_work *work, struct super_block *sb,
> + unsigned long nr_pages,
> + enum writeback_sync_modes sync_mode)
> +{
> + INIT_RCU_HEAD(&work->rcu_head);
> + work->sb_data = (unsigned long) sb;
> + work->nr_pages = nr_pages;
> + work->sync_mode = sync_mode;
> + work->state = 1;
> +
> + /*
> + * state must not be reordered around the insert
> + */
> + smp_mb();
> +}
>
> - return !test_and_set_bit(wb->nr, &bdi->wb_active);
> +static inline void bdi_work_init_on_stack(struct bdi_work *work,
> + struct super_block *sb,
> + unsigned long nr_pages,
> + enum writeback_sync_modes sync_mode)
> +{
> + bdi_work_init(work, sb, nr_pages, sync_mode);
> + work->sb_data |= 1UL;
> }
>
> /**
> * writeback_in_progress - determine whether there is writeback in progress
> * @bdi: the device's backing_dev_info structure.
> *
> - * Determine whether there is writeback in progress against a backing device.
> + * Determine whether there is writeback waiting to be handled against a
> + * backing device.
> */
> int writeback_in_progress(struct backing_dev_info *bdi)
> {
> - return bdi->wb_active != 0;
> + return !list_empty(&bdi->work_list);
> }
>
> -/**
> - * writeback_release - relinquish exclusive writeback access against a device.
> - * @bdi: the device's backing_dev_info structure
> - */
> -static void writeback_release(struct bdi_writeback *wb)
> +static void bdi_work_clear(struct bdi_work *work)
> {
> - struct backing_dev_info *bdi = wb->bdi;
> + clear_bit(0, &work->state);
> + smp_mb__after_clear_bit();
> + wake_up_bit(&work->state, 0);
> +}
> +
> +static void bdi_work_free(struct rcu_head *head)
> +{
> + struct bdi_work *work = container_of(head, struct bdi_work, rcu_head);
>
> - wb->nr_pages = 0;
> - wb->sb = NULL;
> - clear_bit(wb->nr, &bdi->wb_active);
> + if (!bdi_work_on_stack(work))
> + kfree(work);
> + else
> + bdi_work_clear(work);
> }
>
> -static void wb_start_writeback(struct bdi_writeback *wb, struct super_block *sb,
> - long nr_pages,
> - enum writeback_sync_modes sync_mode)
> +static void wb_work_complete(struct bdi_work *work)
> {
> - if (!wb_has_dirty_io(wb))
> - return;
> + if (!bdi_work_on_stack(work)) {
> + bdi_work_clear(work);
> +
> + if (work->sync_mode == WB_SYNC_NONE)
> + call_rcu(&work->rcu_head, bdi_work_free);
> + } else
> + call_rcu(&work->rcu_head, bdi_work_free);
> +}
> +
> +static void wb_clear_pending(struct bdi_writeback *wb, struct bdi_work *work)
> +{
> + /*
> + * The caller has retrieved the work arguments from this work,
> + * drop our reference. If this is the last ref, delete and free it
> + */
> + if (atomic_dec_and_test(&work->pending)) {
> + struct backing_dev_info *bdi = wb->bdi;
>
> - if (writeback_acquire(wb)) {
> - wb->nr_pages = nr_pages;
> - wb->sb = sb;
> - wb->sync_mode = sync_mode;
> + spin_lock(&bdi->wb_lock);
> + list_del_rcu(&work->list);
> + spin_unlock(&bdi->wb_lock);
> +
> + wb_work_complete(work);
> + }
> +}
> +
> +static void wb_start_writeback(struct bdi_writeback *wb, struct bdi_work *work)
> +{
> + /*
> + * If we failed allocating the bdi work item, wake up the wb thread
> + * always. As a safety precaution, it'll flush out everything
> + */
> + if (!wb_has_dirty_io(wb) && work)
> + wb_clear_pending(wb, work);
> + else
> + wake_up(&wb->wait);
> +}
> +
> +static void bdi_queue_work(struct backing_dev_info *bdi, struct bdi_work *work)
> +{
> + if (work) {
> + work->seen = bdi->wb_mask;
> + atomic_set(&work->pending, bdi->wb_cnt);
>
> /*
> - * make above store seen before the task is woken
> + * Make sure stores are seen before it appears on the list
> */
> smp_mb();
> - wake_up(&wb->wait);
> +
> + spin_lock(&bdi->wb_lock);
> + list_add_tail_rcu(&work->list, &bdi->work_list);
> + spin_unlock(&bdi->wb_lock);
> }
> }
>
> -int bdi_start_writeback(struct backing_dev_info *bdi, struct super_block *sb,
> - long nr_pages, enum writeback_sync_modes sync_mode)
> +static void bdi_sched_work(struct backing_dev_info *bdi, struct bdi_work *work)
> +{
> + if (!bdi_wblist_needs_lock(bdi))
> + wb_start_writeback(&bdi->wb, work);
> + else {
> + struct bdi_writeback *wb;
> + int idx;
> +
> + idx = srcu_read_lock(&bdi->srcu);
> +
> + list_for_each_entry_rcu(wb, &bdi->wb_list, list)
> + wb_start_writeback(wb, work);
> +
> + srcu_read_unlock(&bdi->srcu, idx);
> + }
> +}
> +
> +static void __bdi_start_work(struct backing_dev_info *bdi,
> + struct bdi_work *work)
> +{
> + /*
> + * If the default thread isn't there, make sure we add it. When
> + * it gets created and wakes up, we'll run this work.
> + */
> + if (unlikely(list_empty_careful(&bdi->wb_list)))
> + bdi_add_default_flusher_task(bdi);
> + else
> + bdi_sched_work(bdi, work);
> +}
> +
> +static void bdi_start_work(struct backing_dev_info *bdi, struct bdi_work *work)
> {
> /*
> - * This only happens the first time someone kicks this bdi, so put
> - * it out-of-line.
> + * If the default thread isn't there, make sure we add it. When
> + * it gets created and wakes up, we'll run this work.
> */
> - if (unlikely(!bdi->wb.task)) {
> + if (unlikely(list_empty_careful(&bdi->wb_list))) {
> + mutex_lock(&bdi_lock);
> bdi_add_default_flusher_task(bdi);
> - return 1;
> + mutex_unlock(&bdi_lock);
> + } else
> + bdi_sched_work(bdi, work);
> +}
> +
> +/*
> + * Used for on-stack allocated work items. The caller needs to wait until
> + * the wb threads have acked the work before it's safe to continue.
> + */
> +static void bdi_wait_on_work_clear(struct bdi_work *work)
> +{
> + wait_on_bit(&work->state, 0, bdi_sched_wait, TASK_UNINTERRUPTIBLE);
> +}
> +
> +static struct bdi_work *bdi_alloc_work(struct super_block *sb, long nr_pages,
> + enum writeback_sync_modes sync_mode)
> +{
> + struct bdi_work *work;
> +
> + work = kmalloc(sizeof(*work), GFP_ATOMIC);
> + if (work)
> + bdi_work_init(work, sb, nr_pages, sync_mode);
> +
> + return work;
> +}
> +
> +void bdi_start_writeback(struct backing_dev_info *bdi, struct super_block *sb,
> + long nr_pages, enum writeback_sync_modes sync_mode)
> +{
> + const bool must_wait = sync_mode == WB_SYNC_ALL;
> + struct bdi_work work_stack, *work = NULL;
> +
> + if (!must_wait)
> + work = bdi_alloc_work(sb, nr_pages, sync_mode);
> +
> + if (!work) {
> + work = &work_stack;
> + bdi_work_init_on_stack(work, sb, nr_pages, sync_mode);
> }
>
> - wb_start_writeback(&bdi->wb, sb, nr_pages, sync_mode);
> - return 0;
> + bdi_queue_work(bdi, work);
> + bdi_start_work(bdi, work);
> +
> + /*
> + * If the sync mode is WB_SYNC_ALL, block waiting for the work to
> + * complete. If not, we only need to wait for the work to be started,
> + * if we allocated it on-stack. We use the same mechanism, if the
> + * wait bit is set in the bdi_work struct, then threads will not
> + * clear pending until after they are done.
> + *
> + * Note that work == &work_stack if must_wait is true, but that
> + * is implementation detail and we make it explicit here for
> + * ease of reading.
> + */
> + if (work == &work_stack || must_wait) {
> + bdi_wait_on_work_clear(work);
> + if (must_wait)
> + call_rcu(&work->rcu_head, bdi_work_free);
> + }
> }
>
> /*
> @@ -160,7 +324,7 @@ static void wb_kupdated(struct bdi_writeback *wb)
> wbc.more_io = 0;
> wbc.encountered_congestion = 0;
> wbc.nr_to_write = MAX_WRITEBACK_PAGES;
> - generic_sync_bdi_inodes(NULL, &wbc);
> + generic_sync_wb_inodes(wb, NULL, &wbc);
> if (wbc.nr_to_write > 0)
> break; /* All the old data is written */
> nr_to_write -= MAX_WRITEBACK_PAGES;
> @@ -177,22 +341,19 @@ static inline bool over_bground_thresh(void)
> global_page_state(NR_UNSTABLE_NFS) >= background_thresh);
> }
>
> -static void generic_sync_wb_inodes(struct bdi_writeback *wb,
> - struct super_block *sb,
> - struct writeback_control *wbc);
> -
> -static void wb_writeback(struct bdi_writeback *wb)
> +static void __wb_writeback(struct bdi_writeback *wb, long nr_pages,
> + struct super_block *sb,
> + enum writeback_sync_modes sync_mode)
> {
> struct writeback_control wbc = {
> .bdi = wb->bdi,
> - .sync_mode = wb->sync_mode,
> + .sync_mode = sync_mode,
> .older_than_this = NULL,
> .range_cyclic = 1,
> };
> - long nr_pages = wb->nr_pages;
>
> for (;;) {
> - if (wbc.sync_mode == WB_SYNC_NONE && nr_pages <= 0 &&
> + if (sync_mode == WB_SYNC_NONE && nr_pages <= 0 &&
> !over_bground_thresh())
> break;
>
> @@ -200,7 +361,7 @@ static void wb_writeback(struct bdi_writeback *wb)
> wbc.encountered_congestion = 0;
> wbc.nr_to_write = MAX_WRITEBACK_PAGES;
> wbc.pages_skipped = 0;
> - generic_sync_wb_inodes(wb, wb->sb, &wbc);
> + generic_sync_wb_inodes(wb, sb, &wbc);
> nr_pages -= MAX_WRITEBACK_PAGES - wbc.nr_to_write;
> /*
> * If we ran out of stuff to write, bail unless more_io got set
> @@ -214,68 +375,175 @@ static void wb_writeback(struct bdi_writeback *wb)
> }
>
> /*
> + * Return the next bdi_work struct that hasn't been processed by this
> + * wb thread yet
> + */
> +static struct bdi_work *get_next_work_item(struct backing_dev_info *bdi,
> + struct bdi_writeback *wb)
> +{
> + struct bdi_work *work, *ret = NULL;
> +
> + rcu_read_lock();
> +
> + list_for_each_entry_rcu(work, &bdi->work_list, list) {
> + if (!test_and_clear_bit(wb->nr, &work->seen))
> + continue;
> +
> + ret = work;
> + break;
> + }
> +
> + rcu_read_unlock();
> + return ret;
> +}
> +
> +/*
> + * Retrieve work items and do the writeback they describe
> + */
> +static void wb_writeback(struct bdi_writeback *wb)
> +{
> + struct backing_dev_info *bdi = wb->bdi;
> + struct bdi_work *work;
> +
> + while ((work = get_next_work_item(bdi, wb)) != NULL) {
> + struct super_block *sb = bdi_work_sb(work);
> + long nr_pages = work->nr_pages;
> + enum writeback_sync_modes sync_mode = work->sync_mode;
> +
> + /*
> + * If this isn't a data integrity operation, just notify
> + * that we have seen this work and we are now starting it.
> + */
> + if (sync_mode == WB_SYNC_NONE)
> + wb_clear_pending(wb, work);
> +
> + __wb_writeback(wb, nr_pages, sb, sync_mode);
> +
> + /*
> + * This is a data integrity writeback, so only do the
> + * notification when we have completed the work.
> + */
> + if (sync_mode == WB_SYNC_ALL)
> + wb_clear_pending(wb, work);
> + }
> +}
> +
> +/*
> + * This will be inlined in bdi_writeback_task() once we get rid of any
> + * dirty inodes on the default_backing_dev_info
> + */
> +static void wb_do_writeback(struct bdi_writeback *wb)
> +{
> + /*
> + * We get here in two cases:
> + *
> + * schedule_timeout() returned because the dirty writeback
> + * interval has elapsed. If that happens, the work item list
> + * will be empty and we will proceed to do kupdated style writeout.
> + *
> + * Someone called bdi_start_writeback(), which put one/more work
> + * items on the work_list. Process those.
> + */
> + if (list_empty(&wb->bdi->work_list))
> + wb_kupdated(wb);
> + else
> + wb_writeback(wb);
> +}
> +
> +/*
> * Handle writeback of dirty data for the device backed by this bdi. Also
> * wakes up periodically and does kupdated style flushing.
> */
> int bdi_writeback_task(struct bdi_writeback *wb)
> {
> + DEFINE_WAIT(wait);
> +
> while (!kthread_should_stop()) {
> unsigned long wait_jiffies;
> - DEFINE_WAIT(wait);
> +
> + wb_do_writeback(wb);
>
> prepare_to_wait(&wb->wait, &wait, TASK_INTERRUPTIBLE);
> wait_jiffies = msecs_to_jiffies(dirty_writeback_interval * 10);
> schedule_timeout(wait_jiffies);
> try_to_freeze();
> -
> - /*
> - * We get here in two cases:
> - *
> - * schedule_timeout() returned because the dirty writeback
> - * interval has elapsed. If that happens, we will be able
> - * to acquire the writeback lock and will proceed to do
> - * kupdated style writeout.
> - *
> - * Someone called bdi_start_writeback(), which will acquire
> - * the writeback lock. This means our writeback_acquire()
> - * below will fail and we call into bdi_pdflush() for
> - * pdflush style writeout.
> - *
> - */
> - if (writeback_acquire(wb))
> - wb_kupdated(wb);
> - else
> - wb_writeback(wb);
> -
> - writeback_release(wb);
> - finish_wait(&wb->wait, &wait);
> }
>
> + finish_wait(&wb->wait, &wait);
> return 0;
> }
>
> +/*
> + * Schedule writeback for all backing devices. Expensive! If this is a data
> + * integrity operation, writeback will be complete when this returns. If
> + * we are simply called for WB_SYNC_NONE, then writeback will merely be
> + * scheduled to run.
> + */
> void bdi_writeback_all(struct super_block *sb, long nr_pages,
> enum writeback_sync_modes sync_mode)
> {
> + const bool must_wait = sync_mode == WB_SYNC_ALL;
> struct backing_dev_info *bdi, *tmp;
> + struct bdi_work *work;
> + LIST_HEAD(list);
>
> mutex_lock(&bdi_lock);
>
> list_for_each_entry_safe(bdi, tmp, &bdi_list, bdi_list) {
> + struct bdi_work *work, work_stack;
> +
> if (!bdi_has_dirty_io(bdi))
> continue;
> - bdi_start_writeback(bdi, sb, nr_pages, sync_mode);
> +
> + work = bdi_alloc_work(sb, nr_pages, sync_mode);
> + if (!work) {
> + work = &work_stack;
> + bdi_work_init_on_stack(work, sb, nr_pages, sync_mode);
> + } else if (must_wait)
> + list_add_tail(&work->wait_list, &list);
> +
> + bdi_queue_work(bdi, work);
> + __bdi_start_work(bdi, work);
> +
> + /*
> + * Do the wait inline if this came from the stack. This
> + * only happens if we ran out of memory, so should very
> + * rarely trigger.
> + */
> + if (work == &work_stack) {
> + bdi_wait_on_work_clear(work);
> + if (must_wait)
> + call_rcu(&work->rcu_head, bdi_work_free);
> + }
> }
>
> mutex_unlock(&bdi_lock);
> +
> + /*
> + * If this is for WB_SYNC_ALL, wait for pending work to complete
> + * before returning.
> + */
> + while (!list_empty(&list)) {
> + work = list_entry(list.next, struct bdi_work, wait_list);
> + list_del(&work->wait_list);
> + bdi_wait_on_work_clear(work);
> + call_rcu(&work->rcu_head, bdi_work_free);
> + }
> }
>
> /*
> - * We have only a single wb per bdi, so just return that.
> + * If the filesystem didn't provide a way to map an inode to a dedicated
> + * flusher thread, it doesn't support more than 1 thread. So we know it's
> + * the default thread, return that.
> */
> static inline struct bdi_writeback *inode_get_wb(struct inode *inode)
> {
> - return &inode_to_bdi(inode)->wb;
> + const struct super_operations *sop = inode->i_sb->s_op;
> +
> + if (!sop->inode_get_wb)
> + return &inode_to_bdi(inode)->wb;
> +
> + return sop->inode_get_wb(inode);
> }
>
> /**
> @@ -729,8 +997,24 @@ void generic_sync_bdi_inodes(struct super_block *sb,
> struct writeback_control *wbc)
> {
> struct backing_dev_info *bdi = wbc->bdi;
> + struct bdi_writeback *wb;
>
> - generic_sync_wb_inodes(&bdi->wb, sb, wbc);
> + /*
> + * Common case is just a single wb thread and that is embedded in
> + * the bdi, so it doesn't need locking
> + */
> + if (!bdi_wblist_needs_lock(bdi))
> + generic_sync_wb_inodes(&bdi->wb, sb, wbc);
> + else {
> + int idx;
> +
> + idx = srcu_read_lock(&bdi->srcu);
> +
> + list_for_each_entry_rcu(wb, &bdi->wb_list, list)
> + generic_sync_wb_inodes(wb, sb, wbc);
> +
> + srcu_read_unlock(&bdi->srcu, idx);
> + }
> }
>
> /*
> @@ -757,7 +1041,7 @@ void generic_sync_sb_inodes(struct super_block *sb,
> struct writeback_control *wbc)
> {
> if (wbc->bdi)
> - generic_sync_bdi_inodes(sb, wbc);
> + bdi_start_writeback(wbc->bdi, sb, wbc->nr_to_write, wbc->sync_mode);
> else
> bdi_writeback_all(sb, wbc->nr_to_write, wbc->sync_mode);
>
> diff --git a/include/linux/backing-dev.h b/include/linux/backing-dev.h
> index 77dc62c..0559cf8 100644
> --- a/include/linux/backing-dev.h
> +++ b/include/linux/backing-dev.h
> @@ -13,6 +13,8 @@
> #include <linux/proportions.h>
> #include <linux/kernel.h>
> #include <linux/fs.h>
> +#include <linux/sched.h>
> +#include <linux/srcu.h>
> #include <linux/writeback.h>
> #include <asm/atomic.h>
>
> @@ -26,6 +28,7 @@ struct dentry;
> enum bdi_state {
> BDI_pending, /* On its way to being activated */
> BDI_wb_alloc, /* Default embedded wb allocated */
> + BDI_wblist_lock, /* bdi->wb_list now needs locking */
> BDI_async_congested, /* The async (write) queue is getting full */
> BDI_sync_congested, /* The sync queue is getting full */
> BDI_unused, /* Available bits start here */
> @@ -42,6 +45,8 @@ enum bdi_stat_item {
> #define BDI_STAT_BATCH (8*(1+ilog2(nr_cpu_ids)))
>
> struct bdi_writeback {
> + struct list_head list; /* hangs off the bdi */
> +
> struct backing_dev_info *bdi; /* our parent bdi */
> unsigned int nr;
>
> @@ -50,13 +55,12 @@ struct bdi_writeback {
> struct list_head b_dirty; /* dirty inodes */
> struct list_head b_io; /* parked for writeback */
> struct list_head b_more_io; /* parked for more writeback */
> -
> - unsigned long nr_pages;
> - struct super_block *sb;
> - enum writeback_sync_modes sync_mode;
> };
>
> +#define BDI_MAX_FLUSHERS 32
> +
> struct backing_dev_info {
> + struct srcu_struct srcu; /* for wb_list read side protection */
> struct list_head bdi_list;
> unsigned long ra_pages; /* max readahead in PAGE_CACHE_SIZE units */
> unsigned long state; /* Always use atomic bitops on this */
> @@ -75,8 +79,12 @@ struct backing_dev_info {
> unsigned int max_ratio, max_prop_frac;
>
> struct bdi_writeback wb; /* default writeback info for this bdi */
> - unsigned long wb_active; /* bitmap of active tasks */
> - unsigned long wb_mask; /* number of registered tasks */
> + spinlock_t wb_lock; /* protects update side of wb_list */
> + struct list_head wb_list; /* the flusher threads hanging off this bdi */
> + unsigned long wb_mask; /* bitmask of registered tasks */
> + unsigned int wb_cnt; /* number of registered tasks */
> +
> + struct list_head work_list;
>
> struct device *dev;
>
> @@ -93,17 +101,23 @@ int bdi_register(struct backing_dev_info *bdi, struct device *parent,
> const char *fmt, ...);
> int bdi_register_dev(struct backing_dev_info *bdi, dev_t dev);
> void bdi_unregister(struct backing_dev_info *bdi);
> -int bdi_start_writeback(struct backing_dev_info *bdi, struct super_block *sb,
> +void bdi_start_writeback(struct backing_dev_info *bdi, struct super_block *sb,
> long nr_pages, enum writeback_sync_modes sync_mode);
> int bdi_writeback_task(struct bdi_writeback *wb);
> void bdi_writeback_all(struct super_block *sb, long nr_pages,
> enum writeback_sync_modes sync_mode);
> void bdi_add_default_flusher_task(struct backing_dev_info *bdi);
> +void bdi_add_flusher_task(struct backing_dev_info *bdi);
> int bdi_has_dirty_io(struct backing_dev_info *bdi);
>
> extern struct mutex bdi_lock;
> extern struct list_head bdi_list;
>
> +static inline int bdi_wblist_needs_lock(struct backing_dev_info *bdi)
> +{
> + return test_bit(BDI_wblist_lock, &bdi->state);
> +}
> +
> static inline int wb_has_dirty_io(struct bdi_writeback *wb)
> {
> return !list_empty(&wb->b_dirty) ||
> @@ -316,4 +330,10 @@ static inline bool mapping_cap_swap_backed(struct address_space *mapping)
> return bdi_cap_swap_backed(mapping->backing_dev_info);
> }
>
> +static inline int bdi_sched_wait(void *word)
> +{
> + schedule();
> + return 0;
> +}
> +
> #endif /* _LINUX_BACKING_DEV_H */
> diff --git a/include/linux/fs.h b/include/linux/fs.h
> index ecdc544..d3bda5d 100644
> --- a/include/linux/fs.h
> +++ b/include/linux/fs.h
> @@ -1550,11 +1550,14 @@ extern ssize_t vfs_readv(struct file *, const struct iovec __user *,
> extern ssize_t vfs_writev(struct file *, const struct iovec __user *,
> unsigned long, loff_t *);
>
> +struct bdi_writeback;
> +
> struct super_operations {
> struct inode *(*alloc_inode)(struct super_block *sb);
> void (*destroy_inode)(struct inode *);
>
> void (*dirty_inode) (struct inode *);
> + struct bdi_writeback *(*inode_get_wb) (struct inode *);
> int (*write_inode) (struct inode *, int);
> void (*drop_inode) (struct inode *);
> void (*delete_inode) (struct inode *);
> diff --git a/mm/backing-dev.c b/mm/backing-dev.c
> index c8201f0..57e44e3 100644
> --- a/mm/backing-dev.c
> +++ b/mm/backing-dev.c
> @@ -199,53 +199,96 @@ static int __init default_bdi_init(void)
> }
> subsys_initcall(default_bdi_init);
>
> -static void bdi_wb_init(struct bdi_writeback *wb, struct backing_dev_info *bdi)
> +static int wb_assign_nr(struct backing_dev_info *bdi, struct bdi_writeback *wb)
> {
> - memset(wb, 0, sizeof(*wb));
> + unsigned long mask = BDI_MAX_FLUSHERS - 1;
> + unsigned int nr;
>
> - wb->bdi = bdi;
> - init_waitqueue_head(&wb->wait);
> - INIT_LIST_HEAD(&wb->b_dirty);
> - INIT_LIST_HEAD(&wb->b_io);
> - INIT_LIST_HEAD(&wb->b_more_io);
> -}
> + do {
> + if ((bdi->wb_mask & mask) == mask)
> + return 1;
> +
> + nr = find_first_zero_bit(&bdi->wb_mask, BDI_MAX_FLUSHERS);
> + } while (test_and_set_bit(nr, &bdi->wb_mask));
> +
> + wb->nr = nr;
> +
> + spin_lock(&bdi->wb_lock);
> + bdi->wb_cnt++;
> + spin_unlock(&bdi->wb_lock);
>
> -static int wb_assign_nr(struct backing_dev_info *bdi, struct bdi_writeback *wb)
> -{
> - set_bit(0, &bdi->wb_mask);
> - wb->nr = 0;
> return 0;
> }
>
> static void bdi_put_wb(struct backing_dev_info *bdi, struct bdi_writeback *wb)
> {
> clear_bit(wb->nr, &bdi->wb_mask);
> - clear_bit(BDI_wb_alloc, &bdi->state);
> +
> + if (wb == &bdi->wb)
> + clear_bit(BDI_wb_alloc, &bdi->state);
> + else
> + kfree(wb);
> +
> + spin_lock(&bdi->wb_lock);
> + bdi->wb_cnt--;
> + spin_unlock(&bdi->wb_lock);
> +}
> +
> +static int bdi_wb_init(struct bdi_writeback *wb, struct backing_dev_info *bdi)
> +{
> + memset(wb, 0, sizeof(*wb));
> +
> + wb->bdi = bdi;
> + init_waitqueue_head(&wb->wait);
> + INIT_LIST_HEAD(&wb->b_dirty);
> + INIT_LIST_HEAD(&wb->b_io);
> + INIT_LIST_HEAD(&wb->b_more_io);
> +
> + return wb_assign_nr(bdi, wb);
> }
>
> static struct bdi_writeback *bdi_new_wb(struct backing_dev_info *bdi)
> {
> struct bdi_writeback *wb;
>
> - set_bit(BDI_wb_alloc, &bdi->state);
> - wb = &bdi->wb;
> - wb_assign_nr(bdi, wb);
> + /*
> + * Default bdi->wb is already assigned, so just return it
> + */
> + if (!test_and_set_bit(BDI_wb_alloc, &bdi->state))
> + wb = &bdi->wb;
> + else {
> + wb = kmalloc(sizeof(struct bdi_writeback), GFP_KERNEL);
> + if (wb) {
> + if (bdi_wb_init(wb, bdi)) {
> + kfree(wb);
> + wb = NULL;
> + }
> + }
> + }
> +
> return wb;
> }
>
> -static int bdi_start_fn(void *ptr)
> +static void bdi_task_init(struct backing_dev_info *bdi,
> + struct bdi_writeback *wb)
> {
> - struct bdi_writeback *wb = ptr;
> - struct backing_dev_info *bdi = wb->bdi;
> struct task_struct *tsk = current;
> - int ret;
> + int was_empty;
>
> /*
> - * Add us to the active bdi_list
> + * Add us to the active bdi_list. If we are adding threads beyond
> + * the default embedded bdi_writeback, then we need to start using
> + * proper locking. Check the list for empty first, then set the
> + * BDI_wblist_lock flag if there's > 1 entry on the list now
> */
> - mutex_lock(&bdi_lock);
> - list_add(&bdi->bdi_list, &bdi_list);
> - mutex_unlock(&bdi_lock);
> + spin_lock(&bdi->wb_lock);
> +
> + was_empty = list_empty(&bdi->wb_list);
> + list_add_tail_rcu(&wb->list, &bdi->wb_list);
> + if (!was_empty)
> + set_bit(BDI_wblist_lock, &bdi->state);
> +
> + spin_unlock(&bdi->wb_lock);
>
> tsk->flags |= PF_FLUSHER | PF_SWAPWRITE;
> set_freezable();
> @@ -254,6 +297,22 @@ static int bdi_start_fn(void *ptr)
> * Our parent may run at a different priority, just set us to normal
> */
> set_user_nice(tsk, 0);
> +}
> +
> +static int bdi_start_fn(void *ptr)
> +{
> + struct bdi_writeback *wb = ptr;
> + struct backing_dev_info *bdi = wb->bdi;
> + int ret;
> +
> + /*
> + * Add us to the active bdi_list
> + */
> + mutex_lock(&bdi_lock);
> + list_add(&bdi->bdi_list, &bdi_list);
> + mutex_unlock(&bdi_lock);
> +
> + bdi_task_init(bdi, wb);
>
> /*
> * Clear pending bit and wakeup anybody waiting to tear us down
> @@ -264,13 +323,44 @@ static int bdi_start_fn(void *ptr)
>
> ret = bdi_writeback_task(wb);
>
> + /*
> + * Remove us from the list
> + */
> + spin_lock(&bdi->wb_lock);
> + list_del_rcu(&wb->list);
> + spin_unlock(&bdi->wb_lock);
> +
> + /*
> + * wait for rcu grace period to end, so we can free wb
> + */
> + synchronize_srcu(&bdi->srcu);
> +
> bdi_put_wb(bdi, wb);
> return ret;
> }
>
> int bdi_has_dirty_io(struct backing_dev_info *bdi)
> {
> - return wb_has_dirty_io(&bdi->wb);
> + struct bdi_writeback *wb;
> + int ret = 0;
> +
> + if (!bdi_wblist_needs_lock(bdi))
> + ret = wb_has_dirty_io(&bdi->wb);
> + else {
> + int idx;
> +
> + idx = srcu_read_lock(&bdi->srcu);
> +
> + list_for_each_entry_rcu(wb, &bdi->wb_list, list) {
> + ret = wb_has_dirty_io(wb);
> + if (ret)
> + break;
> + }
> +
> + srcu_read_unlock(&bdi->srcu, idx);
> + }
> +
> + return ret;
> }
>
> static void bdi_flush_io(struct backing_dev_info *bdi)
> @@ -291,6 +381,8 @@ static int bdi_forker_task(void *ptr)
> struct bdi_writeback *me = ptr;
> DEFINE_WAIT(wait);
>
> + bdi_task_init(me->bdi, me);
> +
> for (;;) {
> struct backing_dev_info *bdi, *tmp;
> struct bdi_writeback *wb;
> @@ -371,27 +463,70 @@ readd_flush:
> }
>
> /*
> - * Add a new flusher task that gets created for any bdi
> - * that has dirty data pending writeout
> + * bdi_lock held on entry
> */
> -void bdi_add_default_flusher_task(struct backing_dev_info *bdi)
> +static void bdi_add_one_flusher_task(struct backing_dev_info *bdi,
> + int(*func)(struct backing_dev_info *))
> {
> if (!bdi_cap_writeback_dirty(bdi))
> return;
>
> /*
> - * Someone already marked this pending for task creation
> + * Check with the helper whether to proceed adding a task. Will only
> + * abort if we two or more simultanous calls to
> + * bdi_add_default_flusher_task() occured, further additions will block
> + * waiting for previous additions to finish.
> */
> - if (test_and_set_bit(BDI_pending, &bdi->state))
> - return;
> + if (!func(bdi)) {
> + list_move_tail(&bdi->bdi_list, &bdi_pending_list);
>
> - mutex_lock(&bdi_lock);
> - list_move_tail(&bdi->bdi_list, &bdi_pending_list);
> + /*
> + * We are now on the pending list, wake up bdi_forker_task()
> + * to finish the job and add us back to the active bdi_list
> + */
> + wake_up(&default_backing_dev_info.wb.wait);
> + }
> +}
> +
> +static int flusher_add_helper_block(struct backing_dev_info *bdi)
> +{
> mutex_unlock(&bdi_lock);
> + wait_on_bit_lock(&bdi->state, BDI_pending, bdi_sched_wait,
> + TASK_UNINTERRUPTIBLE);
> + mutex_lock(&bdi_lock);
> + return 0;
> +}
>
> - wake_up(&default_backing_dev_info.wb.wait);
> +static int flusher_add_helper_test(struct backing_dev_info *bdi)
> +{
> + return test_and_set_bit(BDI_pending, &bdi->state);
> +}
> +
> +/*
> + * Add the default flusher task that gets created for any bdi
> + * that has dirty data pending writeout
> + */
> +void bdi_add_default_flusher_task(struct backing_dev_info *bdi)
> +{
> + bdi_add_one_flusher_task(bdi, flusher_add_helper_test);
> }
>
> +/**
> + * bdi_add_flusher_task - add one more flusher task to this @bdi
> + * @bdi: the bdi
> + *
> + * Add an additional flusher task to this @bdi. Will block waiting on
> + * previous additions, if any.
> + *
> + */
> +void bdi_add_flusher_task(struct backing_dev_info *bdi)
> +{
> + mutex_lock(&bdi_lock);
> + bdi_add_one_flusher_task(bdi, flusher_add_helper_block);
> + mutex_unlock(&bdi_lock);
> +}
> +EXPORT_SYMBOL(bdi_add_flusher_task);
> +
> int bdi_register(struct backing_dev_info *bdi, struct device *parent,
> const char *fmt, ...)
> {
> @@ -455,24 +590,21 @@ int bdi_register_dev(struct backing_dev_info *bdi, dev_t dev)
> }
> EXPORT_SYMBOL(bdi_register_dev);
>
> -static int sched_wait(void *word)
> -{
> - schedule();
> - return 0;
> -}
> -
> /*
> * Remove bdi from global list and shutdown any threads we have running
> */
> static void bdi_wb_shutdown(struct backing_dev_info *bdi)
> {
> + struct bdi_writeback *wb;
> +
> if (!bdi_cap_writeback_dirty(bdi))
> return;
>
> /*
> * If setup is pending, wait for that to complete first
> */
> - wait_on_bit(&bdi->state, BDI_pending, sched_wait, TASK_UNINTERRUPTIBLE);
> + wait_on_bit(&bdi->state, BDI_pending, bdi_sched_wait,
> + TASK_UNINTERRUPTIBLE);
>
> /*
> * Make sure nobody finds us on the bdi_list anymore
> @@ -482,9 +614,11 @@ static void bdi_wb_shutdown(struct backing_dev_info *bdi)
> mutex_unlock(&bdi_lock);
>
> /*
> - * Finally, kill the kernel thread
> + * Finally, kill the kernel threads. We don't need to be RCU
> + * safe anymore, since the bdi is gone from visibility.
> */
> - kthread_stop(bdi->wb.task);
> + list_for_each_entry(wb, &bdi->wb_list, list)
> + kthread_stop(wb->task);
> }
>
> void bdi_unregister(struct backing_dev_info *bdi)
> @@ -508,8 +642,12 @@ int bdi_init(struct backing_dev_info *bdi)
> bdi->min_ratio = 0;
> bdi->max_ratio = 100;
> bdi->max_prop_frac = PROP_FRAC_BASE;
> + spin_lock_init(&bdi->wb_lock);
> + bdi->wb_mask = 0;
> + bdi->wb_cnt = 0;
> INIT_LIST_HEAD(&bdi->bdi_list);
> - bdi->wb_mask = bdi->wb_active = 0;
> + INIT_LIST_HEAD(&bdi->wb_list);
> + INIT_LIST_HEAD(&bdi->work_list);
>
> bdi_wb_init(&bdi->wb, bdi);
>
> @@ -519,10 +657,15 @@ int bdi_init(struct backing_dev_info *bdi)
> goto err;
> }
>
> + err = init_srcu_struct(&bdi->srcu);
> + if (err)
> + goto err;
> +
> bdi->dirty_exceeded = 0;
> err = prop_local_init_percpu(&bdi->completions);
>
> if (err) {
> + cleanup_srcu_struct(&bdi->srcu);
> err:
> while (i--)
> percpu_counter_destroy(&bdi->bdi_stat[i]);
> @@ -540,6 +683,8 @@ void bdi_destroy(struct backing_dev_info *bdi)
>
> bdi_unregister(bdi);
>
> + cleanup_srcu_struct(&bdi->srcu);
> +
> for (i = 0; i < NR_BDI_STAT_ITEMS; i++)
> percpu_counter_destroy(&bdi->bdi_stat[i]);
>
> diff --git a/mm/page-writeback.c b/mm/page-writeback.c
> index 54a4a65..7dd7de7 100644
> --- a/mm/page-writeback.c
> +++ b/mm/page-writeback.c
> @@ -665,8 +665,7 @@ void throttle_vm_writeout(gfp_t gfp_mask)
>
> /*
> * Start writeback of `nr_pages' pages. If `nr_pages' is zero, write back
> - * the whole world. Returns 0 if a pdflush thread was dispatched. Returns
> - * -1 if all pdflush threads were busy.
> + * the whole world.
> */
> void wakeup_flusher_threads(long nr_pages)
> {
> @@ -674,7 +673,6 @@ void wakeup_flusher_threads(long nr_pages)
> nr_pages = global_page_state(NR_FILE_DIRTY) +
> global_page_state(NR_UNSTABLE_NFS);
> bdi_writeback_all(NULL, nr_pages, WB_SYNC_NONE);
> - return;
> }
>
> static void laptop_timer_fn(unsigned long unused);
> --
> 1.6.3.rc0.1.gf800
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCH 07/12] writeback: support > 1 flusher thread per bdi
2009-06-04 17:44 ` Paul E. McKenney
@ 2009-06-04 19:48 ` Jens Axboe
0 siblings, 0 replies; 33+ messages in thread
From: Jens Axboe @ 2009-06-04 19:48 UTC (permalink / raw)
To: Paul E. McKenney
Cc: linux-kernel, linux-fsdevel, chris.mason, david, hch, akpm, jack,
yanmin_zhang, richard
On Thu, Jun 04 2009, Paul E. McKenney wrote:
> On Tue, May 26, 2009 at 11:33:45AM +0200, Jens Axboe wrote:
> > Build on the bdi_writeback support by allowing registration of
> > more than 1 flusher thread. File systems can call bdi_add_flusher_task(bdi)
> > to add more flusher threads to the device. If they do so, they must also
> > provide a super_operations function to return the suitable bdi_writeback
> > struct from any given inode.
>
> Looks good from an RCU perspective. SRCU used for wb_list, RCU for the
> other RCU-protected data. ;-)
Great, thanks for reviewing!
--
Jens Axboe
^ permalink raw reply [flat|nested] 33+ messages in thread
end of thread, other threads:[~2009-06-04 19:48 UTC | newest]
Thread overview: 33+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2009-05-26 9:33 [PATCH 0/12] Per-bdi writeback flusher threads v7 Jens Axboe
2009-05-26 9:33 ` [PATCH 01/12] ntfs: remove old debug check for dirty data in ntfs_put_super() Jens Axboe
2009-05-26 9:33 ` [PATCH 02/12] btrfs: properly register fs backing device Jens Axboe
2009-05-26 9:33 ` [PATCH 03/12] writeback: move dirty inodes from super_block to backing_dev_info Jens Axboe
2009-05-26 9:33 ` [PATCH 04/12] writeback: switch to per-bdi threads for flushing data Jens Axboe
2009-05-26 9:33 ` [PATCH 05/12] writeback: get rid of pdflush completely Jens Axboe
2009-05-26 9:33 ` [PATCH 06/12] writeback: separate the flushing state/task from the bdi Jens Axboe
2009-05-26 9:33 ` [PATCH 07/12] writeback: support > 1 flusher thread per bdi Jens Axboe
2009-06-04 17:44 ` Paul E. McKenney
2009-06-04 19:48 ` Jens Axboe
2009-05-26 9:33 ` [PATCH 08/12] writeback: include default_backing_dev_info in writeback Jens Axboe
2009-05-26 9:33 ` [PATCH 09/12] writeback: allow sleepy exit of default writeback task Jens Axboe
2009-05-26 9:33 ` [PATCH 10/12] writeback: add some debug inode list counters to bdi stats Jens Axboe
2009-05-26 9:33 ` [PATCH 11/12] writeback: add name to backing_dev_info Jens Axboe
2009-05-26 9:33 ` [PATCH 12/12] writeback: check for registered bdi in flusher add and inode dirty Jens Axboe
2009-05-26 15:25 ` [PATCH 0/12] Per-bdi writeback flusher threads v7 Damien Wyart
2009-05-26 16:41 ` Jens Axboe
2009-05-26 17:08 ` Damien Wyart
2009-05-26 17:10 ` Damien Wyart
2009-05-26 20:47 ` Jens Axboe
2009-05-26 21:11 ` Jens Axboe
2009-05-27 5:21 ` Damien Wyart
2009-05-27 5:49 ` Damien Wyart
2009-05-27 9:20 ` Jens Axboe
2009-05-27 13:15 ` Damien Wyart
2009-05-27 15:05 ` Jens Axboe
2009-05-27 21:06 ` Andrew Morton
2009-05-28 10:20 ` Jens Axboe
2009-05-27 6:17 ` Jens Axboe
2009-05-27 5:27 ` Zhang, Yanmin
2009-05-27 6:17 ` Jens Axboe
2009-06-02 2:07 ` Zhang, Yanmin
2009-06-02 11:53 ` Jens Axboe
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.