regression in page writeback

regression in page writeback

[Shaohua Li]

Hi,
Commit d7831a0bdf06b9f722b947bb0c205ff7d77cebd8 causes disk io regression
in my test.
My system has 12 disks, each disk has two partitions. System runs fio sequence
write on all partitions, each partion has 8 jobs.
2.6.31-rc1, fio gives 460m/s disk io
2.6.31-rc2, fio gives about 400m/s disk io. Revert the patch, speed back to
460m/s

Under latest git: fio gives 450m/s disk io; If reverting the patch, the speed
is 484m/s.

With the patch, fio reports less io merge and more interrupts. My naive
analysis is the patch makes balance_dirty_pages_ratelimited_nr() limits
write chunk to 8 pages and then soon go to sleep in balance_dirty_pages(),
because most time the bdi_nr_reclaimable < bdi_thresh, and so when write
the pages out, the chunk is 8 pages long instead of 4M long. Without the patch,
thread can write 8 pages and then move some pages to writeback, and then
continue doing write. The patch seems to break this.

Unfortunatelly I can't figure out a fix for this issue, hopefully you have more
ideas.

Thanks,
Shaohua

Re: regression in page writeback

[Peter Zijlstra]

On Tue, 2009-09-22 at 13:49 +0800, Shaohua Li wrote:
> Hi,
> Commit d7831a0bdf06b9f722b947bb0c205ff7d77cebd8 causes disk io regression
> in my test.
> My system has 12 disks, each disk has two partitions. System runs fio sequence
> write on all partitions, each partion has 8 jobs.
> 2.6.31-rc1, fio gives 460m/s disk io
> 2.6.31-rc2, fio gives about 400m/s disk io. Revert the patch, speed back to
> 460m/s
> 
> Under latest git: fio gives 450m/s disk io; If reverting the patch, the speed
> is 484m/s.
> 
> With the patch, fio reports less io merge and more interrupts. My naive
> analysis is the patch makes balance_dirty_pages_ratelimited_nr() limits
> write chunk to 8 pages and then soon go to sleep in balance_dirty_pages(),
> because most time the bdi_nr_reclaimable < bdi_thresh, and so when write
> the pages out, the chunk is 8 pages long instead of 4M long. Without the patch,
> thread can write 8 pages and then move some pages to writeback, and then
> continue doing write. The patch seems to break this.
> 
> Unfortunatelly I can't figure out a fix for this issue, hopefully you have more
> ideas.

This whole writeback business is very fragile, the patch does indeed
cure a few cases and compounds a few other cases, typical trade off.

People are looking at it.

Re: regression in page writeback

[Wu Fengguang]

On Tue, Sep 22, 2009 at 02:40:12PM +0800, Peter Zijlstra wrote:
> On Tue, 2009-09-22 at 13:49 +0800, Shaohua Li wrote:
> > Hi,
> > Commit d7831a0bdf06b9f722b947bb0c205ff7d77cebd8 causes disk io regression
> > in my test.
> > My system has 12 disks, each disk has two partitions. System runs fio sequence
> > write on all partitions, each partion has 8 jobs.
> > 2.6.31-rc1, fio gives 460m/s disk io
> > 2.6.31-rc2, fio gives about 400m/s disk io. Revert the patch, speed back to
> > 460m/s
> > 
> > Under latest git: fio gives 450m/s disk io; If reverting the patch, the speed
> > is 484m/s.
> > 
> > With the patch, fio reports less io merge and more interrupts. My naive
> > analysis is the patch makes balance_dirty_pages_ratelimited_nr() limits
> > write chunk to 8 pages and then soon go to sleep in balance_dirty_pages(),
> > because most time the bdi_nr_reclaimable < bdi_thresh, and so when write
> > the pages out, the chunk is 8 pages long instead of 4M long. Without the patch,
> > thread can write 8 pages and then move some pages to writeback, and then
> > continue doing write. The patch seems to break this.
> > 
> > Unfortunatelly I can't figure out a fix for this issue, hopefully you have more
> > ideas.
> 
> This whole writeback business is very fragile,

Agreed, sorry..

> the patch does indeed cure a few cases and compounds a few other
> cases, typical trade off.
> 
> People are looking at it.

Staring at the changelog, I don't think balance_dirty_pages() could
"overshoot its limits and move all the dirty pages to writeback".
Because it will break when enough pages are written:

                if (pages_written >= write_chunk)
                        break;          /* We've done our duty */

The observed "overshooting" may well be the background_writeout()
behavior, which will hit the dirty numbers all the way down to 0.


    mm: prevent balance_dirty_pages() from doing too much work

    balance_dirty_pages can overreact and move all of the dirty pages to
    writeback unnecessarily.

    balance_dirty_pages makes its decision to throttle based on the number of
    dirty plus writeback pages that are over the calculated limit,so it will
    continue to move pages even when there are plenty of pages in writeback
    and less than the threshold still dirty.

    This allows it to overshoot its limits and move all the dirty pages to
    writeback while waiting for the drives to catch up and empty the writeback
    list.


I'm not sure how this patch stopped the "overshooting" behavior.
Maybe it managed to not start the background pdflush, or the started
pdflush thread exited because it found writeback is in progress by
someone else?

-               if (bdi_nr_reclaimable) {
+               if (bdi_nr_reclaimable > bdi_thresh) {

Thanks,
Fengguang

Re: regression in page writeback

[Peter Zijlstra]

On Tue, 2009-09-22 at 16:05 +0800, Wu Fengguang wrote:
> 
> I'm not sure how this patch stopped the "overshooting" behavior.
> Maybe it managed to not start the background pdflush, or the started
> pdflush thread exited because it found writeback is in progress by
> someone else?
> 
> -               if (bdi_nr_reclaimable) {
> +               if (bdi_nr_reclaimable > bdi_thresh) {

The idea is that we shouldn't move more pages from dirty -> writeback
when there's not actually that much dirty left.

Now, I'm not sure about the > bdi_thresh part, I've suggested to maybe
use bdi_thresh/2 a few times, but it generally didn't seem to make much
of a difference.

Re: regression in page writeback

[Wu Fengguang]

On Tue, Sep 22, 2009 at 04:09:25PM +0800, Peter Zijlstra wrote:
> On Tue, 2009-09-22 at 16:05 +0800, Wu Fengguang wrote:
> > 
> > I'm not sure how this patch stopped the "overshooting" behavior.
> > Maybe it managed to not start the background pdflush, or the started
> > pdflush thread exited because it found writeback is in progress by
> > someone else?
> > 
> > -               if (bdi_nr_reclaimable) {
> > +               if (bdi_nr_reclaimable > bdi_thresh) {
> 
> The idea is that we shouldn't move more pages from dirty -> writeback
> when there's not actually that much dirty left.

IMHO this makes little sense given that pdflush will move all dirty
pages anyway. pdflush should already be started to do background
writeback before the process is throttled, and it is designed to sync
all current dirty pages as quick as possible and as much as possible.

> Now, I'm not sure about the > bdi_thresh part, I've suggested to maybe
> use bdi_thresh/2 a few times, but it generally didn't seem to make much
> of a difference.

One possible difference is, the process may end up waiting longer time
in order to sync write_chunk pages and quit the throttle. This could
hurt the responsiveness of the throttled process.

Thanks,
Fengguang

Re: regression in page writeback

[Peter Zijlstra]

On Tue, 2009-09-22 at 16:24 +0800, Wu Fengguang wrote:
> On Tue, Sep 22, 2009 at 04:09:25PM +0800, Peter Zijlstra wrote:
> > On Tue, 2009-09-22 at 16:05 +0800, Wu Fengguang wrote:
> > > 
> > > I'm not sure how this patch stopped the "overshooting" behavior.
> > > Maybe it managed to not start the background pdflush, or the started
> > > pdflush thread exited because it found writeback is in progress by
> > > someone else?
> > > 
> > > -               if (bdi_nr_reclaimable) {
> > > +               if (bdi_nr_reclaimable > bdi_thresh) {
> > 
> > The idea is that we shouldn't move more pages from dirty -> writeback
> > when there's not actually that much dirty left.
> 
> IMHO this makes little sense given that pdflush will move all dirty
> pages anyway. pdflush should already be started to do background
> writeback before the process is throttled, and it is designed to sync
> all current dirty pages as quick as possible and as much as possible.

Not so, pdflush (or now the bdi writer thread thingies) should not
deplete all dirty pages but should stop writing once they are below the
background limit.

> > Now, I'm not sure about the > bdi_thresh part, I've suggested to maybe
> > use bdi_thresh/2 a few times, but it generally didn't seem to make much
> > of a difference.
> 
> One possible difference is, the process may end up waiting longer time
> in order to sync write_chunk pages and quit the throttle. This could
> hurt the responsiveness of the throttled process.

Well, that's all because this congestion_wait stuff is borken..

Re: regression in page writeback

[Wu Fengguang]

On Tue, Sep 22, 2009 at 04:32:14PM +0800, Peter Zijlstra wrote:
> On Tue, 2009-09-22 at 16:24 +0800, Wu Fengguang wrote:
> > On Tue, Sep 22, 2009 at 04:09:25PM +0800, Peter Zijlstra wrote:
> > > On Tue, 2009-09-22 at 16:05 +0800, Wu Fengguang wrote:
> > > > 
> > > > I'm not sure how this patch stopped the "overshooting" behavior.
> > > > Maybe it managed to not start the background pdflush, or the started
> > > > pdflush thread exited because it found writeback is in progress by
> > > > someone else?
> > > > 
> > > > -               if (bdi_nr_reclaimable) {
> > > > +               if (bdi_nr_reclaimable > bdi_thresh) {
> > > 
> > > The idea is that we shouldn't move more pages from dirty -> writeback
> > > when there's not actually that much dirty left.
> > 
> > IMHO this makes little sense given that pdflush will move all dirty
> > pages anyway. pdflush should already be started to do background
> > writeback before the process is throttled, and it is designed to sync
> > all current dirty pages as quick as possible and as much as possible.
> 
> Not so, pdflush (or now the bdi writer thread thingies) should not
> deplete all dirty pages but should stop writing once they are below the
> background limit.

(add CC to fs people for more thoughts :)

> > > Now, I'm not sure about the > bdi_thresh part, I've suggested to maybe
> > > use bdi_thresh/2 a few times, but it generally didn't seem to make much
> > > of a difference.
> > 
> > One possible difference is, the process may end up waiting longer time
> > in order to sync write_chunk pages and quit the throttle. This could
> > hurt the responsiveness of the throttled process.
> 
> Well, that's all because this congestion_wait stuff is borken..

Yes congestion_wait is bad.. I do like the idea of lowering
bdi_thresh to help reduce the uncertainty of throttle time :)

Thanks,
Fengguang

Re: regression in page writeback

[Richard Kennedy]

On Tue, 2009-09-22 at 10:32 +0200, Peter Zijlstra wrote:
> On Tue, 2009-09-22 at 16:24 +0800, Wu Fengguang wrote:
> > On Tue, Sep 22, 2009 at 04:09:25PM +0800, Peter Zijlstra wrote:
> > > On Tue, 2009-09-22 at 16:05 +0800, Wu Fengguang wrote:
> > > > 
> > > > I'm not sure how this patch stopped the "overshooting" behavior.
> > > > Maybe it managed to not start the background pdflush, or the started
> > > > pdflush thread exited because it found writeback is in progress by
> > > > someone else?
> > > > 
> > > > -               if (bdi_nr_reclaimable) {
> > > > +               if (bdi_nr_reclaimable > bdi_thresh) {
> > > 
> > > The idea is that we shouldn't move more pages from dirty -> writeback
> > > when there's not actually that much dirty left.
> > 
> > IMHO this makes little sense given that pdflush will move all dirty
> > pages anyway. pdflush should already be started to do background
> > writeback before the process is throttled, and it is designed to sync
> > all current dirty pages as quick as possible and as much as possible.
> 
> Not so, pdflush (or now the bdi writer thread thingies) should not
> deplete all dirty pages but should stop writing once they are below the
> background limit.
> 
> > > Now, I'm not sure about the > bdi_thresh part, I've suggested to maybe
> > > use bdi_thresh/2 a few times, but it generally didn't seem to make much
> > > of a difference.
> > 
> > One possible difference is, the process may end up waiting longer time
> > in order to sync write_chunk pages and quit the throttle. This could
> > hurt the responsiveness of the throttled process.
> 
> Well, that's all because this congestion_wait stuff is borken..
> 

The problem occurred as pdflush stopped when the number of dirty pages
reached the background threshold but balance_dirty_pages kept moving
pages to writeback because the total of dirty + writeback was over the
limit. 

I tried Peter's suggestion of using bdi_thresh/2 but I didn't see any
difference on my desktop hardware, but it may help RAID setups. I don't
think anyone tried it though.

Since Jens Axboe's per-bdi code got merged in the latest kernel tree,
there's a lot of change in these code paths so I'm not sure how that
reacts and if this change is still needed or relevant. 

regards
Richard

Re: regression in page writeback

[Wu Fengguang]

On Tue, Sep 22, 2009 at 04:52:48PM +0800, Richard Kennedy wrote:
> On Tue, 2009-09-22 at 10:32 +0200, Peter Zijlstra wrote:
> > On Tue, 2009-09-22 at 16:24 +0800, Wu Fengguang wrote:
> > > On Tue, Sep 22, 2009 at 04:09:25PM +0800, Peter Zijlstra wrote:
> > > > On Tue, 2009-09-22 at 16:05 +0800, Wu Fengguang wrote:
> > > > > 
> > > > > I'm not sure how this patch stopped the "overshooting" behavior.
> > > > > Maybe it managed to not start the background pdflush, or the started
> > > > > pdflush thread exited because it found writeback is in progress by
> > > > > someone else?
> > > > > 
> > > > > -               if (bdi_nr_reclaimable) {
> > > > > +               if (bdi_nr_reclaimable > bdi_thresh) {
> > > > 
> > > > The idea is that we shouldn't move more pages from dirty -> writeback
> > > > when there's not actually that much dirty left.
> > > 
> > > IMHO this makes little sense given that pdflush will move all dirty
> > > pages anyway. pdflush should already be started to do background
> > > writeback before the process is throttled, and it is designed to sync
> > > all current dirty pages as quick as possible and as much as possible.
> > 
> > Not so, pdflush (or now the bdi writer thread thingies) should not
> > deplete all dirty pages but should stop writing once they are below the
> > background limit.
> > 
> > > > Now, I'm not sure about the > bdi_thresh part, I've suggested to maybe
> > > > use bdi_thresh/2 a few times, but it generally didn't seem to make much
> > > > of a difference.
> > > 
> > > One possible difference is, the process may end up waiting longer time
> > > in order to sync write_chunk pages and quit the throttle. This could
> > > hurt the responsiveness of the throttled process.
> > 
> > Well, that's all because this congestion_wait stuff is borken..
> > 
> 
> The problem occurred as pdflush stopped when the number of dirty pages
> reached the background threshold but balance_dirty_pages kept moving
> pages to writeback because the total of dirty + writeback was over the
> limit. 

Ah yes it is possible. The pdflush started by balance_dirty_pages()
does stop at the background threshold (sorry for the confusion!),
and then balance_dirty_pages() continue to sync pages in _smaller_
chunk sizes, which should be suboptimal..

Thanks,
Fengguang

Re: regression in page writeback

[Wu Fengguang]

[-- Attachment #1: Type: text/plain, Size: 1305 bytes --]

Shaohua,

On Tue, Sep 22, 2009 at 01:49:13PM +0800, Li, Shaohua wrote:
> Hi,
> Commit d7831a0bdf06b9f722b947bb0c205ff7d77cebd8 causes disk io regression
> in my test.
> My system has 12 disks, each disk has two partitions. System runs fio sequence
> write on all partitions, each partion has 8 jobs.
> 2.6.31-rc1, fio gives 460m/s disk io
> 2.6.31-rc2, fio gives about 400m/s disk io. Revert the patch, speed back to
> 460m/s
> 
> Under latest git: fio gives 450m/s disk io; If reverting the patch, the speed
> is 484m/s.
> 
> With the patch, fio reports less io merge and more interrupts. My naive
> analysis is the patch makes balance_dirty_pages_ratelimited_nr() limits
> write chunk to 8 pages and then soon go to sleep in balance_dirty_pages(),
> because most time the bdi_nr_reclaimable < bdi_thresh, and so when write
> the pages out, the chunk is 8 pages long instead of 4M long. Without the patch,
> thread can write 8 pages and then move some pages to writeback, and then
> continue doing write. The patch seems to break this.

Do you have trace/numbers for above descriptions?

> Unfortunatelly I can't figure out a fix for this issue, hopefully
> you have more ideas.

Attached is a very verbose writeback debug patch, hope it helps and
won't disturb the workload a lot :)

Thanks,
Fengguang


[-- Attachment #2: writeback-debug-2.6.31.patch --]
[-- Type: text/x-diff, Size: 5468 bytes --]

--- linux-2.6.orig/fs/fs-writeback.c	2009-08-23 14:44:22.000000000 +0800
+++ linux-2.6/fs/fs-writeback.c	2009-09-22 18:45:06.000000000 +0800
@@ -26,6 +26,9 @@
 #include "internal.h"
 
 
+int sysctl_dirty_debug __read_mostly;
+
+
 /**
  * writeback_acquire - attempt to get exclusive writeback access to a device
  * @bdi: the device's backing_dev_info structure
@@ -186,6 +189,11 @@ static int write_inode(struct inode *ino
 	return 0;
 }
 
+#define redirty_tail(inode)						\
+	do {								\
+		__redirty_tail(inode, __LINE__);			\
+	} while (0)
+
 /*
  * Redirty an inode: set its when-it-was dirtied timestamp and move it to the
  * furthest end of its superblock's dirty-inode list.
@@ -195,10 +203,15 @@ static int write_inode(struct inode *ino
  * the case then the inode must have been redirtied while it was being written
  * out and we don't reset its dirtied_when.
  */
-static void redirty_tail(struct inode *inode)
+static void __redirty_tail(struct inode *inode, int line)
 {
 	struct super_block *sb = inode->i_sb;
 
+	if (sysctl_dirty_debug) {
+		printk(KERN_DEBUG "redirty_tail +%d: inode %lu\n",
+				line, inode->i_ino);
+	}
+
 	if (!list_empty(&sb->s_dirty)) {
 		struct inode *tail_inode;
 
@@ -210,12 +223,22 @@ static void redirty_tail(struct inode *i
 	list_move(&inode->i_list, &sb->s_dirty);
 }
 
+#define requeue_io(inode)						\
+	do {								\
+		__requeue_io(inode, __LINE__);				\
+	} while (0)
+
 /*
  * requeue inode for re-scanning after sb->s_io list is exhausted.
  */
-static void requeue_io(struct inode *inode)
+static void __requeue_io(struct inode *inode, int line)
 {
 	list_move(&inode->i_list, &inode->i_sb->s_more_io);
+
+	if (sysctl_dirty_debug) {
+		printk(KERN_DEBUG "requeue_io +%d: inode %lu\n",
+				line, inode->i_ino);
+	}
 }
 
 static void inode_sync_complete(struct inode *inode)
--- linux-2.6.orig/include/linux/writeback.h	2009-08-23 14:44:23.000000000 +0800
+++ linux-2.6/include/linux/writeback.h	2009-09-22 18:29:05.000000000 +0800
@@ -168,5 +168,6 @@ void writeback_set_ratelimit(void);
 extern int nr_pdflush_threads;	/* Global so it can be exported to sysctl
 				   read-only. */
 
+extern int sysctl_dirty_debug;
 
 #endif		/* WRITEBACK_H */
--- linux-2.6.orig/kernel/sysctl.c	2009-08-23 14:44:23.000000000 +0800
+++ linux-2.6/kernel/sysctl.c	2009-09-22 18:29:05.000000000 +0800
@@ -1516,6 +1516,14 @@ static struct ctl_table fs_table[] = {
 		.extra1		= &zero,
 		.extra2		= &two,
 	},
+	{
+		.ctl_name	= CTL_UNNUMBERED,
+		.procname	= "dirty_debug",
+		.data		= &sysctl_dirty_debug,
+		.maxlen		= sizeof(int),
+		.mode		= 0644,
+		.proc_handler	= &proc_dointvec,
+	},
 #if defined(CONFIG_BINFMT_MISC) || defined(CONFIG_BINFMT_MISC_MODULE)
 	{
 		.ctl_name	= CTL_UNNUMBERED,
--- linux-2.6.orig/mm/page-writeback.c	2009-08-23 14:44:23.000000000 +0800
+++ linux-2.6/mm/page-writeback.c	2009-09-22 18:45:50.000000000 +0800
@@ -116,6 +116,35 @@ EXPORT_SYMBOL(laptop_mode);
 
 /* End of sysctl-exported parameters */
 
+#define writeback_debug_report(n, wbc) do {                             \
+	if(sysctl_dirty_debug)						\
+		__writeback_debug_report(n, wbc,			\
+				__FILE__, __LINE__, __FUNCTION__);	\
+} while (0)
+
+void print_writeback_control(struct writeback_control *wbc)
+{
+	printk(KERN_DEBUG
+			"global dirty=%lu writeback=%lu nfs=%lu "
+			"flags=%c%c towrite=%ld skipped=%ld\n",
+			global_page_state(NR_FILE_DIRTY),
+			global_page_state(NR_WRITEBACK),
+			global_page_state(NR_UNSTABLE_NFS),
+			wbc->encountered_congestion ? 'C':'_',
+			wbc->more_io ? 'M':'_',
+			wbc->nr_to_write,
+			wbc->pages_skipped);
+}
+
+void __writeback_debug_report(long n, struct writeback_control *wbc,
+		const char *file, int line, const char *func)
+{
+	printk(KERN_DEBUG "%s %d %s: comm=%s pid=%d n=%ld\n",
+			file, line, func,
+			current->comm, current->pid,
+			n);
+	print_writeback_control(wbc);
+}
 
 static void background_writeout(unsigned long _min_pages);
 
@@ -550,7 +579,12 @@ static void balance_dirty_pages(struct a
 			pages_written += write_chunk - wbc.nr_to_write;
 			get_dirty_limits(&background_thresh, &dirty_thresh,
 				       &bdi_thresh, bdi);
+			writeback_debug_report(pages_written, &wbc);
 		}
+		printk("bdi_nr_reclaimable=%lu, bdi_thresh=%lu, "
+		       "background_thresh=%lu, dirty_thresh=%lu\n",
+		       bdi_nr_reclaimable, bdi_thresh,
+		       background_thresh, dirty_thresh);
 
 		/*
 		 * In order to avoid the stacked BDI deadlock we need
@@ -670,6 +704,11 @@ void throttle_vm_writeout(gfp_t gfp_mask
 			global_page_state(NR_WRITEBACK) <= dirty_thresh)
                         	break;
                 congestion_wait(BLK_RW_ASYNC, HZ/10);
+		printk(KERN_DEBUG "throttle_vm_writeout: "
+				"congestion_wait on %lu+%lu > %lu\n",
+				global_page_state(NR_UNSTABLE_NFS),
+				global_page_state(NR_WRITEBACK),
+				dirty_thresh);
 
 		/*
 		 * The caller might hold locks which can prevent IO completion
@@ -719,7 +758,9 @@ static void background_writeout(unsigned
 			else
 				break;
 		}
+		writeback_debug_report(min_pages, &wbc);
 	}
+	writeback_debug_report(min_pages, &wbc);
 }
 
 /*
@@ -792,7 +833,9 @@ static void wb_kupdate(unsigned long arg
 				break;	/* All the old data is written */
 		}
 		nr_to_write -= MAX_WRITEBACK_PAGES - wbc.nr_to_write;
+		writeback_debug_report(nr_to_write, &wbc);
 	}
+	writeback_debug_report(nr_to_write, &wbc);
 	if (time_before(next_jif, jiffies + HZ))
 		next_jif = jiffies + HZ;
 	if (dirty_writeback_interval)

Re: regression in page writeback

[Shaohua Li]

On Tue, Sep 22, 2009 at 05:05:01PM +0800, Wu, Fengguang wrote:
> On Tue, Sep 22, 2009 at 04:52:48PM +0800, Richard Kennedy wrote:
> > On Tue, 2009-09-22 at 10:32 +0200, Peter Zijlstra wrote:
> > > On Tue, 2009-09-22 at 16:24 +0800, Wu Fengguang wrote:
> > > > On Tue, Sep 22, 2009 at 04:09:25PM +0800, Peter Zijlstra wrote:
> > > > > On Tue, 2009-09-22 at 16:05 +0800, Wu Fengguang wrote:
> > > > > > 
> > > > > > I'm not sure how this patch stopped the "overshooting" behavior.
> > > > > > Maybe it managed to not start the background pdflush, or the started
> > > > > > pdflush thread exited because it found writeback is in progress by
> > > > > > someone else?
> > > > > > 
> > > > > > -               if (bdi_nr_reclaimable) {
> > > > > > +               if (bdi_nr_reclaimable > bdi_thresh) {
> > > > > 
> > > > > The idea is that we shouldn't move more pages from dirty -> writeback
> > > > > when there's not actually that much dirty left.
> > > > 
> > > > IMHO this makes little sense given that pdflush will move all dirty
> > > > pages anyway. pdflush should already be started to do background
> > > > writeback before the process is throttled, and it is designed to sync
> > > > all current dirty pages as quick as possible and as much as possible.
> > > 
> > > Not so, pdflush (or now the bdi writer thread thingies) should not
> > > deplete all dirty pages but should stop writing once they are below the
> > > background limit.
> > > 
> > > > > Now, I'm not sure about the > bdi_thresh part, I've suggested to maybe
> > > > > use bdi_thresh/2 a few times, but it generally didn't seem to make much
> > > > > of a difference.
> > > > 
> > > > One possible difference is, the process may end up waiting longer time
> > > > in order to sync write_chunk pages and quit the throttle. This could
> > > > hurt the responsiveness of the throttled process.
> > > 
> > > Well, that's all because this congestion_wait stuff is borken..
> > > 
> > 
> > The problem occurred as pdflush stopped when the number of dirty pages
> > reached the background threshold but balance_dirty_pages kept moving
> > pages to writeback because the total of dirty + writeback was over the
> > limit. 
> 
> Ah yes it is possible. The pdflush started by balance_dirty_pages()
> does stop at the background threshold (sorry for the confusion!),
> and then balance_dirty_pages() continue to sync pages in _smaller_
> chunk sizes, which should be suboptimal..
This is possible. Without the patch, balance_dirty_pages() can move some pages
to writeback and don't need do congestion_wait(), so the task can continue doing
write. The patch seems to break this.
I tried to set dirty_exceeded only when bdi_nr_reclaimable > bdi_thresh, this helps
a little in my test, but still not reach the best.

Re: regression in page writeback

[Shaohua Li]

On Tue, Sep 22, 2009 at 06:49:15PM +0800, Wu, Fengguang wrote:
> Shaohua,
> 
> On Tue, Sep 22, 2009 at 01:49:13PM +0800, Li, Shaohua wrote:
> > Hi,
> > Commit d7831a0bdf06b9f722b947bb0c205ff7d77cebd8 causes disk io regression
> > in my test.
> > My system has 12 disks, each disk has two partitions. System runs fio sequence
> > write on all partitions, each partion has 8 jobs.
> > 2.6.31-rc1, fio gives 460m/s disk io
> > 2.6.31-rc2, fio gives about 400m/s disk io. Revert the patch, speed back to
> > 460m/s
> > 
> > Under latest git: fio gives 450m/s disk io; If reverting the patch, the speed
> > is 484m/s.
> > 
> > With the patch, fio reports less io merge and more interrupts. My naive
> > analysis is the patch makes balance_dirty_pages_ratelimited_nr() limits
> > write chunk to 8 pages and then soon go to sleep in balance_dirty_pages(),
> > because most time the bdi_nr_reclaimable < bdi_thresh, and so when write
> > the pages out, the chunk is 8 pages long instead of 4M long. Without the patch,
> > thread can write 8 pages and then move some pages to writeback, and then
> > continue doing write. The patch seems to break this.
> 
> Do you have trace/numbers for above descriptions?
No. Just guess, because there is less io merge. And watch each bdi's states,
bdi_nr_reclaimable < bdi_thresh seems always true.
 
> > Unfortunatelly I can't figure out a fix for this issue, hopefully
> > you have more ideas.
> 
> Attached is a very verbose writeback debug patch, hope it helps and
> won't disturb the workload a lot :)
Hmm, the log buf will get overflowed soon, there is > 400m/s io. I tried
to produce this issue in a system with two disks, but fail. Anyway, I'll try
it out tomorrow.

Re: regression in page writeback

[Wu Fengguang]

On Tue, Sep 22, 2009 at 07:50:15PM +0800, Li, Shaohua wrote:
> On Tue, Sep 22, 2009 at 06:49:15PM +0800, Wu, Fengguang wrote:
> > Shaohua,
> > 
> > On Tue, Sep 22, 2009 at 01:49:13PM +0800, Li, Shaohua wrote:
> > > Hi,
> > > Commit d7831a0bdf06b9f722b947bb0c205ff7d77cebd8 causes disk io regression
> > > in my test.
> > > My system has 12 disks, each disk has two partitions. System runs fio sequence
> > > write on all partitions, each partion has 8 jobs.
> > > 2.6.31-rc1, fio gives 460m/s disk io
> > > 2.6.31-rc2, fio gives about 400m/s disk io. Revert the patch, speed back to
> > > 460m/s
> > > 
> > > Under latest git: fio gives 450m/s disk io; If reverting the patch, the speed
> > > is 484m/s.
> > > 
> > > With the patch, fio reports less io merge and more interrupts. My naive
> > > analysis is the patch makes balance_dirty_pages_ratelimited_nr() limits
> > > write chunk to 8 pages and then soon go to sleep in balance_dirty_pages(),
> > > because most time the bdi_nr_reclaimable < bdi_thresh, and so when write
> > > the pages out, the chunk is 8 pages long instead of 4M long. Without the patch,
> > > thread can write 8 pages and then move some pages to writeback, and then
> > > continue doing write. The patch seems to break this.
> > 
> > Do you have trace/numbers for above descriptions?
> No. Just guess, because there is less io merge. And watch each bdi's states,
> bdi_nr_reclaimable < bdi_thresh seems always true.

Ah OK.

> > > Unfortunatelly I can't figure out a fix for this issue, hopefully
> > > you have more ideas.
> > 
> > Attached is a very verbose writeback debug patch, hope it helps and
> > won't disturb the workload a lot :)
> Hmm, the log buf will get overflowed soon, there is > 400m/s io. I tried
> to produce this issue in a system with two disks, but fail. Anyway, I'll try
> it out tomorrow.

Thank you~  I'd recommend to use netconsole or serial line, and stop
local klogd because the write of log messages could add noises. 

Thanks,
Fengguang

Re: regression in page writeback

[Chris Mason]

On Tue, Sep 22, 2009 at 10:32:14AM +0200, Peter Zijlstra wrote:
> On Tue, 2009-09-22 at 16:24 +0800, Wu Fengguang wrote:
> > On Tue, Sep 22, 2009 at 04:09:25PM +0800, Peter Zijlstra wrote:
> > > On Tue, 2009-09-22 at 16:05 +0800, Wu Fengguang wrote:
> > > > 
> > > > I'm not sure how this patch stopped the "overshooting" behavior.
> > > > Maybe it managed to not start the background pdflush, or the started
> > > > pdflush thread exited because it found writeback is in progress by
> > > > someone else?
> > > > 
> > > > -               if (bdi_nr_reclaimable) {
> > > > +               if (bdi_nr_reclaimable > bdi_thresh) {
> > > 
> > > The idea is that we shouldn't move more pages from dirty -> writeback
> > > when there's not actually that much dirty left.
> > 
> > IMHO this makes little sense given that pdflush will move all dirty
> > pages anyway. pdflush should already be started to do background
> > writeback before the process is throttled, and it is designed to sync
> > all current dirty pages as quick as possible and as much as possible.
> 
> Not so, pdflush (or now the bdi writer thread thingies) should not
> deplete all dirty pages but should stop writing once they are below the
> background limit.
> 
> > > Now, I'm not sure about the > bdi_thresh part, I've suggested to maybe
> > > use bdi_thresh/2 a few times, but it generally didn't seem to make much
> > > of a difference.
> > 
> > One possible difference is, the process may end up waiting longer time
> > in order to sync write_chunk pages and quit the throttle. This could
> > hurt the responsiveness of the throttled process.
> 
> Well, that's all because this congestion_wait stuff is borken..
> 

I'd suggest retesting with a new baseline against the code in Linus' git
today.  Overall I think the change to make balance_dirty_pages() sleep
instead of kick more IO out is a very good one.  It helps in most
workloads here.

The congestion_wait() from 2.6.31 may just be too long to sleep waiting
for progress on very fast IO rigs.  Try switching to
schedule_timeout_interruptible(1);

-chris

Re: regression in page writeback

[Wu Fengguang]

On Tue, Sep 22, 2009 at 11:52:59PM +0800, Chris Mason wrote:
> On Tue, Sep 22, 2009 at 10:32:14AM +0200, Peter Zijlstra wrote:
> > On Tue, 2009-09-22 at 16:24 +0800, Wu Fengguang wrote:
> > > On Tue, Sep 22, 2009 at 04:09:25PM +0800, Peter Zijlstra wrote:
> > > > On Tue, 2009-09-22 at 16:05 +0800, Wu Fengguang wrote:
> > > > > 
> > > > > I'm not sure how this patch stopped the "overshooting" behavior.
> > > > > Maybe it managed to not start the background pdflush, or the started
> > > > > pdflush thread exited because it found writeback is in progress by
> > > > > someone else?
> > > > > 
> > > > > -               if (bdi_nr_reclaimable) {
> > > > > +               if (bdi_nr_reclaimable > bdi_thresh) {
> > > > 
> > > > The idea is that we shouldn't move more pages from dirty -> writeback
> > > > when there's not actually that much dirty left.
> > > 
> > > IMHO this makes little sense given that pdflush will move all dirty
> > > pages anyway. pdflush should already be started to do background
> > > writeback before the process is throttled, and it is designed to sync
> > > all current dirty pages as quick as possible and as much as possible.
> > 
> > Not so, pdflush (or now the bdi writer thread thingies) should not
> > deplete all dirty pages but should stop writing once they are below the
> > background limit.
> > 
> > > > Now, I'm not sure about the > bdi_thresh part, I've suggested to maybe
> > > > use bdi_thresh/2 a few times, but it generally didn't seem to make much
> > > > of a difference.
> > > 
> > > One possible difference is, the process may end up waiting longer time
> > > in order to sync write_chunk pages and quit the throttle. This could
> > > hurt the responsiveness of the throttled process.
> > 
> > Well, that's all because this congestion_wait stuff is borken..
> > 
> 
> I'd suggest retesting with a new baseline against the code in Linus' git
> today.  Overall I think the change to make balance_dirty_pages() sleep
> instead of kick more IO out is a very good one.  It helps in most
> workloads here.
> 
> The congestion_wait() from 2.6.31 may just be too long to sleep waiting
> for progress on very fast IO rigs.  Try switching to
> schedule_timeout_interruptible(1);

Jens' per-bdi writeback has another improvement. In 2.6.31, when
superblocks A and B both have 100000 dirty pages, it will first
exhaust A's 100000 dirty pages before going on to sync B's.
In latest git, A and B will roughly make progress at the same time.

So for 2.6.31 without this patch, it is possible for pdflush to sync
A's most dirty pages and for balance_dirty_pages() to sync B's most
dirty pages because B is over its bdi thresh.

Thanks,
Fengguang

Re: regression in page writeback

[Andrew Morton]

On Wed, 23 Sep 2009 08:22:20 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:

> Jens' per-bdi writeback has another improvement. In 2.6.31, when
> superblocks A and B both have 100000 dirty pages, it will first
> exhaust A's 100000 dirty pages before going on to sync B's.

That would only be true if someone broke 2.6.31.  Did they?

SYSCALL_DEFINE0(sync)
{
	wakeup_pdflush(0);
	sync_filesystems(0);
	sync_filesystems(1);
	if (unlikely(laptop_mode))
		laptop_sync_completion();
	return 0;
}

the sync_filesystems(0) is supposed to non-blockingly start IO against
all devices.  It used to do that correctly.  But people mucked with it
so perhaps it no longer does.

Re: regression in page writeback

[Wu Fengguang]

On Wed, Sep 23, 2009 at 08:54:52AM +0800, Andrew Morton wrote:
> On Wed, 23 Sep 2009 08:22:20 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> 
> > Jens' per-bdi writeback has another improvement. In 2.6.31, when
> > superblocks A and B both have 100000 dirty pages, it will first
> > exhaust A's 100000 dirty pages before going on to sync B's.
> 
> That would only be true if someone broke 2.6.31.  Did they?
> 
> SYSCALL_DEFINE0(sync)
> {
> 	wakeup_pdflush(0);
> 	sync_filesystems(0);
> 	sync_filesystems(1);
> 	if (unlikely(laptop_mode))
> 		laptop_sync_completion();
> 	return 0;
> }
> 
> the sync_filesystems(0) is supposed to non-blockingly start IO against
> all devices.  It used to do that correctly.  But people mucked with it
> so perhaps it no longer does.

I'm referring to writeback_inodes(). Each invocation of which (to sync
4MB) will do the same iteration over superblocks A => B => C ... So if
A has dirty pages, it will always be served first.

So if wbc->bdi == NULL (which is true for kupdate/background sync), it
will have to first exhaust A before going on to B and C.

There are no "cursor" in the superblock level iterations.

sync wants to exhaust all new inodes in A,B,C anyway, and it has the
live lock prevention logic based on dirtied_when, so that's not a big
problem for sync.

Thanks,
Fengguang

Re: regression in page writeback

[Wu Fengguang]

On Wed, Sep 23, 2009 at 09:17:58AM +0800, Wu Fengguang wrote:
> On Wed, Sep 23, 2009 at 08:54:52AM +0800, Andrew Morton wrote:
> > On Wed, 23 Sep 2009 08:22:20 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> > 
> > > Jens' per-bdi writeback has another improvement. In 2.6.31, when
> > > superblocks A and B both have 100000 dirty pages, it will first
> > > exhaust A's 100000 dirty pages before going on to sync B's.
> > 
> > That would only be true if someone broke 2.6.31.  Did they?
> > 
> > SYSCALL_DEFINE0(sync)
> > {
> > 	wakeup_pdflush(0);
> > 	sync_filesystems(0);
> > 	sync_filesystems(1);
> > 	if (unlikely(laptop_mode))
> > 		laptop_sync_completion();
> > 	return 0;
> > }
> > 
> > the sync_filesystems(0) is supposed to non-blockingly start IO against
> > all devices.  It used to do that correctly.  But people mucked with it
> > so perhaps it no longer does.
> 
> I'm referring to writeback_inodes(). Each invocation of which (to sync
> 4MB) will do the same iteration over superblocks A => B => C ... So if
> A has dirty pages, it will always be served first.
> 
> So if wbc->bdi == NULL (which is true for kupdate/background sync), it
> will have to first exhaust A before going on to B and C.
> 
> There are no "cursor" in the superblock level iterations.

I even have an old patch for it. But Jens' patches are more general solution.

Thanks,
Fengguang
---
writeback: continue from the last super_block in syncing

Cc: David Chinner <dgc@sgi.com>
Cc: Michael Rubin <mrubin@google.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Fengguang Wu <wfg@mail.ustc.edu.cn>
---
 fs/fs-writeback.c         |   12 ++++++++++++
 include/linux/writeback.h |    2 ++
 2 files changed, 14 insertions(+)

--- linux-2.6.orig/fs/fs-writeback.c
+++ linux-2.6/fs/fs-writeback.c
@@ -494,11 +494,19 @@ void
 writeback_inodes(struct writeback_control *wbc)
 {
 	struct super_block *sb;
+	int i;
+
+	if (wbc->sb_index)
+		wbc->more_io = 1;
 
 	might_sleep();
 	spin_lock(&sb_lock);
 restart:
+	i = -1;
 	list_for_each_entry_reverse(sb, &super_blocks, s_list) {
+		i++;
+		if (i < wbc->sb_index)
+			continue;
 		if (sb_has_dirty_inodes(sb)) {
 			/* we're making our own get_super here */
 			sb->s_count++;
@@ -520,9 +528,13 @@ restart:
 			if (__put_super_and_need_restart(sb))
 				goto restart;
 		}
+		if (list_empty(&sb->s_io))
+			wbc->sb_index++;
 		if (wbc->nr_to_write <= 0)
 			break;
 	}
+	if (&sb->s_list == &super_blocks)
+		wbc->sb_index = 0;
 	spin_unlock(&sb_lock);
 }
 
--- linux-2.6.orig/include/linux/writeback.h
+++ linux-2.6/include/linux/writeback.h
@@ -48,6 +48,8 @@ struct writeback_control {
 					   this for each page written */
 	long pages_skipped;		/* Pages which were not written */
 
+	int sb_index;			/* the superblock to continue from */
+
 	/*
 	 * For a_ops->writepages(): is start or end are non-zero then this is
 	 * a hint that the filesystem need only write out the pages inside that

Re: regression in page writeback

[Andrew Morton]

On Wed, 23 Sep 2009 09:17:58 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:

> On Wed, Sep 23, 2009 at 08:54:52AM +0800, Andrew Morton wrote:
> > On Wed, 23 Sep 2009 08:22:20 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> > 
> > > Jens' per-bdi writeback has another improvement. In 2.6.31, when
> > > superblocks A and B both have 100000 dirty pages, it will first
> > > exhaust A's 100000 dirty pages before going on to sync B's.
> > 
> > That would only be true if someone broke 2.6.31.  Did they?
> > 
> > SYSCALL_DEFINE0(sync)
> > {
> > 	wakeup_pdflush(0);
> > 	sync_filesystems(0);
> > 	sync_filesystems(1);
> > 	if (unlikely(laptop_mode))
> > 		laptop_sync_completion();
> > 	return 0;
> > }
> > 
> > the sync_filesystems(0) is supposed to non-blockingly start IO against
> > all devices.  It used to do that correctly.  But people mucked with it
> > so perhaps it no longer does.
> 
> I'm referring to writeback_inodes(). Each invocation of which (to sync
> 4MB) will do the same iteration over superblocks A => B => C ... So if
> A has dirty pages, it will always be served first.
> 
> So if wbc->bdi == NULL (which is true for kupdate/background sync), it
> will have to first exhaust A before going on to B and C.

But that works OK.  We fill the first device's queue, then it gets
congested and sync_sb_inodes() does nothing and we advance to the next
queue.

If a device has more than a queue's worth of dirty data then we'll
probably leave some of that dirty memory un-queued, so there's some
lack of concurrency in that situation.

Re: regression in page writeback

[Wu Fengguang]

On Wed, Sep 23, 2009 at 09:28:32AM +0800, Andrew Morton wrote:
> On Wed, 23 Sep 2009 09:17:58 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> 
> > On Wed, Sep 23, 2009 at 08:54:52AM +0800, Andrew Morton wrote:
> > > On Wed, 23 Sep 2009 08:22:20 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> > > 
> > > > Jens' per-bdi writeback has another improvement. In 2.6.31, when
> > > > superblocks A and B both have 100000 dirty pages, it will first
> > > > exhaust A's 100000 dirty pages before going on to sync B's.
> > > 
> > > That would only be true if someone broke 2.6.31.  Did they?
> > > 
> > > SYSCALL_DEFINE0(sync)
> > > {
> > > 	wakeup_pdflush(0);
> > > 	sync_filesystems(0);
> > > 	sync_filesystems(1);
> > > 	if (unlikely(laptop_mode))
> > > 		laptop_sync_completion();
> > > 	return 0;
> > > }
> > > 
> > > the sync_filesystems(0) is supposed to non-blockingly start IO against
> > > all devices.  It used to do that correctly.  But people mucked with it
> > > so perhaps it no longer does.
> > 
> > I'm referring to writeback_inodes(). Each invocation of which (to sync
> > 4MB) will do the same iteration over superblocks A => B => C ... So if
> > A has dirty pages, it will always be served first.
> > 
> > So if wbc->bdi == NULL (which is true for kupdate/background sync), it
> > will have to first exhaust A before going on to B and C.
> 
> But that works OK.  We fill the first device's queue, then it gets
> congested and sync_sb_inodes() does nothing and we advance to the next
> queue.
> 
> If a device has more than a queue's worth of dirty data then we'll
> probably leave some of that dirty memory un-queued, so there's some
> lack of concurrency in that situation.

Yes, exactly if block device is not fast enough. In that case
balance_dirty_pages() may also kick in with non-NULL bdi.

Thanks,
Fengguang

Re: regression in page writeback

[Wu Fengguang]

On Wed, Sep 23, 2009 at 09:28:32AM +0800, Andrew Morton wrote:
> On Wed, 23 Sep 2009 09:17:58 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> 
> > On Wed, Sep 23, 2009 at 08:54:52AM +0800, Andrew Morton wrote:
> > > On Wed, 23 Sep 2009 08:22:20 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> > > 
> > > > Jens' per-bdi writeback has another improvement. In 2.6.31, when
> > > > superblocks A and B both have 100000 dirty pages, it will first
> > > > exhaust A's 100000 dirty pages before going on to sync B's.
> > > 
> > > That would only be true if someone broke 2.6.31.  Did they?
> > > 
> > > SYSCALL_DEFINE0(sync)
> > > {
> > > 	wakeup_pdflush(0);
> > > 	sync_filesystems(0);
> > > 	sync_filesystems(1);
> > > 	if (unlikely(laptop_mode))
> > > 		laptop_sync_completion();
> > > 	return 0;
> > > }
> > > 
> > > the sync_filesystems(0) is supposed to non-blockingly start IO against
> > > all devices.  It used to do that correctly.  But people mucked with it
> > > so perhaps it no longer does.
> > 
> > I'm referring to writeback_inodes(). Each invocation of which (to sync
> > 4MB) will do the same iteration over superblocks A => B => C ... So if
> > A has dirty pages, it will always be served first.
> > 
> > So if wbc->bdi == NULL (which is true for kupdate/background sync), it
> > will have to first exhaust A before going on to B and C.
> 
> But that works OK.  We fill the first device's queue, then it gets
> congested and sync_sb_inodes() does nothing and we advance to the next
> queue.

So in common cases "exhaust" is a bit exaggerated, but A does receive
much more opportunity than B. Computation resources for IO submission
are unbalanced for A, and there are pointless overheads in rechecking A.

> If a device has more than a queue's worth of dirty data then we'll
> probably leave some of that dirty memory un-queued, so there's some
> lack of concurrency in that situation.

Good insight. That possibly explains one major factor of the
performance gains of Jens' per-bdi writeback.

Thanks,
Fengguang

Re: regression in page writeback

[Andrew Morton]

On Wed, 23 Sep 2009 09:32:36 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:

> On Wed, Sep 23, 2009 at 09:28:32AM +0800, Andrew Morton wrote:
> > On Wed, 23 Sep 2009 09:17:58 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> > 
> > > On Wed, Sep 23, 2009 at 08:54:52AM +0800, Andrew Morton wrote:
> > > > On Wed, 23 Sep 2009 08:22:20 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> > > > 
> > > > > Jens' per-bdi writeback has another improvement. In 2.6.31, when
> > > > > superblocks A and B both have 100000 dirty pages, it will first
> > > > > exhaust A's 100000 dirty pages before going on to sync B's.
> > > > 
> > > > That would only be true if someone broke 2.6.31.  Did they?
> > > > 
> > > > SYSCALL_DEFINE0(sync)
> > > > {
> > > > 	wakeup_pdflush(0);
> > > > 	sync_filesystems(0);
> > > > 	sync_filesystems(1);
> > > > 	if (unlikely(laptop_mode))
> > > > 		laptop_sync_completion();
> > > > 	return 0;
> > > > }
> > > > 
> > > > the sync_filesystems(0) is supposed to non-blockingly start IO against
> > > > all devices.  It used to do that correctly.  But people mucked with it
> > > > so perhaps it no longer does.
> > > 
> > > I'm referring to writeback_inodes(). Each invocation of which (to sync
> > > 4MB) will do the same iteration over superblocks A => B => C ... So if
> > > A has dirty pages, it will always be served first.
> > > 
> > > So if wbc->bdi == NULL (which is true for kupdate/background sync), it
> > > will have to first exhaust A before going on to B and C.
> > 
> > But that works OK.  We fill the first device's queue, then it gets
> > congested and sync_sb_inodes() does nothing and we advance to the next
> > queue.
> > 
> > If a device has more than a queue's worth of dirty data then we'll
> > probably leave some of that dirty memory un-queued, so there's some
> > lack of concurrency in that situation.
> 
> Yes, exactly if block device is not fast enough.

Actually, no.

If there's still outstanding dirty data for any of those queues, both
wb_kupdate() and background_writeout() will take a teeny sleep and then
will re-poll the queues.

Did that logic get broken?

Re: regression in page writeback

[Shaohua Li]

[-- Attachment #1: Type: text/plain, Size: 2400 bytes --]

On Tue, 2009-09-22 at 21:39 +0800, Wu, Fengguang wrote:
> On Tue, Sep 22, 2009 at 07:50:15PM +0800, Li, Shaohua wrote:
> > On Tue, Sep 22, 2009 at 06:49:15PM +0800, Wu, Fengguang wrote:
> > > Shaohua,
> > > 
> > > On Tue, Sep 22, 2009 at 01:49:13PM +0800, Li, Shaohua wrote:
> > > > Hi,
> > > > Commit d7831a0bdf06b9f722b947bb0c205ff7d77cebd8 causes disk io regression
> > > > in my test.
> > > > My system has 12 disks, each disk has two partitions. System runs fio sequence
> > > > write on all partitions, each partion has 8 jobs.
> > > > 2.6.31-rc1, fio gives 460m/s disk io
> > > > 2.6.31-rc2, fio gives about 400m/s disk io. Revert the patch, speed back to
> > > > 460m/s
> > > > 
> > > > Under latest git: fio gives 450m/s disk io; If reverting the patch, the speed
> > > > is 484m/s.
> > > > 
> > > > With the patch, fio reports less io merge and more interrupts. My naive
> > > > analysis is the patch makes balance_dirty_pages_ratelimited_nr() limits
> > > > write chunk to 8 pages and then soon go to sleep in balance_dirty_pages(),
> > > > because most time the bdi_nr_reclaimable < bdi_thresh, and so when write
> > > > the pages out, the chunk is 8 pages long instead of 4M long. Without the patch,
> > > > thread can write 8 pages and then move some pages to writeback, and then
> > > > continue doing write. The patch seems to break this.
> > > 
> > > Do you have trace/numbers for above descriptions?
> > No. Just guess, because there is less io merge. And watch each bdi's states,
> > bdi_nr_reclaimable < bdi_thresh seems always true.
> 
> Ah OK.
> 
> > > > Unfortunatelly I can't figure out a fix for this issue, hopefully
> > > > you have more ideas.
> > > 
> > > Attached is a very verbose writeback debug patch, hope it helps and
> > > won't disturb the workload a lot :)
> > Hmm, the log buf will get overflowed soon, there is > 400m/s io. I tried
> > to produce this issue in a system with two disks, but fail. Anyway, I'll try
> > it out tomorrow.
> 
> Thank you~  I'd recommend to use netconsole or serial line, and stop
> local klogd because the write of log messages could add noises. 
attached is a short log. I'll try to get a full log after finish latest
git test.
bdi_nr_reclaimable is always less than bdi_thresh in the log. because
when bdi_nr_reclaimable + bdi_nr_writeback > bdi_thresh, background
writeback is already started, so bdi_nr_writeback should be > 0.

[-- Attachment #2: msg2 --]
[-- Type: text/plain, Size: 97611 bytes --]

0, bdi_thresh=23810, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=24160, bdi_thresh=27860, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=23080, bdi_thresh=26626, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=23920, bdi_thresh=27422, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145141, dirty_thresh=290283
mm/page-writeback.c 762 background_writeout: comm=pdflush pid=599 n=-6696602
bdi_nr_reclaimable=21040, bdi_thresh=24479, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=20480, bdi_thresh=23802, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=21480, bdi_thresh=24906, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=24160, bdi_thresh=27862, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=23920, bdi_thresh=27416, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=15240, bdi_thresh=16824, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=20480, bdi_thresh=23809, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=21480, bdi_thresh=24903, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=21480, bdi_thresh=24907, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=23080, bdi_thresh=26623, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145141, dirty_thresh=290283
global dirty=246756 writeback=43800 nfs=0 flags=C_ towrite=1018 skipped=0
bdi_nr_reclaimable=15640, bdi_thresh=19030, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=21040, bdi_thresh=24481, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=24160, bdi_thresh=27850, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=21040, bdi_thresh=24493, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=21320, bdi_thresh=24944, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=21320, bdi_thresh=24940, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=15880, bdi_thresh=18244, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=21040, bdi_thresh=24493, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=24160, bdi_thresh=27864, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145141, dirty_thresh=290283
redirty_tail +435: inode 174623
bdi_nr_reclaimable=21480, bdi_thresh=24894, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=23920, bdi_thresh=27412, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=21480, bdi_thresh=24900, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=24160, bdi_thresh=27862, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=24160, bdi_thresh=27855, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=15640, bdi_thresh=19026, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=21040, bdi_thresh=24484, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=22840, bdi_thresh=26420, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=21320, bdi_thresh=24943, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=15240, bdi_thresh=16824, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=23080, bdi_thresh=26611, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=21480, bdi_thresh=24907, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=20480, bdi_thresh=23793, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=21040, bdi_thresh=24479, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=20480, bdi_thresh=23802, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=24160, bdi_thresh=27862, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=15240, bdi_thresh=16825, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=21040, bdi_thresh=24491, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=23920, bdi_thresh=27425, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=23080, bdi_thresh=26615, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=21320, bdi_thresh=24942, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=23080, bdi_thresh=26625, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=24160, bdi_thresh=27860, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=22840, bdi_thresh=26421, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=19720, bdi_thresh=23425, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=21480, bdi_thresh=24904, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=21320, bdi_thresh=24946, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=15880, bdi_thresh=18246, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=21480, bdi_thresh=24906, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=23920, bdi_thresh=27422, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=23920, bdi_thresh=27416, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=22840, bdi_thresh=26418, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=20480, bdi_thresh=23809, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=15880, bdi_thresh=18246, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=21480, bdi_thresh=24907, background_thresh=145141, dirty_thresh=290283
redirty_tail +435: inode 4784181
bdi_nr_reclaimable=21480, bdi_thresh=24902, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=15240, bdi_thresh=16826, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=15240, bdi_thresh=16824, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=21320, bdi_thresh=24944, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=21040, bdi_thresh=24495, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=23920, bdi_thresh=27413, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=15240, bdi_thresh=16824, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=24160, bdi_thresh=27863, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=23080, bdi_thresh=26618, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=21040, bdi_thresh=24492, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=21480, bdi_thresh=24905, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=21480, bdi_thresh=24903, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=21480, bdi_thresh=24903, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=15880, bdi_thresh=18246, background_thresh=145141, dirty_thresh=290283
redirty_tail +435: inode 3474323
bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=21480, bdi_thresh=24864, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=24160, bdi_thresh=27863, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=20480, bdi_thresh=23811, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=15880, bdi_thresh=18245, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=21480, bdi_thresh=24864, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=22840, bdi_thresh=26425, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=15240, bdi_thresh=16826, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=21040, bdi_thresh=24496, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=21320, bdi_thresh=24945, background_thresh=145141, dirty_thresh=290283
requeue_io +527: inode 0
bdi_nr_reclaimable=15640, bdi_thresh=19034, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=23920, bdi_thresh=27426, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=23080, bdi_thresh=26626, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=15880, bdi_thresh=18245, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=24160, bdi_thresh=27861, background_thresh=145141, dirty_thresh=290283
requeue_io +527: inode 0
bdi_nr_reclaimable=23920, bdi_thresh=27427, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145141, dirty_thresh=290283
bdi_nr_reclaimable=22840, bdi_thresh=26421, background_thresh=145150, dirty_thresh=290301
bdi_nr_reclaimable=22840, bdi_thresh=26424, background_thresh=145150, dirty_thresh=290301
bdi_nr_reclaimable=21040, bdi_thresh=24489, background_thresh=145150, dirty_thresh=290301
bdi_nr_reclaimable=21320, bdi_thresh=24906, background_thresh=145150, dirty_thresh=290301
bdi_nr_reclaimable=23920, bdi_thresh=27426, background_thresh=145150, dirty_thresh=290301
redirty_tail +502: inode 12415
bdi_nr_reclaimable=24160, bdi_thresh=27861, background_thresh=145150, dirty_thresh=290301
bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145150, dirty_thresh=290301
bdi_nr_reclaimable=21320, bdi_thresh=24905, background_thresh=145150, dirty_thresh=290301
bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145150, dirty_thresh=290301
bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145150, dirty_thresh=290301
bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145159, dirty_thresh=290319
bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145159, dirty_thresh=290319
bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145159, dirty_thresh=290319
bdi_nr_reclaimable=21640, bdi_thresh=24923, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24919, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24921, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24918, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27421, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24905, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=20480, bdi_thresh=23812, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26623, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24495, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24490, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24903, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27423, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24903, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18246, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27421, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18250, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24919, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24496, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26414, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24160, bdi_thresh=27820, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19030, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24919, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27418, background_thresh=145168, dirty_thresh=290337
mm/page-writeback.c 762 background_writeout: comm=pdflush pid=10090 n=-6621688
bdi_nr_reclaimable=23080, bdi_thresh=26624, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15240, bdi_thresh=16826, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26420, background_thresh=145168, dirty_thresh=290337
global dirty=246847 writeback=43800 nfs=0 flags=C_ towrite=1023 skipped=0
bdi_nr_reclaimable=21320, bdi_thresh=24900, background_thresh=145168, dirty_thresh=290337
redirty_tail +435: inode 174623
bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24160, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
redirty_tail +435: inode 4784181
redirty_tail +435: inode 3474323
requeue_io +527: inode 0
requeue_io +527: inode 0
redirty_tail +502: inode 12396
mm/page-writeback.c 762 background_writeout: comm=pdflush pid=10090 n=-6621689
bdi_nr_reclaimable=24160, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24900, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15240, bdi_thresh=16826, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26624, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24160, bdi_thresh=27820, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24490, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=20480, bdi_thresh=23812, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27421, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24919, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27427, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15240, bdi_thresh=16827, background_thresh=145168, dirty_thresh=290337
global dirty=246847 writeback=43800 nfs=0 flags=C_ towrite=1023 skipped=0
bdi_nr_reclaimable=22840, bdi_thresh=26420, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24910, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=20480, bdi_thresh=23811, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24903, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24904, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24900, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24923, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26611, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24916, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24160, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24160, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18245, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24160, bdi_thresh=27810, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26615, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27412, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24495, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27425, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18248, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27422, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=19720, bdi_thresh=23425, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24160, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24481, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24160, bdi_thresh=27820, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=20480, bdi_thresh=23804, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19026, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24486, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=20480, bdi_thresh=23795, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26625, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24493, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24923, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19030, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15240, bdi_thresh=16825, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24483, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24494, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15240, bdi_thresh=16826, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27416, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26418, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26623, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26623, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24160, bdi_thresh=27815, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24920, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24906, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24920, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24922, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24495, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=20480, bdi_thresh=23811, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15240, bdi_thresh=16825, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26618, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27413, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27421, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26425, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24904, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15240, bdi_thresh=16827, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24498, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27426, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24902, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24497, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24160, bdi_thresh=27823, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24918, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24921, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24920, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18246, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=20480, bdi_thresh=23813, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26414, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26626, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27423, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15240, bdi_thresh=16825, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24160, bdi_thresh=27821, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18249, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24160, bdi_thresh=27823, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24919, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18248, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27426, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18248, background_thresh=145168, dirty_thresh=290337
mm/page-writeback.c 762 background_writeout: comm=pdflush pid=599 n=-6696603
bdi_nr_reclaimable=22840, bdi_thresh=26421, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24905, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24918, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26424, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19034, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26420, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24923, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24906, background_thresh=145168, dirty_thresh=290337
global dirty=246847 writeback=43800 nfs=0 flags=C_ towrite=1023 skipped=0
bdi_nr_reclaimable=21040, bdi_thresh=24491, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26421, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18250, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18248, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24903, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24160, bdi_thresh=27821, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24905, background_thresh=145168, dirty_thresh=290337
requeue_io +527: inode 0
bdi_nr_reclaimable=15880, bdi_thresh=18248, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19030, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24903, background_thresh=145168, dirty_thresh=290337
requeue_io +527: inode 0
bdi_nr_reclaimable=15880, bdi_thresh=18246, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24495, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24919, background_thresh=145168, dirty_thresh=290337
redirty_tail +502: inode 12533
bdi_nr_reclaimable=21040, bdi_thresh=24496, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27418, background_thresh=145168, dirty_thresh=290337
redirty_tail +435: inode 174623
bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
redirty_tail +435: inode 4784181
redirty_tail +435: inode 3474323
requeue_io +527: inode 0
requeue_io +527: inode 0
redirty_tail +502: inode 13474
mm/page-writeback.c 762 background_writeout: comm=pdflush pid=10090 n=-6621691
bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18246, background_thresh=145168, dirty_thresh=290337
mm/page-writeback.c 762 background_writeout: comm=pdflush pid=599 n=-6696603
bdi_nr_reclaimable=15640, bdi_thresh=19030, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24903, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26421, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19034, background_thresh=145168, dirty_thresh=290337
global dirty=246847 writeback=43709 nfs=0 flags=C_ towrite=1024 skipped=0
bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24160, bdi_thresh=27823, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24160, bdi_thresh=27821, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=19720, bdi_thresh=23425, background_thresh=145168, dirty_thresh=290337
global dirty=246847 writeback=43709 nfs=0 flags=C_ towrite=1022 skipped=0
bdi_nr_reclaimable=21640, bdi_thresh=24920, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24160, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27425, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26611, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24489, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24926, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24903, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27421, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24929, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24160, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19026, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27412, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26615, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26624, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=20480, bdi_thresh=23810, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24904, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=20480, bdi_thresh=23811, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27422, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24480, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24900, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24160, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24160, bdi_thresh=27820, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=20480, bdi_thresh=23803, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15240, bdi_thresh=16825, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=20480, bdi_thresh=23794, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24485, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24933, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18248, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24933, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24920, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24903, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26421, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24926, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24160, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24932, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27412, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26615, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=19720, bdi_thresh=23425, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27425, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26611, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24900, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24903, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24492, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19030, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15240, bdi_thresh=16824, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24160, bdi_thresh=27821, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19026, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27416, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24904, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26625, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18246, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24920, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24480, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24160, bdi_thresh=27820, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=20480, bdi_thresh=23794, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24485, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=20480, bdi_thresh=23803, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15240, bdi_thresh=16825, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24933, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18248, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26611, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24903, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24160, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24926, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27422, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18246, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27412, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26615, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24160, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24160, bdi_thresh=27810, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=19720, bdi_thresh=23425, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24160, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27425, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24933, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24900, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24900, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24930, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15240, bdi_thresh=16824, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24493, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24481, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24930, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26611, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24864, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24863, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24918, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18245, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24924, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15240, bdi_thresh=16823, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24860, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27412, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27416, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24160, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19026, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18243, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18246, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24490, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26625, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24931, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24160, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24478, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24160, bdi_thresh=27820, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=20480, bdi_thresh=23792, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26615, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=20480, bdi_thresh=23801, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24160, bdi_thresh=27810, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24491, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=20480, bdi_thresh=23809, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24918, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27425, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24863, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=19720, bdi_thresh=23425, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24924, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15240, bdi_thresh=16823, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24860, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27412, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24483, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24160, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24930, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24160, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26611, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24864, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24160, bdi_thresh=27815, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27422, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24928, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15240, bdi_thresh=16823, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18246, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27416, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24480, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15240, bdi_thresh=16824, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24931, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24492, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24864, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26623, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15240, bdi_thresh=16825, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18245, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26418, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26623, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24492, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24866, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19026, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18243, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15240, bdi_thresh=16823, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=20480, bdi_thresh=23808, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18245, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27421, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19030, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24160, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26618, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=20480, bdi_thresh=23791, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26615, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24490, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26611, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24160, bdi_thresh=27810, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=20480, bdi_thresh=23800, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24863, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24160, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24941, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24477, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24927, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27412, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24481, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=20480, bdi_thresh=23807, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27425, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24200, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24200, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26625, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=19720, bdi_thresh=23425, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24200, bdi_thresh=27810, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24933, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18245, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18242, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26611, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26623, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=20480, bdi_thresh=23790, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26615, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15240, bdi_thresh=16822, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=20480, bdi_thresh=23799, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24200, bdi_thresh=27820, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24863, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18244, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27421, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19026, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24476, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26618, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24927, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24200, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27422, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27416, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24864, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24866, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15240, bdi_thresh=16822, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=20480, bdi_thresh=23806, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24200, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24940, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15240, bdi_thresh=16823, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19030, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24860, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24489, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24940, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24488, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24490, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24478, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18244, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24200, bdi_thresh=27815, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24939, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24937, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24935, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27413, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24200, bdi_thresh=27823, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15240, bdi_thresh=16822, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24490, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18244, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26623, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24492, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15240, bdi_thresh=16824, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26418, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24937, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26425, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=20480, bdi_thresh=23808, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24200, bdi_thresh=27823, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24937, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27426, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24936, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24938, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24864, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26414, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27423, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18243, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24862, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26626, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15240, bdi_thresh=16824, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18246, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18245, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24200, bdi_thresh=27821, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24936, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27421, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26624, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19034, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24935, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27426, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18246, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26421, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15240, bdi_thresh=16824, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24927, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24481, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27425, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24863, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24200, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24200, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26420, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=19720, bdi_thresh=23425, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18245, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24200, bdi_thresh=27810, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26611, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24933, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27412, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26615, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24864, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26625, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24200, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=20480, bdi_thresh=23799, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24200, bdi_thresh=27820, background_thresh=145168, dirty_thresh=290337
requeue_io +527: inode 0
bdi_nr_reclaimable=21640, bdi_thresh=24940, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24860, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15240, bdi_thresh=16823, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24200, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=20480, bdi_thresh=23807, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24476, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27422, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=20480, bdi_thresh=23790, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27416, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24200, bdi_thresh=27815, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24478, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15240, bdi_thresh=16822, background_thresh=145168, dirty_thresh=290337
requeue_io +527: inode 0
bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24939, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18242, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24940, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24937, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24488, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24200, bdi_thresh=27823, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24490, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18244, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24866, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18244, background_thresh=145168, dirty_thresh=290337
redirty_tail +502: inode 13925
bdi_nr_reclaimable=23080, bdi_thresh=26623, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26418, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19026, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24489, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15240, bdi_thresh=16824, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24935, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24490, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15240, bdi_thresh=16821, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=20480, bdi_thresh=23805, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=20480, bdi_thresh=23798, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27421, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26420, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=19720, bdi_thresh=23425, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18245, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26611, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15240, bdi_thresh=16822, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27426, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24932, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27412, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26615, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27425, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24823, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24200, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24200, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26424, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24200, bdi_thresh=27820, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26624, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26421, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24938, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18244, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24475, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=20480, bdi_thresh=23789, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24479, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18243, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24943, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18244, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24200, bdi_thresh=27821, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18243, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15240, bdi_thresh=16820, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24947, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24200, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24947, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24200, bdi_thresh=27810, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24820, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24942, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15240, bdi_thresh=16822, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27416, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24824, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26625, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27422, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24476, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18242, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18242, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15240, bdi_thresh=16820, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18240, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=20480, bdi_thresh=23805, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24490, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24944, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26623, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24825, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18241, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24943, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27421, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24826, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26425, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19034, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26618, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27423, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24200, bdi_thresh=27823, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24488, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27413, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24200, bdi_thresh=27815, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24489, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24200, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26414, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24944, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18241, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19030, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24200, bdi_thresh=27823, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24944, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24945, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=20480, bdi_thresh=23806, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24825, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24822, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24491, background_thresh=145168, dirty_thresh=290337
redirty_tail +435: inode 174623
bdi_nr_reclaimable=20480, bdi_thresh=23804, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27426, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26626, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27427, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24947, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18245, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26420, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24826, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26421, background_thresh=145168, dirty_thresh=290337
redirty_tail +435: inode 4784181
bdi_nr_reclaimable=21040, bdi_thresh=24484, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24483, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15240, bdi_thresh=16821, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24200, bdi_thresh=27821, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24823, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24825, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18243, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19030, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24823, background_thresh=145168, dirty_thresh=290337
redirty_tail +442: inode 3474323
bdi_nr_reclaimable=21040, bdi_thresh=24488, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24943, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24820, background_thresh=145168, dirty_thresh=290337
requeue_io +527: inode 0
bdi_nr_reclaimable=15880, bdi_thresh=18242, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27418, background_thresh=145168, dirty_thresh=290337
redirty_tail +502: inode 14004
bdi_nr_reclaimable=24200, bdi_thresh=27820, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24200, bdi_thresh=27820, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24934, background_thresh=145168, dirty_thresh=290337
mm/page-writeback.c 762 background_writeout: comm=pdflush pid=599 n=-6696607
bdi_nr_reclaimable=21320, bdi_thresh=24780, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19030, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18242, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24481, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27427, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27426, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=20480, bdi_thresh=23803, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24489, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24782, background_thresh=145168, dirty_thresh=290337
global dirty=247029 writeback=43527 nfs=0 flags=CM towrite=1020 skipped=0
bdi_nr_reclaimable=24200, bdi_thresh=27823, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24783, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18241, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26418, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26611, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24940, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24485, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=20480, bdi_thresh=23796, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19026, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26420, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18241, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=19720, bdi_thresh=23425, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26615, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27412, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=20480, bdi_thresh=23803, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24200, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18242, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24472, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24200, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15240, bdi_thresh=16820, background_thresh=145168, dirty_thresh=290337
mm/page-writeback.c 762 background_writeout: comm=pdflush pid=10090 n=-6621691
bdi_nr_reclaimable=24200, bdi_thresh=27820, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27422, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26623, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24477, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=20480, bdi_thresh=23787, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26625, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24781, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24947, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15240, bdi_thresh=16821, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24200, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24474, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24946, background_thresh=145168, dirty_thresh=290337
global dirty=247029 writeback=43527 nfs=0 flags=C_ towrite=1024 skipped=0
bdi_nr_reclaimable=15240, bdi_thresh=16819, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24484, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27416, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24942, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24785, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=20480, bdi_thresh=23804, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24944, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18239, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21320, bdi_thresh=24826, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24200, bdi_thresh=27810, background_thresh=145168, dirty_thresh=290337
redirty_tail +435: inode 174623
bdi_nr_reclaimable=15240, bdi_thresh=16819, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26623, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24486, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24486, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18241, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26618, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26425, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27423, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21040, bdi_thresh=24488, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24943, background_thresh=145168, dirty_thresh=290337
redirty_tail +435: inode 4784181
bdi_nr_reclaimable=23920, bdi_thresh=27413, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24200, bdi_thresh=27823, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24945, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24200, bdi_thresh=27815, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15880, bdi_thresh=18240, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19030, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24944, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24200, bdi_thresh=27821, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=24200, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15240, bdi_thresh=16821, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21240, bdi_thresh=24825, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=20480, bdi_thresh=23804, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=23920, bdi_thresh=27426, background_thresh=145168, dirty_thresh=290337
redirty_tail +442: inode 3474323
bdi_nr_reclaimable=23080, bdi_thresh=26626, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=21640, bdi_thresh=24902, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145168, dirty_thresh=290337
bdi_nr_reclaimable=15640, bdi_thresh=19034, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=23920, bdi_thresh=27421, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15880, bdi_thresh=18243, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21640, bdi_thresh=24907, background_thresh=145177, dirty_thresh=290355
requeue_io +527: inode 0
bdi_nr_reclaimable=21040, bdi_thresh=24483, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15880, bdi_thresh=18244, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21640, bdi_thresh=24904, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=22840, bdi_thresh=26421, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21080, bdi_thresh=24785, background_thresh=145177, dirty_thresh=290355
redirty_tail +502: inode 14023
bdi_nr_reclaimable=22840, bdi_thresh=26421, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15240, bdi_thresh=16820, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15880, bdi_thresh=18243, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21040, bdi_thresh=24786, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=22840, bdi_thresh=26424, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=22840, bdi_thresh=26420, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15880, bdi_thresh=18241, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15880, bdi_thresh=18240, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21000, bdi_thresh=24783, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21640, bdi_thresh=24903, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21640, bdi_thresh=24907, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15880, bdi_thresh=18242, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21040, bdi_thresh=24488, background_thresh=145177, dirty_thresh=290355
redirty_tail +435: inode 21
bdi_nr_reclaimable=21000, bdi_thresh=24823, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=23080, bdi_thresh=26624, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21640, bdi_thresh=24903, background_thresh=145177, dirty_thresh=290355
redirty_tail +435: inode 174623
bdi_nr_reclaimable=23920, bdi_thresh=27421, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=23920, bdi_thresh=27418, background_thresh=145177, dirty_thresh=290355
redirty_tail +435: inode 4784181
bdi_nr_reclaimable=22840, bdi_thresh=26414, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145177, dirty_thresh=290355
redirty_tail +435: inode 3474323
requeue_io +527: inode 0
bdi_nr_reclaimable=23920, bdi_thresh=27425, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15240, bdi_thresh=16819, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21640, bdi_thresh=24904, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21000, bdi_thresh=24824, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=24200, bdi_thresh=27821, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21040, bdi_thresh=24487, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15880, bdi_thresh=18242, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15880, bdi_thresh=18241, background_thresh=145177, dirty_thresh=290355
redirty_tail +502: inode 14042
bdi_nr_reclaimable=21640, bdi_thresh=24905, background_thresh=145177, dirty_thresh=290355
mm/page-writeback.c 762 background_writeout: comm=pdflush pid=599 n=-6696985
bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21000, bdi_thresh=24823, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21040, bdi_thresh=24486, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15880, bdi_thresh=18241, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=22840, bdi_thresh=26421, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21640, bdi_thresh=24867, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15640, bdi_thresh=19034, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15240, bdi_thresh=16820, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=24200, bdi_thresh=27824, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21040, bdi_thresh=24485, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21640, bdi_thresh=24860, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=24200, bdi_thresh=27820, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=23080, bdi_thresh=26611, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=22840, bdi_thresh=26420, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=20480, bdi_thresh=23801, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21040, bdi_thresh=24480, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21640, bdi_thresh=24854, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15640, bdi_thresh=19026, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=24200, bdi_thresh=27822, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21000, bdi_thresh=24823, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=23080, bdi_thresh=26615, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=23920, bdi_thresh=27412, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21040, bdi_thresh=24488, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21040, bdi_thresh=24471, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15240, bdi_thresh=16818, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15880, bdi_thresh=18241, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21000, bdi_thresh=24822, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=24200, bdi_thresh=27822, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15880, bdi_thresh=18241, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15240, bdi_thresh=16819, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=20480, bdi_thresh=23794, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21000, bdi_thresh=24820, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21040, bdi_thresh=24476, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21040, bdi_thresh=24487, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=20480, bdi_thresh=23785, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=24200, bdi_thresh=27820, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15640, bdi_thresh=19030, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=23920, bdi_thresh=27422, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21640, bdi_thresh=24867, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=22840, bdi_thresh=26424, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=24200, bdi_thresh=27824, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21000, bdi_thresh=24820, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=23920, bdi_thresh=27427, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=23920, bdi_thresh=27426, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=23080, bdi_thresh=26626, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=23920, bdi_thresh=27416, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21640, bdi_thresh=24866, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15240, bdi_thresh=16818, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21640, bdi_thresh=24867, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=23920, bdi_thresh=27425, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=24200, bdi_thresh=27821, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=20480, bdi_thresh=23802, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15880, bdi_thresh=18239, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15880, bdi_thresh=18238, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15880, bdi_thresh=18240, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21040, bdi_thresh=24473, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=24200, bdi_thresh=27823, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=24200, bdi_thresh=27810, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15880, bdi_thresh=18240, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21640, bdi_thresh=24864, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21040, bdi_thresh=24485, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145177, dirty_thresh=290355
mm/page-writeback.c 762 background_writeout: comm=pdflush pid=10090 n=-6621693
bdi_nr_reclaimable=21040, bdi_thresh=24482, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21040, bdi_thresh=24483, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15880, bdi_thresh=18239, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=22840, bdi_thresh=26418, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21000, bdi_thresh=24786, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=20480, bdi_thresh=23800, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=24200, bdi_thresh=27823, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21640, bdi_thresh=24862, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15240, bdi_thresh=16817, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15880, bdi_thresh=18239, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=22840, bdi_thresh=26425, background_thresh=145177, dirty_thresh=290355
global dirty=246665 writeback=43800 nfs=0 flags=C_ towrite=1022 skipped=0
bdi_nr_reclaimable=21000, bdi_thresh=24784, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21000, bdi_thresh=24785, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=23080, bdi_thresh=26618, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21040, bdi_thresh=24480, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21040, bdi_thresh=24484, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21640, bdi_thresh=24863, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15240, bdi_thresh=16819, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=23080, bdi_thresh=26623, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21640, bdi_thresh=24863, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21640, bdi_thresh=24864, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=24200, bdi_thresh=27815, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=23920, bdi_thresh=27426, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=24200, bdi_thresh=27821, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=23920, bdi_thresh=27421, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=23920, bdi_thresh=27413, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=22840, bdi_thresh=26420, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15880, bdi_thresh=18241, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21000, bdi_thresh=24785, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15880, bdi_thresh=18238, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=23080, bdi_thresh=26626, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=23920, bdi_thresh=27423, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=20480, bdi_thresh=23802, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=22840, bdi_thresh=26421, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21440, bdi_thresh=24864, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15880, bdi_thresh=18239, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=23920, bdi_thresh=27421, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15640, bdi_thresh=19030, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15880, bdi_thresh=18241, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15880, bdi_thresh=18240, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21360, bdi_thresh=24862, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21360, bdi_thresh=24903, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=23920, bdi_thresh=27418, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=22840, bdi_thresh=26414, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21320, bdi_thresh=24865, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=23080, bdi_thresh=26624, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=15880, bdi_thresh=18242, background_thresh=145177, dirty_thresh=290355
global dirty=246301 writeback=44164 nfs=0 flags=C_ towrite=646 skipped=0
bdi_nr_reclaimable=15240, bdi_thresh=16818, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=19720, bdi_thresh=23385, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21000, bdi_thresh=24786, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21280, bdi_thresh=24863, background_thresh=145177, dirty_thresh=290355
redirty_tail +435: inode 174623
bdi_nr_reclaimable=21000, bdi_thresh=24785, background_thresh=145177, dirty_thresh=290355
bdi_nr_reclaimable=21000, bdi_thresh=24783, background_thresh=145177, dirty_thresh=290355
redirty_tail +435: inode 4784181
redirty_tail +435: inode 3474323
requeue_io +527: inode 0
redirty_tail +502: inode 14061
bdi_nr_reclaimable=23080, bdi_thresh=26623, background_thresh=145177, dirty_thresh=290355
redirty_tail +442: inode 27
redirty_tail +435: inode 174623
redirty_tail +435: inode 4784181
redirty_tail +435: inode 3474323
requeue_io +527: inode 0
redirty_tail +502: inode 14096

Re: regression in page writeback

[Andrew Morton]

On Wed, 23 Sep 2009 09:45:00 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:

> On Wed, Sep 23, 2009 at 09:28:32AM +0800, Andrew Morton wrote:
> > On Wed, 23 Sep 2009 09:17:58 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> > 
> > > On Wed, Sep 23, 2009 at 08:54:52AM +0800, Andrew Morton wrote:
> > > > On Wed, 23 Sep 2009 08:22:20 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> > > > 
> > > > > Jens' per-bdi writeback has another improvement. In 2.6.31, when
> > > > > superblocks A and B both have 100000 dirty pages, it will first
> > > > > exhaust A's 100000 dirty pages before going on to sync B's.
> > > > 
> > > > That would only be true if someone broke 2.6.31.  Did they?
> > > > 
> > > > SYSCALL_DEFINE0(sync)
> > > > {
> > > > 	wakeup_pdflush(0);
> > > > 	sync_filesystems(0);
> > > > 	sync_filesystems(1);
> > > > 	if (unlikely(laptop_mode))
> > > > 		laptop_sync_completion();
> > > > 	return 0;
> > > > }
> > > > 
> > > > the sync_filesystems(0) is supposed to non-blockingly start IO against
> > > > all devices.  It used to do that correctly.  But people mucked with it
> > > > so perhaps it no longer does.
> > > 
> > > I'm referring to writeback_inodes(). Each invocation of which (to sync
> > > 4MB) will do the same iteration over superblocks A => B => C ... So if
> > > A has dirty pages, it will always be served first.
> > > 
> > > So if wbc->bdi == NULL (which is true for kupdate/background sync), it
> > > will have to first exhaust A before going on to B and C.
> > 
> > But that works OK.  We fill the first device's queue, then it gets
> > congested and sync_sb_inodes() does nothing and we advance to the next
> > queue.
> 
> So in common cases "exhaust" is a bit exaggerated, but A does receive
> much more opportunity than B. Computation resources for IO submission
> are unbalanced for A, and there are pointless overheads in rechecking A.

That's unquantified handwaving.  One CPU can do a *lot* of IO.

> > If a device has more than a queue's worth of dirty data then we'll
> > probably leave some of that dirty memory un-queued, so there's some
> > lack of concurrency in that situation.
> 
> Good insight.

It was wrong.  See the other email.

> That possibly explains one major factor of the
> performance gains of Jens' per-bdi writeback.

I've yet to see any believable and complete explanation for these
gains.  I've asked about these things multiple times and nothing happened.

I suspect that what happened over time was that previously-working code
got broken, then later people noticed the breakage but failed to
analyse and fix it in favour of simply ripping everything out and
starting again.

So for the want of analysing and fixing several possible regressions,
we've tossed away some very sensitive core kernel code which had tens
of millions of machine-years testing.  I find this incredibly rash.

Re: regression in page writeback

[Wu Fengguang]

On Wed, Sep 23, 2009 at 09:47:26AM +0800, Andrew Morton wrote:
> On Wed, 23 Sep 2009 09:32:36 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> 
> > On Wed, Sep 23, 2009 at 09:28:32AM +0800, Andrew Morton wrote:
> > > On Wed, 23 Sep 2009 09:17:58 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> > > 
> > > > On Wed, Sep 23, 2009 at 08:54:52AM +0800, Andrew Morton wrote:
> > > > > On Wed, 23 Sep 2009 08:22:20 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> > > > > 
> > > > > > Jens' per-bdi writeback has another improvement. In 2.6.31, when
> > > > > > superblocks A and B both have 100000 dirty pages, it will first
> > > > > > exhaust A's 100000 dirty pages before going on to sync B's.
> > > > > 
> > > > > That would only be true if someone broke 2.6.31.  Did they?
> > > > > 
> > > > > SYSCALL_DEFINE0(sync)
> > > > > {
> > > > > 	wakeup_pdflush(0);
> > > > > 	sync_filesystems(0);
> > > > > 	sync_filesystems(1);
> > > > > 	if (unlikely(laptop_mode))
> > > > > 		laptop_sync_completion();
> > > > > 	return 0;
> > > > > }
> > > > > 
> > > > > the sync_filesystems(0) is supposed to non-blockingly start IO against
> > > > > all devices.  It used to do that correctly.  But people mucked with it
> > > > > so perhaps it no longer does.
> > > > 
> > > > I'm referring to writeback_inodes(). Each invocation of which (to sync
> > > > 4MB) will do the same iteration over superblocks A => B => C ... So if
> > > > A has dirty pages, it will always be served first.
> > > > 
> > > > So if wbc->bdi == NULL (which is true for kupdate/background sync), it
> > > > will have to first exhaust A before going on to B and C.
> > > 
> > > But that works OK.  We fill the first device's queue, then it gets
> > > congested and sync_sb_inodes() does nothing and we advance to the next
> > > queue.
> > > 
> > > If a device has more than a queue's worth of dirty data then we'll
> > > probably leave some of that dirty memory un-queued, so there's some
> > > lack of concurrency in that situation.
> > 
> > Yes, exactly if block device is not fast enough.
> 
> Actually, no.

Sorry my "yes" is mainly for the first paragraph. The concurrency
problem exists for both fast/slow devices.

> If there's still outstanding dirty data for any of those queues, both
> wb_kupdate() and background_writeout() will take a teeny sleep and then
> will re-poll the queues.
> 
> Did that logic get broken?

No, but the "teeny sleep" is normally much smaller. When io queue is
not congested, every io completion event will wakeup the congestion
waiters. Also A's event could wake up B's waiters.

__freed_request() always calls blk_clear_queue_congested() if under
congestion threshold which in turn wakes up congestion waiters:

        if (rl->count[sync] < queue_congestion_off_threshold(q))
                blk_clear_queue_congested(q, sync);

Thanks,
Fengguang

Re: regression in page writeback

[Andrew Morton]

On Wed, 23 Sep 2009 10:01:04 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:

> > If there's still outstanding dirty data for any of those queues, both
> > wb_kupdate() and background_writeout() will take a teeny sleep and then
> > will re-poll the queues.
> > 
> > Did that logic get broken?
> 
> No, but the "teeny sleep" is normally much smaller. When io queue is
> not congested, every io completion event will wakeup the congestion
> waiters. Also A's event could wake up B's waiters.
> 
> __freed_request() always calls blk_clear_queue_congested() if under
> congestion threshold which in turn wakes up congestion waiters:
> 
>         if (rl->count[sync] < queue_congestion_off_threshold(q))
>                 blk_clear_queue_congested(q, sync);
> 

Yes.  Have any problems been demonstrated due to that?

And what's _sufficiently_ wrong with that to justify adding potentially
thousands of kernel threads?  It was always a design objective to avoid
doing that.

Re: regression in page writeback

[Wu Fengguang]

On Wed, Sep 23, 2009 at 09:59:41AM +0800, Andrew Morton wrote:
> On Wed, 23 Sep 2009 09:45:00 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> 
> > On Wed, Sep 23, 2009 at 09:28:32AM +0800, Andrew Morton wrote:
> > > On Wed, 23 Sep 2009 09:17:58 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> > > 
> > > > On Wed, Sep 23, 2009 at 08:54:52AM +0800, Andrew Morton wrote:
> > > > > On Wed, 23 Sep 2009 08:22:20 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> > > > > 
> > > > > > Jens' per-bdi writeback has another improvement. In 2.6.31, when
> > > > > > superblocks A and B both have 100000 dirty pages, it will first
> > > > > > exhaust A's 100000 dirty pages before going on to sync B's.
> > > > > 
> > > > > That would only be true if someone broke 2.6.31.  Did they?
> > > > > 
> > > > > SYSCALL_DEFINE0(sync)
> > > > > {
> > > > > 	wakeup_pdflush(0);
> > > > > 	sync_filesystems(0);
> > > > > 	sync_filesystems(1);
> > > > > 	if (unlikely(laptop_mode))
> > > > > 		laptop_sync_completion();
> > > > > 	return 0;
> > > > > }
> > > > > 
> > > > > the sync_filesystems(0) is supposed to non-blockingly start IO against
> > > > > all devices.  It used to do that correctly.  But people mucked with it
> > > > > so perhaps it no longer does.
> > > > 
> > > > I'm referring to writeback_inodes(). Each invocation of which (to sync
> > > > 4MB) will do the same iteration over superblocks A => B => C ... So if
> > > > A has dirty pages, it will always be served first.
> > > > 
> > > > So if wbc->bdi == NULL (which is true for kupdate/background sync), it
> > > > will have to first exhaust A before going on to B and C.
> > > 
> > > But that works OK.  We fill the first device's queue, then it gets
> > > congested and sync_sb_inodes() does nothing and we advance to the next
> > > queue.
> > 
> > So in common cases "exhaust" is a bit exaggerated, but A does receive
> > much more opportunity than B. Computation resources for IO submission
> > are unbalanced for A, and there are pointless overheads in rechecking A.
> 
> That's unquantified handwaving.  One CPU can do a *lot* of IO.

Yes.. I had the impression that the writeback submission can be pretty slow.
It should be because of the congestion_wait. Now that it is removed,
things are going faster when queue is not full.

> > > If a device has more than a queue's worth of dirty data then we'll
> > > probably leave some of that dirty memory un-queued, so there's some
> > > lack of concurrency in that situation.
> > 
> > Good insight.
> 
> It was wrong.  See the other email.

No your first insight is correct. Because the (unnecessary) teeny
sleeps is independent of the A=>B=>C traversing order. Only queue
congestion could help skip A.

> > That possibly explains one major factor of the
> > performance gains of Jens' per-bdi writeback.
> 
> I've yet to see any believable and complete explanation for these
> gains.  I've asked about these things multiple times and nothing happened.
 
The per-bdi writeback threads does make things more straight.
But given that Jens also piggy backed some other behavior changes,
it's hard to judge the pure gain of the per-bdi writeback.

> I suspect that what happened over time was that previously-working code
> got broken, then later people noticed the breakage but failed to
> analyse and fix it in favour of simply ripping everything out and
> starting again.
>
> So for the want of analysing and fixing several possible regressions,
> we've tossed away some very sensitive core kernel code which had tens
> of millions of machine-years testing.  I find this incredibly rash.

Sorry.. 

Re: regression in page writeback

[Andrew Morton]

On Wed, 23 Sep 2009 10:26:22 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:

> On Wed, Sep 23, 2009 at 09:59:41AM +0800, Andrew Morton wrote:
> > On Wed, 23 Sep 2009 09:45:00 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> > 
> > > On Wed, Sep 23, 2009 at 09:28:32AM +0800, Andrew Morton wrote:
> > > > On Wed, 23 Sep 2009 09:17:58 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> > > > 
> > > > > On Wed, Sep 23, 2009 at 08:54:52AM +0800, Andrew Morton wrote:
> > > > > > On Wed, 23 Sep 2009 08:22:20 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> > > > > > 
> > > > > > > Jens' per-bdi writeback has another improvement. In 2.6.31, when
> > > > > > > superblocks A and B both have 100000 dirty pages, it will first
> > > > > > > exhaust A's 100000 dirty pages before going on to sync B's.
> > > > > > 
> > > > > > That would only be true if someone broke 2.6.31.  Did they?
> > > > > > 
> > > > > > SYSCALL_DEFINE0(sync)
> > > > > > {
> > > > > > 	wakeup_pdflush(0);
> > > > > > 	sync_filesystems(0);
> > > > > > 	sync_filesystems(1);
> > > > > > 	if (unlikely(laptop_mode))
> > > > > > 		laptop_sync_completion();
> > > > > > 	return 0;
> > > > > > }
> > > > > > 
> > > > > > the sync_filesystems(0) is supposed to non-blockingly start IO against
> > > > > > all devices.  It used to do that correctly.  But people mucked with it
> > > > > > so perhaps it no longer does.
> > > > > 
> > > > > I'm referring to writeback_inodes(). Each invocation of which (to sync
> > > > > 4MB) will do the same iteration over superblocks A => B => C ... So if
> > > > > A has dirty pages, it will always be served first.
> > > > > 
> > > > > So if wbc->bdi == NULL (which is true for kupdate/background sync), it
> > > > > will have to first exhaust A before going on to B and C.
> > > > 
> > > > But that works OK.  We fill the first device's queue, then it gets
> > > > congested and sync_sb_inodes() does nothing and we advance to the next
> > > > queue.
> > > 
> > > So in common cases "exhaust" is a bit exaggerated, but A does receive
> > > much more opportunity than B. Computation resources for IO submission
> > > are unbalanced for A, and there are pointless overheads in rechecking A.
> > 
> > That's unquantified handwaving.  One CPU can do a *lot* of IO.
> 
> Yes.. I had the impression that the writeback submission can be pretty slow.
> It should be because of the congestion_wait. Now that it is removed,
> things are going faster when queue is not full.

What?  The wait is short.  The design intent there is that we repoll
all previously-congested queues well before they start to run empty.

> > > > If a device has more than a queue's worth of dirty data then we'll
> > > > probably leave some of that dirty memory un-queued, so there's some
> > > > lack of concurrency in that situation.
> > > 
> > > Good insight.
> > 
> > It was wrong.  See the other email.
> 
> No your first insight is correct. Because the (unnecessary) teeny
> sleeps is independent of the A=>B=>C traversing order. Only queue
> congestion could help skip A.

The sleeps are completely necessary!  Otherwise we end up busywaiting.

After the sleep we repoll all queues.

Re: regression in page writeback

[Wu Fengguang]

On Wed, Sep 23, 2009 at 10:36:22AM +0800, Andrew Morton wrote:
> On Wed, 23 Sep 2009 10:26:22 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> 
> > On Wed, Sep 23, 2009 at 09:59:41AM +0800, Andrew Morton wrote:
> > > On Wed, 23 Sep 2009 09:45:00 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> > > 
> > > > On Wed, Sep 23, 2009 at 09:28:32AM +0800, Andrew Morton wrote:
> > > > > On Wed, 23 Sep 2009 09:17:58 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> > > > > 
> > > > > > On Wed, Sep 23, 2009 at 08:54:52AM +0800, Andrew Morton wrote:
> > > > > > > On Wed, 23 Sep 2009 08:22:20 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> > > > > > > 
> > > > > > > > Jens' per-bdi writeback has another improvement. In 2.6.31, when
> > > > > > > > superblocks A and B both have 100000 dirty pages, it will first
> > > > > > > > exhaust A's 100000 dirty pages before going on to sync B's.
> > > > > > > 
> > > > > > > That would only be true if someone broke 2.6.31.  Did they?
> > > > > > > 
> > > > > > > SYSCALL_DEFINE0(sync)
> > > > > > > {
> > > > > > > 	wakeup_pdflush(0);
> > > > > > > 	sync_filesystems(0);
> > > > > > > 	sync_filesystems(1);
> > > > > > > 	if (unlikely(laptop_mode))
> > > > > > > 		laptop_sync_completion();
> > > > > > > 	return 0;
> > > > > > > }
> > > > > > > 
> > > > > > > the sync_filesystems(0) is supposed to non-blockingly start IO against
> > > > > > > all devices.  It used to do that correctly.  But people mucked with it
> > > > > > > so perhaps it no longer does.
> > > > > > 
> > > > > > I'm referring to writeback_inodes(). Each invocation of which (to sync
> > > > > > 4MB) will do the same iteration over superblocks A => B => C ... So if
> > > > > > A has dirty pages, it will always be served first.
> > > > > > 
> > > > > > So if wbc->bdi == NULL (which is true for kupdate/background sync), it
> > > > > > will have to first exhaust A before going on to B and C.
> > > > > 
> > > > > But that works OK.  We fill the first device's queue, then it gets
> > > > > congested and sync_sb_inodes() does nothing and we advance to the next
> > > > > queue.
> > > > 
> > > > So in common cases "exhaust" is a bit exaggerated, but A does receive
> > > > much more opportunity than B. Computation resources for IO submission
> > > > are unbalanced for A, and there are pointless overheads in rechecking A.
> > > 
> > > That's unquantified handwaving.  One CPU can do a *lot* of IO.
> > 
> > Yes.. I had the impression that the writeback submission can be pretty slow.
> > It should be because of the congestion_wait. Now that it is removed,
> > things are going faster when queue is not full.
> 
> What?  The wait is short.  The design intent there is that we repoll
> all previously-congested queues well before they start to run empty.

When queue is not congested (in which case congestion_wait is not
necessary), the congestion_wait() degrades io submission speed to near
io completion speed.

> > > > > If a device has more than a queue's worth of dirty data then we'll
> > > > > probably leave some of that dirty memory un-queued, so there's some
> > > > > lack of concurrency in that situation.
> > > > 
> > > > Good insight.
> > > 
> > > It was wrong.  See the other email.
> > 
> > No your first insight is correct. Because the (unnecessary) teeny
> > sleeps is independent of the A=>B=>C traversing order. Only queue
> > congestion could help skip A.
> 
> The sleeps are completely necessary!  Otherwise we end up busywaiting.
> 
> After the sleep we repoll all queues.

I mean, it is not always necessary. Only when _all_ superblocks cannot
writeback their inodes (eg. all in congestion), we should wait.

Just before Jens' work, I had patch to convert

-                       if (wbc.encountered_congestion || wbc.more_io)
-                               congestion_wait(WRITE, HZ/10);
-                       else
-                               break;

to

+       if (wbc->encountered_congestion && wbc->nr_to_write == MAX_WRITEBACK_PAGES)
+               congestion_wait(WRITE, HZ/10);

Note that wbc->encountered_congestion only means "at least one bdi
encountered congestion". We may still make progress in other bdis
hence should not sleep.

Thanks,
Fengguang

Re: regression in page writeback

[Andrew Morton]

On Wed, 23 Sep 2009 10:49:58 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:

> > After the sleep we repoll all queues.
> 
> I mean, it is not always necessary. Only when _all_ superblocks cannot
> writeback their inodes (eg. all in congestion), we should wait.

Well.  The code will still submit MAX_WRITEBACK_PAGES pages to _some_
queue in that situation.

But yup, maybe that's a bug!  Sounds easy to fix.  It doesn't justify
rewriting the whole world and then saying "look, the new code is faster
than the old code which had a bug which I knew about but didn't fix".

Re: regression in page writeback

[Wu Fengguang]

On Wed, Sep 23, 2009 at 10:09:15AM +0800, Andrew Morton wrote:
> On Wed, 23 Sep 2009 10:01:04 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> 
> > > If there's still outstanding dirty data for any of those queues, both
> > > wb_kupdate() and background_writeout() will take a teeny sleep and then
> > > will re-poll the queues.
> > > 
> > > Did that logic get broken?
> > 
> > No, but the "teeny sleep" is normally much smaller. When io queue is
> > not congested, every io completion event will wakeup the congestion
> > waiters. Also A's event could wake up B's waiters.
> > 
> > __freed_request() always calls blk_clear_queue_congested() if under
> > congestion threshold which in turn wakes up congestion waiters:
> > 
> >         if (rl->count[sync] < queue_congestion_off_threshold(q))
> >                 blk_clear_queue_congested(q, sync);
> > 
> 
> Yes.  Have any problems been demonstrated due to that?

Hmm, I was merely clarifying a fact.

Chris Mason listed some reasons to convert the congestion_wait() based
polls to the some queue waiting:
 
  http://lkml.org/lkml/2009/9/8/210

My impression is, it changed obviously too fast without enough discussions.

> And what's _sufficiently_ wrong with that to justify adding potentially
> thousands of kernel threads?  It was always a design objective to avoid
> doing that.

Yeah, the number of threads could be a problem. 

I guess the per-bdi threads and congestion_wait are mostly two
independent changes.  congestion_wait is not the reason to do per-bdi
threads.  Just that Jens piggy backed some changes to congestion_wait.

Thanks,
Fengguang

Re: regression in page writeback

[Shaohua Li]

On Wed, Sep 23, 2009 at 10:49:58AM +0800, Wu, Fengguang wrote:
> On Wed, Sep 23, 2009 at 10:36:22AM +0800, Andrew Morton wrote:
> > On Wed, 23 Sep 2009 10:26:22 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> > 
> > > On Wed, Sep 23, 2009 at 09:59:41AM +0800, Andrew Morton wrote:
> > > > On Wed, 23 Sep 2009 09:45:00 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> > > > 
> > > > > On Wed, Sep 23, 2009 at 09:28:32AM +0800, Andrew Morton wrote:
> > > > > > On Wed, 23 Sep 2009 09:17:58 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> > > > > > 
> > > > > > > On Wed, Sep 23, 2009 at 08:54:52AM +0800, Andrew Morton wrote:
> > > > > > > > On Wed, 23 Sep 2009 08:22:20 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> > > > > > > > 
> > > > > > > > > Jens' per-bdi writeback has another improvement. In 2.6.31, when
> > > > > > > > > superblocks A and B both have 100000 dirty pages, it will first
> > > > > > > > > exhaust A's 100000 dirty pages before going on to sync B's.
> > > > > > > > 
> > > > > > > > That would only be true if someone broke 2.6.31.  Did they?
> > > > > > > > 
> > > > > > > > SYSCALL_DEFINE0(sync)
> > > > > > > > {
> > > > > > > > 	wakeup_pdflush(0);
> > > > > > > > 	sync_filesystems(0);
> > > > > > > > 	sync_filesystems(1);
> > > > > > > > 	if (unlikely(laptop_mode))
> > > > > > > > 		laptop_sync_completion();
> > > > > > > > 	return 0;
> > > > > > > > }
> > > > > > > > 
> > > > > > > > the sync_filesystems(0) is supposed to non-blockingly start IO against
> > > > > > > > all devices.  It used to do that correctly.  But people mucked with it
> > > > > > > > so perhaps it no longer does.
> > > > > > > 
> > > > > > > I'm referring to writeback_inodes(). Each invocation of which (to sync
> > > > > > > 4MB) will do the same iteration over superblocks A => B => C ... So if
> > > > > > > A has dirty pages, it will always be served first.
> > > > > > > 
> > > > > > > So if wbc->bdi == NULL (which is true for kupdate/background sync), it
> > > > > > > will have to first exhaust A before going on to B and C.
> > > > > > 
> > > > > > But that works OK.  We fill the first device's queue, then it gets
> > > > > > congested and sync_sb_inodes() does nothing and we advance to the next
> > > > > > queue.
> > > > > 
> > > > > So in common cases "exhaust" is a bit exaggerated, but A does receive
> > > > > much more opportunity than B. Computation resources for IO submission
> > > > > are unbalanced for A, and there are pointless overheads in rechecking A.
> > > > 
> > > > That's unquantified handwaving.  One CPU can do a *lot* of IO.
> > > 
> > > Yes.. I had the impression that the writeback submission can be pretty slow.
> > > It should be because of the congestion_wait. Now that it is removed,
> > > things are going faster when queue is not full.
> > 
> > What?  The wait is short.  The design intent there is that we repoll
> > all previously-congested queues well before they start to run empty.
> 
> When queue is not congested (in which case congestion_wait is not
> necessary), the congestion_wait() degrades io submission speed to near
> io completion speed.
> 
> > > > > > If a device has more than a queue's worth of dirty data then we'll
> > > > > > probably leave some of that dirty memory un-queued, so there's some
> > > > > > lack of concurrency in that situation.
> > > > > 
> > > > > Good insight.
> > > > 
> > > > It was wrong.  See the other email.
> > > 
> > > No your first insight is correct. Because the (unnecessary) teeny
> > > sleeps is independent of the A=>B=>C traversing order. Only queue
> > > congestion could help skip A.
> > 
> > The sleeps are completely necessary!  Otherwise we end up busywaiting.
> > 
> > After the sleep we repoll all queues.
> 
> I mean, it is not always necessary. Only when _all_ superblocks cannot
> writeback their inodes (eg. all in congestion), we should wait.
> 
> Just before Jens' work, I had patch to convert
> 
> -                       if (wbc.encountered_congestion || wbc.more_io)
> -                               congestion_wait(WRITE, HZ/10);
> -                       else
> -                               break;
> 
> to
> 
> +       if (wbc->encountered_congestion && wbc->nr_to_write == MAX_WRITEBACK_PAGES)
> +               congestion_wait(WRITE, HZ/10);
> 
> Note that wbc->encountered_congestion only means "at least one bdi
> encountered congestion". We may still make progress in other bdis
> hence should not sleep.
Hi,
encountered_congestion only is checked when nr_to_write > 0, if some superblocks
aren't congestions, nr_to_write should be 0, right?

Thanks,
Shaohua

Re: regression in page writeback

[Wu Fengguang]

On Wed, Sep 23, 2009 at 10:56:05AM +0800, Andrew Morton wrote:
> On Wed, 23 Sep 2009 10:49:58 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> 
> > > After the sleep we repoll all queues.
> > 
> > I mean, it is not always necessary. Only when _all_ superblocks cannot
> > writeback their inodes (eg. all in congestion), we should wait.
> 
> Well.  The code will still submit MAX_WRITEBACK_PAGES pages to _some_
> queue in that situation.
> 
> But yup, maybe that's a bug!  Sounds easy to fix.  It doesn't justify
> rewriting the whole world and then saying "look, the new code is faster
> than the old code which had a bug which I knew about but didn't fix".

I didn't submit the patches to avoid conflicts with Jens' work.
Sorry for doing that - that's not good attitude towards bug fixes..

Thanks,
Fengguang

Re: regression in page writeback

[Wu Fengguang]

On Wed, Sep 23, 2009 at 11:10:12AM +0800, Li, Shaohua wrote:
> On Wed, Sep 23, 2009 at 10:49:58AM +0800, Wu, Fengguang wrote:
> > On Wed, Sep 23, 2009 at 10:36:22AM +0800, Andrew Morton wrote:
> > > On Wed, 23 Sep 2009 10:26:22 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> > > 
> > > > On Wed, Sep 23, 2009 at 09:59:41AM +0800, Andrew Morton wrote:
> > > > > On Wed, 23 Sep 2009 09:45:00 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> > > > > 
> > > > > > On Wed, Sep 23, 2009 at 09:28:32AM +0800, Andrew Morton wrote:
> > > > > > > On Wed, 23 Sep 2009 09:17:58 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> > > > > > > 
> > > > > > > > On Wed, Sep 23, 2009 at 08:54:52AM +0800, Andrew Morton wrote:
> > > > > > > > > On Wed, 23 Sep 2009 08:22:20 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> > > > > > > > > 
> > > > > > > > > > Jens' per-bdi writeback has another improvement. In 2.6.31, when
> > > > > > > > > > superblocks A and B both have 100000 dirty pages, it will first
> > > > > > > > > > exhaust A's 100000 dirty pages before going on to sync B's.
> > > > > > > > > 
> > > > > > > > > That would only be true if someone broke 2.6.31.  Did they?
> > > > > > > > > 
> > > > > > > > > SYSCALL_DEFINE0(sync)
> > > > > > > > > {
> > > > > > > > > 	wakeup_pdflush(0);
> > > > > > > > > 	sync_filesystems(0);
> > > > > > > > > 	sync_filesystems(1);
> > > > > > > > > 	if (unlikely(laptop_mode))
> > > > > > > > > 		laptop_sync_completion();
> > > > > > > > > 	return 0;
> > > > > > > > > }
> > > > > > > > > 
> > > > > > > > > the sync_filesystems(0) is supposed to non-blockingly start IO against
> > > > > > > > > all devices.  It used to do that correctly.  But people mucked with it
> > > > > > > > > so perhaps it no longer does.
> > > > > > > > 
> > > > > > > > I'm referring to writeback_inodes(). Each invocation of which (to sync
> > > > > > > > 4MB) will do the same iteration over superblocks A => B => C ... So if
> > > > > > > > A has dirty pages, it will always be served first.
> > > > > > > > 
> > > > > > > > So if wbc->bdi == NULL (which is true for kupdate/background sync), it
> > > > > > > > will have to first exhaust A before going on to B and C.
> > > > > > > 
> > > > > > > But that works OK.  We fill the first device's queue, then it gets
> > > > > > > congested and sync_sb_inodes() does nothing and we advance to the next
> > > > > > > queue.
> > > > > > 
> > > > > > So in common cases "exhaust" is a bit exaggerated, but A does receive
> > > > > > much more opportunity than B. Computation resources for IO submission
> > > > > > are unbalanced for A, and there are pointless overheads in rechecking A.
> > > > > 
> > > > > That's unquantified handwaving.  One CPU can do a *lot* of IO.
> > > > 
> > > > Yes.. I had the impression that the writeback submission can be pretty slow.
> > > > It should be because of the congestion_wait. Now that it is removed,
> > > > things are going faster when queue is not full.
> > > 
> > > What?  The wait is short.  The design intent there is that we repoll
> > > all previously-congested queues well before they start to run empty.
> > 
> > When queue is not congested (in which case congestion_wait is not
> > necessary), the congestion_wait() degrades io submission speed to near
> > io completion speed.
> > 
> > > > > > > If a device has more than a queue's worth of dirty data then we'll
> > > > > > > probably leave some of that dirty memory un-queued, so there's some
> > > > > > > lack of concurrency in that situation.
> > > > > > 
> > > > > > Good insight.
> > > > > 
> > > > > It was wrong.  See the other email.
> > > > 
> > > > No your first insight is correct. Because the (unnecessary) teeny
> > > > sleeps is independent of the A=>B=>C traversing order. Only queue
> > > > congestion could help skip A.
> > > 
> > > The sleeps are completely necessary!  Otherwise we end up busywaiting.
> > > 
> > > After the sleep we repoll all queues.
> > 
> > I mean, it is not always necessary. Only when _all_ superblocks cannot
> > writeback their inodes (eg. all in congestion), we should wait.
> > 
> > Just before Jens' work, I had patch to convert
> > 
> > -                       if (wbc.encountered_congestion || wbc.more_io)
> > -                               congestion_wait(WRITE, HZ/10);
> > -                       else
> > -                               break;
> > 
> > to
> > 
> > +       if (wbc->encountered_congestion && wbc->nr_to_write == MAX_WRITEBACK_PAGES)
> > +               congestion_wait(WRITE, HZ/10);
> > 
> > Note that wbc->encountered_congestion only means "at least one bdi
> > encountered congestion". We may still make progress in other bdis
> > hence should not sleep.
> Hi,
> encountered_congestion only is checked when nr_to_write > 0, if some superblocks
> aren't congestions, nr_to_write should be 0, right?

Yeah, good spot! So the change only helps some corner cases.

Thanks,
Fengguang

Re: regression in page writeback

[Wu Fengguang]

On Wed, Sep 23, 2009 at 11:14:50AM +0800, Wu Fengguang wrote:
> On Wed, Sep 23, 2009 at 11:10:12AM +0800, Li, Shaohua wrote:
> > On Wed, Sep 23, 2009 at 10:49:58AM +0800, Wu, Fengguang wrote:
> > > On Wed, Sep 23, 2009 at 10:36:22AM +0800, Andrew Morton wrote:
> > > > On Wed, 23 Sep 2009 10:26:22 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> > > > 
> > > > > On Wed, Sep 23, 2009 at 09:59:41AM +0800, Andrew Morton wrote:
> > > > > > On Wed, 23 Sep 2009 09:45:00 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> > > > > > 
> > > > > > > On Wed, Sep 23, 2009 at 09:28:32AM +0800, Andrew Morton wrote:
> > > > > > > > On Wed, 23 Sep 2009 09:17:58 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> > > > > > > > 
> > > > > > > > > On Wed, Sep 23, 2009 at 08:54:52AM +0800, Andrew Morton wrote:
> > > > > > > > > > On Wed, 23 Sep 2009 08:22:20 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> > > > > > > > > > 
> > > > > > > > > > > Jens' per-bdi writeback has another improvement. In 2.6.31, when
> > > > > > > > > > > superblocks A and B both have 100000 dirty pages, it will first
> > > > > > > > > > > exhaust A's 100000 dirty pages before going on to sync B's.
> > > > > > > > > > 
> > > > > > > > > > That would only be true if someone broke 2.6.31.  Did they?
> > > > > > > > > > 
> > > > > > > > > > SYSCALL_DEFINE0(sync)
> > > > > > > > > > {
> > > > > > > > > > 	wakeup_pdflush(0);
> > > > > > > > > > 	sync_filesystems(0);
> > > > > > > > > > 	sync_filesystems(1);
> > > > > > > > > > 	if (unlikely(laptop_mode))
> > > > > > > > > > 		laptop_sync_completion();
> > > > > > > > > > 	return 0;
> > > > > > > > > > }
> > > > > > > > > > 
> > > > > > > > > > the sync_filesystems(0) is supposed to non-blockingly start IO against
> > > > > > > > > > all devices.  It used to do that correctly.  But people mucked with it
> > > > > > > > > > so perhaps it no longer does.
> > > > > > > > > 
> > > > > > > > > I'm referring to writeback_inodes(). Each invocation of which (to sync
> > > > > > > > > 4MB) will do the same iteration over superblocks A => B => C ... So if
> > > > > > > > > A has dirty pages, it will always be served first.
> > > > > > > > > 
> > > > > > > > > So if wbc->bdi == NULL (which is true for kupdate/background sync), it
> > > > > > > > > will have to first exhaust A before going on to B and C.
> > > > > > > > 
> > > > > > > > But that works OK.  We fill the first device's queue, then it gets
> > > > > > > > congested and sync_sb_inodes() does nothing and we advance to the next
> > > > > > > > queue.
> > > > > > > 
> > > > > > > So in common cases "exhaust" is a bit exaggerated, but A does receive
> > > > > > > much more opportunity than B. Computation resources for IO submission
> > > > > > > are unbalanced for A, and there are pointless overheads in rechecking A.
> > > > > > 
> > > > > > That's unquantified handwaving.  One CPU can do a *lot* of IO.
> > > > > 
> > > > > Yes.. I had the impression that the writeback submission can be pretty slow.
> > > > > It should be because of the congestion_wait. Now that it is removed,
> > > > > things are going faster when queue is not full.
> > > > 
> > > > What?  The wait is short.  The design intent there is that we repoll
> > > > all previously-congested queues well before they start to run empty.
> > > 
> > > When queue is not congested (in which case congestion_wait is not
> > > necessary), the congestion_wait() degrades io submission speed to near
> > > io completion speed.
> > > 
> > > > > > > > If a device has more than a queue's worth of dirty data then we'll
> > > > > > > > probably leave some of that dirty memory un-queued, so there's some
> > > > > > > > lack of concurrency in that situation.
> > > > > > > 
> > > > > > > Good insight.
> > > > > > 
> > > > > > It was wrong.  See the other email.
> > > > > 
> > > > > No your first insight is correct. Because the (unnecessary) teeny
> > > > > sleeps is independent of the A=>B=>C traversing order. Only queue
> > > > > congestion could help skip A.
> > > > 
> > > > The sleeps are completely necessary!  Otherwise we end up busywaiting.
> > > > 
> > > > After the sleep we repoll all queues.
> > > 
> > > I mean, it is not always necessary. Only when _all_ superblocks cannot
> > > writeback their inodes (eg. all in congestion), we should wait.
> > > 
> > > Just before Jens' work, I had patch to convert
> > > 
> > > -                       if (wbc.encountered_congestion || wbc.more_io)
> > > -                               congestion_wait(WRITE, HZ/10);
> > > -                       else
> > > -                               break;
> > > 
> > > to
> > > 
> > > +       if (wbc->encountered_congestion && wbc->nr_to_write == MAX_WRITEBACK_PAGES)
> > > +               congestion_wait(WRITE, HZ/10);
> > > 
> > > Note that wbc->encountered_congestion only means "at least one bdi
> > > encountered congestion". We may still make progress in other bdis
> > > hence should not sleep.
> > Hi,
> > encountered_congestion only is checked when nr_to_write > 0, if some superblocks
> > aren't congestions, nr_to_write should be 0, right?
> 
> Yeah, good spot! So the change only helps some corner cases.

Then it remains a problem why the io submission is slow when !congested.

For example, this trace shows that the io submission speed is about

        4MB / 0.01s = 400MB/s

Workload is a plain copy on a 2Ghz Intel Core 2 CPU.

[   71.487121] mm/page-writeback.c 761 background_writeout: comm=pdflush pid=343 n=-4096
[   71.489635] global dirty=65513 writeback=5925 nfs=1 flags=_M towrite=0 skipped=0
[   71.496019] redirty_tail +442: inode 79232
[   71.497432] mm/page-writeback.c 761 background_writeout: comm=pdflush pid=343 n=-5120
[   71.498890] global dirty=64490 writeback=6700 nfs=1 flags=_M towrite=0 skipped=0
[   71.506355] redirty_tail +442: inode 79232
[   71.508473] mm/page-writeback.c 761 background_writeout: comm=pdflush pid=343 n=-6144
[   71.510538] global dirty=62475 writeback=7599 nfs=1 flags=_M towrite=0 skipped=0
[   71.511910] redirty_tail +502: inode 3438
[   71.512846] redirty_tail +502: inode 1920

Thanks,
Fengguang

Re: regression in page writeback

[Wu Fengguang]

On Wed, Sep 23, 2009 at 09:52:55AM +0800, Li, Shaohua wrote:
> On Tue, 2009-09-22 at 21:39 +0800, Wu, Fengguang wrote:
> > On Tue, Sep 22, 2009 at 07:50:15PM +0800, Li, Shaohua wrote:
> > > On Tue, Sep 22, 2009 at 06:49:15PM +0800, Wu, Fengguang wrote:
> > > > Shaohua,
> > > > 
> > > > On Tue, Sep 22, 2009 at 01:49:13PM +0800, Li, Shaohua wrote:
> > > > > Hi,
> > > > > Commit d7831a0bdf06b9f722b947bb0c205ff7d77cebd8 causes disk io regression
> > > > > in my test.
> > > > > My system has 12 disks, each disk has two partitions. System runs fio sequence
> > > > > write on all partitions, each partion has 8 jobs.
> > > > > 2.6.31-rc1, fio gives 460m/s disk io
> > > > > 2.6.31-rc2, fio gives about 400m/s disk io. Revert the patch, speed back to
> > > > > 460m/s
> > > > > 
> > > > > Under latest git: fio gives 450m/s disk io; If reverting the patch, the speed
> > > > > is 484m/s.
> > > > > 
> > > > > With the patch, fio reports less io merge and more interrupts. My naive
> > > > > analysis is the patch makes balance_dirty_pages_ratelimited_nr() limits
> > > > > write chunk to 8 pages and then soon go to sleep in balance_dirty_pages(),
> > > > > because most time the bdi_nr_reclaimable < bdi_thresh, and so when write
> > > > > the pages out, the chunk is 8 pages long instead of 4M long. Without the patch,
> > > > > thread can write 8 pages and then move some pages to writeback, and then
> > > > > continue doing write. The patch seems to break this.
> > > > 
> > > > Do you have trace/numbers for above descriptions?
> > > No. Just guess, because there is less io merge. And watch each bdi's states,
> > > bdi_nr_reclaimable < bdi_thresh seems always true.
> > 
> > Ah OK.
> > 
> > > > > Unfortunatelly I can't figure out a fix for this issue, hopefully
> > > > > you have more ideas.
> > > > 
> > > > Attached is a very verbose writeback debug patch, hope it helps and
> > > > won't disturb the workload a lot :)
> > > Hmm, the log buf will get overflowed soon, there is > 400m/s io. I tried
> > > to produce this issue in a system with two disks, but fail. Anyway, I'll try
> > > it out tomorrow.
> > 
> > Thank you~  I'd recommend to use netconsole or serial line, and stop
> > local klogd because the write of log messages could add noises. 
> attached is a short log. I'll try to get a full log after finish latest
> git test.
> bdi_nr_reclaimable is always less than bdi_thresh in the log. because

Yes. Only background_writeout() is working and queue is congested.

> when bdi_nr_reclaimable + bdi_nr_writeback > bdi_thresh, background
> writeback is already started, so bdi_nr_writeback should be > 0.

Yes, when process is throttled, bdi_nr_writeback > (bdi_thresh - bdi_nr_reclaimable).

For each background_writeout() loop (each takes about 0.1s), there are
~100 balance_dirty_pages() loops.  The latter makes pretty much loop overheads.

Most importantly, most background_writeout() loops only wrote several pages
(the large towrite values), which is extremely inefficient. I wonder why it
cannot write more?

> global dirty=246756 writeback=43800 nfs=0 flags=C_ towrite=1018 skipped=0
> global dirty=246847 writeback=43800 nfs=0 flags=C_ towrite=1023 skipped=0
> global dirty=246847 writeback=43800 nfs=0 flags=C_ towrite=1023 skipped=0
> global dirty=246847 writeback=43800 nfs=0 flags=C_ towrite=1023 skipped=0
> global dirty=246847 writeback=43709 nfs=0 flags=C_ towrite=1024 skipped=0
> global dirty=246847 writeback=43709 nfs=0 flags=C_ towrite=1022 skipped=0
> global dirty=247029 writeback=43527 nfs=0 flags=CM towrite=1020 skipped=0
> global dirty=247029 writeback=43527 nfs=0 flags=C_ towrite=1024 skipped=0
> global dirty=246665 writeback=43800 nfs=0 flags=C_ towrite=1022 skipped=0
> global dirty=246301 writeback=44164 nfs=0 flags=C_ towrite=646 skipped=0

Thanks,
Fengguang

Content-Description: msg2
> 0, bdi_thresh=23810, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=24160, bdi_thresh=27860, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=23080, bdi_thresh=26626, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=23920, bdi_thresh=27422, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145141, dirty_thresh=290283
> mm/page-writeback.c 762 background_writeout: comm=pdflush pid=599 n=-6696602
> bdi_nr_reclaimable=21040, bdi_thresh=24479, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=20480, bdi_thresh=23802, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=21480, bdi_thresh=24906, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=24160, bdi_thresh=27862, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=23920, bdi_thresh=27416, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=15240, bdi_thresh=16824, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=20480, bdi_thresh=23809, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=21480, bdi_thresh=24903, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=21480, bdi_thresh=24907, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=23080, bdi_thresh=26623, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145141, dirty_thresh=290283
> global dirty=246756 writeback=43800 nfs=0 flags=C_ towrite=1018 skipped=0
> bdi_nr_reclaimable=15640, bdi_thresh=19030, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=21040, bdi_thresh=24481, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=24160, bdi_thresh=27850, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=21040, bdi_thresh=24493, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=21320, bdi_thresh=24944, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=21320, bdi_thresh=24940, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=15880, bdi_thresh=18244, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=21040, bdi_thresh=24493, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=24160, bdi_thresh=27864, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145141, dirty_thresh=290283
> redirty_tail +435: inode 174623
> bdi_nr_reclaimable=21480, bdi_thresh=24894, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=23920, bdi_thresh=27412, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=21480, bdi_thresh=24900, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=24160, bdi_thresh=27862, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=24160, bdi_thresh=27855, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=15640, bdi_thresh=19026, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=21040, bdi_thresh=24484, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=22840, bdi_thresh=26420, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=21320, bdi_thresh=24943, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=15240, bdi_thresh=16824, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=23080, bdi_thresh=26611, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=21480, bdi_thresh=24907, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=20480, bdi_thresh=23793, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=21040, bdi_thresh=24479, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=20480, bdi_thresh=23802, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=24160, bdi_thresh=27862, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=15240, bdi_thresh=16825, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=21040, bdi_thresh=24491, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=23920, bdi_thresh=27425, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=23080, bdi_thresh=26615, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=21320, bdi_thresh=24942, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=23080, bdi_thresh=26625, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=24160, bdi_thresh=27860, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=22840, bdi_thresh=26421, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=19720, bdi_thresh=23425, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=21480, bdi_thresh=24904, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=21320, bdi_thresh=24946, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=15880, bdi_thresh=18246, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=21480, bdi_thresh=24906, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=23920, bdi_thresh=27422, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=23920, bdi_thresh=27416, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=22840, bdi_thresh=26418, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=20480, bdi_thresh=23809, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=15880, bdi_thresh=18246, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=21480, bdi_thresh=24907, background_thresh=145141, dirty_thresh=290283
> redirty_tail +435: inode 4784181
> bdi_nr_reclaimable=21480, bdi_thresh=24902, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=15240, bdi_thresh=16826, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=15240, bdi_thresh=16824, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=21320, bdi_thresh=24944, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=21040, bdi_thresh=24495, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=23920, bdi_thresh=27413, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=15240, bdi_thresh=16824, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=24160, bdi_thresh=27863, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=23080, bdi_thresh=26618, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=21040, bdi_thresh=24492, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=21480, bdi_thresh=24905, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=21480, bdi_thresh=24903, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=21480, bdi_thresh=24903, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=15880, bdi_thresh=18246, background_thresh=145141, dirty_thresh=290283
> redirty_tail +435: inode 3474323
> bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=21480, bdi_thresh=24864, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=24160, bdi_thresh=27863, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=20480, bdi_thresh=23811, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=15880, bdi_thresh=18245, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=21480, bdi_thresh=24864, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=22840, bdi_thresh=26425, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=15240, bdi_thresh=16826, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=21040, bdi_thresh=24496, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=21320, bdi_thresh=24945, background_thresh=145141, dirty_thresh=290283
> requeue_io +527: inode 0
> bdi_nr_reclaimable=15640, bdi_thresh=19034, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=23920, bdi_thresh=27426, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=23080, bdi_thresh=26626, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=15880, bdi_thresh=18245, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=24160, bdi_thresh=27861, background_thresh=145141, dirty_thresh=290283
> requeue_io +527: inode 0
> bdi_nr_reclaimable=23920, bdi_thresh=27427, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145141, dirty_thresh=290283
> bdi_nr_reclaimable=22840, bdi_thresh=26421, background_thresh=145150, dirty_thresh=290301
> bdi_nr_reclaimable=22840, bdi_thresh=26424, background_thresh=145150, dirty_thresh=290301
> bdi_nr_reclaimable=21040, bdi_thresh=24489, background_thresh=145150, dirty_thresh=290301
> bdi_nr_reclaimable=21320, bdi_thresh=24906, background_thresh=145150, dirty_thresh=290301
> bdi_nr_reclaimable=23920, bdi_thresh=27426, background_thresh=145150, dirty_thresh=290301
> redirty_tail +502: inode 12415
> bdi_nr_reclaimable=24160, bdi_thresh=27861, background_thresh=145150, dirty_thresh=290301
> bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145150, dirty_thresh=290301
> bdi_nr_reclaimable=21320, bdi_thresh=24905, background_thresh=145150, dirty_thresh=290301
> bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145150, dirty_thresh=290301
> bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145150, dirty_thresh=290301
> bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145159, dirty_thresh=290319
> bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145159, dirty_thresh=290319
> bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145159, dirty_thresh=290319
> bdi_nr_reclaimable=21640, bdi_thresh=24923, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24919, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24921, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24918, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27421, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24905, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=20480, bdi_thresh=23812, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26623, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24495, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24490, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24903, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27423, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24903, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18246, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27421, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18250, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24919, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24496, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26414, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24160, bdi_thresh=27820, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19030, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24919, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27418, background_thresh=145168, dirty_thresh=290337
> mm/page-writeback.c 762 background_writeout: comm=pdflush pid=10090 n=-6621688
> bdi_nr_reclaimable=23080, bdi_thresh=26624, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15240, bdi_thresh=16826, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26420, background_thresh=145168, dirty_thresh=290337
> global dirty=246847 writeback=43800 nfs=0 flags=C_ towrite=1023 skipped=0
> bdi_nr_reclaimable=21320, bdi_thresh=24900, background_thresh=145168, dirty_thresh=290337
> redirty_tail +435: inode 174623
> bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24160, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
> redirty_tail +435: inode 4784181
> redirty_tail +435: inode 3474323
> requeue_io +527: inode 0
> requeue_io +527: inode 0
> redirty_tail +502: inode 12396
> mm/page-writeback.c 762 background_writeout: comm=pdflush pid=10090 n=-6621689
> bdi_nr_reclaimable=24160, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24900, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15240, bdi_thresh=16826, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26624, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24160, bdi_thresh=27820, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24490, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=20480, bdi_thresh=23812, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27421, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24919, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27427, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15240, bdi_thresh=16827, background_thresh=145168, dirty_thresh=290337
> global dirty=246847 writeback=43800 nfs=0 flags=C_ towrite=1023 skipped=0
> bdi_nr_reclaimable=22840, bdi_thresh=26420, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24910, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=20480, bdi_thresh=23811, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24903, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24904, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24900, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24923, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26611, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24916, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24160, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24160, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18245, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24160, bdi_thresh=27810, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26615, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27412, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24495, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27425, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18248, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27422, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=19720, bdi_thresh=23425, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24160, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24481, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24160, bdi_thresh=27820, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=20480, bdi_thresh=23804, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19026, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24486, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=20480, bdi_thresh=23795, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26625, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24493, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24923, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19030, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15240, bdi_thresh=16825, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24483, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24494, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15240, bdi_thresh=16826, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27416, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26418, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26623, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26623, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24160, bdi_thresh=27815, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24920, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24906, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24920, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24922, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24495, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=20480, bdi_thresh=23811, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15240, bdi_thresh=16825, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26618, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27413, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27421, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26425, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24904, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15240, bdi_thresh=16827, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24498, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27426, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24902, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24497, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24160, bdi_thresh=27823, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24918, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24921, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24920, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18246, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=20480, bdi_thresh=23813, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26414, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26626, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27423, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15240, bdi_thresh=16825, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24160, bdi_thresh=27821, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18249, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24160, bdi_thresh=27823, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24919, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18248, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27426, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18248, background_thresh=145168, dirty_thresh=290337
> mm/page-writeback.c 762 background_writeout: comm=pdflush pid=599 n=-6696603
> bdi_nr_reclaimable=22840, bdi_thresh=26421, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24905, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24918, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26424, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19034, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26420, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24923, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24906, background_thresh=145168, dirty_thresh=290337
> global dirty=246847 writeback=43800 nfs=0 flags=C_ towrite=1023 skipped=0
> bdi_nr_reclaimable=21040, bdi_thresh=24491, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26421, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18250, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18248, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24903, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24160, bdi_thresh=27821, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24905, background_thresh=145168, dirty_thresh=290337
> requeue_io +527: inode 0
> bdi_nr_reclaimable=15880, bdi_thresh=18248, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19030, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24903, background_thresh=145168, dirty_thresh=290337
> requeue_io +527: inode 0
> bdi_nr_reclaimable=15880, bdi_thresh=18246, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24495, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24919, background_thresh=145168, dirty_thresh=290337
> redirty_tail +502: inode 12533
> bdi_nr_reclaimable=21040, bdi_thresh=24496, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27418, background_thresh=145168, dirty_thresh=290337
> redirty_tail +435: inode 174623
> bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
> redirty_tail +435: inode 4784181
> redirty_tail +435: inode 3474323
> requeue_io +527: inode 0
> requeue_io +527: inode 0
> redirty_tail +502: inode 13474
> mm/page-writeback.c 762 background_writeout: comm=pdflush pid=10090 n=-6621691
> bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18246, background_thresh=145168, dirty_thresh=290337
> mm/page-writeback.c 762 background_writeout: comm=pdflush pid=599 n=-6696603
> bdi_nr_reclaimable=15640, bdi_thresh=19030, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24903, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26421, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19034, background_thresh=145168, dirty_thresh=290337
> global dirty=246847 writeback=43709 nfs=0 flags=C_ towrite=1024 skipped=0
> bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24160, bdi_thresh=27823, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24160, bdi_thresh=27821, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=19720, bdi_thresh=23425, background_thresh=145168, dirty_thresh=290337
> global dirty=246847 writeback=43709 nfs=0 flags=C_ towrite=1022 skipped=0
> bdi_nr_reclaimable=21640, bdi_thresh=24920, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24160, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27425, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26611, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24489, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24926, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24903, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27421, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24929, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24160, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19026, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27412, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26615, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26624, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=20480, bdi_thresh=23810, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24904, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=20480, bdi_thresh=23811, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27422, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24480, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24900, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24160, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24160, bdi_thresh=27820, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=20480, bdi_thresh=23803, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15240, bdi_thresh=16825, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=20480, bdi_thresh=23794, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24485, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24933, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18248, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24933, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24920, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24903, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26421, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24926, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24160, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24932, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27412, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26615, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=19720, bdi_thresh=23425, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27425, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26611, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24900, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24903, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24492, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19030, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15240, bdi_thresh=16824, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24160, bdi_thresh=27821, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19026, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27416, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24904, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26625, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18246, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24920, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24480, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24160, bdi_thresh=27820, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=20480, bdi_thresh=23794, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24485, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=20480, bdi_thresh=23803, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15240, bdi_thresh=16825, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24933, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18248, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26611, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24903, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24160, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24926, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27422, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18246, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27412, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26615, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24160, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24160, bdi_thresh=27810, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=19720, bdi_thresh=23425, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24160, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27425, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24933, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24900, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24900, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24930, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15240, bdi_thresh=16824, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24493, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24481, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24930, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26611, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24864, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24863, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24918, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18245, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24924, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15240, bdi_thresh=16823, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24860, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27412, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27416, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24160, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19026, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18243, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18246, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24490, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26625, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24931, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24160, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24478, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24160, bdi_thresh=27820, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=20480, bdi_thresh=23792, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26615, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=20480, bdi_thresh=23801, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24160, bdi_thresh=27810, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24491, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=20480, bdi_thresh=23809, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24918, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27425, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24863, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=19720, bdi_thresh=23425, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24924, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15240, bdi_thresh=16823, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24860, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27412, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24483, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24160, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24930, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24160, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26611, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24864, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24160, bdi_thresh=27815, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27422, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24928, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15240, bdi_thresh=16823, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18246, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27416, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24480, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15240, bdi_thresh=16824, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24931, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24492, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24864, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26623, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15240, bdi_thresh=16825, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18245, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26418, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26623, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24492, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24866, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19026, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18243, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15240, bdi_thresh=16823, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=20480, bdi_thresh=23808, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18245, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27421, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19030, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24160, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26618, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=20480, bdi_thresh=23791, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26615, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24490, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26611, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24160, bdi_thresh=27810, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=20480, bdi_thresh=23800, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24863, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24160, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24941, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24477, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24927, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27412, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24481, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=20480, bdi_thresh=23807, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27425, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24200, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24200, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26625, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=19720, bdi_thresh=23425, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24200, bdi_thresh=27810, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24933, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18245, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18242, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26611, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26623, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=20480, bdi_thresh=23790, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26615, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15240, bdi_thresh=16822, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=20480, bdi_thresh=23799, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24200, bdi_thresh=27820, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24863, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18244, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27421, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19026, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24476, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26618, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24927, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24200, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27422, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27416, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24864, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24866, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15240, bdi_thresh=16822, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=20480, bdi_thresh=23806, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24200, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24940, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15240, bdi_thresh=16823, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19030, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24860, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24489, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24940, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24488, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24490, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24478, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18244, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24200, bdi_thresh=27815, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24939, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24937, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24935, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27413, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24200, bdi_thresh=27823, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15240, bdi_thresh=16822, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24490, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18244, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26623, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24492, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15240, bdi_thresh=16824, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26418, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24937, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26425, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=20480, bdi_thresh=23808, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24200, bdi_thresh=27823, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24937, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27426, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24936, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24938, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24864, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26414, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27423, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18243, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24862, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26626, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15240, bdi_thresh=16824, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18246, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18245, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24200, bdi_thresh=27821, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24936, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27421, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26624, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19034, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24935, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27426, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18246, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26421, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15240, bdi_thresh=16824, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24927, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24481, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27425, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24863, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24200, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24200, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26420, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=19720, bdi_thresh=23425, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18245, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24200, bdi_thresh=27810, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26611, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24933, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27412, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26615, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24864, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26625, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24200, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=20480, bdi_thresh=23799, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24200, bdi_thresh=27820, background_thresh=145168, dirty_thresh=290337
> requeue_io +527: inode 0
> bdi_nr_reclaimable=21640, bdi_thresh=24940, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24860, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15240, bdi_thresh=16823, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24200, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=20480, bdi_thresh=23807, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24476, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27422, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=20480, bdi_thresh=23790, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27416, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24200, bdi_thresh=27815, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24478, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15240, bdi_thresh=16822, background_thresh=145168, dirty_thresh=290337
> requeue_io +527: inode 0
> bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24939, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18242, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24940, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24937, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24488, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24200, bdi_thresh=27823, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24490, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18244, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24866, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18244, background_thresh=145168, dirty_thresh=290337
> redirty_tail +502: inode 13925
> bdi_nr_reclaimable=23080, bdi_thresh=26623, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26418, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19026, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24489, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15240, bdi_thresh=16824, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24935, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24490, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15240, bdi_thresh=16821, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=20480, bdi_thresh=23805, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=20480, bdi_thresh=23798, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27421, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26420, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=19720, bdi_thresh=23425, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18245, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26611, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15240, bdi_thresh=16822, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27426, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24932, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27412, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26615, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27425, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24823, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24200, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24200, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26424, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24200, bdi_thresh=27820, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26624, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26421, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24938, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18244, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24475, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=20480, bdi_thresh=23789, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24479, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18243, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24943, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18244, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24200, bdi_thresh=27821, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18243, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15240, bdi_thresh=16820, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24947, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24200, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24947, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24200, bdi_thresh=27810, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24820, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24942, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15240, bdi_thresh=16822, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27416, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24824, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26625, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27422, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24476, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18242, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18242, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15240, bdi_thresh=16820, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18240, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=20480, bdi_thresh=23805, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24490, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24944, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26623, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24825, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18241, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24943, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27421, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24826, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26425, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19034, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26618, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27423, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24200, bdi_thresh=27823, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24488, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27413, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24200, bdi_thresh=27815, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24489, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24200, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26414, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24944, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18241, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19030, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24200, bdi_thresh=27823, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24944, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24945, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=20480, bdi_thresh=23806, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24825, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24822, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24491, background_thresh=145168, dirty_thresh=290337
> redirty_tail +435: inode 174623
> bdi_nr_reclaimable=20480, bdi_thresh=23804, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27426, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26626, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27427, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24947, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18245, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26420, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24826, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26421, background_thresh=145168, dirty_thresh=290337
> redirty_tail +435: inode 4784181
> bdi_nr_reclaimable=21040, bdi_thresh=24484, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24483, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15240, bdi_thresh=16821, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24200, bdi_thresh=27821, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24823, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24825, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18243, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19030, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24823, background_thresh=145168, dirty_thresh=290337
> redirty_tail +442: inode 3474323
> bdi_nr_reclaimable=21040, bdi_thresh=24488, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24943, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24820, background_thresh=145168, dirty_thresh=290337
> requeue_io +527: inode 0
> bdi_nr_reclaimable=15880, bdi_thresh=18242, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27418, background_thresh=145168, dirty_thresh=290337
> redirty_tail +502: inode 14004
> bdi_nr_reclaimable=24200, bdi_thresh=27820, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24200, bdi_thresh=27820, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24934, background_thresh=145168, dirty_thresh=290337
> mm/page-writeback.c 762 background_writeout: comm=pdflush pid=599 n=-6696607
> bdi_nr_reclaimable=21320, bdi_thresh=24780, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19030, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18242, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24481, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27427, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27426, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=20480, bdi_thresh=23803, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24489, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24782, background_thresh=145168, dirty_thresh=290337
> global dirty=247029 writeback=43527 nfs=0 flags=CM towrite=1020 skipped=0
> bdi_nr_reclaimable=24200, bdi_thresh=27823, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24783, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18241, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26418, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26611, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24940, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24485, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=20480, bdi_thresh=23796, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19026, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26420, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18241, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=19720, bdi_thresh=23425, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26615, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27412, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=20480, bdi_thresh=23803, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24200, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18242, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24472, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24200, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15240, bdi_thresh=16820, background_thresh=145168, dirty_thresh=290337
> mm/page-writeback.c 762 background_writeout: comm=pdflush pid=10090 n=-6621691
> bdi_nr_reclaimable=24200, bdi_thresh=27820, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27422, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26623, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24477, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=20480, bdi_thresh=23787, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26625, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24781, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24947, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15240, bdi_thresh=16821, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24200, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24474, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24946, background_thresh=145168, dirty_thresh=290337
> global dirty=247029 writeback=43527 nfs=0 flags=C_ towrite=1024 skipped=0
> bdi_nr_reclaimable=15240, bdi_thresh=16819, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24484, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27416, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24942, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24785, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=20480, bdi_thresh=23804, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24944, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18239, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21320, bdi_thresh=24826, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24200, bdi_thresh=27810, background_thresh=145168, dirty_thresh=290337
> redirty_tail +435: inode 174623
> bdi_nr_reclaimable=15240, bdi_thresh=16819, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26623, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24486, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24486, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18241, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26618, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26425, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27423, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21040, bdi_thresh=24488, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24943, background_thresh=145168, dirty_thresh=290337
> redirty_tail +435: inode 4784181
> bdi_nr_reclaimable=23920, bdi_thresh=27413, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24200, bdi_thresh=27823, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24945, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24200, bdi_thresh=27815, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15880, bdi_thresh=18240, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19030, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24944, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24200, bdi_thresh=27821, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=24200, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15240, bdi_thresh=16821, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21240, bdi_thresh=24825, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=20480, bdi_thresh=23804, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=23920, bdi_thresh=27426, background_thresh=145168, dirty_thresh=290337
> redirty_tail +442: inode 3474323
> bdi_nr_reclaimable=23080, bdi_thresh=26626, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=21640, bdi_thresh=24902, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145168, dirty_thresh=290337
> bdi_nr_reclaimable=15640, bdi_thresh=19034, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=23920, bdi_thresh=27421, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15880, bdi_thresh=18243, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21640, bdi_thresh=24907, background_thresh=145177, dirty_thresh=290355
> requeue_io +527: inode 0
> bdi_nr_reclaimable=21040, bdi_thresh=24483, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15880, bdi_thresh=18244, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21640, bdi_thresh=24904, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=22840, bdi_thresh=26421, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21080, bdi_thresh=24785, background_thresh=145177, dirty_thresh=290355
> redirty_tail +502: inode 14023
> bdi_nr_reclaimable=22840, bdi_thresh=26421, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15240, bdi_thresh=16820, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15880, bdi_thresh=18243, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21040, bdi_thresh=24786, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=22840, bdi_thresh=26424, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=22840, bdi_thresh=26420, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15880, bdi_thresh=18241, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15880, bdi_thresh=18240, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21000, bdi_thresh=24783, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21640, bdi_thresh=24903, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21640, bdi_thresh=24907, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15880, bdi_thresh=18242, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21040, bdi_thresh=24488, background_thresh=145177, dirty_thresh=290355
> redirty_tail +435: inode 21
> bdi_nr_reclaimable=21000, bdi_thresh=24823, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=23080, bdi_thresh=26624, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21640, bdi_thresh=24903, background_thresh=145177, dirty_thresh=290355
> redirty_tail +435: inode 174623
> bdi_nr_reclaimable=23920, bdi_thresh=27421, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=23920, bdi_thresh=27418, background_thresh=145177, dirty_thresh=290355
> redirty_tail +435: inode 4784181
> bdi_nr_reclaimable=22840, bdi_thresh=26414, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145177, dirty_thresh=290355
> redirty_tail +435: inode 3474323
> requeue_io +527: inode 0
> bdi_nr_reclaimable=23920, bdi_thresh=27425, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15240, bdi_thresh=16819, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21640, bdi_thresh=24904, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21000, bdi_thresh=24824, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=24200, bdi_thresh=27821, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21040, bdi_thresh=24487, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15880, bdi_thresh=18242, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15880, bdi_thresh=18241, background_thresh=145177, dirty_thresh=290355
> redirty_tail +502: inode 14042
> bdi_nr_reclaimable=21640, bdi_thresh=24905, background_thresh=145177, dirty_thresh=290355
> mm/page-writeback.c 762 background_writeout: comm=pdflush pid=599 n=-6696985
> bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21000, bdi_thresh=24823, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21040, bdi_thresh=24486, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15880, bdi_thresh=18241, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=22840, bdi_thresh=26421, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21640, bdi_thresh=24867, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15640, bdi_thresh=19034, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15240, bdi_thresh=16820, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=24200, bdi_thresh=27824, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21040, bdi_thresh=24485, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21640, bdi_thresh=24860, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=24200, bdi_thresh=27820, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=23080, bdi_thresh=26611, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=22840, bdi_thresh=26420, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=20480, bdi_thresh=23801, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21040, bdi_thresh=24480, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21640, bdi_thresh=24854, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15640, bdi_thresh=19026, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=24200, bdi_thresh=27822, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21000, bdi_thresh=24823, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=23080, bdi_thresh=26615, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=23920, bdi_thresh=27412, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21040, bdi_thresh=24488, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21040, bdi_thresh=24471, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15240, bdi_thresh=16818, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15880, bdi_thresh=18241, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21000, bdi_thresh=24822, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=24200, bdi_thresh=27822, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15880, bdi_thresh=18241, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15240, bdi_thresh=16819, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=20480, bdi_thresh=23794, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21000, bdi_thresh=24820, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21040, bdi_thresh=24476, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21040, bdi_thresh=24487, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=20480, bdi_thresh=23785, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=24200, bdi_thresh=27820, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15640, bdi_thresh=19030, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=23920, bdi_thresh=27422, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21640, bdi_thresh=24867, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=22840, bdi_thresh=26424, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=24200, bdi_thresh=27824, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21000, bdi_thresh=24820, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=23920, bdi_thresh=27427, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=23920, bdi_thresh=27426, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=23080, bdi_thresh=26626, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=23920, bdi_thresh=27416, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21640, bdi_thresh=24866, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15240, bdi_thresh=16818, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21640, bdi_thresh=24867, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=23920, bdi_thresh=27425, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=24200, bdi_thresh=27821, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=20480, bdi_thresh=23802, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15880, bdi_thresh=18239, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15880, bdi_thresh=18238, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15880, bdi_thresh=18240, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21040, bdi_thresh=24473, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=24200, bdi_thresh=27823, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=24200, bdi_thresh=27810, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15880, bdi_thresh=18240, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21640, bdi_thresh=24864, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21040, bdi_thresh=24485, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145177, dirty_thresh=290355
> mm/page-writeback.c 762 background_writeout: comm=pdflush pid=10090 n=-6621693
> bdi_nr_reclaimable=21040, bdi_thresh=24482, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21040, bdi_thresh=24483, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15880, bdi_thresh=18239, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=22840, bdi_thresh=26418, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21000, bdi_thresh=24786, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=20480, bdi_thresh=23800, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=24200, bdi_thresh=27823, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21640, bdi_thresh=24862, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15240, bdi_thresh=16817, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15880, bdi_thresh=18239, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=22840, bdi_thresh=26425, background_thresh=145177, dirty_thresh=290355
> global dirty=246665 writeback=43800 nfs=0 flags=C_ towrite=1022 skipped=0
> bdi_nr_reclaimable=21000, bdi_thresh=24784, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21000, bdi_thresh=24785, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=23080, bdi_thresh=26618, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21040, bdi_thresh=24480, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21040, bdi_thresh=24484, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21640, bdi_thresh=24863, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15240, bdi_thresh=16819, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=23080, bdi_thresh=26623, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21640, bdi_thresh=24863, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21640, bdi_thresh=24864, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=24200, bdi_thresh=27815, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=23920, bdi_thresh=27426, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=24200, bdi_thresh=27821, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=23920, bdi_thresh=27421, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=23920, bdi_thresh=27413, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=22840, bdi_thresh=26420, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15880, bdi_thresh=18241, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21000, bdi_thresh=24785, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15880, bdi_thresh=18238, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=23080, bdi_thresh=26626, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=23920, bdi_thresh=27423, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=20480, bdi_thresh=23802, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=22840, bdi_thresh=26421, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21440, bdi_thresh=24864, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15880, bdi_thresh=18239, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=23920, bdi_thresh=27421, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15640, bdi_thresh=19030, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15880, bdi_thresh=18241, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15880, bdi_thresh=18240, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21360, bdi_thresh=24862, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21360, bdi_thresh=24903, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=23920, bdi_thresh=27418, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=22840, bdi_thresh=26414, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21320, bdi_thresh=24865, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=23080, bdi_thresh=26624, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=15880, bdi_thresh=18242, background_thresh=145177, dirty_thresh=290355
> global dirty=246301 writeback=44164 nfs=0 flags=C_ towrite=646 skipped=0
> bdi_nr_reclaimable=15240, bdi_thresh=16818, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=19720, bdi_thresh=23385, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21000, bdi_thresh=24786, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21280, bdi_thresh=24863, background_thresh=145177, dirty_thresh=290355
> redirty_tail +435: inode 174623
> bdi_nr_reclaimable=21000, bdi_thresh=24785, background_thresh=145177, dirty_thresh=290355
> bdi_nr_reclaimable=21000, bdi_thresh=24783, background_thresh=145177, dirty_thresh=290355
> redirty_tail +435: inode 4784181
> redirty_tail +435: inode 3474323
> requeue_io +527: inode 0
> redirty_tail +502: inode 14061
> bdi_nr_reclaimable=23080, bdi_thresh=26623, background_thresh=145177, dirty_thresh=290355
> redirty_tail +442: inode 27
> redirty_tail +435: inode 174623
> redirty_tail +435: inode 4784181
> redirty_tail +435: inode 3474323
> requeue_io +527: inode 0
> redirty_tail +502: inode 14096

Re: regression in page writeback

[Shaohua Li]

On Tue, Sep 22, 2009 at 11:52:59PM +0800, Chris Mason wrote:
> On Tue, Sep 22, 2009 at 10:32:14AM +0200, Peter Zijlstra wrote:
> > On Tue, 2009-09-22 at 16:24 +0800, Wu Fengguang wrote:
> > > On Tue, Sep 22, 2009 at 04:09:25PM +0800, Peter Zijlstra wrote:
> > > > On Tue, 2009-09-22 at 16:05 +0800, Wu Fengguang wrote:
> > > > > 
> > > > > I'm not sure how this patch stopped the "overshooting" behavior.
> > > > > Maybe it managed to not start the background pdflush, or the started
> > > > > pdflush thread exited because it found writeback is in progress by
> > > > > someone else?
> > > > > 
> > > > > -               if (bdi_nr_reclaimable) {
> > > > > +               if (bdi_nr_reclaimable > bdi_thresh) {
> > > > 
> > > > The idea is that we shouldn't move more pages from dirty -> writeback
> > > > when there's not actually that much dirty left.
> > > 
> > > IMHO this makes little sense given that pdflush will move all dirty
> > > pages anyway. pdflush should already be started to do background
> > > writeback before the process is throttled, and it is designed to sync
> > > all current dirty pages as quick as possible and as much as possible.
> > 
> > Not so, pdflush (or now the bdi writer thread thingies) should not
> > deplete all dirty pages but should stop writing once they are below the
> > background limit.
> > 
> > > > Now, I'm not sure about the > bdi_thresh part, I've suggested to maybe
> > > > use bdi_thresh/2 a few times, but it generally didn't seem to make much
> > > > of a difference.
> > > 
> > > One possible difference is, the process may end up waiting longer time
> > > in order to sync write_chunk pages and quit the throttle. This could
> > > hurt the responsiveness of the throttled process.
> > 
> > Well, that's all because this congestion_wait stuff is borken..
> > 
> 
> I'd suggest retesting with a new baseline against the code in Linus' git
> today.  Overall I think the change to make balance_dirty_pages() sleep
> instead of kick more IO out is a very good one.  It helps in most
> workloads here.
I tested today's git tree v2.6.31-7068-g43c1266, looks the regression
disappears. the disk io is almost stable to about 480m/s with/without the patch.

Thanks,
Shaohua

Re: regression in page writeback

[Richard Kennedy]

On Tue, 2009-09-22 at 18:59 -0700, Andrew Morton wrote:
> On Wed, 23 Sep 2009 09:45:00 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> 
> > On Wed, Sep 23, 2009 at 09:28:32AM +0800, Andrew Morton wrote:
> > > On Wed, 23 Sep 2009 09:17:58 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> > > 
> > > > On Wed, Sep 23, 2009 at 08:54:52AM +0800, Andrew Morton wrote:
> > > > > On Wed, 23 Sep 2009 08:22:20 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> > > > > 
> > > > > > Jens' per-bdi writeback has another improvement. In 2.6.31, when
> > > > > > superblocks A and B both have 100000 dirty pages, it will first
> > > > > > exhaust A's 100000 dirty pages before going on to sync B's.
> > > > > 
> > > > > That would only be true if someone broke 2.6.31.  Did they?
> > > > > 
> > > > > SYSCALL_DEFINE0(sync)
> > > > > {
> > > > > 	wakeup_pdflush(0);
> > > > > 	sync_filesystems(0);
> > > > > 	sync_filesystems(1);
> > > > > 	if (unlikely(laptop_mode))
> > > > > 		laptop_sync_completion();
> > > > > 	return 0;
> > > > > }
> > > > > 
> > > > > the sync_filesystems(0) is supposed to non-blockingly start IO against
> > > > > all devices.  It used to do that correctly.  But people mucked with it
> > > > > so perhaps it no longer does.
> > > > 
> > > > I'm referring to writeback_inodes(). Each invocation of which (to sync
> > > > 4MB) will do the same iteration over superblocks A => B => C ... So if
> > > > A has dirty pages, it will always be served first.
> > > > 
> > > > So if wbc->bdi == NULL (which is true for kupdate/background sync), it
> > > > will have to first exhaust A before going on to B and C.
> > > 
> > > But that works OK.  We fill the first device's queue, then it gets
> > > congested and sync_sb_inodes() does nothing and we advance to the next
> > > queue.
> > 
> > So in common cases "exhaust" is a bit exaggerated, but A does receive
> > much more opportunity than B. Computation resources for IO submission
> > are unbalanced for A, and there are pointless overheads in rechecking A.
> 
> That's unquantified handwaving.  One CPU can do a *lot* of IO.
> 
> > > If a device has more than a queue's worth of dirty data then we'll
> > > probably leave some of that dirty memory un-queued, so there's some
> > > lack of concurrency in that situation.
> > 
> > Good insight.
> 
> It was wrong.  See the other email.
> 
> > That possibly explains one major factor of the
> > performance gains of Jens' per-bdi writeback.
> 
> I've yet to see any believable and complete explanation for these
> gains.  I've asked about these things multiple times and nothing happened.
> 
> I suspect that what happened over time was that previously-working code
> got broken, then later people noticed the breakage but failed to
> analyse and fix it in favour of simply ripping everything out and
> starting again.
> 
> So for the want of analysing and fixing several possible regressions,
> we've tossed away some very sensitive core kernel code which had tens
> of millions of machine-years testing.  I find this incredibly rash.

FWIW I agree, I don't think the new per-bdi code has had enough testing
yet.

On my desktop setup I have not been able to measure any significant
change in performance of linear writes.

I am concerned that the background writeout no longer stops when it
reaches the background threshold, as balance_dirty_pages requests all
dirty pages to be written. No doubt this is good for large linear writes
but what about more random write workloads? 

regards
Richard

Re: regression in page writeback

[Peter Zijlstra]

On Wed, 2009-09-23 at 10:19 +0100, Richard Kennedy wrote:
> 
> I am concerned that the background writeout no longer stops when it
> reaches the background threshold, as balance_dirty_pages requests all
> dirty pages to be written. No doubt this is good for large linear writes
> but what about more random write workloads? 

I've not had time to look over the current code, but write-out not
stopping on reaching background threshold is a definite bug and needs to
get fixed.

Re: regression in page writeback

[Wu Fengguang]

On Wed, Sep 23, 2009 at 05:23:31PM +0800, Peter Zijlstra wrote:
> On Wed, 2009-09-23 at 10:19 +0100, Richard Kennedy wrote:
> > 
> > I am concerned that the background writeout no longer stops when it
> > reaches the background threshold, as balance_dirty_pages requests all
> > dirty pages to be written. No doubt this is good for large linear writes
> > but what about more random write workloads? 
> 
> I've not had time to look over the current code, but write-out not
> stopping on reaching background threshold is a definite bug and needs to
> get fixed.

Yes, 2.6.31 code stops writeback when background threshold is reached.
But new behavior in latest git is to writeback all pages.

The code only checks over_bground_thresh() for kupdate works:

                if (args->for_kupdate && args->nr_pages <= 0 &&
                    !over_bground_thresh())
                        break;

However the background work started by balance_dirty_pages() won't check
over_bground_thresh(). So it will move all dirty pages.

I think it's very weird to check over_bground_thresh() for kupdate
instead of background work. Jens must intended for the latter case.

Thanks,
Fengguang

Re: regression in page writeback

[Wu Fengguang]

On Wed, Sep 23, 2009 at 05:37:53PM +0800, Wu Fengguang wrote:
> On Wed, Sep 23, 2009 at 05:23:31PM +0800, Peter Zijlstra wrote:
> > On Wed, 2009-09-23 at 10:19 +0100, Richard Kennedy wrote:
> > > 
> > > I am concerned that the background writeout no longer stops when it
> > > reaches the background threshold, as balance_dirty_pages requests all
> > > dirty pages to be written. No doubt this is good for large linear writes
> > > but what about more random write workloads? 
> > 
> > I've not had time to look over the current code, but write-out not
> > stopping on reaching background threshold is a definite bug and needs to
> > get fixed.
> 
> Yes, 2.6.31 code stops writeback when background threshold is reached.
> But new behavior in latest git is to writeback all pages.
> 
> The code only checks over_bground_thresh() for kupdate works:
> 
>                 if (args->for_kupdate && args->nr_pages <= 0 &&
>                     !over_bground_thresh())
>                         break;
> 
> However the background work started by balance_dirty_pages() won't check
> over_bground_thresh(). So it will move all dirty pages.
> 
> I think it's very weird to check over_bground_thresh() for kupdate
> instead of background work. Jens must intended for the latter case.

Here is the patch to fix it. Tested to work OK. This is an RFC.

Thanks,
Fengguang
---
writeback: stop background writeback when below background threshold

Treat bdi_start_writeback(0) as a special request to do background write,
and stop such work when we are below the background dirty threshold.

Also simplify the (nr_pages <= 0) checks. Since we already pass in
nr_pages=LONG_MAX for WB_SYNC_ALL and background writes, we don't
need to worry about it being decreased to zero.

Reported-by: Richard Kennedy <richard@rsk.demon.co.uk>
CC: Jan Kara <jack@suse.cz>
CC: Jens Axboe <jens.axboe@oracle.com> 
CC: Peter Zijlstra <a.p.zijlstra@chello.nl> 
Signed-off-by: Wu Fengguang <fengguang.wu@intel.com>
---
 fs/fs-writeback.c   |   28 +++++++++++++++++-----------
 mm/page-writeback.c |    6 +++---
 2 files changed, 20 insertions(+), 14 deletions(-)

--- linux.orig/fs/fs-writeback.c	2009-09-23 17:47:23.000000000 +0800
+++ linux/fs/fs-writeback.c	2009-09-23 18:13:36.000000000 +0800
@@ -41,8 +41,9 @@ struct wb_writeback_args {
 	long nr_pages;
 	struct super_block *sb;
 	enum writeback_sync_modes sync_mode;
-	int for_kupdate;
-	int range_cyclic;
+	int for_kupdate:1;
+	int range_cyclic:1;
+	int for_background:1;
 };
 
 /*
@@ -260,6 +261,15 @@ void bdi_start_writeback(struct backing_
 		.range_cyclic	= 1,
 	};
 
+	/*
+	 * We treat @nr_pages=0 as the special case to do background writeback,
+	 * ie. to sync pages until the background dirty threshold is reached.
+	 */
+	if (!nr_pages) {
+		args.nr_pages = LONG_MAX;
+		args.for_background = 1;
+	}
+
 	bdi_alloc_queue_work(bdi, &args);
 }
 
@@ -723,20 +733,16 @@ static long wb_writeback(struct bdi_writ
 
 	for (;;) {
 		/*
-		 * Don't flush anything for non-integrity writeback where
-		 * no nr_pages was given
+		 * Stop writeback when nr_pages has been consumed
 		 */
-		if (!args->for_kupdate && args->nr_pages <= 0 &&
-		     args->sync_mode == WB_SYNC_NONE)
+		if (args->nr_pages <= 0)
 			break;
 
 		/*
-		 * If no specific pages were given and this is just a
-		 * periodic background writeout and we are below the
-		 * background dirty threshold, don't do anything
+		 * For background writeout, stop when we are below the
+		 * background dirty threshold
 		 */
-		if (args->for_kupdate && args->nr_pages <= 0 &&
-		    !over_bground_thresh())
+		if (args->for_background && !over_bground_thresh())
 			break;
 
 		wbc.more_io = 0;
--- linux.orig/mm/page-writeback.c	2009-09-23 17:45:58.000000000 +0800
+++ linux/mm/page-writeback.c	2009-09-23 17:47:17.000000000 +0800
@@ -589,10 +589,10 @@ static void balance_dirty_pages(struct a
 	 * background_thresh, to keep the amount of dirty memory low.
 	 */
 	if ((laptop_mode && pages_written) ||
-	    (!laptop_mode && ((nr_writeback = global_page_state(NR_FILE_DIRTY)
-					  + global_page_state(NR_UNSTABLE_NFS))
+	    (!laptop_mode && ((global_page_state(NR_FILE_DIRTY)
+			       + global_page_state(NR_UNSTABLE_NFS))
 					  > background_thresh)))
-		bdi_start_writeback(bdi, nr_writeback);
+		bdi_start_writeback(bdi, 0);
 }
 
 void set_page_dirty_balance(struct page *page, int page_mkwrite)

Re: regression in page writeback

[Chris Mason]

On Tue, Sep 22, 2009 at 07:36:22PM -0700, Andrew Morton wrote:
> On Wed, 23 Sep 2009 10:26:22 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> 
> > On Wed, Sep 23, 2009 at 09:59:41AM +0800, Andrew Morton wrote:
> > > On Wed, 23 Sep 2009 09:45:00 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> > > 
> > > > On Wed, Sep 23, 2009 at 09:28:32AM +0800, Andrew Morton wrote:
> > > > > On Wed, 23 Sep 2009 09:17:58 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> > > > > 
> > > > > > On Wed, Sep 23, 2009 at 08:54:52AM +0800, Andrew Morton wrote:
> > > > > > > On Wed, 23 Sep 2009 08:22:20 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> > > > > > > 
> > > > > > > > Jens' per-bdi writeback has another improvement. In 2.6.31, when
> > > > > > > > superblocks A and B both have 100000 dirty pages, it will first
> > > > > > > > exhaust A's 100000 dirty pages before going on to sync B's.
> > > > > > > 
> > > > > > > That would only be true if someone broke 2.6.31.  Did they?
> > > > > > > 
> > > > > > > SYSCALL_DEFINE0(sync)
> > > > > > > {
> > > > > > > 	wakeup_pdflush(0);
> > > > > > > 	sync_filesystems(0);
> > > > > > > 	sync_filesystems(1);
> > > > > > > 	if (unlikely(laptop_mode))
> > > > > > > 		laptop_sync_completion();
> > > > > > > 	return 0;
> > > > > > > }
> > > > > > > 
> > > > > > > the sync_filesystems(0) is supposed to non-blockingly start IO against
> > > > > > > all devices.  It used to do that correctly.  But people mucked with it
> > > > > > > so perhaps it no longer does.
> > > > > > 
> > > > > > I'm referring to writeback_inodes(). Each invocation of which (to sync
> > > > > > 4MB) will do the same iteration over superblocks A => B => C ... So if
> > > > > > A has dirty pages, it will always be served first.
> > > > > > 
> > > > > > So if wbc->bdi == NULL (which is true for kupdate/background sync), it
> > > > > > will have to first exhaust A before going on to B and C.
> > > > > 
> > > > > But that works OK.  We fill the first device's queue, then it gets
> > > > > congested and sync_sb_inodes() does nothing and we advance to the next
> > > > > queue.
> > > > 
> > > > So in common cases "exhaust" is a bit exaggerated, but A does receive
> > > > much more opportunity than B. Computation resources for IO submission
> > > > are unbalanced for A, and there are pointless overheads in rechecking A.
> > > 
> > > That's unquantified handwaving.  One CPU can do a *lot* of IO.
> > 
> > Yes.. I had the impression that the writeback submission can be pretty slow.
> > It should be because of the congestion_wait. Now that it is removed,
> > things are going faster when queue is not full.
> 
> What?  The wait is short.  The design intent there is that we repoll
> all previously-congested queues well before they start to run empty.

The congestion code was the big reason I got behind Jens' patches.  When
working on btrfs I tried to tune the existing congestion based setup to
scale well.  What we had before is basically a poll interface hiding
behind a queue flag and a wait.  

The only place that actually honors the congestion flag is pdflush.
It's trivial to get pdflush backed up and make it sit down without
making any progress because once the queue congests, pdflush goes away.

Nothing stops other procs from keeping the queue congested forever.
This can only be fixed by making everyone wait for congestion, at which
point we might as well wait for requests.

Here are some graphs comparing 2.6.31 and 2.6.31 with Jens' latest code.
The workload is two procs doing streaming writes to 32GB files.  I've
used deadline and bumped nr_requests to 2048, so pdflush should be able
to do a significant amount of work between congestion cycles.

The hardware is 5 sata drives pushed into an LVM stripe set.

http://oss.oracle.com/~mason/seekwatcher/bdi-writeback/xfs-streaming-compare.png
http://oss.oracle.com/~mason/seekwatcher/bdi-writeback/btrfs-streaming-compare.png

In the mainline graphs, pdflush is actually doing the vast majority of
the IO thanks to Richard's fix:

http://oss.oracle.com/~mason/seekwatcher/bdi-writeback/xfs-mainline-per-process.png

We can see there are two different pdflush procs hopping in and out
of the work.  This isn't a huge problem, except that we're also hopping
between files.

I don't think this is because anyone broke pdflush, I think this is
because very fast hardware goes from congested to uncongested 
very quickly, even when we bump nr_requests to 2048 like I did in the
graphs above.

The pdflush congestion backoffs skip the batching optimizations done by
the elevator.  pdflush could easily have waited in get_request_wait,
been given a nice fat batch of requests and then said oh no, the queue
is congested, I'll just sleep for a while without submitting any more
IO.

The congestion checks prevent any attempts from the filesystem to write
a whole extent (or a large portion of an extent) at a time.

The pdflush system tried to be async, but at the end of the day it
wasn't async enough to effectively drive the hardware.  Threads did get
tied up in FS locking, metadata reads and get_request_wait.  The end
result is that not enough time was spent keeping all the drives on the
box busy.

This isn't handwaving, my harddrive is littered with pdflush patches
trying to make it scale well, and I just couldn't work it out.

-chris

Re: regression in page writeback

[Wu Fengguang]

Chris,

Thanks for this excellent write up (and the previous one to Peter).
I'm glad to learn about your experiences and rationals behind this
work. As always, comments followed.

On Wed, Sep 23, 2009 at 10:00:58PM +0800, Chris Mason wrote:
> On Tue, Sep 22, 2009 at 07:36:22PM -0700, Andrew Morton wrote:
> > On Wed, 23 Sep 2009 10:26:22 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> > 
> > > On Wed, Sep 23, 2009 at 09:59:41AM +0800, Andrew Morton wrote:
> > > > On Wed, 23 Sep 2009 09:45:00 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> > > > 
> > > > > On Wed, Sep 23, 2009 at 09:28:32AM +0800, Andrew Morton wrote:
> > > > > > On Wed, 23 Sep 2009 09:17:58 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> > > > > > 
> > > > > > > On Wed, Sep 23, 2009 at 08:54:52AM +0800, Andrew Morton wrote:
> > > > > > > > On Wed, 23 Sep 2009 08:22:20 +0800 Wu Fengguang <fengguang.wu@intel.com> wrote:
> > > > > > > > 
> > > > > > > > > Jens' per-bdi writeback has another improvement. In 2.6.31, when
> > > > > > > > > superblocks A and B both have 100000 dirty pages, it will first
> > > > > > > > > exhaust A's 100000 dirty pages before going on to sync B's.
> > > > > > > > 
> > > > > > > > That would only be true if someone broke 2.6.31.  Did they?
> > > > > > > > 
> > > > > > > > SYSCALL_DEFINE0(sync)
> > > > > > > > {
> > > > > > > > 	wakeup_pdflush(0);
> > > > > > > > 	sync_filesystems(0);
> > > > > > > > 	sync_filesystems(1);
> > > > > > > > 	if (unlikely(laptop_mode))
> > > > > > > > 		laptop_sync_completion();
> > > > > > > > 	return 0;
> > > > > > > > }
> > > > > > > > 
> > > > > > > > the sync_filesystems(0) is supposed to non-blockingly start IO against
> > > > > > > > all devices.  It used to do that correctly.  But people mucked with it
> > > > > > > > so perhaps it no longer does.
> > > > > > > 
> > > > > > > I'm referring to writeback_inodes(). Each invocation of which (to sync
> > > > > > > 4MB) will do the same iteration over superblocks A => B => C ... So if
> > > > > > > A has dirty pages, it will always be served first.
> > > > > > > 
> > > > > > > So if wbc->bdi == NULL (which is true for kupdate/background sync), it
> > > > > > > will have to first exhaust A before going on to B and C.
> > > > > > 
> > > > > > But that works OK.  We fill the first device's queue, then it gets
> > > > > > congested and sync_sb_inodes() does nothing and we advance to the next
> > > > > > queue.
> > > > > 
> > > > > So in common cases "exhaust" is a bit exaggerated, but A does receive
> > > > > much more opportunity than B. Computation resources for IO submission
> > > > > are unbalanced for A, and there are pointless overheads in rechecking A.
> > > > 
> > > > That's unquantified handwaving.  One CPU can do a *lot* of IO.
> > > 
> > > Yes.. I had the impression that the writeback submission can be pretty slow.
> > > It should be because of the congestion_wait. Now that it is removed,
> > > things are going faster when queue is not full.
> > 
> > What?  The wait is short.  The design intent there is that we repoll
> > all previously-congested queues well before they start to run empty.
> 
> The congestion code was the big reason I got behind Jens' patches.  When
> working on btrfs I tried to tune the existing congestion based setup to
> scale well.  What we had before is basically a poll interface hiding
> behind a queue flag and a wait.  
 
So it's mainly about fast array writeback performance. 

> The only place that actually honors the congestion flag is pdflush.
> It's trivial to get pdflush backed up and make it sit down without
> making any progress because once the queue congests, pdflush goes away.

Right. I guess that's more or less intentional - to give lowest priority
to periodic/background writeback.

> Nothing stops other procs from keeping the queue congested forever.
> This can only be fixed by making everyone wait for congestion, at which
> point we might as well wait for requests.

Yes. That gives everyone somehow equal opportunity, this is a policy change
that may lead to interesting effects, as well as present a challenge to
get_request_wait(). That said, I'm not against the change to a wait queue
in general.

> Here are some graphs comparing 2.6.31 and 2.6.31 with Jens' latest code.
> The workload is two procs doing streaming writes to 32GB files.  I've
> used deadline and bumped nr_requests to 2048, so pdflush should be able
> to do a significant amount of work between congestion cycles.

The graphs show near 400MB/s throughput and about 4000-17000IO/s.

Writeback traces show that my 2Ghz laptop CPU can do IO submissions
up to 400MB/s. It takes about 0.01s to sync 4MB (one wb_kupdate =>
write_cache_pages traverse).

Given nr_requests=2048 and IOPS=10000, a congestion on-off cycle would
take (2048/16)/10000 = 0.0128s

The 0.0128s vs. 0.01s means that CPU returns just in time to see a
still congested but will soon become !congested queue. It then returns
to do congestion_wait, and be wakeup by the io completion events when
queue goes !congested. The return to write_cache_pages will again take
some time. So the end result may be, queue falls to 6/8 full, much below
the congestion off threshold 13/16.

Without congestion_wait, you get 100% full queue with get_request_wait. 

However I don't think the queue fullness can explain the performance
gain. It's sequential IO. It will only hurt performance if the queue
sometimes endangers starvation - which could happen when CPU is 100%
utilized so that IO submission cannot keep up. The congestion_wait
polls do eat more CPU power. It might impact the response to hard/soft
interrupts.

> The hardware is 5 sata drives pushed into an LVM stripe set.
> 
> http://oss.oracle.com/~mason/seekwatcher/bdi-writeback/xfs-streaming-compare.png
> http://oss.oracle.com/~mason/seekwatcher/bdi-writeback/btrfs-streaming-compare.png
> 
> In the mainline graphs, pdflush is actually doing the vast majority of
> the IO thanks to Richard's fix:
> 
> http://oss.oracle.com/~mason/seekwatcher/bdi-writeback/xfs-mainline-per-process.png
> 
> We can see there are two different pdflush procs hopping in and out
> of the work.

Yeah, that's expected. May eat some CPU cycles (race and locality issues).

> This isn't a huge problem, except that we're also hopping between
> files.

Yes, this is a problem. When encountered congestion, it may happen
that the file be synced only a dozen pages (which is very inefficient)
and then get redirty_tail (which may further delay this inode).

> I don't think this is because anyone broke pdflush, I think this is
> because very fast hardware goes from congested to uncongested 
> very quickly, even when we bump nr_requests to 2048 like I did in the
> graphs above.

What's typical CPU utilization during the test? It would be
interesting to do a comparison on %system numbers between the
poll/wait approaches.

> The pdflush congestion backoffs skip the batching optimizations done by
> the elevator.  pdflush could easily have waited in get_request_wait,
> been given a nice fat batch of requests and then said oh no, the queue
> is congested, I'll just sleep for a while without submitting any more
> IO.

I'd be surprised if the deadline batching is affected by the
interleaveness of incoming requests. Unless there are many expired
requests, which could happen when nr_requests is too large for the
device, which is not in your case.

I noticed that XFS's IOPS is almost doubled. While btrfs's IOPS and
throughput scales up by the same factor. The numbers show that the
average IO size for btrfs is near 64KB, is this your max_sectors_kb?
XFS's avg io size is a smaller 24kb, does that mean many small
metadata ios?

> The congestion checks prevent any attempts from the filesystem to write
> a whole extent (or a large portion of an extent) at a time.

Since writepage is called one by one for each page, will its
interleaveness impact filesystem decisions? Ie. between these two
writepage sequences.

        A1, B1, A2, B2, A3, B3, A4, B4
        A1, A2, A3, A4, B1, B2, B3, B4

Where each An/Bn stands for one page of file A/B, n is page index.

> The pdflush system tried to be async, but at the end of the day it
> wasn't async enough to effectively drive the hardware.  Threads did get
> tied up in FS locking, metadata reads and get_request_wait.  The end
> result is that not enough time was spent keeping all the drives on the
> box busy.

Yes, too much processing overheads with congestion_wait may be enough
to kill performance of fast arrays.

> This isn't handwaving, my harddrive is littered with pdflush patches
> trying to make it scale well, and I just couldn't work it out.

Thanks,
Fengguang

Re: regression in page writeback

[Chris Mason]

On Thu, Sep 24, 2009 at 11:15:08AM +0800, Wu Fengguang wrote:

[ why do the bdi-writeback work? ]

> > 
> > The congestion code was the big reason I got behind Jens' patches.  When
> > working on btrfs I tried to tune the existing congestion based setup to
> > scale well.  What we had before is basically a poll interface hiding
> > behind a queue flag and a wait.  
>  
> So it's mainly about fast array writeback performance. 

You can see the difference on single disks, at least in the two writer
case w/XFS.  But, each FS has its own tweaks in there, and the bigger
arrays show it better across all the filesystems.

> 
> > The only place that actually honors the congestion flag is pdflush.
> > It's trivial to get pdflush backed up and make it sit down without
> > making any progress because once the queue congests, pdflush goes away.
> 
> Right. I guess that's more or less intentional - to give lowest priority
> to periodic/background writeback.

The problem is that when we let pdflush back off, we do all of our IO
from balance_dirty_pages(), which makes much more seeky IO patterns for
the streaming writers.

> 
> > Nothing stops other procs from keeping the queue congested forever.
> > This can only be fixed by making everyone wait for congestion, at which
> > point we might as well wait for requests.
> 
> Yes. That gives everyone somehow equal opportunity, this is a policy change
> that may lead to interesting effects, as well as present a challenge to
> get_request_wait(). That said, I'm not against the change to a wait queue
> in general.

I very much agree here, relying more on get_request_wait is going to
expose some latency differences.

> 
> > Here are some graphs comparing 2.6.31 and 2.6.31 with Jens' latest code.
> > The workload is two procs doing streaming writes to 32GB files.  I've
> > used deadline and bumped nr_requests to 2048, so pdflush should be able
> > to do a significant amount of work between congestion cycles.
> 
> The graphs show near 400MB/s throughput and about 4000-17000IO/s.
> 
> Writeback traces show that my 2Ghz laptop CPU can do IO submissions
> up to 400MB/s. It takes about 0.01s to sync 4MB (one wb_kupdate =>
> write_cache_pages traverse).
> 
> Given nr_requests=2048 and IOPS=10000, a congestion on-off cycle would
> take (2048/16)/10000 = 0.0128s
> 
> The 0.0128s vs. 0.01s means that CPU returns just in time to see a
> still congested but will soon become !congested queue. It then returns
> to do congestion_wait, and be wakeup by the io completion events when
> queue goes !congested. The return to write_cache_pages will again take
> some time. So the end result may be, queue falls to 6/8 full, much below
> the congestion off threshold 13/16.
> 
> Without congestion_wait, you get 100% full queue with get_request_wait. 
> 
> However I don't think the queue fullness can explain the performance
> gain. It's sequential IO. It will only hurt performance if the queue
> sometimes endangers starvation - which could happen when CPU is 100%
> utilized so that IO submission cannot keep up. The congestion_wait
> polls do eat more CPU power. It might impact the response to hard/soft
> interrupts.

I think you're right that queue fullness alone can't explain things,
especially with streaming writes where the requests tend to be very
large.  LVM devices are a bit strange because they go in and out of
congestion based on any of the devices in the stripe set, so
things are less predictable.

The interesting difference between the XFS graph and the
btrfs graph is that btrfs has removed all congestion checks from its
write_cache_pages(), so btrfs is forcing pdflush to hang around even
when the queue is initially congested so that it can write a large
portion of an extent in each call.

This is why the btrfs IO graphs look the same for the two runs, the IO
submitted is basically the same.  The bdi thread is just submitting it
more often.

> 
> > The hardware is 5 sata drives pushed into an LVM stripe set.
> > 
> > http://oss.oracle.com/~mason/seekwatcher/bdi-writeback/xfs-streaming-compare.png
> > http://oss.oracle.com/~mason/seekwatcher/bdi-writeback/btrfs-streaming-compare.png
> > 
> > In the mainline graphs, pdflush is actually doing the vast majority of
> > the IO thanks to Richard's fix:
> > 
> > http://oss.oracle.com/~mason/seekwatcher/bdi-writeback/xfs-mainline-per-process.png
> > 
> > We can see there are two different pdflush procs hopping in and out
> > of the work.
> 
> Yeah, that's expected. May eat some CPU cycles (race and locality issues).
> 
> > This isn't a huge problem, except that we're also hopping between
> > files.
> 
> Yes, this is a problem. When encountered congestion, it may happen
> that the file be synced only a dozen pages (which is very inefficient)
> and then get redirty_tail (which may further delay this inode).
> 
> > I don't think this is because anyone broke pdflush, I think this is
> > because very fast hardware goes from congested to uncongested 
> > very quickly, even when we bump nr_requests to 2048 like I did in the
> > graphs above.
> 
> What's typical CPU utilization during the test? It would be
> interesting to do a comparison on %system numbers between the
> poll/wait approaches.

XFS averages about 20-30% CPU utilization.  Btrfs is much higher because
it is checksumming.

> 
> > The pdflush congestion backoffs skip the batching optimizations done by
> > the elevator.  pdflush could easily have waited in get_request_wait,
> > been given a nice fat batch of requests and then said oh no, the queue
> > is congested, I'll just sleep for a while without submitting any more
> > IO.
> 
> I'd be surprised if the deadline batching is affected by the
> interleaveness of incoming requests. Unless there are many expired
> requests, which could happen when nr_requests is too large for the
> device, which is not in your case.
> 
> I noticed that XFS's IOPS is almost doubled. While btrfs's IOPS and
> throughput scales up by the same factor. The numbers show that the
> average IO size for btrfs is near 64KB, is this your max_sectors_kb?
> XFS's avg io size is a smaller 24kb, does that mean many small
> metadata ios?

Since the traces were done on LVM, the IOPS come from the blktrace Q
events.  This means the IOPS graph basically reflects calls to
submit_bio and does not include any merging.

Btrfs does have an internal max of 64KB, but I'm not sure why xfs is
building smaller bios.  ext4 only builds 4k bios, and it is able to
perform just as well ;)

> 
> > The congestion checks prevent any attempts from the filesystem to write
> > a whole extent (or a large portion of an extent) at a time.
> 
> Since writepage is called one by one for each page, will its
> interleaveness impact filesystem decisions? Ie. between these two
> writepage sequences.
> 
>         A1, B1, A2, B2, A3, B3, A4, B4
>         A1, A2, A3, A4, B1, B2, B3, B4
> 
> Where each An/Bn stands for one page of file A/B, n is page index.

For XFS this is the key question.  We're doing streaming writes, so the
delayed allocation code is responsible for allocating extents, and this
is triggered from writepage.  Your first example becomes:

         A1 [allocate extent A1-A50 ], submit A1
	 B1 [allocate extent B1-B50 ], submit B1 (seek)
	 A2, (seek back to A1's extent)
	 B2, (seek back to B1's extent)
	 ...

This is why the XFS graph for pdflush isn't a straight line.   When we
back off file A and switch to file B, we seek between extents created by
delalloc.

Thanks for spending time reading through all of this.  It's a ton of data
and your improvements are much appreciated!

-chris

Re: regression in page writeback

[Dave Chinner]

On Thu, Sep 24, 2009 at 11:15:08AM +0800, Wu Fengguang wrote:
> On Wed, Sep 23, 2009 at 10:00:58PM +0800, Chris Mason wrote:
> > The only place that actually honors the congestion flag is pdflush.
> > It's trivial to get pdflush backed up and make it sit down without
> > making any progress because once the queue congests, pdflush goes away.
> 
> Right. I guess that's more or less intentional - to give lowest priority
> to periodic/background writeback.

IMO, this is the wrong design. Background writeback should
have higher CPU/scheduler priority than normal tasks. If there is
sufficient dirty pages in the system for background writeback to
be active, it should be running *now* to start as much IO as it can
without being held up by other, lower priority tasks.

Cleaning pages is important to keeping the system running smoothly.
Given that IO takes time to clean pages, it is therefore important
to issue as much as possible as quickly as possible without delays
before going back to sleep. Delaying issue of the IO or doing
sub-optimal issue simply reduces performance of the system because
it takes longer to clean the same number of dirty pages.

> > Nothing stops other procs from keeping the queue congested forever.
> > This can only be fixed by making everyone wait for congestion, at which
> > point we might as well wait for requests.
> 
> Yes. That gives everyone somehow equal opportunity, this is a policy change
> that may lead to interesting effects, as well as present a challenge to
> get_request_wait(). That said, I'm not against the change to a wait queue
> in general.

If you block all threads doing _writebehind caching_ (synchronous IO
is self-throttling) to the same BDI on the same queue as the bdi
flusher then when congestion clears the higher priority background
flusher thread should run first and issue more IO.  This should
happen as a natural side-effect of our scheduling algorithms and it
gives preference to efficient background writeback over in-efficient
foreground writeback. Indeed, with this approach we can even avoid
foreground writeback altogether...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: regression in page writeback

[Chris Mason]

On Fri, Sep 25, 2009 at 10:11:17AM +1000, Dave Chinner wrote:
> On Thu, Sep 24, 2009 at 11:15:08AM +0800, Wu Fengguang wrote:
> > On Wed, Sep 23, 2009 at 10:00:58PM +0800, Chris Mason wrote:
> > > The only place that actually honors the congestion flag is pdflush.
> > > It's trivial to get pdflush backed up and make it sit down without
> > > making any progress because once the queue congests, pdflush goes away.
> > 
> > Right. I guess that's more or less intentional - to give lowest priority
> > to periodic/background writeback.
> 
> IMO, this is the wrong design. Background writeback should
> have higher CPU/scheduler priority than normal tasks. If there is
> sufficient dirty pages in the system for background writeback to
> be active, it should be running *now* to start as much IO as it can
> without being held up by other, lower priority tasks.

I'd say that an fsync from mutt or vi should be done at a higher prio
than a background streaming writer.

-chris

Re: regression in page writeback

[Wu Fengguang]

On Fri, Sep 25, 2009 at 08:11:17AM +0800, Dave Chinner wrote:
> On Thu, Sep 24, 2009 at 11:15:08AM +0800, Wu Fengguang wrote:
> > On Wed, Sep 23, 2009 at 10:00:58PM +0800, Chris Mason wrote:
> > > The only place that actually honors the congestion flag is pdflush.
> > > It's trivial to get pdflush backed up and make it sit down without
> > > making any progress because once the queue congests, pdflush goes away.
> > 
> > Right. I guess that's more or less intentional - to give lowest priority
> > to periodic/background writeback.
> 
> IMO, this is the wrong design. Background writeback should
> have higher CPU/scheduler priority than normal tasks. If there is
> sufficient dirty pages in the system for background writeback to
> be active, it should be running *now* to start as much IO as it can
> without being held up by other, lower priority tasks.
> 
> Cleaning pages is important to keeping the system running smoothly.
> Given that IO takes time to clean pages, it is therefore important
> to issue as much as possible as quickly as possible without delays
> before going back to sleep. Delaying issue of the IO or doing
> sub-optimal issue simply reduces performance of the system because
> it takes longer to clean the same number of dirty pages.
> 
> > > Nothing stops other procs from keeping the queue congested forever.
> > > This can only be fixed by making everyone wait for congestion, at which
> > > point we might as well wait for requests.
> > 
> > Yes. That gives everyone somehow equal opportunity, this is a policy change
> > that may lead to interesting effects, as well as present a challenge to
> > get_request_wait(). That said, I'm not against the change to a wait queue
> > in general.
> 
> If you block all threads doing _writebehind caching_ (synchronous IO
> is self-throttling) to the same BDI on the same queue as the bdi
> flusher then when congestion clears the higher priority background
> flusher thread should run first and issue more IO.  This should
> happen as a natural side-effect of our scheduling algorithms and it
> gives preference to efficient background writeback over in-efficient
> foreground writeback. Indeed, with this approach we can even avoid
> foreground writeback altogether...

I don't see how balance_dirty_pages() writeout is less efficient than
pdflush writeout.

They all called the same routines to do the job.
balance_dirty_pages() sets nr_to_write=1536 at least for ext4 and xfs
(unless memory is tight; btrfs is 1540), which is in fact 50% bigger
than the 1024 pages used by pdflush. And it won't back off on congestion.
The s_io/b_io queues are shared, so a balance_dirty_pages() will just
continue from where the last sync thread exited. So it does not make
much difference who initiates the IO. Did I missed something?

Thanks,
Fengguang

Re: regression in page writeback

[Wu Fengguang]

On Thu, Sep 24, 2009 at 08:10:34PM +0800, Chris Mason wrote:
> On Thu, Sep 24, 2009 at 11:15:08AM +0800, Wu Fengguang wrote:
> 
> [ why do the bdi-writeback work? ]
> 
[snip]
> > > The congestion checks prevent any attempts from the filesystem to write
> > > a whole extent (or a large portion of an extent) at a time.
> > 
> > Since writepage is called one by one for each page, will its
> > interleaveness impact filesystem decisions? Ie. between these two
> > writepage sequences.
> > 
> >         A1, B1, A2, B2, A3, B3, A4, B4
> >         A1, A2, A3, A4, B1, B2, B3, B4
> > 
> > Where each An/Bn stands for one page of file A/B, n is page index.
> 
> For XFS this is the key question.  We're doing streaming writes, so the
> delayed allocation code is responsible for allocating extents, and this
> is triggered from writepage.  Your first example becomes:
> 
>          A1 [allocate extent A1-A50 ], submit A1
> 	 B1 [allocate extent B1-B50 ], submit B1 (seek)
> 	 A2, (seek back to A1's extent)
> 	 B2, (seek back to B1's extent)
> 	 ...
> 
> This is why the XFS graph for pdflush isn't a straight line.   When we
> back off file A and switch to file B, we seek between extents created by
> delalloc.

Does it mean XFS writeback is somehow serialized, so that the elevator
cannot do request merges well?  Hope that's not true..

> Thanks for spending time reading through all of this.  It's a ton of data
> and your improvements are much appreciated!

Thank you :)

Regards,
Fengguang

Re: regression in page writeback

[Dave Chinner]

On Thu, Sep 24, 2009 at 08:38:20PM -0400, Chris Mason wrote:
> On Fri, Sep 25, 2009 at 10:11:17AM +1000, Dave Chinner wrote:
> > On Thu, Sep 24, 2009 at 11:15:08AM +0800, Wu Fengguang wrote:
> > > On Wed, Sep 23, 2009 at 10:00:58PM +0800, Chris Mason wrote:
> > > > The only place that actually honors the congestion flag is pdflush.
> > > > It's trivial to get pdflush backed up and make it sit down without
> > > > making any progress because once the queue congests, pdflush goes away.
> > > 
> > > Right. I guess that's more or less intentional - to give lowest priority
> > > to periodic/background writeback.
> > 
> > IMO, this is the wrong design. Background writeback should
> > have higher CPU/scheduler priority than normal tasks. If there is
> > sufficient dirty pages in the system for background writeback to
> > be active, it should be running *now* to start as much IO as it can
> > without being held up by other, lower priority tasks.
> 
> I'd say that an fsync from mutt or vi should be done at a higher prio
> than a background streaming writer.

I don't think you caught everything I said - synchronous IO is
un-throttled. Background writeback should dump async IO to the
elevator as fast as it can, then get the hell out of the way. If
you've got a UP system, then the fsync can't be issued at the same
time pdflush is running (same as right now), and if you've got a MP
system then fsync can run at the same time. On the premise that sync
IO is unthrottled and given that elevators queue and issue sync IO
sperately to async writes, fsync latency would be entirely derived
from the elevator queuing behaviour, not the CPU priority of
pdflush.

Look at it this way - it is the responsibility of pdflush to keep
the elevator full of background IO. It is the responsibility of
the elevator to ensure that background IO doesn't starve all other
types of IO. If pdflush doesn't run because it can't get CPU time,
then background IO does not get issued, and system performance
suffers as a result.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: regression in page writeback

[Wu Fengguang]

On Wed, Sep 23, 2009 at 12:00:24PM +0800, Wu Fengguang wrote:
[snip]
> > attached is a short log. I'll try to get a full log after finish latest
> > git test.
> > bdi_nr_reclaimable is always less than bdi_thresh in the log. because
> 
> Yes. Only background_writeout() is working and queue is congested.
> 
> > when bdi_nr_reclaimable + bdi_nr_writeback > bdi_thresh, background
> > writeback is already started, so bdi_nr_writeback should be > 0.
> 
> Yes, when process is throttled, bdi_nr_writeback > (bdi_thresh - bdi_nr_reclaimable).
> 
> For each background_writeout() loop (each takes about 0.1s), there are
> ~100 balance_dirty_pages() loops.  The latter makes pretty much loop overheads.
> 
> Most importantly, most background_writeout() loops only wrote several pages
> (the large towrite values), which is extremely inefficient. I wonder why it
> cannot write more?

Ah when queue is congested, the congestion_on => congestion_off
threshold is small. For nr_requests=128, that threshold would be
about 128/16 = 8 pages, a pretty small number.

Thanks,
Fengguang

> > global dirty=246756 writeback=43800 nfs=0 flags=C_ towrite=1018 skipped=0
> > global dirty=246847 writeback=43800 nfs=0 flags=C_ towrite=1023 skipped=0
> > global dirty=246847 writeback=43800 nfs=0 flags=C_ towrite=1023 skipped=0
> > global dirty=246847 writeback=43800 nfs=0 flags=C_ towrite=1023 skipped=0
> > global dirty=246847 writeback=43709 nfs=0 flags=C_ towrite=1024 skipped=0
> > global dirty=246847 writeback=43709 nfs=0 flags=C_ towrite=1022 skipped=0
> > global dirty=247029 writeback=43527 nfs=0 flags=CM towrite=1020 skipped=0
> > global dirty=247029 writeback=43527 nfs=0 flags=C_ towrite=1024 skipped=0
> > global dirty=246665 writeback=43800 nfs=0 flags=C_ towrite=1022 skipped=0
> > global dirty=246301 writeback=44164 nfs=0 flags=C_ towrite=646 skipped=0
> 
> Thanks,
> Fengguang
> 
> Content-Description: msg2
> > 0, bdi_thresh=23810, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=24160, bdi_thresh=27860, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=23080, bdi_thresh=26626, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=23920, bdi_thresh=27422, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145141, dirty_thresh=290283
> > mm/page-writeback.c 762 background_writeout: comm=pdflush pid=599 n=-6696602
> > bdi_nr_reclaimable=21040, bdi_thresh=24479, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=20480, bdi_thresh=23802, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=21480, bdi_thresh=24906, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=24160, bdi_thresh=27862, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=23920, bdi_thresh=27416, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=15240, bdi_thresh=16824, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=20480, bdi_thresh=23809, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=21480, bdi_thresh=24903, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=21480, bdi_thresh=24907, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=23080, bdi_thresh=26623, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145141, dirty_thresh=290283
> > global dirty=246756 writeback=43800 nfs=0 flags=C_ towrite=1018 skipped=0
> > bdi_nr_reclaimable=15640, bdi_thresh=19030, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=21040, bdi_thresh=24481, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=24160, bdi_thresh=27850, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=21040, bdi_thresh=24493, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=21320, bdi_thresh=24944, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=21320, bdi_thresh=24940, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=15880, bdi_thresh=18244, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=21040, bdi_thresh=24493, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=24160, bdi_thresh=27864, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145141, dirty_thresh=290283
> > redirty_tail +435: inode 174623
> > bdi_nr_reclaimable=21480, bdi_thresh=24894, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=23920, bdi_thresh=27412, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=21480, bdi_thresh=24900, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=24160, bdi_thresh=27862, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=24160, bdi_thresh=27855, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=15640, bdi_thresh=19026, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=21040, bdi_thresh=24484, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=22840, bdi_thresh=26420, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=21320, bdi_thresh=24943, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=15240, bdi_thresh=16824, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=23080, bdi_thresh=26611, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=21480, bdi_thresh=24907, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=20480, bdi_thresh=23793, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=21040, bdi_thresh=24479, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=20480, bdi_thresh=23802, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=24160, bdi_thresh=27862, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=15240, bdi_thresh=16825, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=21040, bdi_thresh=24491, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=23920, bdi_thresh=27425, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=23080, bdi_thresh=26615, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=21320, bdi_thresh=24942, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=23080, bdi_thresh=26625, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=24160, bdi_thresh=27860, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=22840, bdi_thresh=26421, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=19720, bdi_thresh=23425, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=21480, bdi_thresh=24904, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=21320, bdi_thresh=24946, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=15880, bdi_thresh=18246, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=21480, bdi_thresh=24906, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=23920, bdi_thresh=27422, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=23920, bdi_thresh=27416, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=22840, bdi_thresh=26418, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=20480, bdi_thresh=23809, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=15880, bdi_thresh=18246, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=21480, bdi_thresh=24907, background_thresh=145141, dirty_thresh=290283
> > redirty_tail +435: inode 4784181
> > bdi_nr_reclaimable=21480, bdi_thresh=24902, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=15240, bdi_thresh=16826, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=15240, bdi_thresh=16824, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=21320, bdi_thresh=24944, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=21040, bdi_thresh=24495, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=23920, bdi_thresh=27413, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=15240, bdi_thresh=16824, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=24160, bdi_thresh=27863, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=23080, bdi_thresh=26618, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=21040, bdi_thresh=24492, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=21480, bdi_thresh=24905, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=21480, bdi_thresh=24903, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=21480, bdi_thresh=24903, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=15880, bdi_thresh=18246, background_thresh=145141, dirty_thresh=290283
> > redirty_tail +435: inode 3474323
> > bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=21480, bdi_thresh=24864, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=24160, bdi_thresh=27863, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=20480, bdi_thresh=23811, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=15880, bdi_thresh=18245, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=21480, bdi_thresh=24864, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=22840, bdi_thresh=26425, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=15240, bdi_thresh=16826, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=21040, bdi_thresh=24496, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=21320, bdi_thresh=24945, background_thresh=145141, dirty_thresh=290283
> > requeue_io +527: inode 0
> > bdi_nr_reclaimable=15640, bdi_thresh=19034, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=23920, bdi_thresh=27426, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=23080, bdi_thresh=26626, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=15880, bdi_thresh=18245, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=24160, bdi_thresh=27861, background_thresh=145141, dirty_thresh=290283
> > requeue_io +527: inode 0
> > bdi_nr_reclaimable=23920, bdi_thresh=27427, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145141, dirty_thresh=290283
> > bdi_nr_reclaimable=22840, bdi_thresh=26421, background_thresh=145150, dirty_thresh=290301
> > bdi_nr_reclaimable=22840, bdi_thresh=26424, background_thresh=145150, dirty_thresh=290301
> > bdi_nr_reclaimable=21040, bdi_thresh=24489, background_thresh=145150, dirty_thresh=290301
> > bdi_nr_reclaimable=21320, bdi_thresh=24906, background_thresh=145150, dirty_thresh=290301
> > bdi_nr_reclaimable=23920, bdi_thresh=27426, background_thresh=145150, dirty_thresh=290301
> > redirty_tail +502: inode 12415
> > bdi_nr_reclaimable=24160, bdi_thresh=27861, background_thresh=145150, dirty_thresh=290301
> > bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145150, dirty_thresh=290301
> > bdi_nr_reclaimable=21320, bdi_thresh=24905, background_thresh=145150, dirty_thresh=290301
> > bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145150, dirty_thresh=290301
> > bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145150, dirty_thresh=290301
> > bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145159, dirty_thresh=290319
> > bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145159, dirty_thresh=290319
> > bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145159, dirty_thresh=290319
> > bdi_nr_reclaimable=21640, bdi_thresh=24923, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24919, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24921, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24918, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27421, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24905, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=20480, bdi_thresh=23812, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26623, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24495, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24490, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24903, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27423, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24903, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18246, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27421, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18250, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24919, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24496, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26414, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24160, bdi_thresh=27820, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19030, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24919, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27418, background_thresh=145168, dirty_thresh=290337
> > mm/page-writeback.c 762 background_writeout: comm=pdflush pid=10090 n=-6621688
> > bdi_nr_reclaimable=23080, bdi_thresh=26624, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15240, bdi_thresh=16826, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26420, background_thresh=145168, dirty_thresh=290337
> > global dirty=246847 writeback=43800 nfs=0 flags=C_ towrite=1023 skipped=0
> > bdi_nr_reclaimable=21320, bdi_thresh=24900, background_thresh=145168, dirty_thresh=290337
> > redirty_tail +435: inode 174623
> > bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24160, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
> > redirty_tail +435: inode 4784181
> > redirty_tail +435: inode 3474323
> > requeue_io +527: inode 0
> > requeue_io +527: inode 0
> > redirty_tail +502: inode 12396
> > mm/page-writeback.c 762 background_writeout: comm=pdflush pid=10090 n=-6621689
> > bdi_nr_reclaimable=24160, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24900, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15240, bdi_thresh=16826, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26624, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24160, bdi_thresh=27820, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24490, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=20480, bdi_thresh=23812, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27421, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24919, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27427, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15240, bdi_thresh=16827, background_thresh=145168, dirty_thresh=290337
> > global dirty=246847 writeback=43800 nfs=0 flags=C_ towrite=1023 skipped=0
> > bdi_nr_reclaimable=22840, bdi_thresh=26420, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24910, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=20480, bdi_thresh=23811, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24903, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24904, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24900, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24923, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26611, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24916, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24160, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24160, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18245, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24160, bdi_thresh=27810, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26615, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27412, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24495, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27425, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18248, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27422, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=19720, bdi_thresh=23425, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24160, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24481, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24160, bdi_thresh=27820, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=20480, bdi_thresh=23804, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19026, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24486, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=20480, bdi_thresh=23795, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26625, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24493, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24923, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19030, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15240, bdi_thresh=16825, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24483, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24494, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15240, bdi_thresh=16826, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27416, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26418, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26623, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26623, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24160, bdi_thresh=27815, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24920, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24906, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24920, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24922, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24495, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=20480, bdi_thresh=23811, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15240, bdi_thresh=16825, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26618, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27413, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27421, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26425, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24904, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15240, bdi_thresh=16827, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24498, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27426, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24902, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24497, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24160, bdi_thresh=27823, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24918, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24921, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24920, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18246, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=20480, bdi_thresh=23813, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26414, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26626, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27423, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15240, bdi_thresh=16825, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24160, bdi_thresh=27821, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18249, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24160, bdi_thresh=27823, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24919, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18248, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27426, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18248, background_thresh=145168, dirty_thresh=290337
> > mm/page-writeback.c 762 background_writeout: comm=pdflush pid=599 n=-6696603
> > bdi_nr_reclaimable=22840, bdi_thresh=26421, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24905, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24918, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26424, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19034, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26420, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24923, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24906, background_thresh=145168, dirty_thresh=290337
> > global dirty=246847 writeback=43800 nfs=0 flags=C_ towrite=1023 skipped=0
> > bdi_nr_reclaimable=21040, bdi_thresh=24491, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26421, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18250, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18248, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24903, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24160, bdi_thresh=27821, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24905, background_thresh=145168, dirty_thresh=290337
> > requeue_io +527: inode 0
> > bdi_nr_reclaimable=15880, bdi_thresh=18248, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19030, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24903, background_thresh=145168, dirty_thresh=290337
> > requeue_io +527: inode 0
> > bdi_nr_reclaimable=15880, bdi_thresh=18246, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24495, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24919, background_thresh=145168, dirty_thresh=290337
> > redirty_tail +502: inode 12533
> > bdi_nr_reclaimable=21040, bdi_thresh=24496, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27418, background_thresh=145168, dirty_thresh=290337
> > redirty_tail +435: inode 174623
> > bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
> > redirty_tail +435: inode 4784181
> > redirty_tail +435: inode 3474323
> > requeue_io +527: inode 0
> > requeue_io +527: inode 0
> > redirty_tail +502: inode 13474
> > mm/page-writeback.c 762 background_writeout: comm=pdflush pid=10090 n=-6621691
> > bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18246, background_thresh=145168, dirty_thresh=290337
> > mm/page-writeback.c 762 background_writeout: comm=pdflush pid=599 n=-6696603
> > bdi_nr_reclaimable=15640, bdi_thresh=19030, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24903, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26421, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19034, background_thresh=145168, dirty_thresh=290337
> > global dirty=246847 writeback=43709 nfs=0 flags=C_ towrite=1024 skipped=0
> > bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24160, bdi_thresh=27823, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24160, bdi_thresh=27821, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=19720, bdi_thresh=23425, background_thresh=145168, dirty_thresh=290337
> > global dirty=246847 writeback=43709 nfs=0 flags=C_ towrite=1022 skipped=0
> > bdi_nr_reclaimable=21640, bdi_thresh=24920, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24160, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27425, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26611, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24489, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24926, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24903, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27421, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24929, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24160, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19026, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27412, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26615, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26624, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=20480, bdi_thresh=23810, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24904, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=20480, bdi_thresh=23811, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27422, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24480, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24900, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24160, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24160, bdi_thresh=27820, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=20480, bdi_thresh=23803, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15240, bdi_thresh=16825, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=20480, bdi_thresh=23794, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24485, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24933, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18248, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24933, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24920, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24903, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26421, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24926, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24160, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24932, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27412, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26615, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=19720, bdi_thresh=23425, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27425, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26611, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24900, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24903, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24492, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19030, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15240, bdi_thresh=16824, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24160, bdi_thresh=27821, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19026, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27416, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24904, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26625, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18246, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24920, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24480, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24160, bdi_thresh=27820, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=20480, bdi_thresh=23794, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24485, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=20480, bdi_thresh=23803, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15240, bdi_thresh=16825, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24933, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18248, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26611, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18247, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24903, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24160, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24926, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27422, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18246, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27412, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26615, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24160, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24160, bdi_thresh=27810, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=19720, bdi_thresh=23425, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24160, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27425, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24933, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24900, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24900, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24930, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15240, bdi_thresh=16824, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24493, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24481, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24930, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26611, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24864, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24863, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24918, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18245, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24924, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15240, bdi_thresh=16823, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24860, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27412, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27416, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24160, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19026, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18243, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18246, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24490, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26625, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24931, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24160, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24478, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24160, bdi_thresh=27820, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=20480, bdi_thresh=23792, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26615, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=20480, bdi_thresh=23801, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24160, bdi_thresh=27810, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24491, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=20480, bdi_thresh=23809, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24918, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27425, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24863, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=19720, bdi_thresh=23425, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24924, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15240, bdi_thresh=16823, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24860, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27412, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24483, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24160, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24930, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24160, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26611, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24864, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24160, bdi_thresh=27815, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27422, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24928, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15240, bdi_thresh=16823, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18246, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27416, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24480, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15240, bdi_thresh=16824, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24931, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24492, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24864, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26623, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15240, bdi_thresh=16825, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18245, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26418, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26623, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24492, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24866, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19026, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18243, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15240, bdi_thresh=16823, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=20480, bdi_thresh=23808, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18245, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27421, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19030, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24160, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26618, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=20480, bdi_thresh=23791, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26615, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24490, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26611, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24160, bdi_thresh=27810, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=20480, bdi_thresh=23800, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24863, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24160, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24941, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24477, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24927, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27412, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24481, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=20480, bdi_thresh=23807, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27425, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24200, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24200, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26625, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=19720, bdi_thresh=23425, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24200, bdi_thresh=27810, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24933, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18245, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18242, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26611, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26623, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=20480, bdi_thresh=23790, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26615, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15240, bdi_thresh=16822, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=20480, bdi_thresh=23799, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24200, bdi_thresh=27820, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24863, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18244, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27421, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19026, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24476, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26618, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24927, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24200, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27422, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27416, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24864, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24866, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15240, bdi_thresh=16822, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=20480, bdi_thresh=23806, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24200, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24940, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15240, bdi_thresh=16823, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19030, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24860, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24489, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24940, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24488, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24490, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24478, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18244, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24200, bdi_thresh=27815, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24939, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24937, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24935, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27413, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24200, bdi_thresh=27823, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15240, bdi_thresh=16822, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24490, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18244, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26623, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24492, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15240, bdi_thresh=16824, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26418, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24937, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26425, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=20480, bdi_thresh=23808, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24200, bdi_thresh=27823, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24937, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27426, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24936, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24938, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24864, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26414, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27423, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18243, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24862, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26626, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15240, bdi_thresh=16824, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18246, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18245, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24200, bdi_thresh=27821, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24936, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27421, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26624, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19034, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24935, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27426, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18246, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26421, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15240, bdi_thresh=16824, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24927, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24481, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27425, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24863, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24200, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24200, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26420, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=19720, bdi_thresh=23425, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18245, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24200, bdi_thresh=27810, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26611, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24933, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27412, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26615, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24864, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26625, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24200, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=20480, bdi_thresh=23799, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24200, bdi_thresh=27820, background_thresh=145168, dirty_thresh=290337
> > requeue_io +527: inode 0
> > bdi_nr_reclaimable=21640, bdi_thresh=24940, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24860, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15240, bdi_thresh=16823, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24200, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=20480, bdi_thresh=23807, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24476, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27422, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=20480, bdi_thresh=23790, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27416, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24200, bdi_thresh=27815, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24478, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15240, bdi_thresh=16822, background_thresh=145168, dirty_thresh=290337
> > requeue_io +527: inode 0
> > bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24939, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18242, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24940, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24937, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24488, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24200, bdi_thresh=27823, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24490, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18244, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24866, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18244, background_thresh=145168, dirty_thresh=290337
> > redirty_tail +502: inode 13925
> > bdi_nr_reclaimable=23080, bdi_thresh=26623, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26418, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19026, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24489, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15240, bdi_thresh=16824, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24935, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24490, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15240, bdi_thresh=16821, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=20480, bdi_thresh=23805, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=20480, bdi_thresh=23798, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27421, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26420, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=19720, bdi_thresh=23425, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18245, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26611, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15240, bdi_thresh=16822, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27426, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24932, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27412, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26615, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27425, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24823, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24200, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24200, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26424, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24200, bdi_thresh=27820, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26624, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26421, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24938, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18244, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24475, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=20480, bdi_thresh=23789, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24479, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18243, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24943, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18244, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24200, bdi_thresh=27821, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18243, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15240, bdi_thresh=16820, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24947, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24200, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24947, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24200, bdi_thresh=27810, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24820, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24942, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15240, bdi_thresh=16822, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27416, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24824, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26625, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27422, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24476, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18242, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18242, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15240, bdi_thresh=16820, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18240, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=20480, bdi_thresh=23805, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24490, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24944, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26623, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24825, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18241, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24943, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27421, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24826, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26425, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19034, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26618, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27423, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24200, bdi_thresh=27823, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24488, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27413, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24200, bdi_thresh=27815, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24489, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24200, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26414, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24944, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18241, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19030, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24200, bdi_thresh=27823, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24944, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24945, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=20480, bdi_thresh=23806, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24825, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24822, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24491, background_thresh=145168, dirty_thresh=290337
> > redirty_tail +435: inode 174623
> > bdi_nr_reclaimable=20480, bdi_thresh=23804, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27426, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26626, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27427, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24947, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18245, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26420, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24826, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26421, background_thresh=145168, dirty_thresh=290337
> > redirty_tail +435: inode 4784181
> > bdi_nr_reclaimable=21040, bdi_thresh=24484, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24483, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15240, bdi_thresh=16821, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24200, bdi_thresh=27821, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24823, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24825, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18243, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19030, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24823, background_thresh=145168, dirty_thresh=290337
> > redirty_tail +442: inode 3474323
> > bdi_nr_reclaimable=21040, bdi_thresh=24488, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24943, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24820, background_thresh=145168, dirty_thresh=290337
> > requeue_io +527: inode 0
> > bdi_nr_reclaimable=15880, bdi_thresh=18242, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27418, background_thresh=145168, dirty_thresh=290337
> > redirty_tail +502: inode 14004
> > bdi_nr_reclaimable=24200, bdi_thresh=27820, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24200, bdi_thresh=27820, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24934, background_thresh=145168, dirty_thresh=290337
> > mm/page-writeback.c 762 background_writeout: comm=pdflush pid=599 n=-6696607
> > bdi_nr_reclaimable=21320, bdi_thresh=24780, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19030, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18242, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24481, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27427, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27426, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=20480, bdi_thresh=23803, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24489, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24782, background_thresh=145168, dirty_thresh=290337
> > global dirty=247029 writeback=43527 nfs=0 flags=CM towrite=1020 skipped=0
> > bdi_nr_reclaimable=24200, bdi_thresh=27823, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24783, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18241, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26418, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26611, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24940, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24485, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=20480, bdi_thresh=23796, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19026, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26420, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18241, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=19720, bdi_thresh=23425, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26615, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27412, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=20480, bdi_thresh=23803, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24200, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18242, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24472, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24200, bdi_thresh=27822, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15240, bdi_thresh=16820, background_thresh=145168, dirty_thresh=290337
> > mm/page-writeback.c 762 background_writeout: comm=pdflush pid=10090 n=-6621691
> > bdi_nr_reclaimable=24200, bdi_thresh=27820, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27422, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26623, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24477, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=20480, bdi_thresh=23787, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26625, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24781, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24947, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15240, bdi_thresh=16821, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24200, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24474, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24946, background_thresh=145168, dirty_thresh=290337
> > global dirty=247029 writeback=43527 nfs=0 flags=C_ towrite=1024 skipped=0
> > bdi_nr_reclaimable=15240, bdi_thresh=16819, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24484, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27416, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24942, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24785, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=20480, bdi_thresh=23804, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24944, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18239, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21320, bdi_thresh=24826, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24200, bdi_thresh=27810, background_thresh=145168, dirty_thresh=290337
> > redirty_tail +435: inode 174623
> > bdi_nr_reclaimable=15240, bdi_thresh=16819, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26623, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24486, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24486, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18241, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26618, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26425, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27423, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21040, bdi_thresh=24488, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24943, background_thresh=145168, dirty_thresh=290337
> > redirty_tail +435: inode 4784181
> > bdi_nr_reclaimable=23920, bdi_thresh=27413, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24200, bdi_thresh=27823, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24945, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24200, bdi_thresh=27815, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15880, bdi_thresh=18240, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19030, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24944, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24200, bdi_thresh=27821, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=24200, bdi_thresh=27824, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15240, bdi_thresh=16821, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21240, bdi_thresh=24825, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=20480, bdi_thresh=23804, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=23920, bdi_thresh=27426, background_thresh=145168, dirty_thresh=290337
> > redirty_tail +442: inode 3474323
> > bdi_nr_reclaimable=23080, bdi_thresh=26626, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=21640, bdi_thresh=24902, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145168, dirty_thresh=290337
> > bdi_nr_reclaimable=15640, bdi_thresh=19034, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=23920, bdi_thresh=27421, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15880, bdi_thresh=18243, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21640, bdi_thresh=24907, background_thresh=145177, dirty_thresh=290355
> > requeue_io +527: inode 0
> > bdi_nr_reclaimable=21040, bdi_thresh=24483, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15880, bdi_thresh=18244, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21640, bdi_thresh=24904, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=22840, bdi_thresh=26421, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21080, bdi_thresh=24785, background_thresh=145177, dirty_thresh=290355
> > redirty_tail +502: inode 14023
> > bdi_nr_reclaimable=22840, bdi_thresh=26421, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15240, bdi_thresh=16820, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15880, bdi_thresh=18243, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21040, bdi_thresh=24786, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=22840, bdi_thresh=26424, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=22840, bdi_thresh=26420, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15880, bdi_thresh=18241, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15880, bdi_thresh=18240, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21000, bdi_thresh=24783, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21640, bdi_thresh=24903, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21640, bdi_thresh=24907, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15880, bdi_thresh=18242, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21040, bdi_thresh=24488, background_thresh=145177, dirty_thresh=290355
> > redirty_tail +435: inode 21
> > bdi_nr_reclaimable=21000, bdi_thresh=24823, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=23080, bdi_thresh=26624, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21640, bdi_thresh=24903, background_thresh=145177, dirty_thresh=290355
> > redirty_tail +435: inode 174623
> > bdi_nr_reclaimable=23920, bdi_thresh=27421, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=23920, bdi_thresh=27418, background_thresh=145177, dirty_thresh=290355
> > redirty_tail +435: inode 4784181
> > bdi_nr_reclaimable=22840, bdi_thresh=26414, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145177, dirty_thresh=290355
> > redirty_tail +435: inode 3474323
> > requeue_io +527: inode 0
> > bdi_nr_reclaimable=23920, bdi_thresh=27425, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15240, bdi_thresh=16819, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21640, bdi_thresh=24904, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21000, bdi_thresh=24824, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=24200, bdi_thresh=27821, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21040, bdi_thresh=24487, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15880, bdi_thresh=18242, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15880, bdi_thresh=18241, background_thresh=145177, dirty_thresh=290355
> > redirty_tail +502: inode 14042
> > bdi_nr_reclaimable=21640, bdi_thresh=24905, background_thresh=145177, dirty_thresh=290355
> > mm/page-writeback.c 762 background_writeout: comm=pdflush pid=599 n=-6696985
> > bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21000, bdi_thresh=24823, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21040, bdi_thresh=24486, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15880, bdi_thresh=18241, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=22840, bdi_thresh=26421, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21640, bdi_thresh=24867, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15640, bdi_thresh=19034, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15240, bdi_thresh=16820, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=24200, bdi_thresh=27824, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21040, bdi_thresh=24485, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21640, bdi_thresh=24860, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=24200, bdi_thresh=27820, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=23080, bdi_thresh=26611, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=22840, bdi_thresh=26420, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=20480, bdi_thresh=23801, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21040, bdi_thresh=24480, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21640, bdi_thresh=24854, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15640, bdi_thresh=19026, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=24200, bdi_thresh=27822, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21000, bdi_thresh=24823, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=23080, bdi_thresh=26615, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=23920, bdi_thresh=27412, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21040, bdi_thresh=24488, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21040, bdi_thresh=24471, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=23080, bdi_thresh=26620, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15240, bdi_thresh=16818, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15880, bdi_thresh=18241, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21000, bdi_thresh=24822, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=24200, bdi_thresh=27822, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15880, bdi_thresh=18241, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15240, bdi_thresh=16819, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=20480, bdi_thresh=23794, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21000, bdi_thresh=24820, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21040, bdi_thresh=24476, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21040, bdi_thresh=24487, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=20480, bdi_thresh=23785, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=24200, bdi_thresh=27820, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15640, bdi_thresh=19030, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=23920, bdi_thresh=27422, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21640, bdi_thresh=24867, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=22840, bdi_thresh=26424, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=24200, bdi_thresh=27824, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21000, bdi_thresh=24820, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=23920, bdi_thresh=27427, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=23920, bdi_thresh=27426, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=23080, bdi_thresh=26626, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=23920, bdi_thresh=27416, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21640, bdi_thresh=24866, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15640, bdi_thresh=19031, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15240, bdi_thresh=16818, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21640, bdi_thresh=24867, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=23920, bdi_thresh=27425, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=24200, bdi_thresh=27821, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=20480, bdi_thresh=23802, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15880, bdi_thresh=18239, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15880, bdi_thresh=18238, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15880, bdi_thresh=18240, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21040, bdi_thresh=24473, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=24200, bdi_thresh=27823, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=24200, bdi_thresh=27810, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15880, bdi_thresh=18240, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21640, bdi_thresh=24864, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21040, bdi_thresh=24485, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145177, dirty_thresh=290355
> > mm/page-writeback.c 762 background_writeout: comm=pdflush pid=10090 n=-6621693
> > bdi_nr_reclaimable=21040, bdi_thresh=24482, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21040, bdi_thresh=24483, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15880, bdi_thresh=18239, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=22840, bdi_thresh=26418, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21000, bdi_thresh=24786, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=20480, bdi_thresh=23800, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=24200, bdi_thresh=27823, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21640, bdi_thresh=24862, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15240, bdi_thresh=16817, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15640, bdi_thresh=19027, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15880, bdi_thresh=18239, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=22840, bdi_thresh=26425, background_thresh=145177, dirty_thresh=290355
> > global dirty=246665 writeback=43800 nfs=0 flags=C_ towrite=1022 skipped=0
> > bdi_nr_reclaimable=21000, bdi_thresh=24784, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=22840, bdi_thresh=26422, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21000, bdi_thresh=24785, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=23080, bdi_thresh=26618, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21040, bdi_thresh=24480, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21040, bdi_thresh=24484, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=23080, bdi_thresh=26621, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21640, bdi_thresh=24863, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15240, bdi_thresh=16819, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=23080, bdi_thresh=26623, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21640, bdi_thresh=24863, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15640, bdi_thresh=19033, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21640, bdi_thresh=24864, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=24200, bdi_thresh=27815, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=23920, bdi_thresh=27426, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=24200, bdi_thresh=27821, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=22840, bdi_thresh=26423, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=23920, bdi_thresh=27421, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=23920, bdi_thresh=27413, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=22840, bdi_thresh=26420, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15880, bdi_thresh=18241, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21000, bdi_thresh=24785, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15880, bdi_thresh=18238, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=23080, bdi_thresh=26626, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=23920, bdi_thresh=27423, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=20480, bdi_thresh=23802, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=22840, bdi_thresh=26421, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21440, bdi_thresh=24864, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15880, bdi_thresh=18239, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=23920, bdi_thresh=27421, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15640, bdi_thresh=19030, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15880, bdi_thresh=18241, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15880, bdi_thresh=18240, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21360, bdi_thresh=24862, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21360, bdi_thresh=24903, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=23920, bdi_thresh=27418, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=22840, bdi_thresh=26414, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21320, bdi_thresh=24865, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=23080, bdi_thresh=26624, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15640, bdi_thresh=19032, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=15880, bdi_thresh=18242, background_thresh=145177, dirty_thresh=290355
> > global dirty=246301 writeback=44164 nfs=0 flags=C_ towrite=646 skipped=0
> > bdi_nr_reclaimable=15240, bdi_thresh=16818, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=19720, bdi_thresh=23385, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21000, bdi_thresh=24786, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21280, bdi_thresh=24863, background_thresh=145177, dirty_thresh=290355
> > redirty_tail +435: inode 174623
> > bdi_nr_reclaimable=21000, bdi_thresh=24785, background_thresh=145177, dirty_thresh=290355
> > bdi_nr_reclaimable=21000, bdi_thresh=24783, background_thresh=145177, dirty_thresh=290355
> > redirty_tail +435: inode 4784181
> > redirty_tail +435: inode 3474323
> > requeue_io +527: inode 0
> > redirty_tail +502: inode 14061
> > bdi_nr_reclaimable=23080, bdi_thresh=26623, background_thresh=145177, dirty_thresh=290355
> > redirty_tail +442: inode 27
> > redirty_tail +435: inode 174623
> > redirty_tail +435: inode 4784181
> > redirty_tail +435: inode 3474323
> > requeue_io +527: inode 0
> > redirty_tail +502: inode 14096
> 

Re: regression in page writeback

[Wu Fengguang]

On Fri, Sep 25, 2009 at 01:04:13PM +0800, Dave Chinner wrote:
> On Thu, Sep 24, 2009 at 08:38:20PM -0400, Chris Mason wrote:
> > On Fri, Sep 25, 2009 at 10:11:17AM +1000, Dave Chinner wrote:
> > > On Thu, Sep 24, 2009 at 11:15:08AM +0800, Wu Fengguang wrote:
> > > > On Wed, Sep 23, 2009 at 10:00:58PM +0800, Chris Mason wrote:
> > > > > The only place that actually honors the congestion flag is pdflush.
> > > > > It's trivial to get pdflush backed up and make it sit down without
> > > > > making any progress because once the queue congests, pdflush goes away.
> > > > 
> > > > Right. I guess that's more or less intentional - to give lowest priority
> > > > to periodic/background writeback.
> > > 
> > > IMO, this is the wrong design. Background writeback should
> > > have higher CPU/scheduler priority than normal tasks. If there is
> > > sufficient dirty pages in the system for background writeback to
> > > be active, it should be running *now* to start as much IO as it can
> > > without being held up by other, lower priority tasks.
> > 
> > I'd say that an fsync from mutt or vi should be done at a higher prio
> > than a background streaming writer.
> 
> I don't think you caught everything I said - synchronous IO is
> un-throttled.

O_SYNC writes may be un-throttled in theory, however it seems to be
throttled in practice:

  generic_file_aio_write
    __generic_file_aio_write
      generic_file_buffered_write
        generic_perform_write
          balance_dirty_pages_ratelimited
    generic_write_sync

Do you mean some other code path?

> Background writeback should dump async IO to the elevator as fast as
> it can, then get the hell out of the way. If you've got a UP system,
> then the fsync can't be issued at the same time pdflush is running
> (same as right now), and if you've got a MP system then fsync can
> run at the same time.

I think you are right for system wide sync.

System wide sync seems to always wait for the queued bdi writeback
works to finish, which should be fine in terms of efficiency, except
that sync could end up do more works and even live lock.

> On the premise that sync IO is unthrottled and given that elevators
> queue and issue sync IO sperately to async writes, fsync latency
> would be entirely derived from the elevator queuing behaviour, not
> the CPU priority of pdflush.

It's not exactly CPU priority, but queue fullness priority.

fsync operations always use nonblocking=0, so in fact they _used to_
enjoy better priority than pdflush. Same is vmscan pageout, which
calls writepage directly. Both won't back off on congested bdi.

So when there comes fsync/pageout, they will always be served first.

> Look at it this way - it is the responsibility of pdflush to keep
> the elevator full of background IO. It is the responsibility of
> the elevator to ensure that background IO doesn't starve all other
> types of IO.

Agreed.

> If pdflush doesn't run because it can't get CPU time,
> then background IO does not get issued, and system performance
> suffers as a result.

pdflush is able to make 80% queue fullness, which should be enough
for efficient streaming IOs. Small random IOs may hurt a bit though.

Thanks,
Fengguang

Re: regression in page writeback

[Chris Mason]

On Fri, Sep 25, 2009 at 03:04:13PM +1000, Dave Chinner wrote:
> On Thu, Sep 24, 2009 at 08:38:20PM -0400, Chris Mason wrote:
> > On Fri, Sep 25, 2009 at 10:11:17AM +1000, Dave Chinner wrote:
> > > On Thu, Sep 24, 2009 at 11:15:08AM +0800, Wu Fengguang wrote:
> > > > On Wed, Sep 23, 2009 at 10:00:58PM +0800, Chris Mason wrote:
> > > > > The only place that actually honors the congestion flag is pdflush.
> > > > > It's trivial to get pdflush backed up and make it sit down without
> > > > > making any progress because once the queue congests, pdflush goes away.
> > > > 
> > > > Right. I guess that's more or less intentional - to give lowest priority
> > > > to periodic/background writeback.
> > > 
> > > IMO, this is the wrong design. Background writeback should
> > > have higher CPU/scheduler priority than normal tasks. If there is
> > > sufficient dirty pages in the system for background writeback to
> > > be active, it should be running *now* to start as much IO as it can
> > > without being held up by other, lower priority tasks.
> > 
> > I'd say that an fsync from mutt or vi should be done at a higher prio
> > than a background streaming writer.
> 
> I don't think you caught everything I said - synchronous IO is
> un-throttled. Background writeback should dump async IO to the
> elevator as fast as it can, then get the hell out of the way. If
> you've got a UP system, then the fsync can't be issued at the same
> time pdflush is running (same as right now), and if you've got a MP
> system then fsync can run at the same time. On the premise that sync
> IO is unthrottled and given that elevators queue and issue sync IO
> sperately to async writes, fsync latency would be entirely derived
> from the elevator queuing behaviour, not the CPU priority of
> pdflush.

I think we've agreed for a long time on this in general.  The congestion
backoff comment was originally about IO priorities (I thought ;) so I
was trying to keep talking around IO priority and not CPU/scheduler
time.  When we get things tuned to the point that process scheduling
matters, I'll be a very happy boy.

The big change from the new code is that we will fill the queue
with async IO.

I think this is good, and I think the congestion backoff didn't really
consistently keep available requests in the queue all the time in a lot
of workloads.  But, its still a change, and so we need to keep an eye on
it as we look at performance reports during .32.

> 
> Look at it this way - it is the responsibility of pdflush to keep
> the elevator full of background IO. It is the responsibility of
> the elevator to ensure that background IO doesn't starve all other
> types of IO. If pdflush doesn't run because it can't get CPU time,
> then background IO does not get issued, and system performance
> suffers as a result.

Most of the time that pdflush didn't get to run in my benchmark it's
because pdflush chose to give up the CPU, not because it was starving.

-chris

Re: regression in page writeback

[Dave Chinner]

On Fri, Sep 25, 2009 at 11:19:20AM +0800, Wu Fengguang wrote:
> On Fri, Sep 25, 2009 at 08:11:17AM +0800, Dave Chinner wrote:
> > On Thu, Sep 24, 2009 at 11:15:08AM +0800, Wu Fengguang wrote:
> > > On Wed, Sep 23, 2009 at 10:00:58PM +0800, Chris Mason wrote:
> > > > The only place that actually honors the congestion flag is pdflush.
> > > > It's trivial to get pdflush backed up and make it sit down without
> > > > making any progress because once the queue congests, pdflush goes away.
> > > 
> > > Right. I guess that's more or less intentional - to give lowest priority
> > > to periodic/background writeback.
> > 
> > IMO, this is the wrong design. Background writeback should
> > have higher CPU/scheduler priority than normal tasks. If there is
> > sufficient dirty pages in the system for background writeback to
> > be active, it should be running *now* to start as much IO as it can
> > without being held up by other, lower priority tasks.
> > 
> > Cleaning pages is important to keeping the system running smoothly.
> > Given that IO takes time to clean pages, it is therefore important
> > to issue as much as possible as quickly as possible without delays
> > before going back to sleep. Delaying issue of the IO or doing
> > sub-optimal issue simply reduces performance of the system because
> > it takes longer to clean the same number of dirty pages.
> > 
> > > > Nothing stops other procs from keeping the queue congested forever.
> > > > This can only be fixed by making everyone wait for congestion, at which
> > > > point we might as well wait for requests.
> > > 
> > > Yes. That gives everyone somehow equal opportunity, this is a policy change
> > > that may lead to interesting effects, as well as present a challenge to
> > > get_request_wait(). That said, I'm not against the change to a wait queue
> > > in general.
> > 
> > If you block all threads doing _writebehind caching_ (synchronous IO
> > is self-throttling) to the same BDI on the same queue as the bdi
> > flusher then when congestion clears the higher priority background
> > flusher thread should run first and issue more IO.  This should
> > happen as a natural side-effect of our scheduling algorithms and it
> > gives preference to efficient background writeback over in-efficient
> > foreground writeback. Indeed, with this approach we can even avoid
> > foreground writeback altogether...
> 
> I don't see how balance_dirty_pages() writeout is less efficient than
> pdflush writeout.
> 
> They all called the same routines to do the job.
> balance_dirty_pages() sets nr_to_write=1536 at least for ext4 and xfs
> (unless memory is tight; btrfs is 1540), which is in fact 50% bigger
> than the 1024 pages used by pdflush.

Sure, but the prёblem now is that you are above the
bdi->dirty_exceeded threshold, foreground writeback tries to issue
1536 pages of IO every 8 pages that are dirtied. That means you'll
block just about every writing process in writeback at the same time
and they will all be blocked in congestion trying to write different
inodes....

> And it won't back off on congestion.

And that is, IMO, a major problem.

> The s_io/b_io queues are shared, so a balance_dirty_pages() will just
> continue from where the last sync thread exited. So it does not make
> much difference who initiates the IO. Did I missed something?

The current implementation uses the request queue to do that
blocking at IO submission time. This is based on the premise that if
we write a certain number of pages, we're guaranteed to have waited
long enough for that many pages to come clean. However, every other
thread doing writes and being throttled does the same thing.  This
leads to N IO submitters from at least N different inodes at the
same time. Which inode gets written when congestion clears is
anyone's guess - it's a thundering herd IIUC the congestion
implementation correctly.

The result is that we end up with N different sets of IO being
issued with potentially zero locality to each other, resulting in
much lower elevator sort/merge efficiency and hence we seek the disk
all over the place to service the different sets of IO.

OTOH, if there is only one submission thread, it doesn't jump
between inodes in the same way when congestion clears - if keeps
writing to the same inode, resulting in large related chunks of
sequential IOs being issued to the disk. This is more efficient than
the above foreground writeback because the elevator works better and
the disk seeks less.

As you can probably guess, I think foreground writeout is the wrong
architecture because of the behaviour it induces under heavy
multithreaded IO patterns. I agree that it works OK if continue
tweaking it to fix problems.

However, my concern is that if it isn't constantly observed, tweaked
and maintained, performance goes backwards as other code changes.
i.e. there is a significant maintenance burden and looking at the
problems once very couple of years (last big maintenance rounds were
2.6.15/16, 2.6.23/24, now 2.6.31/32) isn't good enough to prevent
performance form sliding backwards from release to release.

----

The rest of this is an idea I've been kicking around for a while
which is derived from IO throttling work I've done during a
past life.  I haven't had time to research and prototype it to see
if it performs any better under really heavy load, but I'm going to
throw it out anyway so that everyone can see a little bit more about
what I'm thinking.

My fundamental premise is that throttling does not require IO to be
issued from the thread to be throttled. The essence of write
throttling is waiting for more pages to be cleaned in a period of
time than has been dirtied. i.e.  What we are really waiting on is
pages to be cleaned.

Based on this observation, if we have a flusher thread working in
the background, we don't need to submit more IO when we need to
throttle as all we need to do is wait for a certain number of pages
to transition to the clean state.

If we take a leaf from XFS's book by doing work at IO completion
rather than submission we can keep a count of the number of pages
cleaned on the bdi. This can be used to implement a FIFO-like
throttle. If we add a simple ticket system to the bdi, when a
process needs to be throttled it can submit a ticket with the number
of pages it has dirtied to the bdi, and the bdi can then decide what
the page cleaned count needs to reach before the process can be
woken.

i.e. Take the following ascii art showing the bdi fllusher
thread running and issuing IO in the background:

bdi write thread:   +---X------Y---+-A-----ZB----+------C--------+
1st proc:		o............o
2nd proc:		       o............o
3rd proc:		                   o............o

When the 1st process comes in to be throttled, it samples the page
clean count and gets X. It submits a ticket to be woken at A (X +
some number of pages). If the flusher thread is not running, it gets
kicked.  Process 2 and 3 do the same at Y and Z to be woken at B and
C. At IO completion, the number of pages cleaned is counted and the
tickets that are now under the clean count are pulled from the queue
and the processes that own them are woken.

This avoids the thundering herd problem and applies throttling in
a deterministic, predictable fashion. And by relying on background
writeback, we only have one writeback path to optimise, not two
different paths that interchange unpredictably.

In essence, this mechanism replaces the complex path of IO
submission and congestion with a simple, deterministic counter and
queue system that probably doesn't even require any memory
allocation to implement. I think the simpler a thorttle mechanism
is the more likely it is to work effectively....

I know that words without code aren't going to convince anyone, but I
hope I've given you some food for thought about alternatives to what
we currently do. ;)

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: regression in page writeback

[Wu Fengguang]

On Sat, Sep 26, 2009 at 09:47:15AM +0800, Dave Chinner wrote:
> On Fri, Sep 25, 2009 at 11:19:20AM +0800, Wu Fengguang wrote:
> > On Fri, Sep 25, 2009 at 08:11:17AM +0800, Dave Chinner wrote:
> > > On Thu, Sep 24, 2009 at 11:15:08AM +0800, Wu Fengguang wrote:
> > > > On Wed, Sep 23, 2009 at 10:00:58PM +0800, Chris Mason wrote:
> > > > > The only place that actually honors the congestion flag is pdflush.
> > > > > It's trivial to get pdflush backed up and make it sit down without
> > > > > making any progress because once the queue congests, pdflush goes away.
> > > > 
> > > > Right. I guess that's more or less intentional - to give lowest priority
> > > > to periodic/background writeback.
> > > 
> > > IMO, this is the wrong design. Background writeback should
> > > have higher CPU/scheduler priority than normal tasks. If there is
> > > sufficient dirty pages in the system for background writeback to
> > > be active, it should be running *now* to start as much IO as it can
> > > without being held up by other, lower priority tasks.
> > > 
> > > Cleaning pages is important to keeping the system running smoothly.
> > > Given that IO takes time to clean pages, it is therefore important
> > > to issue as much as possible as quickly as possible without delays
> > > before going back to sleep. Delaying issue of the IO or doing
> > > sub-optimal issue simply reduces performance of the system because
> > > it takes longer to clean the same number of dirty pages.
> > > 
> > > > > Nothing stops other procs from keeping the queue congested forever.
> > > > > This can only be fixed by making everyone wait for congestion, at which
> > > > > point we might as well wait for requests.
> > > > 
> > > > Yes. That gives everyone somehow equal opportunity, this is a policy change
> > > > that may lead to interesting effects, as well as present a challenge to
> > > > get_request_wait(). That said, I'm not against the change to a wait queue
> > > > in general.
> > > 
> > > If you block all threads doing _writebehind caching_ (synchronous IO
> > > is self-throttling) to the same BDI on the same queue as the bdi
> > > flusher then when congestion clears the higher priority background
> > > flusher thread should run first and issue more IO.  This should
> > > happen as a natural side-effect of our scheduling algorithms and it
> > > gives preference to efficient background writeback over in-efficient
> > > foreground writeback. Indeed, with this approach we can even avoid
> > > foreground writeback altogether...
> > 
> > I don't see how balance_dirty_pages() writeout is less efficient than
> > pdflush writeout.
> > 
> > They all called the same routines to do the job.
> > balance_dirty_pages() sets nr_to_write=1536 at least for ext4 and xfs
> > (unless memory is tight; btrfs is 1540), which is in fact 50% bigger
> > than the 1024 pages used by pdflush.
> 
> Sure, but the prёblem now is that you are above the
> bdi->dirty_exceeded threshold, foreground writeback tries to issue
> 1536 pages of IO every 8 pages that are dirtied. That means you'll

I'd suggest to increase that "8 pages". It may be good when
ratelimit_pages is statically set to 32. Now that ratelimit_pages is
dynamically set to a much larger value at boot time, we'd better use
ratelimit_pages/4 for dirty_exceeded case.

> block just about every writing process in writeback at the same time
> and they will all be blocked in congestion trying to write different
> inodes....

Ah got it, balance_dirty_pages could lead to too much concurrency and
thus seeks!

> > And it won't back off on congestion.
> 
> And that is, IMO, a major problem.

Not necessarily, the above concurrency problem occurs even if it backs
off on congestion. But anyway we are switching to get_request_wait now.

> > The s_io/b_io queues are shared, so a balance_dirty_pages() will just
> > continue from where the last sync thread exited. So it does not make
> > much difference who initiates the IO. Did I missed something?
> 
> The current implementation uses the request queue to do that
> blocking at IO submission time. This is based on the premise that if
> we write a certain number of pages, we're guaranteed to have waited
> long enough for that many pages to come clean.

Right.

> However, every other
> thread doing writes and being throttled does the same thing.  This
> leads to N IO submitters from at least N different inodes at the
> same time. Which inode gets written when congestion clears is
> anyone's guess - it's a thundering herd IIUC the congestion
> implementation correctly.

Good insight, thanks for pointing this out!

> The result is that we end up with N different sets of IO being
> issued with potentially zero locality to each other, resulting in
> much lower elevator sort/merge efficiency and hence we seek the disk
> all over the place to service the different sets of IO.

Right. But note that its negative effects are not that common given
the current parameters MAX_WRITEBACK_PAGES=1024, max_sectors_kb=512
and nr_requests=128. As we are going on to the next inode anyway
when 4MB of it have been enqueued. So a request queue could hold up to
128/(4096/512) = 16 inodes.

So the problem will turn up when there are >= 16 throttled processes.

When we increase MAX_WRITEBACK_PAGES to 128MB, even _one_ foreground
writeout will hurt.

> OTOH, if there is only one submission thread, it doesn't jump
> between inodes in the same way when congestion clears - if keeps
> writing to the same inode, resulting in large related chunks of
> sequential IOs being issued to the disk. This is more efficient than
> the above foreground writeback because the elevator works better and
> the disk seeks less.
> 
> As you can probably guess, I think foreground writeout is the wrong
> architecture because of the behaviour it induces under heavy
> multithreaded IO patterns. I agree that it works OK if continue
> tweaking it to fix problems.

Agreed. 

> However, my concern is that if it isn't constantly observed, tweaked
> and maintained, performance goes backwards as other code changes.
> i.e. there is a significant maintenance burden and looking at the
> problems once very couple of years (last big maintenance rounds were
> 2.6.15/16, 2.6.23/24, now 2.6.31/32) isn't good enough to prevent
> performance form sliding backwards from release to release.

One problem is that the queues are very tightly coupled. Every change
of behavior leads to reevaluation of other factors.

> ----
> 
> The rest of this is an idea I've been kicking around for a while
> which is derived from IO throttling work I've done during a
> past life.  I haven't had time to research and prototype it to see
> if it performs any better under really heavy load, but I'm going to
> throw it out anyway so that everyone can see a little bit more about
> what I'm thinking.
> 
> My fundamental premise is that throttling does not require IO to be
> issued from the thread to be throttled. The essence of write
> throttling is waiting for more pages to be cleaned in a period of
> time than has been dirtied. i.e.  What we are really waiting on is
> pages to be cleaned.

Yes, Peter's __bdi_writeout_inc() is a good watch point.

> Based on this observation, if we have a flusher thread working in
> the background, we don't need to submit more IO when we need to
> throttle as all we need to do is wait for a certain number of pages
> to transition to the clean state.

It's great You and Mason (and others) are advocating the same idea :)
Jan and me proposed some possible solutions recently:

        http://lkml.org/lkml/2009/9/14/126

> If we take a leaf from XFS's book by doing work at IO completion
> rather than submission we can keep a count of the number of pages
> cleaned on the bdi. This can be used to implement a FIFO-like
> throttle. If we add a simple ticket system to the bdi, when a
> process needs to be throttled it can submit a ticket with the number
> of pages it has dirtied to the bdi, and the bdi can then decide what
> the page cleaned count needs to reach before the process can be
> woken.

Exactly.

> i.e. Take the following ascii art showing the bdi fllusher
> thread running and issuing IO in the background:
> 
> bdi write thread:   +---X------Y---+-A-----ZB----+------C--------+
> 1st proc:		o............o
> 2nd proc:		       o............o
> 3rd proc:		                   o............o
> 
> When the 1st process comes in to be throttled, it samples the page
> clean count and gets X. It submits a ticket to be woken at A (X +
> some number of pages). If the flusher thread is not running, it gets
> kicked.  Process 2 and 3 do the same at Y and Z to be woken at B and
> C. At IO completion, the number of pages cleaned is counted and the
> tickets that are now under the clean count are pulled from the queue
> and the processes that own them are woken.
> 
> This avoids the thundering herd problem and applies throttling in
> a deterministic, predictable fashion. And by relying on background
> writeback, we only have one writeback path to optimise, not two
> different paths that interchange unpredictably.
> 
> In essence, this mechanism replaces the complex path of IO
> submission and congestion with a simple, deterministic counter and
> queue system that probably doesn't even require any memory
> allocation to implement. I think the simpler a thorttle mechanism
> is the more likely it is to work effectively....
> 
> I know that words without code aren't going to convince anyone, but I
> hope I've given you some food for thought about alternatives to what
> we currently do. ;)

Not at all, it's clear enough - your idea is very similar to Jan's
proposal. And I'd like to present an ascii art for another scheme:


                                          \                /
                  one bdi sync thread      \............../
               =========================>   \............/
                 working on dirty pages      \........../
                                              \......../
                                               \....../
                                                \..../
                                                 \../

  throttled    |  |    |  |    |  |     |  |     |  |     wakeup when enough
============>  |  |    |  |    |  |     |  |     |::| ==> pages are put to io
  task list    +--+    +--+    +--+     +--+     +--+     on behalf of this task
              task 5  task 4  task 3   task 2   task 1

               [.] dirty page  [:] writeback page

One benefit of this scheme is, when necessary, the task list could
be converted to some tree to do priorities and thus IO controller for
buffered writes :)

Thanks,
Fengguang

Re: regression in page writeback

[Wu Fengguang]

On Sat, Sep 26, 2009 at 09:47:15AM +0800, Dave Chinner wrote:
> On Fri, Sep 25, 2009 at 11:19:20AM +0800, Wu Fengguang wrote:
> > On Fri, Sep 25, 2009 at 08:11:17AM +0800, Dave Chinner wrote:
> > > On Thu, Sep 24, 2009 at 11:15:08AM +0800, Wu Fengguang wrote:
> > > > On Wed, Sep 23, 2009 at 10:00:58PM +0800, Chris Mason wrote:
> > > > > The only place that actually honors the congestion flag is pdflush.
> > > > > It's trivial to get pdflush backed up and make it sit down without
> > > > > making any progress because once the queue congests, pdflush goes away.
> > > > 
> > > > Right. I guess that's more or less intentional - to give lowest priority
> > > > to periodic/background writeback.
> > > 
> > > IMO, this is the wrong design. Background writeback should
> > > have higher CPU/scheduler priority than normal tasks. If there is
> > > sufficient dirty pages in the system for background writeback to
> > > be active, it should be running *now* to start as much IO as it can
> > > without being held up by other, lower priority tasks.
> > > 
> > > Cleaning pages is important to keeping the system running smoothly.
> > > Given that IO takes time to clean pages, it is therefore important
> > > to issue as much as possible as quickly as possible without delays
> > > before going back to sleep. Delaying issue of the IO or doing
> > > sub-optimal issue simply reduces performance of the system because
> > > it takes longer to clean the same number of dirty pages.
> > > 
> > > > > Nothing stops other procs from keeping the queue congested forever.
> > > > > This can only be fixed by making everyone wait for congestion, at which
> > > > > point we might as well wait for requests.
> > > > 
> > > > Yes. That gives everyone somehow equal opportunity, this is a policy change
> > > > that may lead to interesting effects, as well as present a challenge to
> > > > get_request_wait(). That said, I'm not against the change to a wait queue
> > > > in general.
> > > 
> > > If you block all threads doing _writebehind caching_ (synchronous IO
> > > is self-throttling) to the same BDI on the same queue as the bdi
> > > flusher then when congestion clears the higher priority background
> > > flusher thread should run first and issue more IO.  This should
> > > happen as a natural side-effect of our scheduling algorithms and it
> > > gives preference to efficient background writeback over in-efficient
> > > foreground writeback. Indeed, with this approach we can even avoid
> > > foreground writeback altogether...
> > 
> > I don't see how balance_dirty_pages() writeout is less efficient than
> > pdflush writeout.
> > 
> > They all called the same routines to do the job.
> > balance_dirty_pages() sets nr_to_write=1536 at least for ext4 and xfs
> > (unless memory is tight; btrfs is 1540), which is in fact 50% bigger
> > than the 1024 pages used by pdflush.
> 
> Sure, but the prёblem now is that you are above the
> bdi->dirty_exceeded threshold, foreground writeback tries to issue
> 1536 pages of IO every 8 pages that are dirtied. That means you'll

I'd suggest to increase that "8 pages". It may be good when
ratelimit_pages is statically set to 32. Now that ratelimit_pages is
dynamically set to a much larger value at boot time, we'd better use
ratelimit_pages/4 for dirty_exceeded case.

> block just about every writing process in writeback at the same time
> and they will all be blocked in congestion trying to write different
> inodes....

Ah got it, balance_dirty_pages could lead to too much concurrency and
thus seeks!

> > And it won't back off on congestion.
> 
> And that is, IMO, a major problem.

Not necessarily, the above concurrency problem occurs even if it backs
off on congestion. But anyway we are switching to get_request_wait now.

> > The s_io/b_io queues are shared, so a balance_dirty_pages() will just
> > continue from where the last sync thread exited. So it does not make
> > much difference who initiates the IO. Did I missed something?
> 
> The current implementation uses the request queue to do that
> blocking at IO submission time. This is based on the premise that if
> we write a certain number of pages, we're guaranteed to have waited
> long enough for that many pages to come clean.

Right.

> However, every other
> thread doing writes and being throttled does the same thing.  This
> leads to N IO submitters from at least N different inodes at the
> same time. Which inode gets written when congestion clears is
> anyone's guess - it's a thundering herd IIUC the congestion
> implementation correctly.

Good insight, thanks for pointing this out!

> The result is that we end up with N different sets of IO being
> issued with potentially zero locality to each other, resulting in
> much lower elevator sort/merge efficiency and hence we seek the disk
> all over the place to service the different sets of IO.

Right. But note that its negative effects are not that common given
the current parameters MAX_WRITEBACK_PAGES=1024, max_sectors_kb=512
and nr_requests=128. As we are going on to the next inode anyway
when 4MB of it have been enqueued. So a request queue could hold up to
128/(4096/512) = 16 inodes.

So the problem will turn up when there are >= 16 throttled processes.

When we increase MAX_WRITEBACK_PAGES to 128MB, even _one_ foreground
writeout will hurt.

> OTOH, if there is only one submission thread, it doesn't jump
> between inodes in the same way when congestion clears - if keeps
> writing to the same inode, resulting in large related chunks of
> sequential IOs being issued to the disk. This is more efficient than
> the above foreground writeback because the elevator works better and
> the disk seeks less.
> 
> As you can probably guess, I think foreground writeout is the wrong
> architecture because of the behaviour it induces under heavy
> multithreaded IO patterns. I agree that it works OK if continue
> tweaking it to fix problems.

Agreed. 

> However, my concern is that if it isn't constantly observed, tweaked
> and maintained, performance goes backwards as other code changes.
> i.e. there is a significant maintenance burden and looking at the
> problems once very couple of years (last big maintenance rounds were
> 2.6.15/16, 2.6.23/24, now 2.6.31/32) isn't good enough to prevent
> performance form sliding backwards from release to release.

One problem is that the queues are very tightly coupled. Every change
of behavior leads to reevaluation of other factors.

> ----
> 
> The rest of this is an idea I've been kicking around for a while
> which is derived from IO throttling work I've done during a
> past life.  I haven't had time to research and prototype it to see
> if it performs any better under really heavy load, but I'm going to
> throw it out anyway so that everyone can see a little bit more about
> what I'm thinking.
> 
> My fundamental premise is that throttling does not require IO to be
> issued from the thread to be throttled. The essence of write
> throttling is waiting for more pages to be cleaned in a period of
> time than has been dirtied. i.e.  What we are really waiting on is
> pages to be cleaned.

Yes, Peter's __bdi_writeout_inc() is a good watch point.

> Based on this observation, if we have a flusher thread working in
> the background, we don't need to submit more IO when we need to
> throttle as all we need to do is wait for a certain number of pages
> to transition to the clean state.

It's great You and Mason (and others) are advocating the same idea :)
Jan and me proposed some possible solutions recently:

        http://lkml.org/lkml/2009/9/14/126

> If we take a leaf from XFS's book by doing work at IO completion
> rather than submission we can keep a count of the number of pages
> cleaned on the bdi. This can be used to implement a FIFO-like
> throttle. If we add a simple ticket system to the bdi, when a
> process needs to be throttled it can submit a ticket with the number
> of pages it has dirtied to the bdi, and the bdi can then decide what
> the page cleaned count needs to reach before the process can be
> woken.

Exactly.

> i.e. Take the following ascii art showing the bdi fllusher
> thread running and issuing IO in the background:
> 
> bdi write thread:   +---X------Y---+-A-----ZB----+------C--------+
> 1st proc:		o............o
> 2nd proc:		       o............o
> 3rd proc:		                   o............o
> 
> When the 1st process comes in to be throttled, it samples the page
> clean count and gets X. It submits a ticket to be woken at A (X +
> some number of pages). If the flusher thread is not running, it gets
> kicked.  Process 2 and 3 do the same at Y and Z to be woken at B and
> C. At IO completion, the number of pages cleaned is counted and the
> tickets that are now under the clean count are pulled from the queue
> and the processes that own them are woken.
> 
> This avoids the thundering herd problem and applies throttling in
> a deterministic, predictable fashion. And by relying on background
> writeback, we only have one writeback path to optimise, not two
> different paths that interchange unpredictably.
> 
> In essence, this mechanism replaces the complex path of IO
> submission and congestion with a simple, deterministic counter and
> queue system that probably doesn't even require any memory
> allocation to implement. I think the simpler a thorttle mechanism
> is the more likely it is to work effectively....
> 
> I know that words without code aren't going to convince anyone, but I
> hope I've given you some food for thought about alternatives to what
> we currently do. ;)

Not at all, it's clear enough - your idea is very similar to Jan's
proposal. And I'd like to present an ascii art for another scheme:


                                          \                /
                  one bdi sync thread      \............../
               =========================>   \............/
                 working on dirty pages      \........../
                                              \......../
                                               \....../
                                                \..../
                                                 \../

  throttled    |  |    |  |    |  |     |  |     |  |     wakeup when enough
============>  |  |    |  |    |  |     |  |     |::| ==> pages are put to io
  task list    +--+    +--+    +--+     +--+     +--+     on behalf of this task
              task 5  task 4  task 3   task 2   task 1

               [.] dirty page  [:] writeback page

One benefit of this scheme is, when necessary, the task list could
be converted to some tree to do priorities and thus IO controller for
buffered writes :)

Thanks,
Fengguang
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: regression in page writeback

[Dave Chinner]

On Fri, Sep 25, 2009 at 02:45:03PM +0800, Wu Fengguang wrote:
> On Fri, Sep 25, 2009 at 01:04:13PM +0800, Dave Chinner wrote:
> > On Thu, Sep 24, 2009 at 08:38:20PM -0400, Chris Mason wrote:
> > > On Fri, Sep 25, 2009 at 10:11:17AM +1000, Dave Chinner wrote:
> > > > On Thu, Sep 24, 2009 at 11:15:08AM +0800, Wu Fengguang wrote:
> > > > > On Wed, Sep 23, 2009 at 10:00:58PM +0800, Chris Mason wrote:
> > > > > > The only place that actually honors the congestion flag is pdflush.
> > > > > > It's trivial to get pdflush backed up and make it sit down without
> > > > > > making any progress because once the queue congests, pdflush goes away.
> > > > > 
> > > > > Right. I guess that's more or less intentional - to give lowest priority
> > > > > to periodic/background writeback.
> > > > 
> > > > IMO, this is the wrong design. Background writeback should
> > > > have higher CPU/scheduler priority than normal tasks. If there is
> > > > sufficient dirty pages in the system for background writeback to
> > > > be active, it should be running *now* to start as much IO as it can
> > > > without being held up by other, lower priority tasks.
> > > 
> > > I'd say that an fsync from mutt or vi should be done at a higher prio
> > > than a background streaming writer.
> > 
> > I don't think you caught everything I said - synchronous IO is
> > un-throttled.
> 
> O_SYNC writes may be un-throttled in theory, however it seems to be
> throttled in practice:
> 
>   generic_file_aio_write
>     __generic_file_aio_write
>       generic_file_buffered_write
>         generic_perform_write
>           balance_dirty_pages_ratelimited
>     generic_write_sync
> 
> Do you mean some other code path?

In the context of the setup I was talking about, I meant is that sync
IO _should_ be unthrottled because it is self-throttling by it's
very nature. The current code makes no differentiation between the
two.

> > Background writeback should dump async IO to the elevator as fast as
> > it can, then get the hell out of the way. If you've got a UP system,
> > then the fsync can't be issued at the same time pdflush is running
> > (same as right now), and if you've got a MP system then fsync can
> > run at the same time.
> 
> I think you are right for system wide sync.
> 
> System wide sync seems to always wait for the queued bdi writeback
> works to finish, which should be fine in terms of efficiency, except
> that sync could end up do more works and even live lock.
> 
> > On the premise that sync IO is unthrottled and given that elevators
> > queue and issue sync IO sperately to async writes, fsync latency
> > would be entirely derived from the elevator queuing behaviour, not
> > the CPU priority of pdflush.
> 
> It's not exactly CPU priority, but queue fullness priority.

That's exactly what I implied. The elevator manages the
queue fullness and when it decides when to block background or
foreground writes. The problem is, the elevator can't make a sane
scheduling decision because it can't tell the difference between
async and sync IO because we don't propagate that information to
THE Block layer from the VFS.

We have all the smarts in the block layer interface to distinguish
between sync and async IO and the elevators do smart stuff with this
information. But by throwing away that information at the VFS level,
we hamstring the elevator scheduler because it never sees any
"synchronous" write IO for data writes. Hence any synchronous data
write gets stuck in the same queue with all the background stuff
and doesn't get priority.

Hence right now if you issue an fsync or pageout, it's a crap shoot
as to whether the elevator will schedule it first or last behind
other IO. The fact that they then ignore congestion is relying on a
side effect to stop background writeback and allow the fsync to
monopolise the elevator. It is not predictable and hence IO patterns
under load will change all the time regardless of whether the system
is in a steady state or not.

IMO there are architectural failings from top to bottom in the
writeback stack - while people are interested in fixing stuff, I
figured that they should be pointed out to give y'all something to
think about...

> fsync operations always use nonblocking=0, so in fact they _used to_
> enjoy better priority than pdflush. Same is vmscan pageout, which
> calls writepage directly. Both won't back off on congested bdi.
> 
> So when there comes fsync/pageout, they will always be served first.

pageout is so horribly inefficient from an IO perspective it is not
funny. It is one of the reasons Linux sucks so much when under
memory pressure. It basically causes the system to do random 4k
writeback of dirty pages (and lumpy reclaim can make it
synchronous!). 

pageout needs an enema, and preferably it should defer to background
writeback to clean pages. background writeback will clean pages
much, much faster than the random crap that pageout spews at the
disk right now.

Given that I can basically lock up my 2.6.30-based laptop for 10-15
minutes at a time with the disk running flat out in low memory
situations simply by starting to copy a large file(*), I think that
the way we currently handle dirty page writeback needs a bit of a
rethink.

(*) I had this happen 4-5 times last week moving VM images around on
my laptop, and it involved the Linux VM switching between pageout
and swapping to make more memory available while the copy was was
hammering the same drive with dirty pages from foreground writeback.
It made for extremely fragmented files when the machine finally
recovered because of the non-sequential writeback patterns on the
single file being copied.  You can't tell me that this is sane,
desirable behaviour, and this is the sort of problem that I want
sorted out. I don't beleive it can be fixed by maintaining the
number of uncoordinated, competing writeback mechanisms we currently
have.

> Small random IOs may hurt a bit though.

They *always* hurt, and under load, that appears to be the common IO
pattern that Linux is generating....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: regression in page writeback

[Wu Fengguang]

On Mon, Sep 28, 2009 at 09:07:00AM +0800, Dave Chinner wrote:
> On Fri, Sep 25, 2009 at 02:45:03PM +0800, Wu Fengguang wrote:
> > On Fri, Sep 25, 2009 at 01:04:13PM +0800, Dave Chinner wrote:
> > > On Thu, Sep 24, 2009 at 08:38:20PM -0400, Chris Mason wrote:
> > > > On Fri, Sep 25, 2009 at 10:11:17AM +1000, Dave Chinner wrote:
> > > > > On Thu, Sep 24, 2009 at 11:15:08AM +0800, Wu Fengguang wrote:
> > > > > > On Wed, Sep 23, 2009 at 10:00:58PM +0800, Chris Mason wrote:
> > > > > > > The only place that actually honors the congestion flag is pdflush.
> > > > > > > It's trivial to get pdflush backed up and make it sit down without
> > > > > > > making any progress because once the queue congests, pdflush goes away.
> > > > > > 
> > > > > > Right. I guess that's more or less intentional - to give lowest priority
> > > > > > to periodic/background writeback.
> > > > > 
> > > > > IMO, this is the wrong design. Background writeback should
> > > > > have higher CPU/scheduler priority than normal tasks. If there is
> > > > > sufficient dirty pages in the system for background writeback to
> > > > > be active, it should be running *now* to start as much IO as it can
> > > > > without being held up by other, lower priority tasks.
> > > > 
> > > > I'd say that an fsync from mutt or vi should be done at a higher prio
> > > > than a background streaming writer.
> > > 
> > > I don't think you caught everything I said - synchronous IO is
> > > un-throttled.
> > 
> > O_SYNC writes may be un-throttled in theory, however it seems to be
> > throttled in practice:
> > 
> >   generic_file_aio_write
> >     __generic_file_aio_write
> >       generic_file_buffered_write
> >         generic_perform_write
> >           balance_dirty_pages_ratelimited
> >     generic_write_sync
> > 
> > Do you mean some other code path?
> 
> In the context of the setup I was talking about, I meant is that sync
> IO _should_ be unthrottled because it is self-throttling by it's
> very nature. The current code makes no differentiation between the
> two.

Yes, O_SYNC writers are double throttled now..

> > > Background writeback should dump async IO to the elevator as fast as
> > > it can, then get the hell out of the way. If you've got a UP system,
> > > then the fsync can't be issued at the same time pdflush is running
> > > (same as right now), and if you've got a MP system then fsync can
> > > run at the same time.
> > 
> > I think you are right for system wide sync.
> > 
> > System wide sync seems to always wait for the queued bdi writeback
> > works to finish, which should be fine in terms of efficiency, except
> > that sync could end up do more works and even live lock.
> > 
> > > On the premise that sync IO is unthrottled and given that elevators
> > > queue and issue sync IO sperately to async writes, fsync latency
> > > would be entirely derived from the elevator queuing behaviour, not
> > > the CPU priority of pdflush.
> > 
> > It's not exactly CPU priority, but queue fullness priority.
> 
> That's exactly what I implied. The elevator manages the
> queue fullness and when it decides when to block background or
> foreground writes. The problem is, the elevator can't make a sane
> scheduling decision because it can't tell the difference between
> async and sync IO because we don't propagate that information to
> THE Block layer from the VFS.
> 
> We have all the smarts in the block layer interface to distinguish
> between sync and async IO and the elevators do smart stuff with this
> information. But by throwing away that information at the VFS level,
> we hamstring the elevator scheduler because it never sees any
> "synchronous" write IO for data writes. Hence any synchronous data
> write gets stuck in the same queue with all the background stuff
> and doesn't get priority.

Yes this is a problem. We may also need to add priority awareness to
get_request_wait() to get a complete solution.

> Hence right now if you issue an fsync or pageout, it's a crap shoot
> as to whether the elevator will schedule it first or last behind
> other IO. The fact that they then ignore congestion is relying on a
> side effect to stop background writeback and allow the fsync to
> monopolise the elevator. It is not predictable and hence IO patterns
> under load will change all the time regardless of whether the system
> is in a steady state or not.
>
> IMO there are architectural failings from top to bottom in the
> writeback stack - while people are interested in fixing stuff, I
> figured that they should be pointed out to give y'all something to
> think about...

Thanks, your information helps a lot.

> > fsync operations always use nonblocking=0, so in fact they _used to_
> > enjoy better priority than pdflush. Same is vmscan pageout, which
> > calls writepage directly. Both won't back off on congested bdi.
> > 
> > So when there comes fsync/pageout, they will always be served first.
> 
> pageout is so horribly inefficient from an IO perspective it is not
> funny. It is one of the reasons Linux sucks so much when under
> memory pressure. It basically causes the system to do random 4k
> writeback of dirty pages (and lumpy reclaim can make it
> synchronous!). 
> 
> pageout needs an enema, and preferably it should defer to background
> writeback to clean pages. background writeback will clean pages
> much, much faster than the random crap that pageout spews at the
> disk right now.
> 
> Given that I can basically lock up my 2.6.30-based laptop for 10-15
> minutes at a time with the disk running flat out in low memory
> situations simply by starting to copy a large file(*), I think that
> the way we currently handle dirty page writeback needs a bit of a
> rethink.
> 
> (*) I had this happen 4-5 times last week moving VM images around on
> my laptop, and it involved the Linux VM switching between pageout
> and swapping to make more memory available while the copy was was
> hammering the same drive with dirty pages from foreground writeback.
> It made for extremely fragmented files when the machine finally
> recovered because of the non-sequential writeback patterns on the
> single file being copied.  You can't tell me that this is sane,
> desirable behaviour, and this is the sort of problem that I want
> sorted out. I don't beleive it can be fixed by maintaining the
> number of uncoordinated, competing writeback mechanisms we currently
> have.

I imagined some lumpy pageout policy would help, but didn't realize
it's such a severe problem that can happen in daily desktop workload..

Below is a quick patch. Any comments?

> > Small random IOs may hurt a bit though.
> 
> They *always* hurt, and under load, that appears to be the common IO
> pattern that Linux is generating....

Thanks,
Fengguang
---

vmscan: lumpy pageout

Signed-off-by: Wu Fengguang <fengguang.wu@intel.com>
---
 mm/vmscan.c |   72 +++++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 63 insertions(+), 9 deletions(-)

--- linux.orig/mm/vmscan.c	2009-09-28 11:45:48.000000000 +0800
+++ linux/mm/vmscan.c	2009-09-28 14:45:19.000000000 +0800
@@ -344,6 +344,64 @@ typedef enum {
 	PAGE_CLEAN,
 } pageout_t;
 
+#define LUMPY_PAGEOUT_PAGES	(512 * 1024 / PAGE_CACHE_SIZE)
+
+static pageout_t try_lumpy_pageout(struct page *page,
+				   struct address_space *mapping,
+				   struct writeback_control *wbc)
+{
+	struct page *pages[PAGEVEC_SIZE];
+	pgoff_t start;
+	int total;
+	int count;
+	int i;
+	int err;
+	int res = 0;
+
+	page_cache_get(page);
+	pages[0] = page;
+	i = 0;
+	count = 1;
+	start = page->index + 1;
+
+	for (total = LUMPY_PAGEOUT_PAGES; total > 0; total--) {
+		if (i >= count) {
+			i = 0;
+			count = find_get_pages(mapping, start,
+					       min(total, PAGEVEC_SIZE), pages);
+			if (!count)
+				break;
+
+			/* continuous? */
+			if (start + count - 1 != pages[count - 1]->index)
+				break;
+
+			start += count;
+		}
+
+		page = pages[i];
+		if (!PageDirty(page))
+			break;
+		if (!PageActive(page))
+			SetPageReclaim(page);
+		err = mapping->a_ops->writepage(page, wbc);
+		if (err < 0)
+			handle_write_error(mapping, page, res);
+		if (err == AOP_WRITEPAGE_ACTIVATE) {
+			ClearPageReclaim(page);
+			res = PAGE_ACTIVATE;
+			break;
+		}
+		page_cache_release(page);
+		i++;
+	}
+
+	for (; i < count; i++)
+		page_cache_release(pages[i]);
+
+	return res;
+}
+
 /*
  * pageout is called by shrink_page_list() for each dirty page.
  * Calls ->writepage().
@@ -392,21 +450,17 @@ static pageout_t pageout(struct page *pa
 		int res;
 		struct writeback_control wbc = {
 			.sync_mode = WB_SYNC_NONE,
-			.nr_to_write = SWAP_CLUSTER_MAX,
+			.nr_to_write = LUMPY_PAGEOUT_PAGES,
 			.range_start = 0,
 			.range_end = LLONG_MAX,
 			.nonblocking = 1,
 			.for_reclaim = 1,
 		};
 
-		SetPageReclaim(page);
-		res = mapping->a_ops->writepage(page, &wbc);
-		if (res < 0)
-			handle_write_error(mapping, page, res);
-		if (res == AOP_WRITEPAGE_ACTIVATE) {
-			ClearPageReclaim(page);
-			return PAGE_ACTIVATE;
-		}
+
+		res = try_lumpy_pageout(page, mapping, &wbc);
+		if (res == PAGE_ACTIVATE)
+			return res;
 
 		/*
 		 * Wait on writeback if requested to. This happens when

Re: regression in page writeback

[Christoph Hellwig]

On Mon, Sep 28, 2009 at 03:15:07PM +0800, Wu Fengguang wrote:
> +		if (!PageActive(page))
> +			SetPageReclaim(page);
> +		err = mapping->a_ops->writepage(page, wbc);
> +		if (err < 0)
> +			handle_write_error(mapping, page, res);
> +		if (err == AOP_WRITEPAGE_ACTIVATE) {
> +			ClearPageReclaim(page);
> +			res = PAGE_ACTIVATE;
> +			break;
> +		}

This should help a bit for XFS as it historically does multi-page
writeouts from ->writepages (and apprently btrfs that added some
write-around recently?) but not those brave filesystems only
implementing the multi-page writeout from writepages as designed.

But really, the best would be to leave the writeout to the flusher
threads and just reclaim the clean pages from the VM.

Re: regression in page writeback

[Theodore Tso]

On Mon, Sep 28, 2009 at 09:08:04AM -0400, Christoph Hellwig wrote:
> This should help a bit for XFS as it historically does multi-page
> writeouts from ->writepages (and apprently btrfs that added some
> write-around recently?) but not those brave filesystems only
> implementing the multi-page writeout from writepages as designed.

Here's the hack which I'm currently working on to work around the
writeback code limiting writebacks to 1024 pages.  I'm assuming this
is going to be short-term hack assuming the writeback code gets more
intelligent, but I thought I would throw this into the mix....

	     	   	     	   	      - Ted

ext4: Adjust ext4_da_writepages() to write out larger contiguous chunks

Work around problems in the writeback code to force out writebacks in
larger chunks than just 4mb, which is just too small.  This also works
around limitations in the ext4 block allocator, which can't allocate
more than 2048 blocks at a time.  So we need to defeat the round-robin
characteristics of the writeback code and try to write out as many
blocks in one inode before allowing the writeback code to move on to
another inode.  We add a a new per-filesystem tunable,
max_contig_writeback_mb, which caps this to a default of 128mb per
inode.

Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
---
 fs/ext4/ext4.h              |    1 +
 fs/ext4/inode.c             |  100 +++++++++++++++++++++++++++++++++++++-----
 fs/ext4/super.c             |    3 +
 include/trace/events/ext4.h |   14 ++++--
 4 files changed, 102 insertions(+), 16 deletions(-)

diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h
index e227eea..9f99427 100644
--- a/fs/ext4/ext4.h
+++ b/fs/ext4/ext4.h
@@ -942,6 +942,7 @@ struct ext4_sb_info {
 	unsigned int s_mb_stats;
 	unsigned int s_mb_order2_reqs;
 	unsigned int s_mb_group_prealloc;
+	unsigned int s_max_contig_writeback_mb;
 	/* where last allocation was done - for stream allocation */
 	unsigned long s_mb_last_group;
 	unsigned long s_mb_last_start;
diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
index 5fb72a9..9e0acb7 100644
--- a/fs/ext4/inode.c
+++ b/fs/ext4/inode.c
@@ -1145,6 +1145,60 @@ static int check_block_validity(struct inode *inode, const char *msg,
 }
 
 /*
+ * Return the number of dirty pages in the given inode starting at
+ * page frame idx.
+ */
+static pgoff_t ext4_num_dirty_pages(struct inode *inode, pgoff_t idx)
+{
+	struct address_space *mapping = inode->i_mapping;
+	pgoff_t	index;
+	struct pagevec pvec;
+	pgoff_t num = 0;
+	int i, nr_pages, done = 0;
+
+	pagevec_init(&pvec, 0);
+
+	while (!done) {
+		index = idx;
+		nr_pages = pagevec_lookup_tag(&pvec, mapping, &index,
+					      PAGECACHE_TAG_DIRTY,
+					      (pgoff_t)PAGEVEC_SIZE);
+		if (nr_pages == 0)
+			break;
+		for (i = 0; i < nr_pages; i++) {
+			struct page *page = pvec.pages[i];
+			struct buffer_head *bh, *head;
+
+			lock_page(page);
+			if (unlikely(page->mapping != mapping) ||
+			    !PageDirty(page) ||
+			    PageWriteback(page) ||
+			    page->index != idx) {
+				done = 1;
+				unlock_page(page);
+				break;
+			}
+			head = page_buffers(page);
+			bh = head;
+			do {
+				if (!buffer_delay(bh) &&
+				    !buffer_unwritten(bh)) {
+					done = 1;
+					break;
+				}
+			} while ((bh = bh->b_this_page) != head);
+			unlock_page(page);
+			if (done)
+				break;
+			idx++;
+			num++;
+		}
+		pagevec_release(&pvec);
+	}
+	return num;
+}
+
+/*
  * The ext4_get_blocks() function tries to look up the requested blocks,
  * and returns if the blocks are already mapped.
  *
@@ -2744,7 +2798,8 @@ static int ext4_da_writepages(struct address_space *mapping,
 	int pages_written = 0;
 	long pages_skipped;
 	int range_cyclic, cycled = 1, io_done = 0;
-	int needed_blocks, ret = 0, nr_to_writebump = 0;
+	int needed_blocks, ret = 0;
+	long desired_nr_to_write, nr_to_writebump = 0;
 	loff_t range_start = wbc->range_start;
 	struct ext4_sb_info *sbi = EXT4_SB(mapping->host->i_sb);
 
@@ -2771,16 +2826,6 @@ static int ext4_da_writepages(struct address_space *mapping,
 	if (unlikely(sbi->s_mount_flags & EXT4_MF_FS_ABORTED))
 		return -EROFS;
 
-	/*
-	 * Make sure nr_to_write is >= sbi->s_mb_stream_request
-	 * This make sure small files blocks are allocated in
-	 * single attempt. This ensure that small files
-	 * get less fragmented.
-	 */
-	if (wbc->nr_to_write < sbi->s_mb_stream_request) {
-		nr_to_writebump = sbi->s_mb_stream_request - wbc->nr_to_write;
-		wbc->nr_to_write = sbi->s_mb_stream_request;
-	}
 	if (wbc->range_start == 0 && wbc->range_end == LLONG_MAX)
 		range_whole = 1;
 
@@ -2795,6 +2840,36 @@ static int ext4_da_writepages(struct address_space *mapping,
 	} else
 		index = wbc->range_start >> PAGE_CACHE_SHIFT;
 
+	/*
+	 * This works around two forms of stupidity.  The first is in
+	 * the writeback code, which caps the maximum number of pages
+	 * written to be 1024 pages.  This is wrong on multiple
+	 * levels; different architectues have a different page size,
+	 * which changes the maximum amount of data which gets
+	 * written.  Secondly, 4 megabytes is way too small.  XFS
+	 * forces this value to be 16 megabytes by multiplying
+	 * nr_to_write parameter by four, and then relies on its
+	 * allocator to allocate larger extents to make them
+	 * contiguous.  Unfortunately this brings us to the second
+	 * stupidity, which is that ext4's mballoc code only allocates
+	 * at most 2048 blocks.  So we force contiguous writes up to
+	 * the number of dirty blocks in the inode, or
+	 * sbi->max_contig_writeback_mb whichever is smaller.
+	 */
+	if (!range_cyclic && range_whole)
+		desired_nr_to_write = wbc->nr_to_write * 8;
+	else
+		desired_nr_to_write = ext4_num_dirty_pages(inode, index);
+	if (desired_nr_to_write > (sbi->s_max_contig_writeback_mb << 
+				   (20 - PAGE_CACHE_SHIFT)))
+		desired_nr_to_write = (sbi->s_max_contig_writeback_mb << 
+				       (20 - PAGE_CACHE_SHIFT));
+
+	if (wbc->nr_to_write < desired_nr_to_write) {
+		nr_to_writebump = desired_nr_to_write - wbc->nr_to_write;
+		wbc->nr_to_write = desired_nr_to_write;
+	}
+
 	mpd.wbc = wbc;
 	mpd.inode = mapping->host;
 
@@ -2914,7 +2989,8 @@ retry:
 out_writepages:
 	if (!no_nrwrite_index_update)
 		wbc->no_nrwrite_index_update = 0;
-	wbc->nr_to_write -= nr_to_writebump;
+	if (wbc->nr_to_write > nr_to_writebump)
+		wbc->nr_to_write -= nr_to_writebump;
 	wbc->range_start = range_start;
 	trace_ext4_da_writepages_result(inode, wbc, ret, pages_written);
 	return ret;
diff --git a/fs/ext4/super.c b/fs/ext4/super.c
index df539ba..9d04bd9 100644
--- a/fs/ext4/super.c
+++ b/fs/ext4/super.c
@@ -2197,6 +2197,7 @@ EXT4_RW_ATTR_SBI_UI(mb_min_to_scan, s_mb_min_to_scan);
 EXT4_RW_ATTR_SBI_UI(mb_order2_req, s_mb_order2_reqs);
 EXT4_RW_ATTR_SBI_UI(mb_stream_req, s_mb_stream_request);
 EXT4_RW_ATTR_SBI_UI(mb_group_prealloc, s_mb_group_prealloc);
+EXT4_RW_ATTR_SBI_UI(max_contig_writeback, s_max_contig_writeback_mb);
 
 static struct attribute *ext4_attrs[] = {
 	ATTR_LIST(delayed_allocation_blocks),
@@ -2210,6 +2211,7 @@ static struct attribute *ext4_attrs[] = {
 	ATTR_LIST(mb_order2_req),
 	ATTR_LIST(mb_stream_req),
 	ATTR_LIST(mb_group_prealloc),
+	ATTR_LIST(max_contig_writeback),
 	NULL,
 };
 
@@ -2679,6 +2681,7 @@ static int ext4_fill_super(struct super_block *sb, void *data, int silent)
 	}
 
 	sbi->s_stripe = ext4_get_stripe_size(sbi);
+	sbi->s_max_contig_writeback_mb = 128;
 
 	/*
 	 * set up enough so that it can read an inode
diff --git a/include/trace/events/ext4.h b/include/trace/events/ext4.h
index c1bd8f1..7c6bbb7 100644
--- a/include/trace/events/ext4.h
+++ b/include/trace/events/ext4.h
@@ -236,6 +236,7 @@ TRACE_EVENT(ext4_da_writepages,
 		__field(	char,	for_kupdate		)
 		__field(	char,	for_reclaim		)
 		__field(	char,	range_cyclic		)
+		__field(       pgoff_t,	writeback_index		)
 	),
 
 	TP_fast_assign(
@@ -249,15 +250,17 @@ TRACE_EVENT(ext4_da_writepages,
 		__entry->for_kupdate	= wbc->for_kupdate;
 		__entry->for_reclaim	= wbc->for_reclaim;
 		__entry->range_cyclic	= wbc->range_cyclic;
+		__entry->writeback_index = inode->i_mapping->writeback_index;
 	),
 
-	TP_printk("dev %s ino %lu nr_to_write %ld pages_skipped %ld range_start %llu range_end %llu nonblocking %d for_kupdate %d for_reclaim %d range_cyclic %d",
+	TP_printk("dev %s ino %lu nr_to_write %ld pages_skipped %ld range_start %llu range_end %llu nonblocking %d for_kupdate %d for_reclaim %d range_cyclic %d writeback_index %lu",
 		  jbd2_dev_to_name(__entry->dev),
 		  (unsigned long) __entry->ino, __entry->nr_to_write,
 		  __entry->pages_skipped, __entry->range_start,
 		  __entry->range_end, __entry->nonblocking,
 		  __entry->for_kupdate, __entry->for_reclaim,
-		  __entry->range_cyclic)
+		  __entry->range_cyclic,
+		  (unsigned long) __entry->writeback_index)
 );
 
 TRACE_EVENT(ext4_da_write_pages,
@@ -309,6 +312,7 @@ TRACE_EVENT(ext4_da_writepages_result,
 		__field(	char,	encountered_congestion	)
 		__field(	char,	more_io			)	
 		__field(	char,	no_nrwrite_index_update )
+		__field(       pgoff_t,	writeback_index		)
 	),
 
 	TP_fast_assign(
@@ -320,14 +324,16 @@ TRACE_EVENT(ext4_da_writepages_result,
 		__entry->encountered_congestion	= wbc->encountered_congestion;
 		__entry->more_io	= wbc->more_io;
 		__entry->no_nrwrite_index_update = wbc->no_nrwrite_index_update;
+		__entry->writeback_index = inode->i_mapping->writeback_index;
 	),
 
-	TP_printk("dev %s ino %lu ret %d pages_written %d pages_skipped %ld congestion %d more_io %d no_nrwrite_index_update %d",
+	TP_printk("dev %s ino %lu ret %d pages_written %d pages_skipped %ld congestion %d more_io %d no_nrwrite_index_update %d writeback_index %lu",
 		  jbd2_dev_to_name(__entry->dev),
 		  (unsigned long) __entry->ino, __entry->ret,
 		  __entry->pages_written, __entry->pages_skipped,
 		  __entry->encountered_congestion, __entry->more_io,
-		  __entry->no_nrwrite_index_update)
+		  __entry->no_nrwrite_index_update,
+		  (unsigned long) __entry->writeback_index)
 );
 
 TRACE_EVENT(ext4_da_write_begin,

Re: regression in page writeback

[Chris Mason]

On Mon, Sep 28, 2009 at 11:07:00AM +1000, Dave Chinner wrote:
> On Fri, Sep 25, 2009 at 02:45:03PM +0800, Wu Fengguang wrote:
> > On Fri, Sep 25, 2009 at 01:04:13PM +0800, Dave Chinner wrote:
> > > On Thu, Sep 24, 2009 at 08:38:20PM -0400, Chris Mason wrote:
> > > > On Fri, Sep 25, 2009 at 10:11:17AM +1000, Dave Chinner wrote:
> > > > > On Thu, Sep 24, 2009 at 11:15:08AM +0800, Wu Fengguang wrote:
> > > > > > On Wed, Sep 23, 2009 at 10:00:58PM +0800, Chris Mason wrote:
> > > > > > > The only place that actually honors the congestion flag is pdflush.
> > > > > > > It's trivial to get pdflush backed up and make it sit down without
> > > > > > > making any progress because once the queue congests, pdflush goes away.
> > > > > > 
> > > > > > Right. I guess that's more or less intentional - to give lowest priority
> > > > > > to periodic/background writeback.
> > > > > 
> > > > > IMO, this is the wrong design. Background writeback should
> > > > > have higher CPU/scheduler priority than normal tasks. If there is
> > > > > sufficient dirty pages in the system for background writeback to
> > > > > be active, it should be running *now* to start as much IO as it can
> > > > > without being held up by other, lower priority tasks.
> > > > 
> > > > I'd say that an fsync from mutt or vi should be done at a higher prio
> > > > than a background streaming writer.
> > > 
> > > I don't think you caught everything I said - synchronous IO is
> > > un-throttled.
> > 
> > O_SYNC writes may be un-throttled in theory, however it seems to be
> > throttled in practice:
> > 
> >   generic_file_aio_write
> >     __generic_file_aio_write
> >       generic_file_buffered_write
> >         generic_perform_write
> >           balance_dirty_pages_ratelimited
> >     generic_write_sync
> > 
> > Do you mean some other code path?
> 
> In the context of the setup I was talking about, I meant is that sync
> IO _should_ be unthrottled because it is self-throttling by it's
> very nature. The current code makes no differentiation between the
> two.

This isn't entirely true anymore.  WB_SYNC_ALL is turned into a sync
bio, which is sent down with higher priority.  There may be a few spots
that still need to be changed for it, but it is much better than it was.

re: pageout() being the worst way to do IO, definitely agree.

-chris

Re: regression in page writeback

[Wu Fengguang]

On Mon, Sep 28, 2009 at 03:15:07PM +0800, Wu Fengguang wrote:
> On Mon, Sep 28, 2009 at 09:07:00AM +0800, Dave Chinner wrote:
> > 
> > pageout is so horribly inefficient from an IO perspective it is not
> > funny. It is one of the reasons Linux sucks so much when under
> > memory pressure. It basically causes the system to do random 4k
> > writeback of dirty pages (and lumpy reclaim can make it
> > synchronous!). 
> > 
> > pageout needs an enema, and preferably it should defer to background
> > writeback to clean pages. background writeback will clean pages
> > much, much faster than the random crap that pageout spews at the
> > disk right now.
> > 
> > Given that I can basically lock up my 2.6.30-based laptop for 10-15
> > minutes at a time with the disk running flat out in low memory
> > situations simply by starting to copy a large file(*), I think that
> > the way we currently handle dirty page writeback needs a bit of a
> > rethink.
> > 
> > (*) I had this happen 4-5 times last week moving VM images around on
> > my laptop, and it involved the Linux VM switching between pageout
> > and swapping to make more memory available while the copy was was
> > hammering the same drive with dirty pages from foreground writeback.
> > It made for extremely fragmented files when the machine finally
> > recovered because of the non-sequential writeback patterns on the
> > single file being copied.  You can't tell me that this is sane,
> > desirable behaviour, and this is the sort of problem that I want
> > sorted out. I don't beleive it can be fixed by maintaining the
> > number of uncoordinated, competing writeback mechanisms we currently
> > have.
> 
> I imagined some lumpy pageout policy would help, but didn't realize
> it's such a severe problem that can happen in daily desktop workload..
> 
> Below is a quick patch. Any comments?

Wow, it's much easier to reuse write_cache_pages for lumpy pageout :)

---
 mm/page-writeback.c |   36 ++++++++++++++++++++++++------------
 mm/shmem.c          |    1 +
 mm/vmscan.c         |    6 ++++++
 3 files changed, 31 insertions(+), 12 deletions(-)

--- linux.orig/mm/vmscan.c	2009-09-29 07:21:51.000000000 +0800
+++ linux/mm/vmscan.c	2009-09-29 07:46:59.000000000 +0800
@@ -344,6 +344,8 @@ typedef enum {
 	PAGE_CLEAN,
 } pageout_t;
 
+#define LUMPY_PAGEOUT_PAGES	(512 * 1024 / PAGE_CACHE_SIZE)
+
 /*
  * pageout is called by shrink_page_list() for each dirty page.
  * Calls ->writepage().
@@ -408,6 +410,10 @@ static pageout_t pageout(struct page *pa
 			return PAGE_ACTIVATE;
 		}
 
+		wbc.range_start = (page->index + 1) << PAGE_CACHE_SHIFT;
+		wbc.nr_to_write = LUMPY_PAGEOUT_PAGES - 1;
+		generic_writepages(mapping, &wbc);
+
 		/*
 		 * Wait on writeback if requested to. This happens when
 		 * direct reclaiming a large contiguous area and the
--- linux.orig/mm/page-writeback.c	2009-09-29 07:33:13.000000000 +0800
+++ linux/mm/page-writeback.c	2009-09-29 08:10:39.000000000 +0800
@@ -799,6 +799,12 @@ retry:
 		if (nr_pages == 0)
 			break;
 
+		if (wbc->for_reclaim && done_index + nr_pages - 1 !=
+					pvec.pages[nr_pages - 1]->index) {
+			pagevec_release(&pvec);
+			break;
+		}
+
 		for (i = 0; i < nr_pages; i++) {
 			struct page *page = pvec.pages[i];
 
@@ -852,24 +858,30 @@ continue_unlock:
 			if (!clear_page_dirty_for_io(page))
 				goto continue_unlock;
 
+			/*
+			 * active and unevictable pages will be checked at
+			 * rotate time
+			 */
+			if (wbc->for_reclaim)
+				SetPageReclaim(page);
+
 			ret = (*writepage)(page, wbc, data);
 			if (unlikely(ret)) {
 				if (ret == AOP_WRITEPAGE_ACTIVATE) {
 					unlock_page(page);
 					ret = 0;
-				} else {
-					/*
-					 * done_index is set past this page,
-					 * so media errors will not choke
-					 * background writeout for the entire
-					 * file. This has consequences for
-					 * range_cyclic semantics (ie. it may
-					 * not be suitable for data integrity
-					 * writeout).
-					 */
-					done = 1;
-					break;
 				}
+				/*
+				 * done_index is set past this page,
+				 * so media errors will not choke
+				 * background writeout for the entire
+				 * file. This has consequences for
+				 * range_cyclic semantics (ie. it may
+				 * not be suitable for data integrity
+				 * writeout).
+				 */
+				done = 1;
+				break;
  			}
 
 			if (nr_to_write > 0) {
--- linux.orig/mm/shmem.c	2009-09-29 08:07:22.000000000 +0800
+++ linux/mm/shmem.c	2009-09-29 08:08:02.000000000 +0800
@@ -1103,6 +1103,7 @@ unlock:
 	 */
 	swapcache_free(swap, NULL);
 redirty:
+	wbc->pages_skipped++;
 	set_page_dirty(page);
 	if (wbc->for_reclaim)
 		return AOP_WRITEPAGE_ACTIVATE;	/* Return with page locked */

Re: regression in page writeback

[Wu Fengguang]

On Mon, Sep 28, 2009 at 09:08:04PM +0800, Christoph Hellwig wrote:
> On Mon, Sep 28, 2009 at 03:15:07PM +0800, Wu Fengguang wrote:
> > +		if (!PageActive(page))
> > +			SetPageReclaim(page);
> > +		err = mapping->a_ops->writepage(page, wbc);
> > +		if (err < 0)
> > +			handle_write_error(mapping, page, res);
> > +		if (err == AOP_WRITEPAGE_ACTIVATE) {
> > +			ClearPageReclaim(page);
> > +			res = PAGE_ACTIVATE;
> > +			break;
> > +		}
> 
> This should help a bit for XFS as it historically does multi-page
> writeouts from ->writepages (and apprently btrfs that added some

->writepage ?

> write-around recently?) but not those brave filesystems only
> implementing the multi-page writeout from writepages as designed.

Thanks.  Just tried write_cache_pages(), looks simple. Need to further
convert all aops->writepages to support lumpy pageout :)

> But really, the best would be to leave the writeout to the flusher
> threads and just reclaim the clean pages from the VM.

Yup, that's much larger behavior change, and could be pursued as a
long term goal.

Thanks,
Fengguang

Re: regression in page writeback

[Chris Mason]

On Tue, Sep 29, 2009 at 10:32:27AM +0800, Wu Fengguang wrote:
> On Mon, Sep 28, 2009 at 09:08:04PM +0800, Christoph Hellwig wrote:
> > On Mon, Sep 28, 2009 at 03:15:07PM +0800, Wu Fengguang wrote:
> > > +		if (!PageActive(page))
> > > +			SetPageReclaim(page);
> > > +		err = mapping->a_ops->writepage(page, wbc);
> > > +		if (err < 0)
> > > +			handle_write_error(mapping, page, res);
> > > +		if (err == AOP_WRITEPAGE_ACTIVATE) {
> > > +			ClearPageReclaim(page);
> > > +			res = PAGE_ACTIVATE;
> > > +			break;
> > > +		}
> > 
> > This should help a bit for XFS as it historically does multi-page
> > writeouts from ->writepages (and apprently btrfs that added some
> 
> ->writepage ?
> 
> > write-around recently?) but not those brave filesystems only
> > implementing the multi-page writeout from writepages as designed.
> 
> Thanks.  Just tried write_cache_pages(), looks simple. Need to further
> convert all aops->writepages to support lumpy pageout :)
> 
> > But really, the best would be to leave the writeout to the flusher
> > threads and just reclaim the clean pages from the VM.
> 
> Yup, that's much larger behavior change, and could be pursued as a
> long term goal.

I don't think we can just change kswapd to wait on flusher thread
progress because the flusher thread can happily spend forever writing
pages that can't actually be freed.  Lumpy pageout is a better middle
ground I think.

-chris

Re: regression in page writeback

[Christoph Hellwig]

On Tue, Sep 29, 2009 at 10:32:27AM +0800, Wu Fengguang wrote:
> > This should help a bit for XFS as it historically does multi-page
> > writeouts from ->writepages (and apprently btrfs that added some
> 
> ->writepage ?

Yes.

> > write-around recently?) but not those brave filesystems only
> > implementing the multi-page writeout from writepages as designed.
> 
> Thanks.  Just tried write_cache_pages(), looks simple. Need to further
> convert all aops->writepages to support lumpy pageout :)

Yeah.  Most ->writepages instances are just copies of write_cache_pages
with local hacks.  If you have any good idea to consolidate that that
would be great.  Also XFS and btrfs do cluster move pages even from
->writepage, I wonder how well that interacts with reclaim.

Re: regression in page writeback

[Dave Chinner]

On Mon, Sep 28, 2009 at 10:25:24AM -0400, Chris Mason wrote:
> On Mon, Sep 28, 2009 at 11:07:00AM +1000, Dave Chinner wrote:
> > In the context of the setup I was talking about, I meant is that sync
> > IO _should_ be unthrottled because it is self-throttling by it's
> > very nature. The current code makes no differentiation between the
> > two.
> 
> This isn't entirely true anymore.  WB_SYNC_ALL is turned into a sync
> bio, which is sent down with higher priority.  There may be a few spots
> that still need to be changed for it, but it is much better than it was.

Oh, I didn't realise that had changed - when did WRITE_SYNC_PLUG get
introduced? FWIW, I notice that __block_write_full_page(), gfs2, btrfs
and jdb use WRITE_SYNC_PLUG to implement this, but it appears that
filesystems that have their own writeback code (e.g. XFS) have not
been converted (gfs2 and btrfs being the exceptions).

Oh well, something else that needs tweaking in XFS...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: regression in page writeback

[Wu Fengguang]

On Wed, Sep 30, 2009 at 07:39:36AM +0800, Dave Chinner wrote:
> On Mon, Sep 28, 2009 at 10:25:24AM -0400, Chris Mason wrote:
> > On Mon, Sep 28, 2009 at 11:07:00AM +1000, Dave Chinner wrote:
> > > In the context of the setup I was talking about, I meant is that sync
> > > IO _should_ be unthrottled because it is self-throttling by it's
> > > very nature. The current code makes no differentiation between the
> > > two.
> > 
> > This isn't entirely true anymore.  WB_SYNC_ALL is turned into a sync
> > bio, which is sent down with higher priority.  There may be a few spots
> > that still need to be changed for it, but it is much better than it was.
> 
> Oh, I didn't realise that had changed - when did WRITE_SYNC_PLUG get
> introduced?

About 5 months before, when Linus complained something similar :)

        http://lkml.org/lkml/2009/4/6/114

Thanks,
Fengguang

> FWIW, I notice that __block_write_full_page(), gfs2, btrfs
> and jdb use WRITE_SYNC_PLUG to implement this, but it appears that
> filesystems that have their own writeback code (e.g. XFS) have not
> been converted (gfs2 and btrfs being the exceptions).
> 
> Oh well, something else that needs tweaking in XFS...
> 
> Cheers,
> 
> Dave.
> -- 
> Dave Chinner
> david@fromorbit.com

Re: regression in page writeback

[Wu Fengguang]

Hi Ted,

On Mon, Sep 28, 2009 at 10:07:57PM +0800, Theodore Ts'o wrote:
> On Mon, Sep 28, 2009 at 09:08:04AM -0400, Christoph Hellwig wrote:
> > This should help a bit for XFS as it historically does multi-page
> > writeouts from ->writepages (and apprently btrfs that added some
> > write-around recently?) but not those brave filesystems only
> > implementing the multi-page writeout from writepages as designed.
> 
> Here's the hack which I'm currently working on to work around the
> writeback code limiting writebacks to 1024 pages.  I'm assuming this
> is going to be short-term hack assuming the writeback code gets more
> intelligent, but I thought I would throw this into the mix....

> ext4: Adjust ext4_da_writepages() to write out larger contiguous chunks
> 
> Work around problems in the writeback code to force out writebacks in
> larger chunks than just 4mb, which is just too small.  This also works
> around limitations in the ext4 block allocator, which can't allocate
> more than 2048 blocks at a time.  So we need to defeat the round-robin
> characteristics of the writeback code and try to write out as many
> blocks in one inode before allowing the writeback code to move on to
> another inode.  We add a a new per-filesystem tunable,
> max_contig_writeback_mb, which caps this to a default of 128mb per
> inode.

It's good to increase MAX_WRITEBACK_PAGES, however I'm afraid
max_contig_writeback_mb may be a burden in future: either it is not
necessary, or a per-bdi counterpart must be introduced for all
filesystems.

And it's preferred to automatically handle slow devices well with the
increased chunk size, instead of adding another parameter.

I scratched up a patch to demo the ideas collected in recent discussions.
Can you check if it serves your needs? Thanks.

> Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
> ---
>  fs/ext4/ext4.h              |    1 +
>  fs/ext4/inode.c             |  100 +++++++++++++++++++++++++++++++++++++-----
>  fs/ext4/super.c             |    3 +
>  include/trace/events/ext4.h |   14 ++++--
>  4 files changed, 102 insertions(+), 16 deletions(-)
> 
> diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h
> index e227eea..9f99427 100644
> --- a/fs/ext4/ext4.h
> +++ b/fs/ext4/ext4.h
> @@ -942,6 +942,7 @@ struct ext4_sb_info {
>  	unsigned int s_mb_stats;
>  	unsigned int s_mb_order2_reqs;
>  	unsigned int s_mb_group_prealloc;
> +	unsigned int s_max_contig_writeback_mb;
>  	/* where last allocation was done - for stream allocation */
>  	unsigned long s_mb_last_group;
>  	unsigned long s_mb_last_start;
> diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
> index 5fb72a9..9e0acb7 100644
> --- a/fs/ext4/inode.c
> +++ b/fs/ext4/inode.c
> @@ -1145,6 +1145,60 @@ static int check_block_validity(struct inode *inode, const char *msg,
>  }
>  
>  /*
> + * Return the number of dirty pages in the given inode starting at
> + * page frame idx.
> + */
> +static pgoff_t ext4_num_dirty_pages(struct inode *inode, pgoff_t idx)
> +{

ext4_num_dirty_pages() may be improved to take a "max_pages" parameter
to avoid unnecessary work.

> +	struct address_space *mapping = inode->i_mapping;
> +	pgoff_t	index;
> +	struct pagevec pvec;
> +	pgoff_t num = 0;
> +	int i, nr_pages, done = 0;
> +
> +	pagevec_init(&pvec, 0);
> +
> +	while (!done) {
> +		index = idx;
> +		nr_pages = pagevec_lookup_tag(&pvec, mapping, &index,
> +					      PAGECACHE_TAG_DIRTY,
> +					      (pgoff_t)PAGEVEC_SIZE);
> +		if (nr_pages == 0)
> +			break;
> +		for (i = 0; i < nr_pages; i++) {
> +			struct page *page = pvec.pages[i];
> +			struct buffer_head *bh, *head;
> +
> +			lock_page(page);
> +			if (unlikely(page->mapping != mapping) ||
> +			    !PageDirty(page) ||
> +			    PageWriteback(page) ||
> +			    page->index != idx) {
> +				done = 1;
> +				unlock_page(page);
> +				break;
> +			}
> +			head = page_buffers(page);
> +			bh = head;
> +			do {
> +				if (!buffer_delay(bh) &&
> +				    !buffer_unwritten(bh)) {
> +					done = 1;
> +					break;
> +				}
> +			} while ((bh = bh->b_this_page) != head);
> +			unlock_page(page);

I guess a rough estimation will suffice, hehe.
There are no guarantee anyway.

> +			if (done)
> +				break;
> +			idx++;
> +			num++;
> +		}
> +		pagevec_release(&pvec);
> +	}
> +	return num;
> +}
> +
> +/*
>   * The ext4_get_blocks() function tries to look up the requested blocks,
>   * and returns if the blocks are already mapped.
>   *
> @@ -2744,7 +2798,8 @@ static int ext4_da_writepages(struct address_space *mapping,
>  	int pages_written = 0;
>  	long pages_skipped;
>  	int range_cyclic, cycled = 1, io_done = 0;
> -	int needed_blocks, ret = 0, nr_to_writebump = 0;
> +	int needed_blocks, ret = 0;
> +	long desired_nr_to_write, nr_to_writebump = 0;
>  	loff_t range_start = wbc->range_start;
>  	struct ext4_sb_info *sbi = EXT4_SB(mapping->host->i_sb);
>  
> @@ -2771,16 +2826,6 @@ static int ext4_da_writepages(struct address_space *mapping,
>  	if (unlikely(sbi->s_mount_flags & EXT4_MF_FS_ABORTED))
>  		return -EROFS;
>  
> -	/*
> -	 * Make sure nr_to_write is >= sbi->s_mb_stream_request
> -	 * This make sure small files blocks are allocated in
> -	 * single attempt. This ensure that small files
> -	 * get less fragmented.
> -	 */
> -	if (wbc->nr_to_write < sbi->s_mb_stream_request) {
> -		nr_to_writebump = sbi->s_mb_stream_request - wbc->nr_to_write;
> -		wbc->nr_to_write = sbi->s_mb_stream_request;
> -	}
>  	if (wbc->range_start == 0 && wbc->range_end == LLONG_MAX)
>  		range_whole = 1;
>  
> @@ -2795,6 +2840,36 @@ static int ext4_da_writepages(struct address_space *mapping,
>  	} else
>  		index = wbc->range_start >> PAGE_CACHE_SHIFT;
>  
> +	/*
> +	 * This works around two forms of stupidity.  The first is in
> +	 * the writeback code, which caps the maximum number of pages
> +	 * written to be 1024 pages.  This is wrong on multiple
> +	 * levels; different architectues have a different page size,
> +	 * which changes the maximum amount of data which gets

Good point.

> +	 * written.  Secondly, 4 megabytes is way too small.  XFS
> +	 * forces this value to be 16 megabytes by multiplying
> +	 * nr_to_write parameter by four, and then relies on its
> +	 * allocator to allocate larger extents to make them
> +	 * contiguous.  Unfortunately this brings us to the second

Hopefully we can make these hacks unnecessary :)

> +	 * stupidity, which is that ext4's mballoc code only allocates
> +	 * at most 2048 blocks.  So we force contiguous writes up to
> +	 * the number of dirty blocks in the inode, or
> +	 * sbi->max_contig_writeback_mb whichever is smaller.
> +	 */
> +	if (!range_cyclic && range_whole)
> +		desired_nr_to_write = wbc->nr_to_write * 8;
> +	else
> +		desired_nr_to_write = ext4_num_dirty_pages(inode, index);
> +	if (desired_nr_to_write > (sbi->s_max_contig_writeback_mb << 
> +				   (20 - PAGE_CACHE_SHIFT)))
> +		desired_nr_to_write = (sbi->s_max_contig_writeback_mb << 
> +				       (20 - PAGE_CACHE_SHIFT));
> +
> +	if (wbc->nr_to_write < desired_nr_to_write) {
> +		nr_to_writebump = desired_nr_to_write - wbc->nr_to_write;
> +		wbc->nr_to_write = desired_nr_to_write;
> +	}
> +
>  	mpd.wbc = wbc;
>  	mpd.inode = mapping->host;
>  
> @@ -2914,7 +2989,8 @@ retry:
>  out_writepages:
>  	if (!no_nrwrite_index_update)
>  		wbc->no_nrwrite_index_update = 0;
> -	wbc->nr_to_write -= nr_to_writebump;
> +	if (wbc->nr_to_write > nr_to_writebump)
> +		wbc->nr_to_write -= nr_to_writebump;
>  	wbc->range_start = range_start;
>  	trace_ext4_da_writepages_result(inode, wbc, ret, pages_written);
>  	return ret;

Thanks,
Fengguang
---
writeback: bump up writeback chunk size to 128MB

Adjust the writeback call stack to support larger writeback chunk size.

- make wbc.nr_to_write a per-file parameter
- init wbc.nr_to_write with MAX_WRITEBACK_PAGES=128MB
  (proposed by Ted)
- add wbc.nr_segments to limit seeks inside sparsely dirtied file
  (proposed by Chris)
- add wbc.timeout which will be used to control IO submission time
  either per-file or globally.
  
The wbc.nr_segments is now determined purely by logical page index
distance: if two pages are 1MB apart, it makes a new segment.

Filesystems could do this better with real extent knowledges.
One possible scheme is to record the previous page index in
wbc.writeback_index, and let ->writepage compare if the current and
previous pages lie in the same extent, and decrease wbc.nr_segments
accordingly. Care should taken to avoid double decreases in writepage
and write_cache_pages.

The wbc.timeout (when used per-file) is mainly a safeguard against slow
devices, which may take too long time to sync 128MB data.

The wbc.timeout (when used globally) could be useful when we decide to
do two sync scans on dirty pages and dirty metadata. XFS could say:
please return to sync dirty metadata after 10s. Would need another
b_io_metadata queue, but that's possible.

This work depends on the balance_dirty_pages() wait queue patch.

CC: Theodore Ts'o <tytso@mit.edu>
CC: Chris Mason <chris.mason@oracle.com>
CC: Dave Chinner <david@fromorbit.com> 
CC: Christoph Hellwig <hch@infradead.org>
CC: Jan Kara <jack@suse.cz> 
CC: Peter Zijlstra <a.p.zijlstra@chello.nl> 
CC: Jens Axboe <jens.axboe@oracle.com> 
Signed-off-by: Wu Fengguang <fengguang.wu@intel.com>
---
 fs/fs-writeback.c         |   60 ++++++++++++++++++++++--------------
 fs/jbd2/commit.c          |    1 
 include/linux/writeback.h |   15 +++++++--
 mm/backing-dev.c          |    2 -
 mm/filemap.c              |    1 
 mm/page-writeback.c       |   13 +++++++
 6 files changed, 67 insertions(+), 25 deletions(-)

--- linux.orig/fs/fs-writeback.c	2009-09-30 12:17:15.000000000 +0800
+++ linux/fs/fs-writeback.c	2009-09-30 12:17:17.000000000 +0800
@@ -31,6 +31,15 @@
 #define inode_to_bdi(inode)	((inode)->i_mapping->backing_dev_info)
 
 /*
+ * The maximum number of pages to writeout in a single bdi flush/kupdate
+ * operation.  We do this so we don't hold I_SYNC against an inode for
+ * enormous amounts of time, which would block a userspace task which has
+ * been forced to throttle against that inode.  Also, the code reevaluates
+ * the dirty each time it has written this many pages.
+ */
+#define MAX_WRITEBACK_PAGES     (128 * (20 - PAGE_CACHE_SHIFT))
+
+/*
  * We don't actually have pdflush, but this one is exported though /proc...
  */
 int nr_pdflush_threads;
@@ -540,6 +549,14 @@ writeback_single_inode(struct inode *ino
 
 	spin_unlock(&inode_lock);
 
+	if (wbc->for_kupdate || wbc->for_background) {
+		wbc->nr_segments = 1;	/* TODO: test blk_queue_nonrot() */
+		wbc->timeout = HZ;
+	} else {
+		wbc->nr_segments = LONG_MAX;
+		wbc->timeout = 0;
+	}
+
 	ret = do_writepages(mapping, wbc);
 
 	/* Don't write the inode if only I_DIRTY_PAGES was set */
@@ -564,7 +581,9 @@ writeback_single_inode(struct inode *ino
 			 * sometimes bales out without doing anything.
 			 */
 			inode->i_state |= I_DIRTY_PAGES;
-			if (wbc->nr_to_write <= 0) {
+			if (wbc->nr_to_write <= 0 ||
+			    wbc->nr_segments <= 0 ||
+			    wbc->timeout < 0) {
 				/*
 				 * slice used up: queue for next turn
 				 */
@@ -659,12 +678,17 @@ pinned:
 	return 0;
 }
 
-static void writeback_inodes_wb(struct bdi_writeback *wb,
+static long writeback_inodes_wb(struct bdi_writeback *wb,
 				struct writeback_control *wbc)
 {
 	struct super_block *sb = wbc->sb, *pin_sb = NULL;
 	const int is_blkdev_sb = sb_is_blkdev_sb(sb);
 	const unsigned long start = jiffies;	/* livelock avoidance */
+	unsigned long stop_time = 0;
+	long wrote = 0;
+
+	if (wbc->timeout)
+		stop_time = (start + wbc->timeout) | 1;
 
 	spin_lock(&inode_lock);
 
@@ -721,7 +745,9 @@ static void writeback_inodes_wb(struct b
 		BUG_ON(inode->i_state & (I_FREEING | I_CLEAR));
 		__iget(inode);
 		pages_skipped = wbc->pages_skipped;
+		wbc->nr_to_write = MAX_WRITEBACK_PAGES;
 		writeback_single_inode(inode, wbc);
+		wrote += MAX_WRITEBACK_PAGES - wbc->nr_to_write;
 		if (wbc->pages_skipped != pages_skipped) {
 			/*
 			 * writeback is not making progress due to locked
@@ -739,12 +765,15 @@ static void writeback_inodes_wb(struct b
 		}
 		if (!list_empty(&wb->b_more_io))
 			wbc->more_io = 1;
+		if (stop_time && time_after(jiffies, stop_time))
+			break;
 	}
 
 	unpin_sb_for_writeback(&pin_sb);
 
 	spin_unlock(&inode_lock);
 	/* Leave any unwritten inodes on b_io */
+	return wrote;
 }
 
 void writeback_inodes_wbc(struct writeback_control *wbc)
@@ -754,15 +783,6 @@ void writeback_inodes_wbc(struct writeba
 	writeback_inodes_wb(&bdi->wb, wbc);
 }
 
-/*
- * The maximum number of pages to writeout in a single bdi flush/kupdate
- * operation.  We do this so we don't hold I_SYNC against an inode for
- * enormous amounts of time, which would block a userspace task which has
- * been forced to throttle against that inode.  Also, the code reevaluates
- * the dirty each time it has written this many pages.
- */
-#define MAX_WRITEBACK_PAGES     1024
-
 static inline bool over_bground_thresh(void)
 {
 	unsigned long background_thresh, dirty_thresh;
@@ -797,10 +817,12 @@ static long wb_writeback(struct bdi_writ
 		.sync_mode		= args->sync_mode,
 		.older_than_this	= NULL,
 		.for_kupdate		= args->for_kupdate,
+		.for_background		= args->for_background,
 		.range_cyclic		= args->range_cyclic,
 	};
 	unsigned long oldest_jif;
 	long wrote = 0;
+	long nr;
 	struct inode *inode;
 
 	if (wbc.for_kupdate) {
@@ -834,26 +856,20 @@ static long wb_writeback(struct bdi_writ
 
 		wbc.more_io = 0;
 		wbc.encountered_congestion = 0;
-		wbc.nr_to_write = MAX_WRITEBACK_PAGES;
 		wbc.pages_skipped = 0;
-		writeback_inodes_wb(wb, &wbc);
-		args->nr_pages -= MAX_WRITEBACK_PAGES - wbc.nr_to_write;
-		wrote += MAX_WRITEBACK_PAGES - wbc.nr_to_write;
+		nr = writeback_inodes_wb(wb, &wbc);
+		args->nr_pages -= nr;
+		wrote += nr;
 
 		/*
-		 * If we consumed everything, see if we have more
-		 */
-		if (wbc.nr_to_write <= 0)
-			continue;
-		/*
-		 * Didn't write everything and we don't have more IO, bail
+		 * Bail if no more IO
 		 */
 		if (!wbc.more_io)
 			break;
 		/*
 		 * Did we write something? Try for more
 		 */
-		if (wbc.nr_to_write < MAX_WRITEBACK_PAGES)
+		if (nr)
 			continue;
 		/*
 		 * Nothing written. Wait for some inode to
--- linux.orig/include/linux/writeback.h	2009-09-30 12:13:00.000000000 +0800
+++ linux/include/linux/writeback.h	2009-09-30 12:17:17.000000000 +0800
@@ -32,10 +32,15 @@ struct writeback_control {
 	struct super_block *sb;		/* if !NULL, only write inodes from
 					   this super_block */
 	enum writeback_sync_modes sync_mode;
+	int timeout;
 	unsigned long *older_than_this;	/* If !NULL, only write back inodes
 					   older than this */
-	long nr_to_write;		/* Write this many pages, and decrement
-					   this for each page written */
+	long nr_to_write;		/* Max pages to write per file, and
+					   decrement this for each page written
+					 */
+	long nr_segments;		/* Max page segments to write per file,
+					   this is also a count down value
+					 */
 	long pages_skipped;		/* Pages which were not written */
 
 	/*
@@ -49,6 +54,7 @@ struct writeback_control {
 	unsigned nonblocking:1;		/* Don't get stuck on request queues */
 	unsigned encountered_congestion:1; /* An output: a queue is full */
 	unsigned for_kupdate:1;		/* A kupdate writeback */
+	unsigned for_background:1;	/* A background writeback */
 	unsigned for_reclaim:1;		/* Invoked from the page allocator */
 	unsigned range_cyclic:1;	/* range_start is cyclic */
 	unsigned stop_on_wrap:1;	/* stop when write index is to wrap */
@@ -65,6 +71,11 @@ struct writeback_control {
 };
 
 /*
+ * if two page ranges are more than 1MB apart, they are taken as two segments.
+ */
+#define WB_SEGMENT_DIST		(1024 >> (PAGE_CACHE_SHIFT - 10))
+
+/*
  * fs/fs-writeback.c
  */	
 struct bdi_writeback;
--- linux.orig/mm/filemap.c	2009-09-30 12:13:00.000000000 +0800
+++ linux/mm/filemap.c	2009-09-30 12:17:17.000000000 +0800
@@ -216,6 +216,7 @@ int __filemap_fdatawrite_range(struct ad
 	struct writeback_control wbc = {
 		.sync_mode = sync_mode,
 		.nr_to_write = LONG_MAX,
+		.nr_segments = LONG_MAX,
 		.range_start = start,
 		.range_end = end,
 	};
--- linux.orig/mm/page-writeback.c	2009-09-30 12:17:15.000000000 +0800
+++ linux/mm/page-writeback.c	2009-09-30 12:17:17.000000000 +0800
@@ -765,6 +765,7 @@ int write_cache_pages(struct address_spa
 	int cycled;
 	int range_whole = 0;
 	long nr_to_write = wbc->nr_to_write;
+	unsigned long start_time = jiffies;
 
 	pagevec_init(&pvec, 0);
 	if (wbc->range_cyclic) {
@@ -818,6 +819,12 @@ retry:
 				break;
 			}
 
+			if (done_index + WB_SEGMENT_DIST > page->index &&
+			    --wbc->nr_segments <= 0) {
+				done = 1;
+				break;
+			}
+
 			done_index = page->index + 1;
 
 			lock_page(page);
@@ -899,6 +906,12 @@ continue_unlock:
 		}
 		pagevec_release(&pvec);
 		cond_resched();
+		if (wbc->timeout &&
+		    time_after(jiffies, start_time + wbc->timeout)) {
+			wbc->timeout = -1;
+			done = 1;
+			break;
+		}
 	}
 	if (wbc->stop_on_wrap)
 		done_index = 0;
--- linux.orig/fs/jbd2/commit.c	2009-09-30 12:13:00.000000000 +0800
+++ linux/fs/jbd2/commit.c	2009-09-30 12:17:17.000000000 +0800
@@ -219,6 +219,7 @@ static int journal_submit_inode_data_buf
 	struct writeback_control wbc = {
 		.sync_mode =  WB_SYNC_ALL,
 		.nr_to_write = mapping->nrpages * 2,
+		.nr_segments = LONG_MAX,
 		.range_start = 0,
 		.range_end = i_size_read(mapping->host),
 	};
--- linux.orig/mm/backing-dev.c	2009-09-30 12:17:26.000000000 +0800
+++ linux/mm/backing-dev.c	2009-09-30 12:17:38.000000000 +0800
@@ -336,7 +336,7 @@ static void bdi_flush_io(struct backing_
 		.sync_mode		= WB_SYNC_NONE,
 		.older_than_this	= NULL,
 		.range_cyclic		= 1,
-		.nr_to_write		= 1024,
+		.timeout		= HZ,
 	};
 
 	writeback_inodes_wbc(&wbc);

Re: regression in page writeback

[Wu Fengguang]

On Wed, Sep 30, 2009 at 01:26:57PM +0800, Wu Fengguang wrote:

> +#define MAX_WRITEBACK_PAGES     (128 * (20 - PAGE_CACHE_SHIFT))

Sorry for the silly mistake!

---
writeback: bump up writeback chunk size to 128MB

Adjust the writeback call stack to support larger writeback chunk size.

- make wbc.nr_to_write a per-file parameter
- init wbc.nr_to_write with MAX_WRITEBACK_PAGES=128MB
  (proposed by Ted)
- add wbc.nr_segments to limit seeks inside sparsely dirtied file
  (proposed by Chris)
- add wbc.timeout which will be used to control IO submission time
  either per-file or globally.
  
The wbc.nr_segments is now determined purely by logical page index
distance: if two pages are 1MB apart, it makes a new segment.

Filesystems could do this better with real extent knowledges.
One possible scheme is to record the previous page index in
wbc.writeback_index, and let ->writepage compare if the current and
previous pages lie in the same extent, and decrease wbc.nr_segments
accordingly. Care should taken to avoid double decreases in writepage
and write_cache_pages.

The wbc.timeout (when used per-file) is mainly a safeguard against slow
devices, which may take too long time to sync 128MB data.

The wbc.timeout (when used globally) could be useful when we decide to
do two sync scans on dirty pages and dirty metadata. XFS could say:
please return to sync dirty metadata after 10s. Would need another
b_io_metadata queue, but that's possible.

This work depends on the balance_dirty_pages() wait queue patch.

CC: Theodore Ts'o <tytso@mit.edu>
CC: Chris Mason <chris.mason@oracle.com>
CC: Dave Chinner <david@fromorbit.com> 
CC: Christoph Hellwig <hch@infradead.org>
CC: Jan Kara <jack@suse.cz> 
CC: Peter Zijlstra <a.p.zijlstra@chello.nl> 
CC: Jens Axboe <jens.axboe@oracle.com> 
Signed-off-by: Wu Fengguang <fengguang.wu@intel.com>
---
 fs/fs-writeback.c         |   60 ++++++++++++++++++++++--------------
 fs/jbd2/commit.c          |    1 
 include/linux/writeback.h |   15 +++++++--
 mm/backing-dev.c          |    2 -
 mm/filemap.c              |    1 
 mm/page-writeback.c       |   13 +++++++
 6 files changed, 67 insertions(+), 25 deletions(-)

--- linux.orig/fs/fs-writeback.c	2009-09-30 12:17:15.000000000 +0800
+++ linux/fs/fs-writeback.c	2009-09-30 13:29:24.000000000 +0800
@@ -31,6 +31,15 @@
 #define inode_to_bdi(inode)	((inode)->i_mapping->backing_dev_info)
 
 /*
+ * The maximum number of pages to writeout in a single bdi flush/kupdate
+ * operation.  We do this so we don't hold I_SYNC against an inode for
+ * enormous amounts of time, which would block a userspace task which has
+ * been forced to throttle against that inode.  Also, the code reevaluates
+ * the dirty each time it has written this many pages.
+ */
+#define MAX_WRITEBACK_PAGES     (128 << (20 - PAGE_CACHE_SHIFT))
+
+/*
  * We don't actually have pdflush, but this one is exported though /proc...
  */
 int nr_pdflush_threads;
@@ -540,6 +549,14 @@ writeback_single_inode(struct inode *ino
 
 	spin_unlock(&inode_lock);
 
+	if (wbc->for_kupdate || wbc->for_background) {
+		wbc->nr_segments = 1;	/* TODO: test blk_queue_nonrot() */
+		wbc->timeout = HZ;
+	} else {
+		wbc->nr_segments = LONG_MAX;
+		wbc->timeout = 0;
+	}
+
 	ret = do_writepages(mapping, wbc);
 
 	/* Don't write the inode if only I_DIRTY_PAGES was set */
@@ -564,7 +581,9 @@ writeback_single_inode(struct inode *ino
 			 * sometimes bales out without doing anything.
 			 */
 			inode->i_state |= I_DIRTY_PAGES;
-			if (wbc->nr_to_write <= 0) {
+			if (wbc->nr_to_write <= 0 ||
+			    wbc->nr_segments <= 0 ||
+			    wbc->timeout < 0) {
 				/*
 				 * slice used up: queue for next turn
 				 */
@@ -659,12 +678,17 @@ pinned:
 	return 0;
 }
 
-static void writeback_inodes_wb(struct bdi_writeback *wb,
+static long writeback_inodes_wb(struct bdi_writeback *wb,
 				struct writeback_control *wbc)
 {
 	struct super_block *sb = wbc->sb, *pin_sb = NULL;
 	const int is_blkdev_sb = sb_is_blkdev_sb(sb);
 	const unsigned long start = jiffies;	/* livelock avoidance */
+	unsigned long stop_time = 0;
+	long wrote = 0;
+
+	if (wbc->timeout)
+		stop_time = (start + wbc->timeout) | 1;
 
 	spin_lock(&inode_lock);
 
@@ -721,7 +745,9 @@ static void writeback_inodes_wb(struct b
 		BUG_ON(inode->i_state & (I_FREEING | I_CLEAR));
 		__iget(inode);
 		pages_skipped = wbc->pages_skipped;
+		wbc->nr_to_write = MAX_WRITEBACK_PAGES;
 		writeback_single_inode(inode, wbc);
+		wrote += MAX_WRITEBACK_PAGES - wbc->nr_to_write;
 		if (wbc->pages_skipped != pages_skipped) {
 			/*
 			 * writeback is not making progress due to locked
@@ -739,12 +765,15 @@ static void writeback_inodes_wb(struct b
 		}
 		if (!list_empty(&wb->b_more_io))
 			wbc->more_io = 1;
+		if (stop_time && time_after(jiffies, stop_time))
+			break;
 	}
 
 	unpin_sb_for_writeback(&pin_sb);
 
 	spin_unlock(&inode_lock);
 	/* Leave any unwritten inodes on b_io */
+	return wrote;
 }
 
 void writeback_inodes_wbc(struct writeback_control *wbc)
@@ -754,15 +783,6 @@ void writeback_inodes_wbc(struct writeba
 	writeback_inodes_wb(&bdi->wb, wbc);
 }
 
-/*
- * The maximum number of pages to writeout in a single bdi flush/kupdate
- * operation.  We do this so we don't hold I_SYNC against an inode for
- * enormous amounts of time, which would block a userspace task which has
- * been forced to throttle against that inode.  Also, the code reevaluates
- * the dirty each time it has written this many pages.
- */
-#define MAX_WRITEBACK_PAGES     1024
-
 static inline bool over_bground_thresh(void)
 {
 	unsigned long background_thresh, dirty_thresh;
@@ -797,10 +817,12 @@ static long wb_writeback(struct bdi_writ
 		.sync_mode		= args->sync_mode,
 		.older_than_this	= NULL,
 		.for_kupdate		= args->for_kupdate,
+		.for_background		= args->for_background,
 		.range_cyclic		= args->range_cyclic,
 	};
 	unsigned long oldest_jif;
 	long wrote = 0;
+	long nr;
 	struct inode *inode;
 
 	if (wbc.for_kupdate) {
@@ -834,26 +856,20 @@ static long wb_writeback(struct bdi_writ
 
 		wbc.more_io = 0;
 		wbc.encountered_congestion = 0;
-		wbc.nr_to_write = MAX_WRITEBACK_PAGES;
 		wbc.pages_skipped = 0;
-		writeback_inodes_wb(wb, &wbc);
-		args->nr_pages -= MAX_WRITEBACK_PAGES - wbc.nr_to_write;
-		wrote += MAX_WRITEBACK_PAGES - wbc.nr_to_write;
+		nr = writeback_inodes_wb(wb, &wbc);
+		args->nr_pages -= nr;
+		wrote += nr;
 
 		/*
-		 * If we consumed everything, see if we have more
-		 */
-		if (wbc.nr_to_write <= 0)
-			continue;
-		/*
-		 * Didn't write everything and we don't have more IO, bail
+		 * Bail if no more IO
 		 */
 		if (!wbc.more_io)
 			break;
 		/*
 		 * Did we write something? Try for more
 		 */
-		if (wbc.nr_to_write < MAX_WRITEBACK_PAGES)
+		if (nr)
 			continue;
 		/*
 		 * Nothing written. Wait for some inode to
--- linux.orig/include/linux/writeback.h	2009-09-30 12:13:00.000000000 +0800
+++ linux/include/linux/writeback.h	2009-09-30 12:17:17.000000000 +0800
@@ -32,10 +32,15 @@ struct writeback_control {
 	struct super_block *sb;		/* if !NULL, only write inodes from
 					   this super_block */
 	enum writeback_sync_modes sync_mode;
+	int timeout;
 	unsigned long *older_than_this;	/* If !NULL, only write back inodes
 					   older than this */
-	long nr_to_write;		/* Write this many pages, and decrement
-					   this for each page written */
+	long nr_to_write;		/* Max pages to write per file, and
+					   decrement this for each page written
+					 */
+	long nr_segments;		/* Max page segments to write per file,
+					   this is also a count down value
+					 */
 	long pages_skipped;		/* Pages which were not written */
 
 	/*
@@ -49,6 +54,7 @@ struct writeback_control {
 	unsigned nonblocking:1;		/* Don't get stuck on request queues */
 	unsigned encountered_congestion:1; /* An output: a queue is full */
 	unsigned for_kupdate:1;		/* A kupdate writeback */
+	unsigned for_background:1;	/* A background writeback */
 	unsigned for_reclaim:1;		/* Invoked from the page allocator */
 	unsigned range_cyclic:1;	/* range_start is cyclic */
 	unsigned stop_on_wrap:1;	/* stop when write index is to wrap */
@@ -65,6 +71,11 @@ struct writeback_control {
 };
 
 /*
+ * if two page ranges are more than 1MB apart, they are taken as two segments.
+ */
+#define WB_SEGMENT_DIST		(1024 >> (PAGE_CACHE_SHIFT - 10))
+
+/*
  * fs/fs-writeback.c
  */	
 struct bdi_writeback;
--- linux.orig/mm/filemap.c	2009-09-30 12:13:00.000000000 +0800
+++ linux/mm/filemap.c	2009-09-30 12:17:17.000000000 +0800
@@ -216,6 +216,7 @@ int __filemap_fdatawrite_range(struct ad
 	struct writeback_control wbc = {
 		.sync_mode = sync_mode,
 		.nr_to_write = LONG_MAX,
+		.nr_segments = LONG_MAX,
 		.range_start = start,
 		.range_end = end,
 	};
--- linux.orig/mm/page-writeback.c	2009-09-30 12:17:15.000000000 +0800
+++ linux/mm/page-writeback.c	2009-09-30 12:17:17.000000000 +0800
@@ -765,6 +765,7 @@ int write_cache_pages(struct address_spa
 	int cycled;
 	int range_whole = 0;
 	long nr_to_write = wbc->nr_to_write;
+	unsigned long start_time = jiffies;
 
 	pagevec_init(&pvec, 0);
 	if (wbc->range_cyclic) {
@@ -818,6 +819,12 @@ retry:
 				break;
 			}
 
+			if (done_index + WB_SEGMENT_DIST > page->index &&
+			    --wbc->nr_segments <= 0) {
+				done = 1;
+				break;
+			}
+
 			done_index = page->index + 1;
 
 			lock_page(page);
@@ -899,6 +906,12 @@ continue_unlock:
 		}
 		pagevec_release(&pvec);
 		cond_resched();
+		if (wbc->timeout &&
+		    time_after(jiffies, start_time + wbc->timeout)) {
+			wbc->timeout = -1;
+			done = 1;
+			break;
+		}
 	}
 	if (wbc->stop_on_wrap)
 		done_index = 0;
--- linux.orig/fs/jbd2/commit.c	2009-09-30 12:13:00.000000000 +0800
+++ linux/fs/jbd2/commit.c	2009-09-30 12:17:17.000000000 +0800
@@ -219,6 +219,7 @@ static int journal_submit_inode_data_buf
 	struct writeback_control wbc = {
 		.sync_mode =  WB_SYNC_ALL,
 		.nr_to_write = mapping->nrpages * 2,
+		.nr_segments = LONG_MAX,
 		.range_start = 0,
 		.range_end = i_size_read(mapping->host),
 	};
--- linux.orig/mm/backing-dev.c	2009-09-30 12:17:26.000000000 +0800
+++ linux/mm/backing-dev.c	2009-09-30 12:17:38.000000000 +0800
@@ -336,7 +336,7 @@ static void bdi_flush_io(struct backing_
 		.sync_mode		= WB_SYNC_NONE,
 		.older_than_this	= NULL,
 		.range_cyclic		= 1,
-		.nr_to_write		= 1024,
+		.timeout		= HZ,
 	};
 
 	writeback_inodes_wbc(&wbc);

Re: regression in page writeback

[Theodore Tso]

On Wed, Sep 30, 2009 at 01:26:57PM +0800, Wu Fengguang wrote:
> It's good to increase MAX_WRITEBACK_PAGES, however I'm afraid
> max_contig_writeback_mb may be a burden in future: either it is not
> necessary, or a per-bdi counterpart must be introduced for all
> filesystems.

The per-filesystem tunable was just a short-term hack; the reason why
I did it that way was it was clear that a global tunable wouldn't fly,
and rightly so --- what might be suitable for a slow USB stick might
be very different than a super-fast RAID array, and someone might very
well have both on the same system.

> And it's preferred to automatically handle slow devices well with the
> increased chunk size, instead of adding another parameter.

Agreed; long-term what we probably need is something which is
automatically tunable.  My thinking was that we should tune the the
initial nr_to_write parameter based on how many blocks could be
written in some time interval, which is tunable.  So if we decide that
1 second is a suitable time period to be writing out one inode's dirty
pages, then for a fast server-class SATA disk, we might want to set
nr_to_write to be around 128mb worth of pages.  For a laptop SATA
disk, it might be around 64mb, and for a really slow USB stick, it
might be more like 16mb.  For super-fast enterprise RAID array, 128mb
might be too small!

If we get timing and/or congestion information from the block layer,
it wouldn't be hard to figure out the optimal number of pages that
should be sent down to the filesystem, and to tune this automatically.

> I scratched up a patch to demo the ideas collected in recent discussions.
> Can you check if it serves your needs? Thanks.

Sure, I'll definitely play with it, thanks.

> The wbc.timeout (when used per-file) is mainly a safeguard against slow
> devices, which may take too long time to sync 128MB data.

Maybe I'm missing something, but I don't think the wbc.timeout
approach is sufficient.  Consider the scenario of someone who is
ripping a DVD disc to an 8 gig USB stick.  The USB stick will be very
slow, but since the file is contiguous the filesystem will very
happily try to push it out there 128MB at a time, and wbc.timeout
value isn't really going to help since a single call to writepages
could easily cause 128MB worth of data to be streamed out to the USB
stick.

This is why the MAX_WRITEBACK_PAGES really needs to be tuned on a
per-bdi basis; either manually, via a sysfs tunable, or automatically,
by auto-tuning based on how fast the storage device is or by some kind
of congestion-based approach.  This is certainly the best long-term
solution; my concern was that it might take a long-time for us to get
the auto-tunable just right, so in the meantime I added a
per-mounted-filesystem tunable and put the hack in the filesystem
layer.  I would like nothing better than to rip it out, once we have a
long-term solution.

Regards,

							- Ted

Re: regression in page writeback

[Wu Fengguang]

Ted,

On Wed, Sep 30, 2009 at 10:11:58PM +0800, Theodore Ts'o wrote:
> On Wed, Sep 30, 2009 at 01:26:57PM +0800, Wu Fengguang wrote:
> > It's good to increase MAX_WRITEBACK_PAGES, however I'm afraid
> > max_contig_writeback_mb may be a burden in future: either it is not
> > necessary, or a per-bdi counterpart must be introduced for all
> > filesystems.
> 
> The per-filesystem tunable was just a short-term hack; the reason why
> I did it that way was it was clear that a global tunable wouldn't fly,
> and rightly so --- what might be suitable for a slow USB stick might
> be very different than a super-fast RAID array, and someone might very
> well have both on the same system.

Ah Yes.

> > And it's preferred to automatically handle slow devices well with the
> > increased chunk size, instead of adding another parameter.
> 
> Agreed; long-term what we probably need is something which is
> automatically tunable.  My thinking was that we should tune the the
> initial nr_to_write parameter based on how many blocks could be
> written in some time interval, which is tunable.  So if we decide that
> 1 second is a suitable time period to be writing out one inode's dirty
> pages, then for a fast server-class SATA disk, we might want to set
> nr_to_write to be around 128mb worth of pages.  For a laptop SATA
> disk, it might be around 64mb, and for a really slow USB stick, it
> might be more like 16mb.  For super-fast enterprise RAID array, 128mb
> might be too small!

Yes, 128MB may be too small :)

> If we get timing and/or congestion information from the block layer,
> it wouldn't be hard to figure out the optimal number of pages that
> should be sent down to the filesystem, and to tune this automatically.

Sure, it's possible.

> > I scratched up a patch to demo the ideas collected in recent discussions.
> > Can you check if it serves your needs? Thanks.
> 
> Sure, I'll definitely play with it, thanks.

Thanks :)

> > The wbc.timeout (when used per-file) is mainly a safeguard against slow
> > devices, which may take too long time to sync 128MB data.
> 
> Maybe I'm missing something, but I don't think the wbc.timeout
> approach is sufficient.  Consider the scenario of someone who is
> ripping a DVD disc to an 8 gig USB stick.  The USB stick will be very
> slow, but since the file is contiguous the filesystem will very
> happily try to push it out there 128MB at a time, and wbc.timeout
> value isn't really going to help since a single call to writepages
> could easily cause 128MB worth of data to be streamed out to the USB
> stick.

Yes and no. Yes if the queue was empty for the slow device. No if the
queue was full, in which case IO submission speed = IO complete speed
for previously queued requests.

So wbc.timeout will be accurate for IO submission time, and mostly
accurate for IO completion time. The transient queue fill up phase
shall not be a big problem?

> This is why the MAX_WRITEBACK_PAGES really needs to be tuned on a
> per-bdi basis; either manually, via a sysfs tunable, or automatically,
> by auto-tuning based on how fast the storage device is or by some kind
> of congestion-based approach.  This is certainly the best long-term
> solution; my concern was that it might take a long-time for us to get
> the auto-tunable just right, so in the meantime I added a
> per-mounted-filesystem tunable and put the hack in the filesystem
> layer.  I would like nothing better than to rip it out, once we have a
> long-term solution.

Thanks,
Fengguang

Re: regression in page writeback

[Theodore Tso]

On Thu, Oct 01, 2009 at 11:14:29PM +0800, Wu Fengguang wrote:
> Yes and no. Yes if the queue was empty for the slow device. No if the
> queue was full, in which case IO submission speed = IO complete speed
> for previously queued requests.
> 
> So wbc.timeout will be accurate for IO submission time, and mostly
> accurate for IO completion time. The transient queue fill up phase
> shall not be a big problem?

So the problem is if we have a mixed workload where there are lots
large contiguous writes, and lots of small writes which are fsync'ed()
--- for example, consider the workload of copying lots of big DVD
images combined with the infamous firefox-we-must-write-out-300-megs-of-
small-random-writes-and-then-fsync-them-on-every-single-url-click-so-
that-every-last-visited-page-is-preserved-for-history-bar-autocompletion
workload.    The big writes, if the are contiguous, could take 1-2 seconds
on a very slow, ancient laptop disk, and that will hold up any kind of 
small synchornous activities --- such as either a disk read or a firefox-
triggered fsync().

That's why the IO completion time matters; it causes latency problems
for slow disks and mixed large and small write workloads.  It was the
original reason for the 1024 MAX_WRITEBACK_PAGES, which might have
made sense 10 years ago back when disks were a lot slower.  One of the
advantages of an auto-tuning algorithm, beyond auto-adjusting for
different types of hardware, is that we don't need to worry about
arbitrary and magic caps beocoming obsolete due to technological
changes.  :-)

							- Ted

Re: regression in page writeback

[Jan Kara]

On Wed 30-09-09 13:32:23, Wu Fengguang wrote:
> writeback: bump up writeback chunk size to 128MB
> 
> Adjust the writeback call stack to support larger writeback chunk size.
> 
> - make wbc.nr_to_write a per-file parameter
> - init wbc.nr_to_write with MAX_WRITEBACK_PAGES=128MB
>   (proposed by Ted)
> - add wbc.nr_segments to limit seeks inside sparsely dirtied file
>   (proposed by Chris)
> - add wbc.timeout which will be used to control IO submission time
>   either per-file or globally.
>   
> The wbc.nr_segments is now determined purely by logical page index
> distance: if two pages are 1MB apart, it makes a new segment.
> 
> Filesystems could do this better with real extent knowledges.
> One possible scheme is to record the previous page index in
> wbc.writeback_index, and let ->writepage compare if the current and
> previous pages lie in the same extent, and decrease wbc.nr_segments
> accordingly. Care should taken to avoid double decreases in writepage
> and write_cache_pages.
> 
> The wbc.timeout (when used per-file) is mainly a safeguard against slow
> devices, which may take too long time to sync 128MB data.
> 
> The wbc.timeout (when used globally) could be useful when we decide to
> do two sync scans on dirty pages and dirty metadata. XFS could say:
> please return to sync dirty metadata after 10s. Would need another
> b_io_metadata queue, but that's possible.
> 
> This work depends on the balance_dirty_pages() wait queue patch.
  I don't know, I think it gets too complicated... I'd either use the
segments idea or the timeout idea but not both (unless you can find real
world tests in which both help). Also when we'll assure fairness via
timeout, maybe nr_to_write isn't needed anymore? WB_SYNC_ALL writeback
doesn't use nr_to_write. WB_SYNC_NONE writeback either sets it to some
large value (like LONG_MAX) or number of dirty pages (to effectively write
back as much as possible) or to MAX_WRITEBACK_PAGES to assure fairness
in kupdate style writeback. There are a few exceptions in btrfs but I
belive nr_to_write isn't really needed there either...

								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

Re: regression in page writeback

[Wu Fengguang]

On Fri, Oct 02, 2009 at 05:54:38AM +0800, Theodore Ts'o wrote:
> On Thu, Oct 01, 2009 at 11:14:29PM +0800, Wu Fengguang wrote:
> > Yes and no. Yes if the queue was empty for the slow device. No if the
> > queue was full, in which case IO submission speed = IO complete speed
> > for previously queued requests.
> > 
> > So wbc.timeout will be accurate for IO submission time, and mostly
> > accurate for IO completion time. The transient queue fill up phase
> > shall not be a big problem?
> 
> So the problem is if we have a mixed workload where there are lots
> large contiguous writes, and lots of small writes which are fsync'ed()
> --- for example, consider the workload of copying lots of big DVD
> images combined with the infamous firefox-we-must-write-out-300-megs-of-
> small-random-writes-and-then-fsync-them-on-every-single-url-click-so-
> that-every-last-visited-page-is-preserved-for-history-bar-autocompletion
> workload.    The big writes, if the are contiguous, could take 1-2 seconds
> on a very slow, ancient laptop disk, and that will hold up any kind of 
> small synchornous activities --- such as either a disk read or a firefox-
> triggered fsync().

Yes, that's a problem. The SYNC/ASYNC elevator queues can help here.

In IO submission paths, fsync writes will not be blocked by non-sync
writes because __filemap_fdatawrite_range() starts foreground sync
for the inode. Without the congestion backoff, it will now have to
compete queue with bdi-flush. Should not be a big problem though.

There's still the problem of IO submission time != IO completion time,
due to fluctuations of randomness and more. However that's a general
and unavoidable problem.  Both the wbc.timeout scheme and the
"wbc.nr_to_write based on estimated throughput" scheme are based on
_past_ requests and it's simply impossible to have a 100% accurate
scheme. In principle, wbc.timeout will only be inferior at IO startup
time. In the steady state of 100% full queue, it is actually estimating
the IO throughput implicitly :)

> That's why the IO completion time matters; it causes latency problems
> for slow disks and mixed large and small write workloads.  It was the
> original reason for the 1024 MAX_WRITEBACK_PAGES, which might have
> made sense 10 years ago back when disks were a lot slower.  One of the
> advantages of an auto-tuning algorithm, beyond auto-adjusting for
> different types of hardware, is that we don't need to worry about
> arbitrary and magic caps beocoming obsolete due to technological
> changes.  :-)

Yeah, I'm a big fan of auto-tuning :)

Thanks,
Fengguang

Re: regression in page writeback

[Wu Fengguang]

On Fri, Oct 02, 2009 at 06:17:39AM +0800, Jan Kara wrote:
> On Wed 30-09-09 13:32:23, Wu Fengguang wrote:
> > writeback: bump up writeback chunk size to 128MB
> > 
> > Adjust the writeback call stack to support larger writeback chunk size.
> > 
> > - make wbc.nr_to_write a per-file parameter
> > - init wbc.nr_to_write with MAX_WRITEBACK_PAGES=128MB
> >   (proposed by Ted)
> > - add wbc.nr_segments to limit seeks inside sparsely dirtied file
> >   (proposed by Chris)
> > - add wbc.timeout which will be used to control IO submission time
> >   either per-file or globally.
> >   
> > The wbc.nr_segments is now determined purely by logical page index
> > distance: if two pages are 1MB apart, it makes a new segment.
> > 
> > Filesystems could do this better with real extent knowledges.
> > One possible scheme is to record the previous page index in
> > wbc.writeback_index, and let ->writepage compare if the current and
> > previous pages lie in the same extent, and decrease wbc.nr_segments
> > accordingly. Care should taken to avoid double decreases in writepage
> > and write_cache_pages.
> > 
> > The wbc.timeout (when used per-file) is mainly a safeguard against slow
> > devices, which may take too long time to sync 128MB data.
> > 
> > The wbc.timeout (when used globally) could be useful when we decide to
> > do two sync scans on dirty pages and dirty metadata. XFS could say:
> > please return to sync dirty metadata after 10s. Would need another
> > b_io_metadata queue, but that's possible.
> > 
> > This work depends on the balance_dirty_pages() wait queue patch.
>   I don't know, I think it gets too complicated... I'd either use the
> segments idea or the timeout idea but not both (unless you can find real
> world tests in which both help).

Maybe complicated, but nr_segments and timeout each has their target
application.  nr_segments serves two major purposes:
- fairness between two large files, one is continuously dirtied,
  another is sparsely dirtied. Given the same amount of dirty pages,
  it could take vastly different time to sync them to the _same_
  device. The nr_segments check helps to favor continuous data.
- avoid seeks/fragmentations. To give each file fair chance of
  writeback, we have to abort a file when some nr_to_write or timeout
  is reached. However they are both not good abort conditions.
  The best is for filesystem to abort earlier in seek boundaries,
  and treat nr_to_write/timeout as large enough bottom lines.
timeout is mainly a safeguard in case nr_to_write is too large for
slow devices. It is not necessary if nr_to_write is auto-computed,
however timeout in itself serves as a simple throughput adapting
scheme.

> Also when we'll assure fairness via
> timeout, maybe nr_to_write isn't needed anymore? WB_SYNC_ALL writeback
> doesn't use nr_to_write. WB_SYNC_NONE writeback either sets it to some
> large value (like LONG_MAX) or number of dirty pages (to effectively write
> back as much as possible) or to MAX_WRITEBACK_PAGES to assure fairness
> in kupdate style writeback. There are a few exceptions in btrfs but I
> belive nr_to_write isn't really needed there either...

Totally agreed. I'd rather remove the top level nr_page/nr_to_write
parameters. They are simply redundant ones.. The meaningful ones are
background threshold, dirty expireness or global timeout, depending on
the mission of the writeback work.

Thanks,
Fengguang

Re: regression in page writeback

[Wu Fengguang]

On Fri, Oct 02, 2009 at 10:55:02AM +0800, Wu Fengguang wrote:
> On Fri, Oct 02, 2009 at 05:54:38AM +0800, Theodore Ts'o wrote:
> > On Thu, Oct 01, 2009 at 11:14:29PM +0800, Wu Fengguang wrote:
> > > Yes and no. Yes if the queue was empty for the slow device. No if the
> > > queue was full, in which case IO submission speed = IO complete speed
> > > for previously queued requests.
> > > 
> > > So wbc.timeout will be accurate for IO submission time, and mostly
> > > accurate for IO completion time. The transient queue fill up phase
> > > shall not be a big problem?
> > 
> > So the problem is if we have a mixed workload where there are lots
> > large contiguous writes, and lots of small writes which are fsync'ed()
> > --- for example, consider the workload of copying lots of big DVD
> > images combined with the infamous firefox-we-must-write-out-300-megs-of-
> > small-random-writes-and-then-fsync-them-on-every-single-url-click-so-
> > that-every-last-visited-page-is-preserved-for-history-bar-autocompletion
> > workload.    The big writes, if the are contiguous, could take 1-2 seconds
> > on a very slow, ancient laptop disk, and that will hold up any kind of 
> > small synchornous activities --- such as either a disk read or a firefox-
> > triggered fsync().
> 
> Yes, that's a problem. The SYNC/ASYNC elevator queues can help here.
> 
> In IO submission paths, fsync writes will not be blocked by non-sync
> writes because __filemap_fdatawrite_range() starts foreground sync
> for the inode.

> Without the congestion backoff, it will now have to
> compete queue with bdi-flush. Should not be a big problem though.

I'd like to correct this: get_request_wait() uses one queue for SYNC
rw and another for ASYNC rw. So fsync won't compete the request queue
with background flush. That's perfect: when fsync comes, CFQ will
honor it a green channel, and somehow block background flushes.

> There's still the problem of IO submission time != IO completion time,
> due to fluctuations of randomness and more. However that's a general
> and unavoidable problem.  Both the wbc.timeout scheme and the
> "wbc.nr_to_write based on estimated throughput" scheme are based on
> _past_ requests and it's simply impossible to have a 100% accurate
> scheme. In principle, wbc.timeout will only be inferior at IO startup
> time. In the steady state of 100% full queue, it is actually estimating
> the IO throughput implicitly :)

Another difference between wbc.timeout and adaptive wbc.nr_to_write
is, when there comes many _read_ requests or fsync, these SYNC rw
requests will significant lower the ASYNC writeback throughput, if
it's not completely stalled. So with timeout, the inode will be
aborted with few pages written; with nr_to_write, the inode will be
written a good number of pages, at the cost of taking up long time.

IMHO the nr_to_write behavior seems more efficient. What do you think?

Thanks,
Fengguang

> > That's why the IO completion time matters; it causes latency problems
> > for slow disks and mixed large and small write workloads.  It was the
> > original reason for the 1024 MAX_WRITEBACK_PAGES, which might have
> > made sense 10 years ago back when disks were a lot slower.  One of the
> > advantages of an auto-tuning algorithm, beyond auto-adjusting for
> > different types of hardware, is that we don't need to worry about
> > arbitrary and magic caps beocoming obsolete due to technological
> > changes.  :-)
> 
> Yeah, I'm a big fan of auto-tuning :)
> 
> Thanks,
> Fengguang

Re: regression in page writeback

[Theodore Tso]

On Fri, Oct 02, 2009 at 04:19:53PM +0800, Wu Fengguang wrote:
> > > The big writes, if they are contiguous, could take 1-2 seconds
> > > on a very slow, ancient laptop disk, and that will hold up any kind of 
> > > small synchornous activities --- such as either a disk read or a firefox-
> > > triggered fsync().
> > 
> > Yes, that's a problem. The SYNC/ASYNC elevator queues can help here.

The SYNC/ASYNC queues will partially help, up to the whatever the
largest I/O that can issued as a single chunk times the queue depth
for those disks that support NCQ. 

> > There's still the problem of IO submission time != IO completion time,
> > due to fluctuations of randomness and more. However that's a general
> > and unavoidable problem.  Both the wbc.timeout scheme and the
> > "wbc.nr_to_write based on estimated throughput" scheme are based on
> > _past_ requests and it's simply impossible to have a 100% accurate
> > scheme. In principle, wbc.timeout will only be inferior at IO startup
> > time. In the steady state of 100% full queue, it is actually estimating
> > the IO throughput implicitly :)
> 
> Another difference between wbc.timeout and adaptive wbc.nr_to_write
> is, when there comes many _read_ requests or fsync, these SYNC rw
> requests will significant lower the ASYNC writeback throughput, if
> it's not completely stalled. So with timeout, the inode will be
> aborted with few pages written; with nr_to_write, the inode will be
> written a good number of pages, at the cost of taking up long time.
> 
> IMHO the nr_to_write behavior seems more efficient. What do you think?

I agree, adaptively changing nr_to_write seems like the right thing to
do.  For bonus points, we could also monitor how often synchronous I/O
operations are happening, allow nr_to_write to go up by some amount if
there aren't many synchronous operations happening at the moment.  So
that might be another opportunity to do auto-tuning, although this
might be a hueristic that might need to be configurable for certain
specialized workloads.  For many other workloads, the it should be
possible to detect regular pattern of reads and/or synchronous writes,
and if so, use a lower nr_to_write versus if there isn't many
synchronous I/O operations happening on that particular block device.

	    		   	     	     - Ted

Re: regression in page writeback

[Wu Fengguang]

On Sat, Oct 03, 2009 at 01:26:20AM +0800, Theodore Ts'o wrote:
> On Fri, Oct 02, 2009 at 04:19:53PM +0800, Wu Fengguang wrote:
> > > > The big writes, if they are contiguous, could take 1-2 seconds
> > > > on a very slow, ancient laptop disk, and that will hold up any kind of 
> > > > small synchornous activities --- such as either a disk read or a firefox-
> > > > triggered fsync().
> > > 
> > > Yes, that's a problem. The SYNC/ASYNC elevator queues can help here.
> 
> The SYNC/ASYNC queues will partially help, up to the whatever the
> largest I/O that can issued as a single chunk times the queue depth
> for those disks that support NCQ. 
> 
> > > There's still the problem of IO submission time != IO completion time,
> > > due to fluctuations of randomness and more. However that's a general
> > > and unavoidable problem.  Both the wbc.timeout scheme and the
> > > "wbc.nr_to_write based on estimated throughput" scheme are based on
> > > _past_ requests and it's simply impossible to have a 100% accurate
> > > scheme. In principle, wbc.timeout will only be inferior at IO startup
> > > time. In the steady state of 100% full queue, it is actually estimating
> > > the IO throughput implicitly :)
> > 
> > Another difference between wbc.timeout and adaptive wbc.nr_to_write
> > is, when there comes many _read_ requests or fsync, these SYNC rw
> > requests will significant lower the ASYNC writeback throughput, if
> > it's not completely stalled. So with timeout, the inode will be
> > aborted with few pages written; with nr_to_write, the inode will be
> > written a good number of pages, at the cost of taking up long time.
> > 
> > IMHO the nr_to_write behavior seems more efficient. What do you think?
> 
> I agree, adaptively changing nr_to_write seems like the right thing to

I'd like to estimate the writeback throughput in bdi_writeback_wakeup(),
where the queue is not starved and the estimation would reflect the max
device capability (unless there are busy reads, in which case we need
lower nr_to_write anyway).

> do.  For bonus points, we could also monitor how often synchronous I/O
> operations are happening, allow nr_to_write to go up by some amount if
> there aren't many synchronous operations happening at the moment.  So
> that might be another opportunity to do auto-tuning, although this
> might be a hueristic that might need to be configurable for certain
> specialized workloads.  For many other workloads, the it should be
> possible to detect regular pattern of reads and/or synchronous writes,
> and if so, use a lower nr_to_write versus if there isn't many
> synchronous I/O operations happening on that particular block device.

It's not easy to get state of the art SYNC read/write busyness.
However it is possible to "feel" them through the progress of ASYNC
writes.

- setup a per-file timeout=3*HZ
- check this in write_cache_pages:

        if (half nr_to_write pages written && timeout)
                break;

In this way we back off to nr_to_write/2 if the writeback is blocked
by some busy READs.

I'd choose to implement this advanced feature some time later :)

Thanks,
Fengguang

Re: regression in page writeback

[Jan Kara]

On Fri 02-10-09 11:27:14, Wu Fengguang wrote:
> On Fri, Oct 02, 2009 at 06:17:39AM +0800, Jan Kara wrote:
> > On Wed 30-09-09 13:32:23, Wu Fengguang wrote:
> > > writeback: bump up writeback chunk size to 128MB
> > > 
> > > Adjust the writeback call stack to support larger writeback chunk size.
> > > 
> > > - make wbc.nr_to_write a per-file parameter
> > > - init wbc.nr_to_write with MAX_WRITEBACK_PAGES=128MB
> > >   (proposed by Ted)
> > > - add wbc.nr_segments to limit seeks inside sparsely dirtied file
> > >   (proposed by Chris)
> > > - add wbc.timeout which will be used to control IO submission time
> > >   either per-file or globally.
> > >   
> > > The wbc.nr_segments is now determined purely by logical page index
> > > distance: if two pages are 1MB apart, it makes a new segment.
> > > 
> > > Filesystems could do this better with real extent knowledges.
> > > One possible scheme is to record the previous page index in
> > > wbc.writeback_index, and let ->writepage compare if the current and
> > > previous pages lie in the same extent, and decrease wbc.nr_segments
> > > accordingly. Care should taken to avoid double decreases in writepage
> > > and write_cache_pages.
> > > 
> > > The wbc.timeout (when used per-file) is mainly a safeguard against slow
> > > devices, which may take too long time to sync 128MB data.
> > > 
> > > The wbc.timeout (when used globally) could be useful when we decide to
> > > do two sync scans on dirty pages and dirty metadata. XFS could say:
> > > please return to sync dirty metadata after 10s. Would need another
> > > b_io_metadata queue, but that's possible.
> > > 
> > > This work depends on the balance_dirty_pages() wait queue patch.
> >   I don't know, I think it gets too complicated... I'd either use the
> > segments idea or the timeout idea but not both (unless you can find real
> > world tests in which both help).
  I'm sorry for a delayed reply but I had to work on something else.

> Maybe complicated, but nr_segments and timeout each has their target
> application.  nr_segments serves two major purposes:
> - fairness between two large files, one is continuously dirtied,
>   another is sparsely dirtied. Given the same amount of dirty pages,
>   it could take vastly different time to sync them to the _same_
>   device. The nr_segments check helps to favor continuous data.
> - avoid seeks/fragmentations. To give each file fair chance of
>   writeback, we have to abort a file when some nr_to_write or timeout
>   is reached. However they are both not good abort conditions.
>   The best is for filesystem to abort earlier in seek boundaries,
>   and treat nr_to_write/timeout as large enough bottom lines.
> timeout is mainly a safeguard in case nr_to_write is too large for
> slow devices. It is not necessary if nr_to_write is auto-computed,
> however timeout in itself serves as a simple throughput adapting
> scheme.
  I understand why you have introduced both segments and timeout value
and a completely agree with your reasons to introduce them. I just think
that when the system gets too complex (there will be several independent
methods of determining when writeback should be terminated, and even
though each method is simple on its own, their interactions needn't be
simple...) it will be hard to debug all the corner cases - even more
because they will manifest "just" by slow or unfair writeback. So I'd
prefer a single metric to determine when to stop writeback of an inode
even though it might be a bit more complicated.
  For example terminating on writeout does not really get a file fair
chance of writeback because it might have been blocked just because we were
writing some heavily fragmented file just before. And your nr_segments
check is just a rough guess of whether a writeback is going to be
fragmented or not.
  So I'd rather implement in mpage_ functions a proper detection of how
fragmented the writeback is and give each inode a limit on number of
fragments which mpage_ functions would obey. We could even use a queue's
NONROT flag (set for solid state disks) to detect whether we should expect
higher or lower seek times.
								Honza

Re: regression in page writeback

[Wu Fengguang]

On Tue, Oct 06, 2009 at 08:55:19PM +0800, Jan Kara wrote:
> On Fri 02-10-09 11:27:14, Wu Fengguang wrote:
> > On Fri, Oct 02, 2009 at 06:17:39AM +0800, Jan Kara wrote:
> > > On Wed 30-09-09 13:32:23, Wu Fengguang wrote:
> > > > writeback: bump up writeback chunk size to 128MB
> > > > 
> > > > Adjust the writeback call stack to support larger writeback chunk size.
> > > > 
> > > > - make wbc.nr_to_write a per-file parameter
> > > > - init wbc.nr_to_write with MAX_WRITEBACK_PAGES=128MB
> > > >   (proposed by Ted)
> > > > - add wbc.nr_segments to limit seeks inside sparsely dirtied file
> > > >   (proposed by Chris)
> > > > - add wbc.timeout which will be used to control IO submission time
> > > >   either per-file or globally.
> > > >   
> > > > The wbc.nr_segments is now determined purely by logical page index
> > > > distance: if two pages are 1MB apart, it makes a new segment.
> > > > 
> > > > Filesystems could do this better with real extent knowledges.
> > > > One possible scheme is to record the previous page index in
> > > > wbc.writeback_index, and let ->writepage compare if the current and
> > > > previous pages lie in the same extent, and decrease wbc.nr_segments
> > > > accordingly. Care should taken to avoid double decreases in writepage
> > > > and write_cache_pages.
> > > > 
> > > > The wbc.timeout (when used per-file) is mainly a safeguard against slow
> > > > devices, which may take too long time to sync 128MB data.
> > > > 
> > > > The wbc.timeout (when used globally) could be useful when we decide to
> > > > do two sync scans on dirty pages and dirty metadata. XFS could say:
> > > > please return to sync dirty metadata after 10s. Would need another
> > > > b_io_metadata queue, but that's possible.
> > > > 
> > > > This work depends on the balance_dirty_pages() wait queue patch.
> > >   I don't know, I think it gets too complicated... I'd either use the
> > > segments idea or the timeout idea but not both (unless you can find real
> > > world tests in which both help).
>   I'm sorry for a delayed reply but I had to work on something else.
> 
> > Maybe complicated, but nr_segments and timeout each has their target
> > application.  nr_segments serves two major purposes:
> > - fairness between two large files, one is continuously dirtied,
> >   another is sparsely dirtied. Given the same amount of dirty pages,
> >   it could take vastly different time to sync them to the _same_
> >   device. The nr_segments check helps to favor continuous data.
> > - avoid seeks/fragmentations. To give each file fair chance of
> >   writeback, we have to abort a file when some nr_to_write or timeout
> >   is reached. However they are both not good abort conditions.
> >   The best is for filesystem to abort earlier in seek boundaries,
> >   and treat nr_to_write/timeout as large enough bottom lines.
> > timeout is mainly a safeguard in case nr_to_write is too large for
> > slow devices. It is not necessary if nr_to_write is auto-computed,
> > however timeout in itself serves as a simple throughput adapting
> > scheme.
>   I understand why you have introduced both segments and timeout value
> and a completely agree with your reasons to introduce them. I just think
> that when the system gets too complex (there will be several independent
> methods of determining when writeback should be terminated, and even
> though each method is simple on its own, their interactions needn't be
> simple...) it will be hard to debug all the corner cases - even more
> because they will manifest "just" by slow or unfair writeback. So I'd

I definitely agree on the complications. There are some known issues
as well as possibly some corner cases to be discovered. One problem I
noticed now is, what if all the files are sparsely dirtied? Then
a small nr_segments can only hurt.  Another problem is, the block
device file tend to have sparsely dirtied pages (with metadata on
them).  Not sure how to detect/handle such conditions..

> prefer a single metric to determine when to stop writeback of an inode
> even though it might be a bit more complicated.
>   For example terminating on writeout does not really get a file fair
> chance of writeback because it might have been blocked just because we were
> writing some heavily fragmented file just before. And your nr_segments

You mean timeout? I've dropped that idea in favor of an nr_to_write
adaptive to the bdi write speed :)

> check is just a rough guess of whether a writeback is going to be
> fragmented or not.

It could be made accurate if btrfs decreases it in its own writepages,
based on the extent info. Should also be possible for ext4.

>   So I'd rather implement in mpage_ functions a proper detection of how
> fragmented the writeback is and give each inode a limit on number of
> fragments which mpage_ functions would obey. We could even use a queue's
> NONROT flag (set for solid state disks) to detect whether we should expect
> higher or lower seek times.

Yes, mpage_* can also utilize nr_segments.

Anyway nr_segments is not perfect, I'll post a patch and let fs
developers decide whether it is convenient/useful :) 

Thanks,
Fengguang