memcg writeback (was Re: [Lsf-pc] [LSF/MM TOPIC] memcg topics.)

memcg writeback (was Re: [Lsf-pc] [LSF/MM TOPIC] memcg topics.)

[Greg Thelen]

On Fri, Feb 3, 2012 at 1:40 AM, Wu Fengguang <fengguang.wu@intel.com> wrote:
> If moving dirty pages out of the memcg to the 20% global dirty pages
> pool on page reclaim, the above OOM can be avoided. It does change the
> meaning of memory.limit_in_bytes in that the memcg tasks can now
> actually consume more pages (up to the shared global 20% dirty limit).

This seems like an easy change, but unfortunately the global 20% pool
has some shortcomings for my needs:

1. the global 20% pool is not moderated.  One cgroup can dominate it
    and deny service to other cgroups.

2. the global 20% pool is free, unaccounted memory.  Ideally cgroups only
    use the amount of memory specified in their memory.limit_in_bytes.  The
    goal is to sell portions of a system.  Global resource like the 20% are an
    undesirable system-wide tax that's shared by jobs that may not even
    perform buffered writes.

3. Setting aside 20% extra memory for system wide dirty buffers is a lot of
    memory.  This becomes a larger issue when the global dirty_ratio is
    higher than 20%.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: memcg writeback (was Re: [Lsf-pc] [LSF/MM TOPIC] memcg topics.)

[Wu Fengguang]

On Tue, Feb 07, 2012 at 11:55:05PM -0800, Greg Thelen wrote:
> On Fri, Feb 3, 2012 at 1:40 AM, Wu Fengguang <fengguang.wu@intel.com> wrote:
> > If moving dirty pages out of the memcg to the 20% global dirty pages
> > pool on page reclaim, the above OOM can be avoided. It does change the
> > meaning of memory.limit_in_bytes in that the memcg tasks can now
> > actually consume more pages (up to the shared global 20% dirty limit).
> 
> This seems like an easy change, but unfortunately the global 20% pool
> has some shortcomings for my needs:
> 
> 1. the global 20% pool is not moderated.  One cgroup can dominate it
>     and deny service to other cgroups.

It is moderated by balance_dirty_pages() -- in terms of dirty ratelimit.
And you have the freedom to control the bandwidth allocation with some
async write I/O controller.

Even though there is no direct control of dirty pages, we can roughly
get it as the side effect of rate control. Given

        ratelimit_cgroup_A = 2 * ratelimit_cgroup_B

There will naturally be more dirty pages for cgroup A to be worked by
the flusher. And the dirty pages will be roughly balanced around

        nr_dirty_cgroup_A = 2 * nr_dirty_cgroup_B

when writeout bandwidths for their dirty pages are equal.

> 2. the global 20% pool is free, unaccounted memory.  Ideally cgroups only
>     use the amount of memory specified in their memory.limit_in_bytes.  The
>     goal is to sell portions of a system.  Global resource like the 20% are an
>     undesirable system-wide tax that's shared by jobs that may not even
>     perform buffered writes.

Right, it is the shortcoming.

> 3. Setting aside 20% extra memory for system wide dirty buffers is a lot of
>     memory.  This becomes a larger issue when the global dirty_ratio is
>     higher than 20%.

Yeah the global pool scheme does mean that you'd better allocate at
most 80% memory to individual memory cgroups, otherwise it's possible
for a tiny memcg doing dd writes to push dirty pages to global LRU and
*squeeze* the size of other memcgs.

However I guess it should be mitigated by the fact that

- we typically already reserve some space for the root memcg

- 20% dirty ratio is mostly an overkill for large memory systems.
  It's often enough to hold 10-30s worth of dirty data for them, which
  is 1-3GB for one 100MB/s disk. This is the reason vm.dirty_bytes is
  introduced: someone wants to do some <1% dirty ratio.

Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: memcg writeback (was Re: [Lsf-pc] [LSF/MM TOPIC] memcg topics.)

[Ying Han]

On Wed, Feb 8, 2012 at 1:31 AM, Wu Fengguang <fengguang.wu@intel.com> wrote:
> On Tue, Feb 07, 2012 at 11:55:05PM -0800, Greg Thelen wrote:
>> On Fri, Feb 3, 2012 at 1:40 AM, Wu Fengguang <fengguang.wu@intel.com> wrote:
>> > If moving dirty pages out of the memcg to the 20% global dirty pages
>> > pool on page reclaim, the above OOM can be avoided. It does change the
>> > meaning of memory.limit_in_bytes in that the memcg tasks can now
>> > actually consume more pages (up to the shared global 20% dirty limit).
>>
>> This seems like an easy change, but unfortunately the global 20% pool
>> has some shortcomings for my needs:
>>
>> 1. the global 20% pool is not moderated.  One cgroup can dominate it
>>     and deny service to other cgroups.
>
> It is moderated by balance_dirty_pages() -- in terms of dirty ratelimit.
> And you have the freedom to control the bandwidth allocation with some
> async write I/O controller.
>
> Even though there is no direct control of dirty pages, we can roughly
> get it as the side effect of rate control. Given
>
>        ratelimit_cgroup_A = 2 * ratelimit_cgroup_B
>
> There will naturally be more dirty pages for cgroup A to be worked by
> the flusher. And the dirty pages will be roughly balanced around
>
>        nr_dirty_cgroup_A = 2 * nr_dirty_cgroup_B
>
> when writeout bandwidths for their dirty pages are equal.
>
>> 2. the global 20% pool is free, unaccounted memory.  Ideally cgroups only
>>     use the amount of memory specified in their memory.limit_in_bytes.  The
>>     goal is to sell portions of a system.  Global resource like the 20% are an
>>     undesirable system-wide tax that's shared by jobs that may not even
>>     perform buffered writes.
>
> Right, it is the shortcoming.
>
>> 3. Setting aside 20% extra memory for system wide dirty buffers is a lot of
>>     memory.  This becomes a larger issue when the global dirty_ratio is
>>     higher than 20%.
>
> Yeah the global pool scheme does mean that you'd better allocate at
> most 80% memory to individual memory cgroups, otherwise it's possible
> for a tiny memcg doing dd writes to push dirty pages to global LRU and
> *squeeze* the size of other memcgs.
>
> However I guess it should be mitigated by the fact that
>
> - we typically already reserve some space for the root memcg

Can you give more details on that? AFAIK, we don't treat root cgroup
differently than other sub-cgroups, except root cgroup doesn't have
limit.

In general, I don't like the idea of shared pool in root for all the
dirty pages.

Imagining a system which has nothing running under root and every
application runs within sub-cgroup. It is easy to track and limit each
cgroup's memory usage, but not the pages being moved to root. We have
been experiencing difficulties of tracking pages being re-parented to
root, and this will make it even harder.

--Ying

>
> - 20% dirty ratio is mostly an overkill for large memory systems.
>  It's often enough to hold 10-30s worth of dirty data for them, which
>  is 1-3GB for one 100MB/s disk. This is the reason vm.dirty_bytes is
>  introduced: someone wants to do some <1% dirty ratio.
>
> Thanks,
> Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: memcg writeback (was Re: [Lsf-pc] [LSF/MM TOPIC] memcg topics.)

[Wu Fengguang]

On Wed, Feb 08, 2012 at 12:54:33PM -0800, Ying Han wrote:
> On Wed, Feb 8, 2012 at 1:31 AM, Wu Fengguang <fengguang.wu@intel.com> wrote:
> > On Tue, Feb 07, 2012 at 11:55:05PM -0800, Greg Thelen wrote:
> >> On Fri, Feb 3, 2012 at 1:40 AM, Wu Fengguang <fengguang.wu@intel.com> wrote:
> >> > If moving dirty pages out of the memcg to the 20% global dirty pages
> >> > pool on page reclaim, the above OOM can be avoided. It does change the
> >> > meaning of memory.limit_in_bytes in that the memcg tasks can now
> >> > actually consume more pages (up to the shared global 20% dirty limit).
> >>
> >> This seems like an easy change, but unfortunately the global 20% pool
> >> has some shortcomings for my needs:
> >>
> >> 1. the global 20% pool is not moderated. A One cgroup can dominate it
> >> A  A  and deny service to other cgroups.
> >
> > It is moderated by balance_dirty_pages() -- in terms of dirty ratelimit.
> > And you have the freedom to control the bandwidth allocation with some
> > async write I/O controller.
> >
> > Even though there is no direct control of dirty pages, we can roughly
> > get it as the side effect of rate control. Given
> >
> > A  A  A  A ratelimit_cgroup_A = 2 * ratelimit_cgroup_B
> >
> > There will naturally be more dirty pages for cgroup A to be worked by
> > the flusher. And the dirty pages will be roughly balanced around
> >
> > A  A  A  A nr_dirty_cgroup_A = 2 * nr_dirty_cgroup_B
> >
> > when writeout bandwidths for their dirty pages are equal.
> >
> >> 2. the global 20% pool is free, unaccounted memory. A Ideally cgroups only
> >> A  A  use the amount of memory specified in their memory.limit_in_bytes. A The
> >> A  A  goal is to sell portions of a system. A Global resource like the 20% are an
> >> A  A  undesirable system-wide tax that's shared by jobs that may not even
> >> A  A  perform buffered writes.
> >
> > Right, it is the shortcoming.
> >
> >> 3. Setting aside 20% extra memory for system wide dirty buffers is a lot of
> >> A  A  memory. A This becomes a larger issue when the global dirty_ratio is
> >> A  A  higher than 20%.
> >
> > Yeah the global pool scheme does mean that you'd better allocate at
> > most 80% memory to individual memory cgroups, otherwise it's possible
> > for a tiny memcg doing dd writes to push dirty pages to global LRU and
> > *squeeze* the size of other memcgs.
> >
> > However I guess it should be mitigated by the fact that
> >
> > - we typically already reserve some space for the root memcg
> 
> Can you give more details on that? AFAIK, we don't treat root cgroup
> differently than other sub-cgroups, except root cgroup doesn't have
> limit.

OK. I'd imagine this to be the typical usage for desktop and quite a
few servers: a few cgroups are employed to limit the resource usage
for selected tasks (such as backups, background GUI tasks, cron tasks,
etc.). These systems are still running mainly in the global context.

> In general, I don't like the idea of shared pool in root for all the
> dirty pages.
> 
> Imagining a system which has nothing running under root and every
> application runs within sub-cgroup. It is easy to track and limit each
> cgroup's memory usage, but not the pages being moved to root. We have
> been experiencing difficulties of tracking pages being re-parented to
> root, and this will make it even harder.

So you want to push memcg allocations to the hardware limits. This is
a worthwhile target for cloud servers that run a number of well
contained jobs.

I guess it can be achieved reasonably well with the global shared
dirty pool.  Let's discuss the two major cases.

1) no change of behavior

For example, when the system memory is divided equally to 10 cgroups
each running 1 dd. In this case, the dirty pages will be contained
within the memcg LRUs. Page reclaim rarely encounters any dirty pages.
There is no moving to the global LRU, so no side effect at all.

2) small memcg squeezing other memcg(s)

When system memory is divided to 1 small memcg A and 1 large memcg B,
each running a dd task. In this case the dirty pages from A will be
moved to the global LRU, and global page reclaims will be triggered.

In the end it will be balanced around

- global LRU: 10% memory (which are A's dirty pages)
- memcg B: 90% memory
- memcg A: a tiny ignorable fraction of memory

Now job B uses 10% less memory than w/o the global dirty pool scheme.
I guess this is bad for some type of jobs.

However my question is, will the typical demand be more flexible?
Something like the "minimal" and "recommended" setup: "this job
requires at least XXX memory and better at YYY memory", rather than
some fixed size memory allocation.

The minimal requirement should be trivially satisfied by adding a
memcg watermark that protects the memcg LRU from being reclaimed
when dropped under it.

Then the cloud server could be configured to

        sum(memcg.limit_in_bytes) / memtotal = 100%
        sum(memcg.minimal_size)   / memtotal < 100% - dirty_ratio

Which makes a simple and flexibly partitioned system.

Thanks,
Fengguang

> > - 20% dirty ratio is mostly an overkill for large memory systems.
> > A It's often enough to hold 10-30s worth of dirty data for them, which
> > A is 1-3GB for one 100MB/s disk. This is the reason vm.dirty_bytes is
> > A introduced: someone wants to do some <1% dirty ratio.
> >
> > Thanks,
> > Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: memcg writeback (was Re: [Lsf-pc] [LSF/MM TOPIC] memcg topics.)

[Ying Han]

On Thu, Feb 9, 2012 at 5:50 AM, Wu Fengguang <fengguang.wu@intel.com> wrote:
> On Wed, Feb 08, 2012 at 12:54:33PM -0800, Ying Han wrote:
>> On Wed, Feb 8, 2012 at 1:31 AM, Wu Fengguang <fengguang.wu@intel.com> wrote:
>> > On Tue, Feb 07, 2012 at 11:55:05PM -0800, Greg Thelen wrote:
>> >> On Fri, Feb 3, 2012 at 1:40 AM, Wu Fengguang <fengguang.wu@intel.com> wrote:
>> >> > If moving dirty pages out of the memcg to the 20% global dirty pages
>> >> > pool on page reclaim, the above OOM can be avoided. It does change the
>> >> > meaning of memory.limit_in_bytes in that the memcg tasks can now
>> >> > actually consume more pages (up to the shared global 20% dirty limit).
>> >>
>> >> This seems like an easy change, but unfortunately the global 20% pool
>> >> has some shortcomings for my needs:
>> >>
>> >> 1. the global 20% pool is not moderated.  One cgroup can dominate it
>> >>     and deny service to other cgroups.
>> >
>> > It is moderated by balance_dirty_pages() -- in terms of dirty ratelimit.
>> > And you have the freedom to control the bandwidth allocation with some
>> > async write I/O controller.
>> >
>> > Even though there is no direct control of dirty pages, we can roughly
>> > get it as the side effect of rate control. Given
>> >
>> >        ratelimit_cgroup_A = 2 * ratelimit_cgroup_B
>> >
>> > There will naturally be more dirty pages for cgroup A to be worked by
>> > the flusher. And the dirty pages will be roughly balanced around
>> >
>> >        nr_dirty_cgroup_A = 2 * nr_dirty_cgroup_B
>> >
>> > when writeout bandwidths for their dirty pages are equal.
>> >
>> >> 2. the global 20% pool is free, unaccounted memory.  Ideally cgroups only
>> >>     use the amount of memory specified in their memory.limit_in_bytes.  The
>> >>     goal is to sell portions of a system.  Global resource like the 20% are an
>> >>     undesirable system-wide tax that's shared by jobs that may not even
>> >>     perform buffered writes.
>> >
>> > Right, it is the shortcoming.
>> >
>> >> 3. Setting aside 20% extra memory for system wide dirty buffers is a lot of
>> >>     memory.  This becomes a larger issue when the global dirty_ratio is
>> >>     higher than 20%.
>> >
>> > Yeah the global pool scheme does mean that you'd better allocate at
>> > most 80% memory to individual memory cgroups, otherwise it's possible
>> > for a tiny memcg doing dd writes to push dirty pages to global LRU and
>> > *squeeze* the size of other memcgs.
>> >
>> > However I guess it should be mitigated by the fact that
>> >
>> > - we typically already reserve some space for the root memcg
>>
>> Can you give more details on that? AFAIK, we don't treat root cgroup
>> differently than other sub-cgroups, except root cgroup doesn't have
>> limit.
>
> OK. I'd imagine this to be the typical usage for desktop and quite a
> few servers: a few cgroups are employed to limit the resource usage
> for selected tasks (such as backups, background GUI tasks, cron tasks,
> etc.). These systems are still running mainly in the global context.

The use case makes senses, but still not sure about the "reservation
for root" part.

For other tasks not running under cgroups, they runs under global
context as you said. However, there is no memory limit for root cgroup
and it will only trigger global reclaim when running short of memory.
It doesn't sounds like a straight-forward configuration for
environments requires memory isolation badly. The worst part is the
unpredictability, which we don't have control of how much
dirty-and-later-clean pages being leaked to root and stays.

--Ying

>
>> In general, I don't like the idea of shared pool in root for all the
>> dirty pages.
>>
>> Imagining a system which has nothing running under root and every
>> application runs within sub-cgroup. It is easy to track and limit each
>> cgroup's memory usage, but not the pages being moved to root. We have
>> been experiencing difficulties of tracking pages being re-parented to
>> root, and this will make it even harder.
>
> So you want to push memcg allocations to the hardware limits. This is
> a worthwhile target for cloud servers that run a number of well
> contained jobs.
>
> I guess it can be achieved reasonably well with the global shared
> dirty pool.  Let's discuss the two major cases.
>
> 1) no change of behavior
>
> For example, when the system memory is divided equally to 10 cgroups
> each running 1 dd. In this case, the dirty pages will be contained
> within the memcg LRUs. Page reclaim rarely encounters any dirty pages.
> There is no moving to the global LRU, so no side effect at all.

>
> 2) small memcg squeezing other memcg(s)
>
> When system memory is divided to 1 small memcg A and 1 large memcg B,
> each running a dd task. In this case the dirty pages from A will be
> moved to the global LRU, and global page reclaims will be triggered.
>
> In the end it will be balanced around
>
> - global LRU: 10% memory (which are A's dirty pages)
> - memcg B: 90% memory
> - memcg A: a tiny ignorable fraction of memory
>
> Now job B uses 10% less memory than w/o the global dirty pool scheme.
> I guess this is bad for some type of jobs.
>
> However my question is, will the typical demand be more flexible?
> Something like the "minimal" and "recommended" setup: "this job
> requires at least XXX memory and better at YYY memory", rather than
> some fixed size memory allocation.
>
> The minimal requirement should be trivially satisfied by adding a
> memcg watermark that protects the memcg LRU from being reclaimed
> when dropped under it.
>
> Then the cloud server could be configured to
>
>        sum(memcg.limit_in_bytes) / memtotal = 100%
>        sum(memcg.minimal_size)   / memtotal < 100% - dirty_ratio
>
> Which makes a simple and flexibly partitioned system.
>
> Thanks,
> Fengguang
>
>> > - 20% dirty ratio is mostly an overkill for large memory systems.
>> >  It's often enough to hold 10-30s worth of dirty data for them, which
>> >  is 1-3GB for one 100MB/s disk. This is the reason vm.dirty_bytes is
>> >  introduced: someone wants to do some <1% dirty ratio.
>> >
>> > Thanks,
>> > Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: memcg writeback (was Re: [Lsf-pc] [LSF/MM TOPIC] memcg topics.)

[Greg Thelen]

(removed lsf-pc@lists.linux-foundation.org because this really isn't
program committee matter)

On Wed, Feb 8, 2012 at 1:31 AM, Wu Fengguang <fengguang.wu@intel.com> wrote:
> On Tue, Feb 07, 2012 at 11:55:05PM -0800, Greg Thelen wrote:
>> On Fri, Feb 3, 2012 at 1:40 AM, Wu Fengguang <fengguang.wu@intel.com> wrote:
>> > If moving dirty pages out of the memcg to the 20% global dirty pages
>> > pool on page reclaim, the above OOM can be avoided. It does change the
>> > meaning of memory.limit_in_bytes in that the memcg tasks can now
>> > actually consume more pages (up to the shared global 20% dirty limit).
>>
>> This seems like an easy change, but unfortunately the global 20% pool
>> has some shortcomings for my needs:
>>
>> 1. the global 20% pool is not moderated.  One cgroup can dominate it
>>     and deny service to other cgroups.
>
> It is moderated by balance_dirty_pages() -- in terms of dirty ratelimit.
> And you have the freedom to control the bandwidth allocation with some
> async write I/O controller.
>
> Even though there is no direct control of dirty pages, we can roughly
> get it as the side effect of rate control. Given
>
>        ratelimit_cgroup_A = 2 * ratelimit_cgroup_B
>
> There will naturally be more dirty pages for cgroup A to be worked by
> the flusher. And the dirty pages will be roughly balanced around
>
>        nr_dirty_cgroup_A = 2 * nr_dirty_cgroup_B
>
> when writeout bandwidths for their dirty pages are equal.
>
>> 2. the global 20% pool is free, unaccounted memory.  Ideally cgroups only
>>     use the amount of memory specified in their memory.limit_in_bytes.  The
>>     goal is to sell portions of a system.  Global resource like the 20% are an
>>     undesirable system-wide tax that's shared by jobs that may not even
>>     perform buffered writes.
>
> Right, it is the shortcoming.
>
>> 3. Setting aside 20% extra memory for system wide dirty buffers is a lot of
>>     memory.  This becomes a larger issue when the global dirty_ratio is
>>     higher than 20%.
>
> Yeah the global pool scheme does mean that you'd better allocate at
> most 80% memory to individual memory cgroups, otherwise it's possible
> for a tiny memcg doing dd writes to push dirty pages to global LRU and
> *squeeze* the size of other memcgs.
>
> However I guess it should be mitigated by the fact that
>
> - we typically already reserve some space for the root memcg
>
> - 20% dirty ratio is mostly an overkill for large memory systems.
>  It's often enough to hold 10-30s worth of dirty data for them, which
>  is 1-3GB for one 100MB/s disk. This is the reason vm.dirty_bytes is
>  introduced: someone wants to do some <1% dirty ratio.

Have you encountered situations where it's desirable to have more than
20% dirty ratio?  I imagine that if the dirty working set is larger
than 20% increasing dirty ratio would prevent rewrites.

Leaking dirty memory to a root global dirty pool is concerning.  I
suspect that under some conditions such pages may remain remain in
root after writeback indefinitely as clean pages.  I admit this may
not be the common case, but having such leaks into root can allow low
priority jobs access entire machine denying service to higher priority
jobs.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: memcg writeback (was Re: [Lsf-pc] [LSF/MM TOPIC] memcg topics.)

[Greg Thelen]

(removed lsf-pc@lists.linux-foundation.org because this really isn't
program committee matter)

On Wed, Feb 1, 2012 at 11:52 PM, Wu Fengguang <fengguang.wu@intel.com> wrote:
> Unfortunately the memcg partitioning could fundamentally make the
> dirty throttling more bumpy.
>
> Imagine 10 memcgs each with
>
> - memcg_dirty_limit=50MB
> - 1 dd dirty task
>
> The flusher thread will be working on 10 inodes in turn, each time
> grabbing the next inode and taking ~0.5s to write ~50MB of its dirty
> pages to the disk. So each inode will be flushed on every ~5s.

Does the flusher thread need to write 50MB/inode in this case?  Would
there be problems interleaving writes by declaring some max write
limit (e.g. 8 MiB/write).  Such interleaving would be beneficial if
there are multiple memcg expecting service from the single bdi flusher
thread.  I suspect certain filesystems might have increased
fragmentation with this, but I am not sure if appending writes can
easily expand an extent.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: memcg writeback (was Re: [Lsf-pc] [LSF/MM TOPIC] memcg topics.)

[Wu Fengguang]

On Thu, Feb 09, 2012 at 09:52:03PM -0800, Greg Thelen wrote:
> (removed lsf-pc@lists.linux-foundation.org because this really isn't
> program committee matter)
> 
> On Wed, Feb 1, 2012 at 11:52 PM, Wu Fengguang <fengguang.wu@intel.com> wrote:
> > Unfortunately the memcg partitioning could fundamentally make the
> > dirty throttling more bumpy.
> >
> > Imagine 10 memcgs each with
> >
> > - memcg_dirty_limit=50MB
> > - 1 dd dirty task
> >
> > The flusher thread will be working on 10 inodes in turn, each time
> > grabbing the next inode and taking ~0.5s to write ~50MB of its dirty
> > pages to the disk. So each inode will be flushed on every ~5s.
> 
> Does the flusher thread need to write 50MB/inode in this case?
> Would there be problems interleaving writes by declaring some max
> write limit (e.g. 8 MiB/write).  

ext4 actually forces write chunk size to be >=128MB for better write
throughput and less fragmentation, which also helps read performance.

Other filesystems use the VFS computed chunk size, which is defined
in writeback_chunk_size() as write_bandwith/2.

> Such interleaving would be beneficial if there are multiple memcg
> expecting service from the single bdi flusher thread.

Right, reducing the writeback chunk size will improve the memcg's
dirty pages smoothness right away.

> I suspect certain filesystems might have increased fragmentation
> with this, but I am not sure if appending writes can easily expand
> an extent.

To be exact, it's ext4 that will suffer from fragmentation with smaller
chunk sizes. Because it uses the size passed by ->writepages() as hint
to allocate extents. Perhaps this heuristic is somehow improvable.

XFS does not have the fragmentation issue (at least not affected by
the chunk size). However my old tests show that it costs much less
seeks and performs noticeably better with raised write chunk size.

Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: memcg writeback (was Re: [Lsf-pc] [LSF/MM TOPIC] memcg topics.)

[Wu Fengguang]

On Thu, Feb 09, 2012 at 09:51:31PM -0800, Greg Thelen wrote:
> On Wed, Feb 8, 2012 at 1:31 AM, Wu Fengguang <fengguang.wu@intel.com> wrote:
> > On Tue, Feb 07, 2012 at 11:55:05PM -0800, Greg Thelen wrote:
> >> On Fri, Feb 3, 2012 at 1:40 AM, Wu Fengguang <fengguang.wu@intel.com> wrote:
> >> > If moving dirty pages out of the memcg to the 20% global dirty pages
> >> > pool on page reclaim, the above OOM can be avoided. It does change the
> >> > meaning of memory.limit_in_bytes in that the memcg tasks can now
> >> > actually consume more pages (up to the shared global 20% dirty limit).
> >>
> >> This seems like an easy change, but unfortunately the global 20% pool
> >> has some shortcomings for my needs:
> >>
> >> 1. the global 20% pool is not moderated. A One cgroup can dominate it
> >> A  A  and deny service to other cgroups.
> >
> > It is moderated by balance_dirty_pages() -- in terms of dirty ratelimit.
> > And you have the freedom to control the bandwidth allocation with some
> > async write I/O controller.
> >
> > Even though there is no direct control of dirty pages, we can roughly
> > get it as the side effect of rate control. Given
> >
> > A  A  A  A ratelimit_cgroup_A = 2 * ratelimit_cgroup_B
> >
> > There will naturally be more dirty pages for cgroup A to be worked by
> > the flusher. And the dirty pages will be roughly balanced around
> >
> > A  A  A  A nr_dirty_cgroup_A = 2 * nr_dirty_cgroup_B
> >
> > when writeout bandwidths for their dirty pages are equal.
> >
> >> 2. the global 20% pool is free, unaccounted memory. A Ideally cgroups only
> >> A  A  use the amount of memory specified in their memory.limit_in_bytes. A The
> >> A  A  goal is to sell portions of a system. A Global resource like the 20% are an
> >> A  A  undesirable system-wide tax that's shared by jobs that may not even
> >> A  A  perform buffered writes.
> >
> > Right, it is the shortcoming.
> >
> >> 3. Setting aside 20% extra memory for system wide dirty buffers is a lot of
> >> A  A  memory. A This becomes a larger issue when the global dirty_ratio is
> >> A  A  higher than 20%.
> >
> > Yeah the global pool scheme does mean that you'd better allocate at
> > most 80% memory to individual memory cgroups, otherwise it's possible
> > for a tiny memcg doing dd writes to push dirty pages to global LRU and
> > *squeeze* the size of other memcgs.
> >
> > However I guess it should be mitigated by the fact that
> >
> > - we typically already reserve some space for the root memcg
> >
> > - 20% dirty ratio is mostly an overkill for large memory systems.
> > A It's often enough to hold 10-30s worth of dirty data for them, which
> > A is 1-3GB for one 100MB/s disk. This is the reason vm.dirty_bytes is
> > A introduced: someone wants to do some <1% dirty ratio.
> 
> Have you encountered situations where it's desirable to have more than
> 20% dirty ratio?  I imagine that if the dirty working set is larger
> than 20% increasing dirty ratio would prevent rewrites.

Not encountered in person, but there will sure be such situations.

One may need to dirty some 40% sized in-memory data set and don't want
to be throttled and trigger lots of I/O. In this case increasing the
dirty ratio to 40% will do the job.

The less obvious condition for this to work is, the workload should
avoid much page allocations when working on the large fixed data set.
Otherwise page reclaim will keep running into dirty pages and act badly.

So it looks still pretty compatible with the "reparent to root memcg
on page reclaim" scheme if that job is put into some memcg. There will
be almost no dirty pages encountered during page reclaim, hence no
move and no any side effects.

But if there is another job doing heavy dirtying, that job will eat up
the global 40% dirty limit and heavily impact the above job. This is
one case the memcg dirty ratio can help a lot.

> Leaking dirty memory to a root global dirty pool is concerning.  I
> suspect that under some conditions such pages may remain remain in
> root after writeback indefinitely as clean pages.  I admit this may
> not be the common case, but having such leaks into root can allow low
> priority jobs access entire machine denying service to higher priority
> jobs.

You are right. DoS can be achieved by

        loop {
                dirty one more page
                access all previously dirtied pages
        }

Assuming only !PG_reference pages are moved to the global dirty pool,
it requires someone to access the page in order for it to stay in the
global LRU for one more cycle, and to access it frequently for keeping
it in global LRU indefinitely.

So yes, it's possible for some evil job to DoS the whole box.  It will
be an issue when hosting jobs from untrusted sources (ie. Amazon style
cloud service), which I guess should be running inside KVM?

It should hardly happen in real workloads. If some job does manage to
do so, it probably means some kind of mis-configuration: the memcg is
configured way too small to hold the job's working set.

Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

reclaim the LRU lists full of dirty/writeback pages

[Wu Fengguang]

On Fri, Feb 10, 2012 at 07:47:06PM +0800, Wu Fengguang wrote:
> On Thu, Feb 09, 2012 at 09:51:31PM -0800, Greg Thelen wrote:

> > Have you encountered situations where it's desirable to have more than
> > 20% dirty ratio?  I imagine that if the dirty working set is larger
> > than 20% increasing dirty ratio would prevent rewrites.

> One may need to dirty some 40% sized in-memory data set and don't want
> to be throttled and trigger lots of I/O. In this case increasing the
> dirty ratio to 40% will do the job.
 
> But if there is another job doing heavy dirtying, that job will eat up
> the global 40% dirty limit and heavily impact the above job. This is
> one case the memcg dirty ratio can help a lot.
> 
> > Leaking dirty memory to a root global dirty pool is concerning.  I
> > suspect that under some conditions such pages may remain remain in
> > root after writeback indefinitely as clean pages.  I admit this may
> > not be the common case, but having such leaks into root can allow low
> > priority jobs access entire machine denying service to higher priority
> > jobs.
> 
> You are right. DoS can be achieved by
> 
>         loop {
>                 dirty one more page
>                 access all previously dirtied pages
>         }

So there are situations that prefer the dirty pages to be strictly
contained within the memcg.

For these use cases it looks worthwhile to improve the page reclaim
algorithms to handle the 100% dirty zone well. I'd regard this as a
much more promising direction than memcg dirty ratio, because efforts
on this is going to benefit the general kernel as a whole.

The below patch aims to be the first step towards the goal.  It turns
out to work pretty well for avoiding OOM, with reasonably good I/O
throughput and low CPU overheads.

Hopefully the page reclaim can be further improved to make the 100%
dirty zone a seriously supported and well performed case.

Thanks,
Fengguang
---
Subject: writeback: introduce the pageout work
Date: Thu Jul 29 14:41:19 CST 2010

This relays file pageout IOs to the flusher threads.

The ultimate target is to gracefully handle the LRU lists full of
dirty/writeback pages.

1) I/O efficiency

The flusher will piggy back the around 1MB dirty pages for I/O (XXX:
make the chunk size adaptive to the bdi write bandwidth).

This takes advantage of the time/spacial locality in most workloads: the
nearby pages of one file are typically populated into the LRU at the same
time, hence will likely be close to each other in the LRU list. Writing
them in one shot helps clean more pages effectively for page reclaim.

2) OOM avoidance and scan rate control

Typically we do LRU scan w/o rate control and quickly get enough clean
pages for the LRU lists not full of dirty pages.

Or we can still get a number of freshly cleaned pages (moved to LRU tail
by end_page_writeback()) when the queued pageout I/O is completed within
tens of milli-seconds.

However if the LRU list is small and full of dirty pages, it can be
quickly fully scanned and go OOM before the flusher manages to clean
enough pages.

Here a simple yet reliable scheme is employed to avoid OOM and keep scan
rate in sync with the I/O rate:

	if (PageReclaim(page))
		congestion_wait();

PG_reclaim plays the key role. When dirty pages are encountered, we
queue I/O for it, set PG_reclaim and put it back to the LRU head.
So if PG_reclaim pages are encountered again, it means the dirty page
has not yet been cleaned by the flusher after a full zone scan. It
indicates we are scanning more fast than I/O and shall take a snap.

The runtime behavior on a fully dirtied small LRU list would be:
It will start with a quick scan of the list, queuing all pages for I/O.
Then the scan will be slowed down by the PG_reclaim pages *adaptively*
to match the I/O bandwidth.

3) writeback work coordinations

To avoid memory allocations at page reclaim, a mempool for struct
wb_writeback_work is created.

wakeup_flusher_threads() is removed because it can easily delay the
more oriented pageout works and even exhaust the mempool reservations.
It's often not I/O efficient by submitting writeback works with small
->nr_pages.

Background/periodic works will quit automatically (as done in another
patch), so as to clean the pages under reclaim ASAP. However for now the
sync work can still block us for long time.

Jan Kara: limit the search scope. Note that the limited search and work
pool is not a big problem: 1000 IOs under flight are typically more than
enough to saturate the disk. And the overheads of searching in the work
list didn't even show up in the perf report.

4) test case

Run 2 dd tasks in a 100MB memcg (a very handy test case from Greg Thelen):

	mkdir /cgroup/x
	echo 100M > /cgroup/x/memory.limit_in_bytes
	echo $$ > /cgroup/x/tasks

	for i in `seq 2`
	do
		dd if=/dev/zero of=/fs/f$i bs=1k count=1M &
	done

Before patch, the dd tasks are quickly OOM killed.
After patch, they run well with reasonably good performance and overheads:

1073741824 bytes (1.1 GB) copied, 22.2196 s, 48.3 MB/s
1073741824 bytes (1.1 GB) copied, 22.4675 s, 47.8 MB/s

iostat -kx 1

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00    0.00  178.00     0.00 89568.00  1006.38    74.35  417.71   4.80  85.40
sda               0.00     2.00    0.00  191.00     0.00 94428.00   988.77    53.34  219.03   4.34  82.90
sda               0.00    20.00    0.00  196.00     0.00 97712.00   997.06    71.11  337.45   4.77  93.50
sda               0.00     5.00    0.00  175.00     0.00 84648.00   967.41    54.03  316.44   5.06  88.60
sda               0.00     0.00    0.00  186.00     0.00 92432.00   993.89    56.22  267.54   5.38 100.00
sda               0.00     1.00    0.00  183.00     0.00 90156.00   985.31    37.99  325.55   4.33  79.20
sda               0.00     0.00    0.00  175.00     0.00 88692.00  1013.62    48.70  218.43   4.69  82.10
sda               0.00     0.00    0.00  196.00     0.00 97528.00   995.18    43.38  236.87   5.10 100.00
sda               0.00     0.00    0.00  179.00     0.00 88648.00   990.48    45.83  285.43   5.59 100.00
sda               0.00     0.00    0.00  178.00     0.00 88500.00   994.38    28.28  158.89   4.99  88.80
sda               0.00     0.00    0.00  194.00     0.00 95852.00   988.16    32.58  167.39   5.15 100.00
sda               0.00     2.00    0.00  215.00     0.00 105996.00   986.01    41.72  201.43   4.65 100.00
sda               0.00     4.00    0.00  173.00     0.00 84332.00   974.94    50.48  260.23   5.76  99.60
sda               0.00     0.00    0.00  182.00     0.00 90312.00   992.44    36.83  212.07   5.49 100.00
sda               0.00     8.00    0.00  195.00     0.00 95940.50   984.01    50.18  221.06   5.13 100.00
sda               0.00     1.00    0.00  220.00     0.00 108852.00   989.56    40.99  202.68   4.55 100.00
sda               0.00     2.00    0.00  161.00     0.00 80384.00   998.56    37.19  268.49   6.21 100.00
sda               0.00     4.00    0.00  182.00     0.00 90830.00   998.13    50.58  239.77   5.49 100.00
sda               0.00     0.00    0.00  197.00     0.00 94877.00   963.22    36.68  196.79   5.08 100.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.25    0.00   15.08   33.92    0.00   50.75
           0.25    0.00   14.54   35.09    0.00   50.13
           0.50    0.00   13.57   32.41    0.00   53.52
           0.50    0.00   11.28   36.84    0.00   51.38
           0.50    0.00   15.75   32.00    0.00   51.75
           0.50    0.00   10.50   34.00    0.00   55.00
           0.50    0.00   17.63   27.46    0.00   54.41
           0.50    0.00   15.08   30.90    0.00   53.52
           0.50    0.00   11.28   32.83    0.00   55.39
           0.75    0.00   16.79   26.82    0.00   55.64
           0.50    0.00   16.08   29.15    0.00   54.27
           0.50    0.00   13.50   30.50    0.00   55.50
           0.50    0.00   14.32   35.18    0.00   50.00
           0.50    0.00   12.06   33.92    0.00   53.52
           0.50    0.00   17.29   30.58    0.00   51.63
           0.50    0.00   15.08   29.65    0.00   54.77
           0.50    0.00   12.53   29.32    0.00   57.64
           0.50    0.00   15.29   31.83    0.00   52.38

The global dd iostat for comparison:

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00    0.00  189.00     0.00 95752.00  1013.25   143.09  684.48   5.29 100.00
sda               0.00     0.00    0.00  208.00     0.00 105480.00  1014.23   143.06  733.29   4.81 100.00
sda               0.00     0.00    0.00  161.00     0.00 81924.00  1017.69   141.71  757.79   6.21 100.00
sda               0.00     0.00    0.00  217.00     0.00 109580.00  1009.95   143.09  749.55   4.61 100.10
sda               0.00     0.00    0.00  187.00     0.00 94728.00  1013.13   144.31  773.67   5.35 100.00
sda               0.00     0.00    0.00  189.00     0.00 95752.00  1013.25   144.14  742.00   5.29 100.00
sda               0.00     0.00    0.00  177.00     0.00 90032.00  1017.31   143.32  656.59   5.65 100.00
sda               0.00     0.00    0.00  215.00     0.00 108640.00  1010.60   142.90  817.54   4.65 100.00
sda               0.00     2.00    0.00  166.00     0.00 83858.00  1010.34   143.64  808.61   6.02 100.00
sda               0.00     0.00    0.00  186.00     0.00 92813.00   997.99   141.18  736.95   5.38 100.00
sda               0.00     0.00    0.00  206.00     0.00 104456.00  1014.14   146.27  729.33   4.85 100.00
sda               0.00     0.00    0.00  213.00     0.00 107024.00  1004.92   143.25  705.70   4.69 100.00
sda               0.00     0.00    0.00  188.00     0.00 95748.00  1018.60   141.82  764.78   5.32 100.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.51    0.00   11.22   52.30    0.00   35.97
           0.25    0.00   10.15   52.54    0.00   37.06
           0.25    0.00    5.01   56.64    0.00   38.10
           0.51    0.00   15.15   43.94    0.00   40.40
           0.25    0.00   12.12   48.23    0.00   39.39
           0.51    0.00   11.20   53.94    0.00   34.35
           0.26    0.00    9.72   51.41    0.00   38.62
           0.76    0.00    9.62   50.63    0.00   38.99
           0.51    0.00   10.46   53.32    0.00   35.71
           0.51    0.00    9.41   51.91    0.00   38.17
           0.25    0.00   10.69   49.62    0.00   39.44
           0.51    0.00   12.21   52.67    0.00   34.61
           0.51    0.00   11.45   53.18    0.00   34.86

Note that it's data for XFS. ext4 seems to have some problem with the
workload: the majority pages are found to be writeback pages, and the
flusher ends up blocking on the unconditional wait_on_page_writeback()
in write_cache_pages_da() from time to time...

XXX: commit NFS unstable pages via write_inode()
XXX: the added congestion_wait() may be undesirable in some situations

CC: Jan Kara <jack@suse.cz>
CC: Mel Gorman <mgorman@suse.de>
CC: Rik van Riel <riel@redhat.com>
CC: Greg Thelen <gthelen@google.com>
CC: Minchan Kim <minchan.kim@gmail.com>
Signed-off-by: Wu Fengguang <fengguang.wu@intel.com>
---
 fs/fs-writeback.c                |  165 ++++++++++++++++++++++++++++-
 include/linux/writeback.h        |    4 
 include/trace/events/writeback.h |   12 +-
 mm/vmscan.c                      |   17 +-
 4 files changed, 184 insertions(+), 14 deletions(-)

--- linux.orig/mm/vmscan.c	2012-02-03 21:42:21.000000000 +0800
+++ linux/mm/vmscan.c	2012-02-11 17:28:54.000000000 +0800
@@ -813,6 +813,8 @@ static unsigned long shrink_page_list(st
 
 		if (PageWriteback(page)) {
 			nr_writeback++;
+			if (PageReclaim(page))
+				congestion_wait(BLK_RW_ASYNC, HZ/10);
 			/*
 			 * Synchronous reclaim cannot queue pages for
 			 * writeback due to the possibility of stack overflow
@@ -874,12 +876,15 @@ static unsigned long shrink_page_list(st
 			nr_dirty++;
 
 			/*
-			 * Only kswapd can writeback filesystem pages to
-			 * avoid risk of stack overflow but do not writeback
-			 * unless under significant pressure.
+			 * run into the visited page again: we are scanning
+			 * faster than the flusher can writeout dirty pages
 			 */
-			if (page_is_file_cache(page) &&
-					(!current_is_kswapd() || priority >= DEF_PRIORITY - 2)) {
+			if (page_is_file_cache(page) && PageReclaim(page)) {
+				congestion_wait(BLK_RW_ASYNC, HZ/10);
+				goto keep_locked;
+			}
+			if (page_is_file_cache(page) && mapping &&
+			    flush_inode_page(mapping, page, true) >= 0) {
 				/*
 				 * Immediately reclaim when written back.
 				 * Similar in principal to deactivate_page()
@@ -2382,8 +2387,6 @@ static unsigned long do_try_to_free_page
 		 */
 		writeback_threshold = sc->nr_to_reclaim + sc->nr_to_reclaim / 2;
 		if (total_scanned > writeback_threshold) {
-			wakeup_flusher_threads(laptop_mode ? 0 : total_scanned,
-						WB_REASON_TRY_TO_FREE_PAGES);
 			sc->may_writepage = 1;
 		}
 
--- linux.orig/fs/fs-writeback.c	2012-02-03 21:42:16.000000000 +0800
+++ linux/fs/fs-writeback.c	2012-02-11 18:24:24.000000000 +0800
@@ -35,12 +35,21 @@
 #define MIN_WRITEBACK_PAGES	(4096UL >> (PAGE_CACHE_SHIFT - 10))
 
 /*
+ * When flushing an inode page (for page reclaim), try to piggy back up to
+ * 1MB nearby pages for IO efficiency. These pages will have good opportunity
+ * to be in the same LRU list.
+ */
+#define WRITE_AROUND_PAGES	(1024UL >> (PAGE_CACHE_SHIFT - 10))
+
+/*
  * Passed into wb_writeback(), essentially a subset of writeback_control
  */
 struct wb_writeback_work {
 	long nr_pages;
 	struct super_block *sb;
 	unsigned long *older_than_this;
+	struct inode *inode;
+	pgoff_t offset;
 	enum writeback_sync_modes sync_mode;
 	unsigned int tagged_writepages:1;
 	unsigned int for_kupdate:1;
@@ -65,6 +74,27 @@ struct wb_writeback_work {
  */
 int nr_pdflush_threads;
 
+static mempool_t *wb_work_mempool;
+
+static void *wb_work_alloc(gfp_t gfp_mask, void *pool_data)
+{
+	/*
+	 * bdi_flush_inode_range() may be called on page reclaim
+	 */
+	if (current->flags & PF_MEMALLOC)
+		return NULL;
+
+	return kmalloc(sizeof(struct wb_writeback_work), gfp_mask);
+}
+
+static __init int wb_work_init(void)
+{
+	wb_work_mempool = mempool_create(1024,
+					 wb_work_alloc, mempool_kfree, NULL);
+	return wb_work_mempool ? 0 : -ENOMEM;
+}
+fs_initcall(wb_work_init);
+
 /**
  * writeback_in_progress - determine whether there is writeback in progress
  * @bdi: the device's backing_dev_info structure.
@@ -129,7 +159,7 @@ __bdi_start_writeback(struct backing_dev
 	 * This is WB_SYNC_NONE writeback, so if allocation fails just
 	 * wakeup the thread for old dirty data writeback
 	 */
-	work = kzalloc(sizeof(*work), GFP_ATOMIC);
+	work = mempool_alloc(wb_work_mempool, GFP_NOWAIT);
 	if (!work) {
 		if (bdi->wb.task) {
 			trace_writeback_nowork(bdi);
@@ -138,6 +168,7 @@ __bdi_start_writeback(struct backing_dev
 		return;
 	}
 
+	memset(work, 0, sizeof(*work));
 	work->sync_mode	= WB_SYNC_NONE;
 	work->nr_pages	= nr_pages;
 	work->range_cyclic = range_cyclic;
@@ -186,6 +217,114 @@ void bdi_start_background_writeback(stru
 	spin_unlock_bh(&bdi->wb_lock);
 }
 
+static bool extend_writeback_range(struct wb_writeback_work *work,
+				   pgoff_t offset)
+{
+	pgoff_t end = work->offset + work->nr_pages;
+
+	if (offset >= work->offset && offset < end)
+		return true;
+
+	if (work->nr_pages >= 8 * WRITE_AROUND_PAGES)
+		return false;
+
+	/* the unsigned comparison helps eliminate one compare */
+	if (work->offset - offset < WRITE_AROUND_PAGES) {
+		work->nr_pages += WRITE_AROUND_PAGES;
+		work->offset -= WRITE_AROUND_PAGES;
+		return true;
+	}
+
+	if (offset - end < WRITE_AROUND_PAGES) {
+		work->nr_pages += WRITE_AROUND_PAGES;
+		return true;
+	}
+
+	return false;
+}
+
+/*
+ * schedule writeback on a range of inode pages.
+ */
+static struct wb_writeback_work *
+bdi_flush_inode_range(struct backing_dev_info *bdi,
+		      struct inode *inode,
+		      pgoff_t offset,
+		      pgoff_t len,
+		      bool wait)
+{
+	struct wb_writeback_work *work;
+
+	if (!igrab(inode))
+		return ERR_PTR(-ENOENT);
+
+	work = mempool_alloc(wb_work_mempool, wait ? GFP_NOIO : GFP_NOWAIT);
+	if (!work)
+		return ERR_PTR(-ENOMEM);
+
+	memset(work, 0, sizeof(*work));
+	work->sync_mode		= WB_SYNC_NONE;
+	work->inode		= inode;
+	work->offset		= offset;
+	work->nr_pages		= len;
+	work->reason		= WB_REASON_PAGEOUT;
+
+	bdi_queue_work(bdi, work);
+
+	return work;
+}
+
+/*
+ * Called by page reclaim code to flush the dirty page ASAP. Do write-around to
+ * improve IO throughput. The nearby pages will have good chance to reside in
+ * the same LRU list that vmscan is working on, and even close to each other
+ * inside the LRU list in the common case of sequential read/write.
+ *
+ * ret > 0: success, found/reused a previous writeback work
+ * ret = 0: success, allocated/queued a new writeback work
+ * ret < 0: failed
+ */
+long flush_inode_page(struct address_space *mapping,
+		      struct page *page,
+		      bool wait)
+{
+	struct backing_dev_info *bdi = mapping->backing_dev_info;
+	struct inode *inode = mapping->host;
+	pgoff_t offset = page->index;
+	pgoff_t len = 0;
+	struct wb_writeback_work *work;
+	long ret = -ENOENT;
+
+	if (unlikely(!inode))
+		goto out;
+
+	len = 1;
+	spin_lock_bh(&bdi->wb_lock);
+	list_for_each_entry_reverse(work, &bdi->work_list, list) {
+		if (work->inode != inode)
+			continue;
+		if (extend_writeback_range(work, offset)) {
+			ret = len;
+			offset = work->offset;
+			len = work->nr_pages;
+			break;
+		}
+		if (len++ > 100)	/* limit search depth */
+			break;
+	}
+	spin_unlock_bh(&bdi->wb_lock);
+
+	if (ret > 0)
+		goto out;
+
+	offset = round_down(offset, WRITE_AROUND_PAGES);
+	len = WRITE_AROUND_PAGES;
+	work = bdi_flush_inode_range(bdi, inode, offset, len, wait);
+	ret = IS_ERR(work) ? PTR_ERR(work) : 0;
+out:
+	return ret;
+}
+
 /*
  * Remove the inode from the writeback list it is on.
  */
@@ -833,6 +972,23 @@ static unsigned long get_nr_dirty_pages(
 		get_nr_dirty_inodes();
 }
 
+static long wb_flush_inode(struct bdi_writeback *wb,
+			   struct wb_writeback_work *work)
+{
+	struct writeback_control wbc = {
+		.sync_mode = WB_SYNC_NONE,
+		.nr_to_write = LONG_MAX,
+		.range_start = work->offset << PAGE_CACHE_SHIFT,
+		.range_end = (work->offset + work->nr_pages - 1)
+						<< PAGE_CACHE_SHIFT,
+	};
+
+	do_writepages(work->inode->i_mapping, &wbc);
+	iput(work->inode);
+
+	return LONG_MAX - wbc.nr_to_write;
+}
+
 static long wb_check_background_flush(struct bdi_writeback *wb)
 {
 	if (over_bground_thresh(wb->bdi)) {
@@ -905,7 +1061,10 @@ long wb_do_writeback(struct bdi_writebac
 
 		trace_writeback_exec(bdi, work);
 
-		wrote += wb_writeback(wb, work);
+		if (work->inode)
+			wrote += wb_flush_inode(wb, work);
+		else
+			wrote += wb_writeback(wb, work);
 
 		/*
 		 * Notify the caller of completion if this is a synchronous
@@ -914,7 +1073,7 @@ long wb_do_writeback(struct bdi_writebac
 		if (work->done)
 			complete(work->done);
 		else
-			kfree(work);
+			mempool_free(work, wb_work_mempool);
 	}
 
 	/*
--- linux.orig/include/trace/events/writeback.h	2012-02-10 21:54:14.000000000 +0800
+++ linux/include/trace/events/writeback.h	2012-02-11 16:49:18.000000000 +0800
@@ -23,7 +23,7 @@
 
 #define WB_WORK_REASON							\
 		{WB_REASON_BACKGROUND,		"background"},		\
-		{WB_REASON_TRY_TO_FREE_PAGES,	"try_to_free_pages"},	\
+		{WB_REASON_PAGEOUT,		"pageout"},		\
 		{WB_REASON_SYNC,		"sync"},		\
 		{WB_REASON_PERIODIC,		"periodic"},		\
 		{WB_REASON_LAPTOP_TIMER,	"laptop_timer"},	\
@@ -45,6 +45,8 @@ DECLARE_EVENT_CLASS(writeback_work_class
 		__field(int, range_cyclic)
 		__field(int, for_background)
 		__field(int, reason)
+		__field(unsigned long, ino)
+		__field(unsigned long, offset)
 	),
 	TP_fast_assign(
 		strncpy(__entry->name, dev_name(bdi->dev), 32);
@@ -55,9 +57,11 @@ DECLARE_EVENT_CLASS(writeback_work_class
 		__entry->range_cyclic = work->range_cyclic;
 		__entry->for_background	= work->for_background;
 		__entry->reason = work->reason;
+		__entry->ino = work->inode ? work->inode->i_ino : 0;
+		__entry->offset = work->offset;
 	),
 	TP_printk("bdi %s: sb_dev %d:%d nr_pages=%ld sync_mode=%d "
-		  "kupdate=%d range_cyclic=%d background=%d reason=%s",
+		  "kupdate=%d range_cyclic=%d background=%d reason=%s ino=%lu offset=%lu",
 		  __entry->name,
 		  MAJOR(__entry->sb_dev), MINOR(__entry->sb_dev),
 		  __entry->nr_pages,
@@ -65,7 +69,9 @@ DECLARE_EVENT_CLASS(writeback_work_class
 		  __entry->for_kupdate,
 		  __entry->range_cyclic,
 		  __entry->for_background,
-		  __print_symbolic(__entry->reason, WB_WORK_REASON)
+		  __print_symbolic(__entry->reason, WB_WORK_REASON),
+		  __entry->ino,
+		  __entry->offset
 	)
 );
 #define DEFINE_WRITEBACK_WORK_EVENT(name) \
--- linux.orig/include/linux/writeback.h	2012-02-11 09:53:53.000000000 +0800
+++ linux/include/linux/writeback.h	2012-02-11 16:49:36.000000000 +0800
@@ -40,7 +40,7 @@ enum writeback_sync_modes {
  */
 enum wb_reason {
 	WB_REASON_BACKGROUND,
-	WB_REASON_TRY_TO_FREE_PAGES,
+	WB_REASON_PAGEOUT,
 	WB_REASON_SYNC,
 	WB_REASON_PERIODIC,
 	WB_REASON_LAPTOP_TIMER,
@@ -94,6 +94,8 @@ long writeback_inodes_wb(struct bdi_writ
 				enum wb_reason reason);
 long wb_do_writeback(struct bdi_writeback *wb, int force_wait);
 void wakeup_flusher_threads(long nr_pages, enum wb_reason reason);
+long flush_inode_page(struct address_space *mapping, struct page *page,
+		      bool wait);
 
 /* writeback.h requires fs.h; it, too, is not included from here. */
 static inline void wait_on_inode(struct inode *inode)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: reclaim the LRU lists full of dirty/writeback pages

[Rik van Riel]

On 02/11/2012 07:44 AM, Wu Fengguang wrote:

> Note that it's data for XFS. ext4 seems to have some problem with the
> workload: the majority pages are found to be writeback pages, and the
> flusher ends up blocking on the unconditional wait_on_page_writeback()
> in write_cache_pages_da() from time to time...
>
> XXX: commit NFS unstable pages via write_inode()
> XXX: the added congestion_wait() may be undesirable in some situations

Even with these caveats, this seems to be the right way forward.

> CC: Jan Kara<jack@suse.cz>
> CC: Mel Gorman<mgorman@suse.de>
> CC: Rik van Riel<riel@redhat.com>
> CC: Greg Thelen<gthelen@google.com>
> CC: Minchan Kim<minchan.kim@gmail.com>
> Signed-off-by: Wu Fengguang<fengguang.wu@intel.com>

Acked-by: Rik van Riel <riel@redhat.com>

-- 
All rights reversed

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: reclaim the LRU lists full of dirty/writeback pages

[Wu Fengguang]

On Sat, Feb 11, 2012 at 09:55:38AM -0500, Rik van Riel wrote:
> On 02/11/2012 07:44 AM, Wu Fengguang wrote:
> 
> >Note that it's data for XFS. ext4 seems to have some problem with the
> >workload: the majority pages are found to be writeback pages, and the
> >flusher ends up blocking on the unconditional wait_on_page_writeback()
> >in write_cache_pages_da() from time to time...

Sorry I overlooked the WB_SYNC_NONE test before the wait_on_page_writeback()
call! And the issue can no longer be reproduce anyway. ext4 performs pretty
good now, here is the result for one single memcg dd:

        dd if=/dev/zero of=/fs/f$i bs=4k count=1M

        4294967296 bytes (4.3 GB) copied, 44.5759 s, 96.4 MB/s

iostat -kx 3

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.25    0.00   11.03   28.54    0.00   60.19
           0.25    0.00   13.71   16.65    0.00   69.39
           0.17    0.00    8.41   24.81    0.00   66.61
           0.25    0.00   15.00   19.63    0.00   65.12

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00    17.00    0.00  178.33     0.00 90694.67  1017.14   111.34  520.88   5.45  97.23
sda               0.00     0.00    0.00  193.67     0.00 98816.00  1020.48    86.22  496.81   4.81  93.07
sda               0.00     3.33    0.00  182.33     0.00 92345.33  1012.93   101.14  623.98   5.49 100.03
sda               0.00     3.00    0.00  187.00     0.00 95586.67  1022.32    89.36  441.70   4.96  92.70

> >XXX: commit NFS unstable pages via write_inode()
> >XXX: the added congestion_wait() may be undesirable in some situations
> 
> Even with these caveats, this seems to be the right way forward.

> Acked-by: Rik van Riel <riel@redhat.com>

Thank you!
 
Here is the updated patch.
- ~10ms write around chunk size, adaptive to the bdi bandwith 
- cleanup flush_inode_page()

Thanks,
Fengguang
---
Subject: writeback: introduce the pageout work
Date: Thu Jul 29 14:41:19 CST 2010

This relays file pageout IOs to the flusher threads.

The ultimate target is to gracefully handle the LRU lists full of
dirty/writeback pages.

1) I/O efficiency

The flusher will piggy back the nearby ~10ms worth of dirty pages for I/O.

This takes advantage of the time/spacial locality in most workloads: the
nearby pages of one file are typically populated into the LRU at the same
time, hence will likely be close to each other in the LRU list. Writing
them in one shot helps clean more pages effectively for page reclaim.

2) OOM avoidance and scan rate control

Typically we do LRU scan w/o rate control and quickly get enough clean
pages for the LRU lists not full of dirty pages.

Or we can still get a number of freshly cleaned pages (moved to LRU tail
by end_page_writeback()) when the queued pageout I/O is completed within
tens of milli-seconds.

However if the LRU list is small and full of dirty pages, it can be
quickly fully scanned and go OOM before the flusher manages to clean
enough pages.

A simple yet reliable scheme is employed to avoid OOM and keep scan rate
in sync with the I/O rate:

	if (PageReclaim(page))
		congestion_wait(HZ/10);

PG_reclaim plays the key role. When dirty pages are encountered, we
queue I/O for it, set PG_reclaim and put it back to the LRU head.
So if PG_reclaim pages are encountered again, it means the dirty page
has not yet been cleaned by the flusher after a full zone scan. It
indicates we are scanning more fast than I/O and shall take a snap.

The runtime behavior on a fully dirtied small LRU list would be:
It will start with a quick scan of the list, queuing all pages for I/O.
Then the scan will be slowed down by the PG_reclaim pages *adaptively*
to match the I/O bandwidth.

3) writeback work coordinations

To avoid memory allocations at page reclaim, a mempool for struct
wb_writeback_work is created.

wakeup_flusher_threads() is removed because it can easily delay the
more oriented pageout works and even exhaust the mempool reservations.
It's also found to not I/O efficient by frequently submitting writeback
works with small ->nr_pages.

Background/periodic works will quit automatically, so as to clean the
pages under reclaim ASAP. However for now the sync work can still block
us for long time.

Jan Kara: limit the search scope. Note that the limited search and work
pool is not a big problem: 1000 IOs under flight are typically more than
enough to saturate the disk. And the overheads of searching in the work
list didn't even show up in the perf report.

4) test case

Run 2 dd tasks in a 100MB memcg (a very handy test case from Greg Thelen):

	mkdir /cgroup/x
	echo 100M > /cgroup/x/memory.limit_in_bytes
	echo $$ > /cgroup/x/tasks

	for i in `seq 2`
	do
		dd if=/dev/zero of=/fs/f$i bs=1k count=1M &
	done

Before patch, the dd tasks are quickly OOM killed.
After patch, they run well with reasonably good performance and overheads:

1073741824 bytes (1.1 GB) copied, 22.2196 s, 48.3 MB/s
1073741824 bytes (1.1 GB) copied, 22.4675 s, 47.8 MB/s

iostat -kx 1

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00    0.00  178.00     0.00 89568.00  1006.38    74.35  417.71   4.80  85.40
sda               0.00     2.00    0.00  191.00     0.00 94428.00   988.77    53.34  219.03   4.34  82.90
sda               0.00    20.00    0.00  196.00     0.00 97712.00   997.06    71.11  337.45   4.77  93.50
sda               0.00     5.00    0.00  175.00     0.00 84648.00   967.41    54.03  316.44   5.06  88.60
sda               0.00     0.00    0.00  186.00     0.00 92432.00   993.89    56.22  267.54   5.38 100.00
sda               0.00     1.00    0.00  183.00     0.00 90156.00   985.31    37.99  325.55   4.33  79.20
sda               0.00     0.00    0.00  175.00     0.00 88692.00  1013.62    48.70  218.43   4.69  82.10
sda               0.00     0.00    0.00  196.00     0.00 97528.00   995.18    43.38  236.87   5.10 100.00
sda               0.00     0.00    0.00  179.00     0.00 88648.00   990.48    45.83  285.43   5.59 100.00
sda               0.00     0.00    0.00  178.00     0.00 88500.00   994.38    28.28  158.89   4.99  88.80
sda               0.00     0.00    0.00  194.00     0.00 95852.00   988.16    32.58  167.39   5.15 100.00
sda               0.00     2.00    0.00  215.00     0.00 105996.00   986.01    41.72  201.43   4.65 100.00
sda               0.00     4.00    0.00  173.00     0.00 84332.00   974.94    50.48  260.23   5.76  99.60
sda               0.00     0.00    0.00  182.00     0.00 90312.00   992.44    36.83  212.07   5.49 100.00
sda               0.00     8.00    0.00  195.00     0.00 95940.50   984.01    50.18  221.06   5.13 100.00
sda               0.00     1.00    0.00  220.00     0.00 108852.00   989.56    40.99  202.68   4.55 100.00
sda               0.00     2.00    0.00  161.00     0.00 80384.00   998.56    37.19  268.49   6.21 100.00
sda               0.00     4.00    0.00  182.00     0.00 90830.00   998.13    50.58  239.77   5.49 100.00
sda               0.00     0.00    0.00  197.00     0.00 94877.00   963.22    36.68  196.79   5.08 100.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.25    0.00   15.08   33.92    0.00   50.75
           0.25    0.00   14.54   35.09    0.00   50.13
           0.50    0.00   13.57   32.41    0.00   53.52
           0.50    0.00   11.28   36.84    0.00   51.38
           0.50    0.00   15.75   32.00    0.00   51.75
           0.50    0.00   10.50   34.00    0.00   55.00
           0.50    0.00   17.63   27.46    0.00   54.41
           0.50    0.00   15.08   30.90    0.00   53.52
           0.50    0.00   11.28   32.83    0.00   55.39
           0.75    0.00   16.79   26.82    0.00   55.64
           0.50    0.00   16.08   29.15    0.00   54.27
           0.50    0.00   13.50   30.50    0.00   55.50
           0.50    0.00   14.32   35.18    0.00   50.00
           0.50    0.00   12.06   33.92    0.00   53.52
           0.50    0.00   17.29   30.58    0.00   51.63
           0.50    0.00   15.08   29.65    0.00   54.77
           0.50    0.00   12.53   29.32    0.00   57.64
           0.50    0.00   15.29   31.83    0.00   52.38

The global dd numbers for comparison:

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00    0.00  189.00     0.00 95752.00  1013.25   143.09  684.48   5.29 100.00
sda               0.00     0.00    0.00  208.00     0.00 105480.00  1014.23   143.06  733.29   4.81 100.00
sda               0.00     0.00    0.00  161.00     0.00 81924.00  1017.69   141.71  757.79   6.21 100.00
sda               0.00     0.00    0.00  217.00     0.00 109580.00  1009.95   143.09  749.55   4.61 100.10
sda               0.00     0.00    0.00  187.00     0.00 94728.00  1013.13   144.31  773.67   5.35 100.00
sda               0.00     0.00    0.00  189.00     0.00 95752.00  1013.25   144.14  742.00   5.29 100.00
sda               0.00     0.00    0.00  177.00     0.00 90032.00  1017.31   143.32  656.59   5.65 100.00
sda               0.00     0.00    0.00  215.00     0.00 108640.00  1010.60   142.90  817.54   4.65 100.00
sda               0.00     2.00    0.00  166.00     0.00 83858.00  1010.34   143.64  808.61   6.02 100.00
sda               0.00     0.00    0.00  186.00     0.00 92813.00   997.99   141.18  736.95   5.38 100.00
sda               0.00     0.00    0.00  206.00     0.00 104456.00  1014.14   146.27  729.33   4.85 100.00
sda               0.00     0.00    0.00  213.00     0.00 107024.00  1004.92   143.25  705.70   4.69 100.00
sda               0.00     0.00    0.00  188.00     0.00 95748.00  1018.60   141.82  764.78   5.32 100.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.51    0.00   11.22   52.30    0.00   35.97
           0.25    0.00   10.15   52.54    0.00   37.06
           0.25    0.00    5.01   56.64    0.00   38.10
           0.51    0.00   15.15   43.94    0.00   40.40
           0.25    0.00   12.12   48.23    0.00   39.39
           0.51    0.00   11.20   53.94    0.00   34.35
           0.26    0.00    9.72   51.41    0.00   38.62
           0.76    0.00    9.62   50.63    0.00   38.99
           0.51    0.00   10.46   53.32    0.00   35.71
           0.51    0.00    9.41   51.91    0.00   38.17
           0.25    0.00   10.69   49.62    0.00   39.44
           0.51    0.00   12.21   52.67    0.00   34.61
           0.51    0.00   11.45   53.18    0.00   34.86

XXX: commit NFS unstable pages via write_inode()
XXX: the added congestion_wait() may be undesirable in some situations

CC: Jan Kara <jack@suse.cz>
CC: Mel Gorman <mgorman@suse.de>
Acked-by: Rik van Riel <riel@redhat.com>
CC: Greg Thelen <gthelen@google.com>
CC: Minchan Kim <minchan.kim@gmail.com>
Signed-off-by: Wu Fengguang <fengguang.wu@intel.com>
---
 fs/fs-writeback.c                |  167 ++++++++++++++++++++++++++++-
 include/linux/writeback.h        |    4 
 include/trace/events/writeback.h |   12 +-
 mm/vmscan.c                      |   17 +-
 4 files changed, 186 insertions(+), 14 deletions(-)

--- linux.orig/mm/vmscan.c	2012-02-11 19:59:40.000000000 +0800
+++ linux/mm/vmscan.c	2012-02-11 20:07:48.000000000 +0800
@@ -813,6 +813,8 @@ static unsigned long shrink_page_list(st
 
 		if (PageWriteback(page)) {
 			nr_writeback++;
+			if (PageReclaim(page))
+				congestion_wait(BLK_RW_ASYNC, HZ/10);
 			/*
 			 * Synchronous reclaim cannot queue pages for
 			 * writeback due to the possibility of stack overflow
@@ -874,12 +876,15 @@ static unsigned long shrink_page_list(st
 			nr_dirty++;
 
 			/*
-			 * Only kswapd can writeback filesystem pages to
-			 * avoid risk of stack overflow but do not writeback
-			 * unless under significant pressure.
+			 * run into the visited page again: we are scanning
+			 * faster than the flusher can writeout dirty pages
 			 */
-			if (page_is_file_cache(page) &&
-					(!current_is_kswapd() || priority >= DEF_PRIORITY - 2)) {
+			if (page_is_file_cache(page) && PageReclaim(page)) {
+				congestion_wait(BLK_RW_ASYNC, HZ/10);
+				goto keep_locked;
+			}
+			if (page_is_file_cache(page) && mapping &&
+			    flush_inode_page(mapping, page, true) >= 0) {
 				/*
 				 * Immediately reclaim when written back.
 				 * Similar in principal to deactivate_page()
@@ -2382,8 +2387,6 @@ static unsigned long do_try_to_free_page
 		 */
 		writeback_threshold = sc->nr_to_reclaim + sc->nr_to_reclaim / 2;
 		if (total_scanned > writeback_threshold) {
-			wakeup_flusher_threads(laptop_mode ? 0 : total_scanned,
-						WB_REASON_TRY_TO_FREE_PAGES);
 			sc->may_writepage = 1;
 		}
 
--- linux.orig/fs/fs-writeback.c	2012-02-11 19:59:40.000000000 +0800
+++ linux/fs/fs-writeback.c	2012-02-12 10:54:05.000000000 +0800
@@ -41,6 +41,8 @@ struct wb_writeback_work {
 	long nr_pages;
 	struct super_block *sb;
 	unsigned long *older_than_this;
+	struct inode *inode;
+	pgoff_t offset;
 	enum writeback_sync_modes sync_mode;
 	unsigned int tagged_writepages:1;
 	unsigned int for_kupdate:1;
@@ -65,6 +67,27 @@ struct wb_writeback_work {
  */
 int nr_pdflush_threads;
 
+static mempool_t *wb_work_mempool;
+
+static void *wb_work_alloc(gfp_t gfp_mask, void *pool_data)
+{
+	/*
+	 * bdi_flush_inode_range() may be called on page reclaim
+	 */
+	if (current->flags & PF_MEMALLOC)
+		return NULL;
+
+	return kmalloc(sizeof(struct wb_writeback_work), gfp_mask);
+}
+
+static __init int wb_work_init(void)
+{
+	wb_work_mempool = mempool_create(1024,
+					 wb_work_alloc, mempool_kfree, NULL);
+	return wb_work_mempool ? 0 : -ENOMEM;
+}
+fs_initcall(wb_work_init);
+
 /**
  * writeback_in_progress - determine whether there is writeback in progress
  * @bdi: the device's backing_dev_info structure.
@@ -129,7 +152,7 @@ __bdi_start_writeback(struct backing_dev
 	 * This is WB_SYNC_NONE writeback, so if allocation fails just
 	 * wakeup the thread for old dirty data writeback
 	 */
-	work = kzalloc(sizeof(*work), GFP_ATOMIC);
+	work = mempool_alloc(wb_work_mempool, GFP_NOWAIT);
 	if (!work) {
 		if (bdi->wb.task) {
 			trace_writeback_nowork(bdi);
@@ -138,6 +161,7 @@ __bdi_start_writeback(struct backing_dev
 		return;
 	}
 
+	memset(work, 0, sizeof(*work));
 	work->sync_mode	= WB_SYNC_NONE;
 	work->nr_pages	= nr_pages;
 	work->range_cyclic = range_cyclic;
@@ -186,6 +210,123 @@ void bdi_start_background_writeback(stru
 	spin_unlock_bh(&bdi->wb_lock);
 }
 
+static bool extend_writeback_range(struct wb_writeback_work *work,
+				   pgoff_t offset,
+				   unsigned long write_around_pages)
+{
+	pgoff_t end = work->offset + work->nr_pages;
+
+	if (offset >= work->offset && offset < end)
+		return true;
+
+	/*
+	 * for sequential workloads with good locality, include up to 8 times
+	 * more data in one chunk
+	 */
+	if (work->nr_pages >= 8 * write_around_pages)
+		return false;
+
+	/* the unsigned comparison helps eliminate one compare */
+	if (work->offset - offset < write_around_pages) {
+		work->nr_pages += write_around_pages;
+		work->offset -= write_around_pages;
+		return true;
+	}
+
+	if (offset - end < write_around_pages) {
+		work->nr_pages += write_around_pages;
+		return true;
+	}
+
+	return false;
+}
+
+/*
+ * schedule writeback on a range of inode pages.
+ */
+static struct wb_writeback_work *
+bdi_flush_inode_range(struct backing_dev_info *bdi,
+		      struct inode *inode,
+		      pgoff_t offset,
+		      pgoff_t len,
+		      bool wait)
+{
+	struct wb_writeback_work *work;
+
+	if (!igrab(inode))
+		return ERR_PTR(-ENOENT);
+
+	work = mempool_alloc(wb_work_mempool, wait ? GFP_NOIO : GFP_NOWAIT);
+	if (!work)
+		return ERR_PTR(-ENOMEM);
+
+	memset(work, 0, sizeof(*work));
+	work->sync_mode		= WB_SYNC_NONE;
+	work->inode		= inode;
+	work->offset		= offset;
+	work->nr_pages		= len;
+	work->reason		= WB_REASON_PAGEOUT;
+
+	bdi_queue_work(bdi, work);
+
+	return work;
+}
+
+/*
+ * Called by page reclaim code to flush the dirty page ASAP. Do write-around to
+ * improve IO throughput. The nearby pages will have good chance to reside in
+ * the same LRU list that vmscan is working on, and even close to each other
+ * inside the LRU list in the common case of sequential read/write.
+ *
+ * ret > 0: success, found/reused a previous writeback work
+ * ret = 0: success, allocated/queued a new writeback work
+ * ret < 0: failed
+ */
+long flush_inode_page(struct address_space *mapping,
+		      struct page *page,
+		      bool wait)
+{
+	struct backing_dev_info *bdi = mapping->backing_dev_info;
+	struct inode *inode = mapping->host;
+	struct wb_writeback_work *work;
+	unsigned long write_around_pages;
+	pgoff_t offset = page->index;
+	int i;
+	long ret = 0;
+
+	if (unlikely(!inode))
+		return -ENOENT;
+
+	/*
+	 * piggy back 8-15ms worth of data
+	 */
+	write_around_pages = bdi->avg_write_bandwidth + MIN_WRITEBACK_PAGES;
+	write_around_pages = rounddown_pow_of_two(write_around_pages) >> 6;
+
+	i = 1;
+	spin_lock_bh(&bdi->wb_lock);
+	list_for_each_entry_reverse(work, &bdi->work_list, list) {
+		if (work->inode != inode)
+			continue;
+		if (extend_writeback_range(work, offset, write_around_pages)) {
+			ret = i;
+			break;
+		}
+		if (i++ > 100)	/* limit search depth */
+			break;
+	}
+	spin_unlock_bh(&bdi->wb_lock);
+
+	if (!ret) {
+		offset = round_down(offset, write_around_pages);
+		work = bdi_flush_inode_range(bdi, inode,
+					     offset, write_around_pages, wait);
+		if (IS_ERR(work))
+			ret = PTR_ERR(work);
+	}
+	return ret;
+}
+
 /*
  * Remove the inode from the writeback list it is on.
  */
@@ -833,6 +974,23 @@ static unsigned long get_nr_dirty_pages(
 		get_nr_dirty_inodes();
 }
 
+static long wb_flush_inode(struct bdi_writeback *wb,
+			   struct wb_writeback_work *work)
+{
+	struct writeback_control wbc = {
+		.sync_mode = WB_SYNC_NONE,
+		.nr_to_write = LONG_MAX,
+		.range_start = work->offset << PAGE_CACHE_SHIFT,
+		.range_end = (work->offset + work->nr_pages - 1)
+						<< PAGE_CACHE_SHIFT,
+	};
+
+	do_writepages(work->inode->i_mapping, &wbc);
+	iput(work->inode);
+
+	return LONG_MAX - wbc.nr_to_write;
+}
+
 static long wb_check_background_flush(struct bdi_writeback *wb)
 {
 	if (over_bground_thresh(wb->bdi)) {
@@ -905,7 +1063,10 @@ long wb_do_writeback(struct bdi_writebac
 
 		trace_writeback_exec(bdi, work);
 
-		wrote += wb_writeback(wb, work);
+		if (work->inode)
+			wrote += wb_flush_inode(wb, work);
+		else
+			wrote += wb_writeback(wb, work);
 
 		/*
 		 * Notify the caller of completion if this is a synchronous
@@ -914,7 +1075,7 @@ long wb_do_writeback(struct bdi_writebac
 		if (work->done)
 			complete(work->done);
 		else
-			kfree(work);
+			mempool_free(work, wb_work_mempool);
 	}
 
 	/*
--- linux.orig/include/trace/events/writeback.h	2012-02-11 19:59:40.000000000 +0800
+++ linux/include/trace/events/writeback.h	2012-02-11 20:07:48.000000000 +0800
@@ -23,7 +23,7 @@
 
 #define WB_WORK_REASON							\
 		{WB_REASON_BACKGROUND,		"background"},		\
-		{WB_REASON_TRY_TO_FREE_PAGES,	"try_to_free_pages"},	\
+		{WB_REASON_PAGEOUT,		"pageout"},		\
 		{WB_REASON_SYNC,		"sync"},		\
 		{WB_REASON_PERIODIC,		"periodic"},		\
 		{WB_REASON_LAPTOP_TIMER,	"laptop_timer"},	\
@@ -45,6 +45,8 @@ DECLARE_EVENT_CLASS(writeback_work_class
 		__field(int, range_cyclic)
 		__field(int, for_background)
 		__field(int, reason)
+		__field(unsigned long, ino)
+		__field(unsigned long, offset)
 	),
 	TP_fast_assign(
 		strncpy(__entry->name, dev_name(bdi->dev), 32);
@@ -55,9 +57,11 @@ DECLARE_EVENT_CLASS(writeback_work_class
 		__entry->range_cyclic = work->range_cyclic;
 		__entry->for_background	= work->for_background;
 		__entry->reason = work->reason;
+		__entry->ino = work->inode ? work->inode->i_ino : 0;
+		__entry->offset = work->offset;
 	),
 	TP_printk("bdi %s: sb_dev %d:%d nr_pages=%ld sync_mode=%d "
-		  "kupdate=%d range_cyclic=%d background=%d reason=%s",
+		  "kupdate=%d range_cyclic=%d background=%d reason=%s ino=%lu offset=%lu",
 		  __entry->name,
 		  MAJOR(__entry->sb_dev), MINOR(__entry->sb_dev),
 		  __entry->nr_pages,
@@ -65,7 +69,9 @@ DECLARE_EVENT_CLASS(writeback_work_class
 		  __entry->for_kupdate,
 		  __entry->range_cyclic,
 		  __entry->for_background,
-		  __print_symbolic(__entry->reason, WB_WORK_REASON)
+		  __print_symbolic(__entry->reason, WB_WORK_REASON),
+		  __entry->ino,
+		  __entry->offset
 	)
 );
 #define DEFINE_WRITEBACK_WORK_EVENT(name) \
--- linux.orig/include/linux/writeback.h	2012-02-11 19:59:40.000000000 +0800
+++ linux/include/linux/writeback.h	2012-02-11 20:07:48.000000000 +0800
@@ -40,7 +40,7 @@ enum writeback_sync_modes {
  */
 enum wb_reason {
 	WB_REASON_BACKGROUND,
-	WB_REASON_TRY_TO_FREE_PAGES,
+	WB_REASON_PAGEOUT,
 	WB_REASON_SYNC,
 	WB_REASON_PERIODIC,
 	WB_REASON_LAPTOP_TIMER,
@@ -94,6 +94,8 @@ long writeback_inodes_wb(struct bdi_writ
 				enum wb_reason reason);
 long wb_do_writeback(struct bdi_writeback *wb, int force_wait);
 void wakeup_flusher_threads(long nr_pages, enum wb_reason reason);
+long flush_inode_page(struct address_space *mapping, struct page *page,
+		      bool wait);
 
 /* writeback.h requires fs.h; it, too, is not included from here. */
 static inline void wait_on_inode(struct inode *inode)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: reclaim the LRU lists full of dirty/writeback pages

[Wu Fengguang]

[-- Attachment #1: Type: text/plain, Size: 409223 bytes --]

> ext4 performs pretty good now, here is the result for one single
> memcg dd:
> 
>         dd if=/dev/zero of=/fs/f$i bs=4k count=1M

Here is one full writeback trace collected with

echo 1 > /debug/tracing/events/writeback/enable
echo 0 > /debug/tracing/events/writeback/global_dirty_state/enable
echo 0 > /debug/tracing/events/writeback/wbc_writepage/enable
echo 0 > /debug/tracing/events/writeback/writeback_wait_iff_congested/enable

Looking at the reason=pageout lines, the chunk size at queue time is
mostly nr_pages=256, and since it's a sequential dd with very good
locality, lots of the pageout works have been extended to
nr_pages=2048 at the time of execution.

The progress is a bit bumpy in that the writeback_queue/writeback_exec
lines and the writeback_congestion_wait lines tend to come together in
batches. According to the attached graphs (generated on a private
task_io trace event), the dd write progress is reasonably smooth.
There are several sudden jump of both x and y values, maybe it's
caused by lost trace samples.

# tracer: nop
#
# entries-in-buffer/entries-written: 127889/168807   #P:4
#
#                              _-----=> irqs-off
#                             / _----=> need-resched
#                            | / _---=> hardirq/softirq
#                            || / _--=> preempt-depth
#                            ||| /     delay
#           TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION
#              | |       |   ||||       |         |
              dd-4267  [002] ....  8250.326788: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8250.335124: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8250.343750: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8250.356916: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [002] ....  8250.361203: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [002] ....  8251.323386: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [002] ....  8251.363306: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [002] ....  8251.405320: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [002] ....  8251.427430: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [002] ....  8251.458557: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [002] ....  8251.476925: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8251.592810: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [002] ....  8251.647229: writeback_congestion_wait: usec_timeout=100000 usec_delayed=54000
              dd-4267  [002] ....  8251.682142: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [002] ....  8251.728420: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [002] ....  8251.754083: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
       flush-8:0-4272  [002] ....  8251.819186: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=24320
       flush-8:0-4272  [002] ....  8251.823270: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=24320
       flush-8:0-4272  [002] ....  8251.826068: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=26368
       flush-8:0-4272  [002] ....  8251.832939: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=28416
       flush-8:0-4272  [002] ....  8251.839785: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=30464
       flush-8:0-4272  [002] ....  8251.846554: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=32512
       flush-8:0-4272  [002] ....  8251.852810: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=34560
       flush-8:0-4272  [002] ....  8251.858865: writeback_start: bdi 8:0: sb_dev 0:0 nr_pages=9223372036854775807 sync_mode=0 kupdate=0 range_cyclic=1 background=1 reason=background ino=0 offset=0
       flush-8:0-4272  [002] ....  8251.858866: writeback_queue_io: bdi 8:0: older=4302923541 age=0 enqueue=0 reason=background
       flush-8:0-4272  [002] ....  8251.860726: writeback_single_inode: bdi 8:0: ino=13 state=I_DIRTY_PAGES dirtied_when=4302921148 age=55 index=37632 to_write=1024 wrote=1024
       flush-8:0-4272  [002] ....  8251.860729: writeback_written: bdi 8:0: sb_dev 0:0 nr_pages=9223372036854774783 sync_mode=0 kupdate=0 range_cyclic=1 background=1 reason=background ino=0 offset=0
       flush-8:0-4272  [002] ....  8251.860730: writeback_start: bdi 8:0: sb_dev 0:0 nr_pages=9223372036854774783 sync_mode=0 kupdate=0 range_cyclic=1 background=1 reason=background ino=0 offset=0
       flush-8:0-4272  [002] ....  8251.860731: writeback_queue_io: bdi 8:0: older=4302923543 age=0 enqueue=0 reason=background
       flush-8:0-4272  [002] ....  8252.607338: writeback_single_inode: bdi 8:0: ino=13 state=I_DIRTY_PAGES dirtied_when=4302921148 age=55 index=51712 to_write=1024 wrote=1024
       flush-8:0-4272  [002] ....  8252.607341: writeback_written: bdi 8:0: sb_dev 0:0 nr_pages=9223372036854774783 sync_mode=0 kupdate=0 range_cyclic=1 background=1 reason=background ino=0 offset=0
       flush-8:0-4272  [002] ....  8252.607342: writeback_start: bdi 8:0: sb_dev 0:0 nr_pages=9223372036854774783 sync_mode=0 kupdate=0 range_cyclic=1 background=1 reason=background ino=0 offset=0
       flush-8:0-4272  [002] ....  8252.607343: writeback_queue_io: bdi 8:0: older=4302924290 age=0 enqueue=0 reason=background
       flush-8:0-4272  [002] ....  8253.026787: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=63232
       flush-8:0-4272  [002] ....  8253.026915: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=512 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=63232
       flush-8:0-4272  [002] ....  8253.027360: writeback_pages_written: 316
       flush-8:0-4272  [002] ....  8253.027437: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=63744
       flush-8:0-4272  [002] ....  8253.029744: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=63744
       flush-8:0-4272  [002] ....  8253.030492: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1024 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=64512
       flush-8:0-4272  [002] ....  8253.033488: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=65280
       flush-8:0-4272  [002] ....  8253.036827: writeback_pages_written: 3584
       flush-8:0-4272  [002] ....  8253.040560: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=67328
       flush-8:0-4272  [002] ....  8253.042129: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=67328
       flush-8:0-4272  [002] ....  8253.045098: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=69376
       flush-8:0-4272  [002] ....  8253.050144: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=71424
##### CPU 3 buffer started ####
       flush-8:0-4272  [002] ....  8253.485155: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=75008
       flush-8:0-4272  [002] ....  8253.488248: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=75008
       flush-8:0-4272  [002] ....  8253.488254: writeback_pages_written: 256
              dd-4267  [003] ....  8253.587708: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [002] ....  8253.616408: writeback_congestion_wait: usec_timeout=100000 usec_delayed=28000
              dd-4267  [002] ....  8253.627826: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
       flush-8:0-4272  [002] ....  8253.678663: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=75264
       flush-8:0-4272  [002] ....  8253.682232: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=75264
       flush-8:0-4272  [002] ....  8253.682240: writeback_start: bdi 8:0: sb_dev 0:0 nr_pages=9223372036854775807 sync_mode=0 kupdate=0 range_cyclic=1 background=1 reason=background ino=0 offset=0
       flush-8:0-4272  [002] ....  8253.682240: writeback_queue_io: bdi 8:0: older=4302925365 age=0 enqueue=0 reason=background
       flush-8:0-4272  [002] ....  8253.684064: writeback_single_inode: bdi 8:0: ino=13 state=I_DIRTY_PAGES dirtied_when=4302921148 age=55 index=76544 to_write=1024 wrote=1024
       flush-8:0-4272  [002] ....  8253.684067: writeback_written: bdi 8:0: sb_dev 0:0 nr_pages=9223372036854774783 sync_mode=0 kupdate=0 range_cyclic=1 background=1 reason=background ino=0 offset=0
       flush-8:0-4272  [002] ....  8253.684069: writeback_pages_written: 1280
              dd-4267  [002] ....  8253.730357: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [002] ....  8253.814751: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [002] ....  8253.850639: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8253.870770: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [002] ....  8253.897749: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [002] ....  8254.014484: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [002] ....  8254.113342: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
       flush-8:0-4272  [002] ....  8254.123898: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=76544
       flush-8:0-4272  [002] ....  8254.128327: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=76544
       flush-8:0-4272  [002] ....  8254.131253: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=78592
       flush-8:0-4272  [002] ....  8254.138062: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=80640
       flush-8:0-4272  [002] ....  8254.144703: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=82688
       flush-8:0-4272  [002] ....  8254.230122: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=84736
       flush-8:0-4272  [002] ....  8254.235649: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=86784
       flush-8:0-4272  [002] ....  8254.241491: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=88832
       flush-8:0-4272  [002] ....  8254.246782: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1024 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=90880
       flush-8:0-4272  [002] ....  8254.249905: writeback_pages_written: 15360
       flush-8:0-4272  [002] ....  8254.282817: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=91904
       flush-8:0-4272  [002] ....  8254.285648: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=91904
       flush-8:0-4272  [002] ....  8254.289279: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1280 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=93952
       flush-8:0-4272  [002] ....  8254.292899: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=94976
       flush-8:0-4272  [002] ....  8254.295917: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=97024
       flush-8:0-4272  [002] ....  8254.299985: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=99072
       flush-8:0-4272  [002] ....  8254.300865: writeback_pages_written: 7424
              dd-4267  [003] ....  8254.393293: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [002] ....  8254.536096: writeback_congestion_wait: usec_timeout=100000 usec_delayed=82000
              dd-4267  [002] ....  8254.625314: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=99328
              dd-4267  [002] ....  8254.625340: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=99328
              dd-4267  [002] ....  8254.626413: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [002] ....  8254.630694: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
      flush-0:16-4223  [003] ....  8255.307759: writeback_start: bdi 0:16: sb_dev 0:0 nr_pages=45399 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
      flush-0:16-4223  [003] ....  8255.307762: writeback_queue_io: bdi 0:16: older=4302896992 age=30000 enqueue=0 reason=periodic
      flush-0:16-4223  [003] ....  8255.307762: writeback_written: bdi 0:16: sb_dev 0:0 nr_pages=45399 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
      flush-0:16-4223  [003] ....  8255.307764: writeback_pages_written: 0
              dd-4267  [002] ....  8255.318648: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=99584
              dd-4267  [002] ....  8255.318664: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=99584
              dd-4267  [002] ....  8255.320138: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=101632
              dd-4267  [002] ....  8255.321638: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=103680
              dd-4267  [002] ....  8255.323080: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=105728
              dd-4267  [002] ....  8255.324538: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=107776
              dd-4267  [002] ....  8255.326007: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=109824
              dd-4267  [002] ....  8255.327498: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=111872
              dd-4267  [002] ....  8255.328994: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=113920
              dd-4267  [002] ....  8255.330478: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=115968
              dd-4267  [002] ....  8255.332007: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=118016
              dd-4267  [002] ....  8255.333557: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=120064
              dd-4267  [002] ....  8255.335096: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=122112
              dd-4267  [002] ....  8255.341094: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8255.345525: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [002] ....  8255.350156: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
       flush-8:0-4272  [003] ....  8255.767124: writeback_single_inode: bdi 8:0: ino=13 state=I_DIRTY_SYNC|I_DIRTY_DATASYNC|I_DIRTY_PAGES dirtied_when=4302921148 age=55 index=127007 to_write=1024 wrote=2079
       flush-8:0-4272  [003] ....  8255.767127: writeback_written: bdi 8:0: sb_dev 0:0 nr_pages=21763 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
       flush-8:0-4272  [003] ....  8255.767128: writeback_start: bdi 8:0: sb_dev 0:0 nr_pages=21763 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
       flush-8:0-4272  [003] ....  8255.767128: writeback_queue_io: bdi 8:0: older=4302897451 age=30000 enqueue=0 reason=periodic
              dd-4267  [002] ....  8255.913486: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
       flush-8:0-4272  [003] ....  8256.148983: writeback_single_inode: bdi 8:0: ino=13 state=I_DIRTY_PAGES dirtied_when=4302921148 age=55 index=141755 to_write=1024 wrote=5901
       flush-8:0-4272  [003] ....  8256.148987: writeback_written: bdi 8:0: sb_dev 0:0 nr_pages=7015 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
       flush-8:0-4272  [003] ....  8256.148988: writeback_start: bdi 8:0: sb_dev 0:0 nr_pages=7015 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
       flush-8:0-4272  [003] ....  8256.148988: writeback_queue_io: bdi 8:0: older=4302897833 age=30000 enqueue=0 reason=periodic
              dd-4267  [002] ....  8256.369200: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
       flush-8:0-4272  [003] ....  8256.628529: writeback_single_inode: bdi 8:0: ino=13 state=I_DIRTY_PAGES dirtied_when=4302921148 age=55 index=150440 to_write=1024 wrote=6452
       flush-8:0-4272  [003] ....  8256.628533: writeback_written: bdi 8:0: sb_dev 0:0 nr_pages=-1670 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
       flush-8:0-4272  [003] ....  8256.628536: writeback_pages_written: 50856
       flush-8:0-4272  [003] ....  8256.628538: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=147456
       flush-8:0-4272  [003] ....  8256.628545: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=149504
       flush-8:0-4272  [003] ....  8256.630190: writeback_pages_written: 1112
              dd-4267  [002] ....  8256.693086: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [003] ....  8256.909199: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [003] ....  8257.016875: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [003] ....  8257.041563: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [003] ....  8257.106900: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [003] ....  8257.246224: writeback_congestion_wait: usec_timeout=100000 usec_delayed=8000
              dd-4267  [003] ....  8257.271030: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [003] ....  8257.348551: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [003] ....  8257.381224: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
       flush-8:0-4272  [003] ....  8257.393446: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=151552
       flush-8:0-4272  [003] ....  8257.398224: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=151552
       flush-8:0-4272  [003] ....  8257.400851: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=153600
              dd-4267  [003] ....  8257.434020: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [003] ....  8257.541621: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [003] ....  8257.840373: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [002] ....  8258.031388: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [002] ....  8258.054534: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
       flush-8:0-4272  [002] ....  8258.104190: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=175872
       flush-8:0-4272  [002] ....  8258.107511: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=512 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=175872
       flush-8:0-4272  [002] ....  8258.107923: writeback_pages_written: 512
              dd-4267  [002] ....  8258.126205: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [002] ....  8258.130305: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=176384
              dd-4267  [002] ....  8258.130881: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [002] ....  8258.133116: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [002] ....  8258.137594: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [002] ....  8258.257151: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [002] ....  8258.357770: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [002] ....  8258.406325: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [002] ....  8258.432124: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [002] ....  8258.452275: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [002] ....  8258.472362: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [002] ....  8258.492576: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
       flush-8:0-4272  [002] ....  8258.518908: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=176640
       flush-8:0-4272  [002] ....  8258.523446: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=176640
       flush-8:0-4272  [002] ....  8258.523453: writeback_pages_written: 256
              dd-4267  [002] ....  8258.545795: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
       flush-8:0-4272  [002] ....  8258.559572: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=176896
       flush-8:0-4272  [002] ....  8258.564143: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=176896
       flush-8:0-4272  [002] ....  8258.567279: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=178944
       flush-8:0-4272  [002] ....  8258.575307: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=180992
              dd-4267  [002] ....  8258.779190: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=192256
              dd-4267  [002] ....  8258.779212: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=192256
              dd-4267  [002] ....  8258.784476: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=193280
              dd-4267  [002] ....  8258.784497: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=193280
              dd-4267  [002] ....  8258.786546: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=195328
              dd-4267  [002] ....  8258.788222: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=197376
              dd-4267  [002] ....  8258.789834: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=199424
              dd-4267  [003] ....  8259.091794: writeback_congestion_wait: usec_timeout=100000 usec_delayed=9000
              dd-4267  [003] ....  8259.103932: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [003] ....  8259.142635: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
       flush-8:0-4272  [003] ....  8259.166727: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=200448
       flush-8:0-4272  [003] ....  8259.169783: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=200448
       flush-8:0-4272  [003] ....  8259.169790: writeback_pages_written: 256
              dd-4267  [003] ....  8259.183714: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
       flush-8:0-4272  [003] ....  8259.211788: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=200704
       flush-8:0-4272  [003] ....  8259.214924: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=200704
       flush-8:0-4272  [003] ....  8259.217800: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=202752
       flush-8:0-4272  [003] ....  8259.222920: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1024 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=204800
       flush-8:0-4272  [003] ....  8259.226184: writeback_pages_written: 5120
              dd-4267  [003] ....  8259.257092: writeback_congestion_wait: usec_timeout=100000 usec_delayed=35000
              dd-4267  [003] ....  8259.275740: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=205824
              dd-4267  [003] ....  8259.275756: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=205824
              dd-4267  [003] ....  8259.277624: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [003] ....  8259.298203: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [003] ....  8259.318745: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [003] ....  8259.451826: writeback_congestion_wait: usec_timeout=100000 usec_delayed=54000
              dd-4267  [003] ....  8259.465856: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [003] ....  8259.499584: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [003] ....  8259.518780: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [003] ....  8259.538457: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [003] ....  8259.558070: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
       flush-8:0-4272  [003] ....  8259.570146: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=206080
       flush-8:0-4272  [003] ....  8259.573737: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=512 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=206080
       flush-8:0-4272  [003] ....  8259.574164: writeback_pages_written: 512
              dd-4267  [003] ....  8259.577670: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
       flush-8:0-4272  [003] ....  8259.594106: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=206592
       flush-8:0-4272  [003] ....  8259.597848: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=206592
       flush-8:0-4272  [003] ....  8259.600521: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=208640
       flush-8:0-4272  [003] ....  8259.606747: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=210688
       flush-8:0-4272  [003] ....  8259.612775: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=212736
       flush-8:0-4272  [003] ....  8259.618277: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1792 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=214784
              dd-4267  [003] ....  8259.656171: writeback_congestion_wait: usec_timeout=100000 usec_delayed=37000
              dd-4267  [003] ....  8259.890390: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=216576
              dd-4267  [003] ....  8259.890404: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=216576
              dd-4267  [003] ....  8259.891900: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=218624
              dd-4267  [003] ....  8259.895754: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=220672
              dd-4267  [003] ....  8259.897308: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=222720
              dd-4267  [003] ....  8259.902780: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [003] ....  8259.908417: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [003] ....  8260.079480: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [003] ....  8260.084487: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=224768
              dd-4267  [003] ....  8260.084506: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=224768
              dd-4267  [003] ....  8260.107416: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [003] ....  8260.173675: writeback_congestion_wait: usec_timeout=100000 usec_delayed=38000
              dd-4267  [003] ....  8261.028667: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [003] ....  8261.148521: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
       flush-8:0-4272  [002] ....  8261.283270: writeback_start: bdi 8:0: sb_dev 0:0 nr_pages=26057 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
       flush-8:0-4272  [002] ....  8261.283272: writeback_queue_io: bdi 8:0: older=4302902970 age=30000 enqueue=0 reason=periodic
       flush-8:0-4272  [002] ....  8261.285234: writeback_single_inode: bdi 8:0: ino=13 state=I_DIRTY_PAGES dirtied_when=4302921148 age=55 index=250624 to_write=1024 wrote=1024
       flush-8:0-4272  [002] ....  8261.285237: writeback_written: bdi 8:0: sb_dev 0:0 nr_pages=25033 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
       flush-8:0-4272  [002] ....  8261.285237: writeback_start: bdi 8:0: sb_dev 0:0 nr_pages=25033 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
       flush-8:0-4272  [002] ....  8261.285238: writeback_queue_io: bdi 8:0: older=4302902972 age=30000 enqueue=0 reason=periodic
       flush-8:0-4272  [002] ....  8261.556566: writeback_single_inode: bdi 8:0: ino=13 state=I_DIRTY_PAGES dirtied_when=4302921148 age=55 index=258435 to_write=1024 wrote=4472
       flush-8:0-4272  [002] ....  8261.556570: writeback_written: bdi 8:0: sb_dev 0:0 nr_pages=17222 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
       flush-8:0-4272  [002] ....  8261.556571: writeback_start: bdi 8:0: sb_dev 0:0 nr_pages=17222 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
       flush-8:0-4272  [002] ....  8261.556571: writeback_queue_io: bdi 8:0: older=4302903244 age=30000 enqueue=0 reason=periodic
              dd-4267  [003] ....  8261.828188: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [003] ....  8261.930822: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=272384
              dd-4267  [003] ....  8261.934217: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=274432
              dd-4267  [003] ....  8262.188991: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
       flush-8:0-4272  [002] ....  8262.287291: writeback_single_inode: bdi 8:0: ino=13 state=I_DIRTY_PAGES dirtied_when=4302921148 age=55 index=276437 to_write=1024 wrote=6356
       flush-8:0-4272  [002] ....  8262.287296: writeback_written: bdi 8:0: sb_dev 0:0 nr_pages=-780 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
       flush-8:0-4272  [002] ....  8262.287298: writeback_pages_written: 51413
       flush-8:0-4272  [002] ....  8262.287300: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=272384
       flush-8:0-4272  [002] ....  8262.287308: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1280 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=274432
       flush-8:0-4272  [002] ....  8262.287312: writeback_pages_written: 0
##### CPU 1 buffer started ####
              dd-4267  [001] ....  8262.412091: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=276224
       flush-8:0-4272  [002] ....  8262.412109: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=276224
              dd-4267  [001] ....  8262.412113: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=276224
       flush-8:0-4272  [002] ....  8262.412273: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=512 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=276224
              dd-4267  [001] ....  8262.412843: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
       flush-8:0-4272  [002] ....  8262.413228: writeback_pages_written: 299
              dd-4267  [002] ....  8262.432804: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [002] ....  8262.452739: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [002] ....  8262.472704: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [002] ....  8262.546051: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [002] ....  8262.585998: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [002] ....  8262.635683: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8262.655672: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8262.675597: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8262.695554: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8262.717581: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [002] ....  8262.738142: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [002] ....  8262.758666: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [002] ....  8262.780860: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [002] ....  8262.801393: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8262.811726: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [002] ....  8262.821976: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [002] ....  8262.842505: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
       flush-8:0-4272  [002] ....  8262.855614: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=276736
       flush-8:0-4272  [002] ....  8262.862079: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=276736
              dd-4267  [001] ....  8262.863088: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=280832
              dd-4267  [001] ....  8262.864811: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=282880
       flush-8:0-4272  [002] ....  8262.865154: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=278784
              dd-4267  [001] ....  8262.866948: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=284928
              dd-4267  [001] ....  8262.868516: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=286976
              dd-4267  [002] ....  8262.873680: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
       flush-8:0-4272  [001] ....  8263.074676: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=282880
       flush-8:0-4272  [001] ....  8263.082377: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=284928
       flush-8:0-4272  [001] ....  8263.088659: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1792 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=286976
       flush-8:0-4272  [001] ....  8263.093872: writeback_pages_written: 12032
              dd-4267  [002] ....  8263.162787: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=288768
       flush-8:0-4272  [001] ....  8263.162806: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=288768
              dd-4267  [002] ....  8263.162807: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=288768
              dd-4267  [002] ....  8263.166798: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=290816
       flush-8:0-4272  [001] ....  8263.167706: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=288768
              dd-4267  [002] ....  8263.168416: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=292864
              dd-4267  [002] ....  8263.170061: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=294912
       flush-8:0-4272  [001] ....  8263.172041: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=290816
              dd-4267  [002] ....  8263.174392: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=296960
              dd-4267  [002] ....  8263.178723: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=299008
       flush-8:0-4272  [001] ....  8263.178800: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=292864
              dd-4267  [002] ....  8263.180744: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [002] ....  8263.184819: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
       flush-8:0-4272  [001] ....  8263.184948: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=294912
       flush-8:0-4272  [001] ....  8263.189635: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=296960
       flush-8:0-4272  [001] ....  8263.193882: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1280 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=299008
       flush-8:0-4272  [001] ....  8263.196897: writeback_pages_written: 11520
              dd-4267  [001] ....  8263.371538: writeback_congestion_wait: usec_timeout=100000 usec_delayed=22000
              dd-4267  [001] ....  8263.391105: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [001] ....  8263.397755: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [001] ....  8263.417164: writeback_congestion_wait: usec_timeout=100000 usec_delayed=15000
              dd-4267  [003] ....  8263.421709: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [001] ....  8263.424214: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [001] ....  8263.466360: writeback_congestion_wait: usec_timeout=100000 usec_delayed=43000
              dd-4267  [001] ....  8263.482457: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=300288
       flush-8:0-4272  [003] ....  8263.482473: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=300288
              dd-4267  [001] ....  8263.482480: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=300288
              dd-4267  [001] ....  8263.483957: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=302336
              dd-4267  [001] ....  8263.485413: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=304384
       flush-8:0-4272  [003] ....  8263.485563: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=300288
       flush-8:0-4272  [003] ....  8263.488444: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=302336
       flush-8:0-4272  [003] ....  8263.493375: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1792 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=304384
       flush-8:0-4272  [003] ....  8263.497569: writeback_pages_written: 5888
              dd-4267  [003] ....  8263.586187: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [003] ....  8263.621915: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=306176
       flush-8:0-4272  [001] ....  8263.621932: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=306176
              dd-4267  [003] ....  8263.621932: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=306176
       flush-8:0-4272  [001] ....  8263.624277: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=306176
       flush-8:0-4272  [001] ....  8263.624284: writeback_pages_written: 256
              dd-4267  [001] ....  8263.713731: writeback_congestion_wait: usec_timeout=100000 usec_delayed=92000
              dd-4267  [003] ....  8263.762825: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [001] ....  8263.772571: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8263.782397: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8263.786346: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [003] ....  8263.792063: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8263.794373: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [001] ....  8263.802026: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [001] ....  8263.823655: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=306432
       flush-8:0-4272  [003] ....  8263.823672: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=306432
              dd-4267  [001] ....  8263.823673: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=306432
              dd-4267  [001] ....  8263.826272: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [001] ....  8263.828813: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [001] ....  8263.831328: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [003] ....  8263.835578: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [001] ....  8263.843013: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8263.852831: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8263.862617: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [003] ....  8263.872450: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8263.894518: writeback_congestion_wait: usec_timeout=100000 usec_delayed=18000
              dd-4267  [001] ....  8263.898954: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [001] ....  8263.942418: writeback_congestion_wait: usec_timeout=100000 usec_delayed=44000
              dd-4267  [001] ....  8263.962011: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=306688
              dd-4267  [001] ....  8263.962030: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=306688
              dd-4267  [001] ....  8263.963721: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=308736
              dd-4267  [001] ....  8263.965316: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=310784
              dd-4267  [001] ....  8263.966932: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=312832
              dd-4267  [003] ....  8263.969872: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [001] ....  8264.003468: writeback_congestion_wait: usec_timeout=100000 usec_delayed=31000
              dd-4267  [001] ....  8264.005330: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [001] ....  8264.007494: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [001] ....  8264.172407: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [003] ....  8264.179889: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [003] ....  8264.185841: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [003] ....  8264.190127: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [001] ....  8264.196118: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [003] ....  8264.200459: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [003] ....  8264.206405: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [003] ....  8264.210702: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [001] ....  8264.216654: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [003] ....  8264.221002: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [003] ....  8264.226948: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [003] ....  8264.231270: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [001] ....  8264.237169: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [003] ....  8264.241569: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [003] ....  8264.247491: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [003] ....  8264.249307: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=313856
              dd-4267  [003] ....  8264.249328: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=313856
              dd-4267  [003] ....  8264.250997: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=315904
              dd-4267  [003] ....  8264.251768: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [003] ....  8264.255542: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=317952
              dd-4267  [003] ....  8264.257148: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=320000
              dd-4267  [001] ....  8264.257726: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [001] ....  8264.262047: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [001] ....  8264.268005: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8264.270266: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=321792
              dd-4267  [001] ....  8264.271910: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=323840
              dd-4267  [003] ....  8264.373787: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [001] ....  8264.530715: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
       flush-8:0-4272  [002] ....  8264.602404: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=325120
       flush-8:0-4272  [002] ....  8264.605571: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=325120
       flush-8:0-4272  [002] ....  8264.608617: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=327168
       flush-8:0-4272  [002] ....  8264.613823: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1280 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=329216
       flush-8:0-4272  [002] ....  8264.617055: writeback_pages_written: 5376
              dd-4267  [003] ....  8264.706635: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [003] ....  8264.777165: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=330496
       flush-8:0-4272  [002] ....  8264.777184: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=330496
              dd-4267  [003] ....  8264.777184: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=330496
       flush-8:0-4272  [002] ....  8264.780903: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=330496
       flush-8:0-4272  [002] ....  8264.780910: writeback_pages_written: 256
              dd-4267  [002] ....  8264.781921: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [002] ....  8264.821810: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [002] ....  8264.845760: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [002] ....  8264.875710: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
       flush-8:0-4272  [002] ....  8264.887380: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=330752
       flush-8:0-4272  [002] ....  8264.890912: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=330752
       flush-8:0-4272  [002] ....  8264.890919: writeback_pages_written: 256
              dd-4267  [002] ....  8264.901596: writeback_congestion_wait: usec_timeout=100000 usec_delayed=8000
              dd-4267  [002] ....  8264.941549: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [002] ....  8264.965702: writeback_congestion_wait: usec_timeout=100000 usec_delayed=8000
       flush-8:0-4272  [002] ....  8264.983529: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=331008
       flush-8:0-4272  [002] ....  8264.987449: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=331008
       flush-8:0-4272  [002] ....  8264.987456: writeback_pages_written: 256
              dd-4267  [002] ....  8264.991277: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
       flush-8:0-4272  [002] ....  8265.017144: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=331264
       flush-8:0-4272  [002] ....  8265.021013: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=331264
       flush-8:0-4272  [002] ....  8265.024087: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=333312
       flush-8:0-4272  [002] ....  8265.030297: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=335360
       flush-8:0-4272  [002] ....  8265.036048: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1024 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=337408
       flush-8:0-4272  [002] ....  8265.039951: writeback_pages_written: 7168
              dd-4267  [002] ....  8265.057223: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8265.077361: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [002] ....  8265.097530: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8265.117633: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8265.143609: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [002] ....  8265.163795: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8265.183921: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8265.204076: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
       flush-8:0-4272  [002] ....  8265.211688: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=338432
       flush-8:0-4272  [002] ....  8265.215220: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=338432
       flush-8:0-4272  [002] ....  8265.215226: writeback_pages_written: 256
              dd-4267  [002] ....  8265.224208: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
       flush-8:0-4272  [002] ....  8265.231866: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=338688
       flush-8:0-4272  [002] ....  8265.235287: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=338688
       flush-8:0-4272  [002] ....  8265.238119: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=340736
       flush-8:0-4272  [002] ....  8265.243774: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=342784
       flush-8:0-4272  [002] ....  8265.249266: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1536 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=344832
      flush-0:16-4223  [003] ....  8265.302236: writeback_start: bdi 0:16: sb_dev 0:0 nr_pages=30929 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
      flush-0:16-4223  [003] ....  8265.302237: writeback_queue_io: bdi 0:16: older=4302906992 age=30000 enqueue=0 reason=periodic
      flush-0:16-4223  [003] ....  8265.302238: writeback_written: bdi 0:16: sb_dev 0:0 nr_pages=30929 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
      flush-0:16-4223  [003] ....  8265.302243: writeback_pages_written: 0
              dd-4267  [002] ....  8265.346236: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [002] ....  8265.415236: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=346368
              dd-4267  [002] ....  8265.415253: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=346368
              dd-4267  [002] ....  8265.416806: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=348416
              dd-4267  [002] ....  8265.593749: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [002] ....  8265.597894: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=349696
              dd-4267  [002] ....  8265.597911: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=349696
              dd-4267  [002] ....  8265.599469: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=351744
              dd-4267  [002] ....  8265.601033: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=353792
              dd-4267  [002] ....  8265.603547: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
       flush-8:0-4272  [001] ....  8265.604475: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=351744
       flush-8:0-4272  [001] ....  8265.612575: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1024 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=353792
       flush-8:0-4272  [001] ....  8265.615654: writeback_pages_written: 5120
       flush-8:0-4272  [001] ....  8265.779243: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=354816
              dd-4267  [002] ....  8265.780668: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
       flush-8:0-4272  [003] ....  8266.597438: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=354816
       flush-8:0-4272  [003] ....  8266.597448: writeback_start: bdi 8:0: sb_dev 0:0 nr_pages=34303 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
       flush-8:0-4272  [003] ....  8266.597449: writeback_queue_io: bdi 8:0: older=4302908287 age=30000 enqueue=0 reason=periodic
       flush-8:0-4272  [003] ....  8266.599938: writeback_single_inode: bdi 8:0: ino=13 state=I_DIRTY_PAGES dirtied_when=4302921148 age=55 index=356096 to_write=1024 wrote=1024
       flush-8:0-4272  [003] ....  8266.599941: writeback_written: bdi 8:0: sb_dev 0:0 nr_pages=33279 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
       flush-8:0-4272  [003] ....  8266.599942: writeback_start: bdi 8:0: sb_dev 0:0 nr_pages=33279 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
       flush-8:0-4272  [003] ....  8266.599942: writeback_queue_io: bdi 8:0: older=4302908290 age=30000 enqueue=0 reason=periodic
       flush-8:0-4272  [003] ....  8266.622191: writeback_single_inode: bdi 8:0: ino=13 state=I_DIRTY_PAGES dirtied_when=4302921148 age=55 index=367701 to_write=1024 wrote=11605
       flush-8:0-4272  [003] ....  8266.622196: writeback_written: bdi 8:0: sb_dev 0:0 nr_pages=21674 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
       flush-8:0-4272  [003] ....  8266.622196: writeback_start: bdi 8:0: sb_dev 0:0 nr_pages=21674 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
       flush-8:0-4272  [003] ....  8266.622197: writeback_queue_io: bdi 8:0: older=4302908312 age=30000 enqueue=0 reason=periodic
       flush-8:0-4272  [003] ....  8266.624595: writeback_single_inode: bdi 8:0: ino=13 state=I_DIRTY_PAGES dirtied_when=4302921148 age=55 index=368861 to_write=1024 wrote=1160
       flush-8:0-4272  [003] ....  8266.624597: writeback_written: bdi 8:0: sb_dev 0:0 nr_pages=20514 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
       flush-8:0-4272  [003] ....  8266.624597: writeback_start: bdi 8:0: sb_dev 0:0 nr_pages=20514 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
       flush-8:0-4272  [003] ....  8266.624598: writeback_queue_io: bdi 8:0: older=4302908315 age=30000 enqueue=0 reason=periodic
       flush-8:0-4272  [003] ....  8266.625314: writeback_single_inode: bdi 8:0: ino=13 state=I_DIRTY_PAGES dirtied_when=4302938315 age=38 index=368861 to_write=1024 wrote=145
       flush-8:0-4272  [003] ....  8266.625316: writeback_written: bdi 8:0: sb_dev 0:0 nr_pages=20369 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
       flush-8:0-4272  [003] ....  8266.625317: writeback_start: bdi 8:0: sb_dev 0:0 nr_pages=20369 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
       flush-8:0-4272  [003] ....  8266.625317: writeback_queue_io: bdi 8:0: older=4302908315 age=30000 enqueue=0 reason=periodic
       flush-8:0-4272  [003] ....  8266.625317: writeback_written: bdi 8:0: sb_dev 0:0 nr_pages=20369 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
       flush-8:0-4272  [003] ....  8266.625319: writeback_pages_written: 14190
       flush-8:0-4272  [003] ....  8266.825892: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=368896
       flush-8:0-4272  [003] ....  8266.826219: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=368896
       flush-8:0-4272  [003] ....  8266.827173: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1536 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=369408
       flush-8:0-4272  [003] ....  8266.829558: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=370688
       flush-8:0-4272  [003] ....  8266.832712: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=372736
       flush-8:0-4272  [003] ....  8266.838252: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=374784
       flush-8:0-4272  [003] ....  8266.842368: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=376832
              dd-4267  [002] ....  8266.935447: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
       flush-8:0-4272  [001] ....  8266.999637: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=378880
       flush-8:0-4272  [001] ....  8267.145618: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1280 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=380928
       flush-8:0-4272  [001] ....  8267.148792: writeback_pages_written: 13202
              dd-4267  [001] ....  8267.351195: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [001] ....  8267.361164: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [003] ....  8267.370981: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8267.380830: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8267.390569: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8267.400401: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [003] ....  8267.410227: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [001] ....  8267.420027: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [001] ....  8267.429869: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8267.439597: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [003] ....  8267.449465: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8267.459294: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [001] ....  8267.469116: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8267.477442: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [003] ....  8267.481950: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [001] ....  8267.486445: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [001] ....  8267.490959: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [001] ....  8267.609943: writeback_congestion_wait: usec_timeout=100000 usec_delayed=97000
              dd-4267  [003] ....  8267.614031: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [001] ....  8267.618624: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [001] ....  8267.628386: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [003] ....  8267.635454: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [001] ....  8267.639944: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [001] ....  8267.644410: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [001] ....  8267.648982: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [003] ....  8267.654300: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [001] ....  8267.658700: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [002] ....  8267.667206: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [001] ....  8267.669674: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [001] ....  8267.674305: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [001] ....  8267.706797: writeback_congestion_wait: usec_timeout=100000 usec_delayed=18000
              dd-4267  [001] ....  8267.711275: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [003] ....  8267.719838: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [001] ....  8267.729680: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [001] ....  8267.737881: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [001] ....  8267.742316: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [001] ....  8267.752567: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=382208
       flush-8:0-4272  [003] ....  8267.752583: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=382208
              dd-4267  [001] ....  8267.752584: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=382208
              dd-4267  [001] ....  8267.754070: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=384256
              dd-4267  [001] ....  8267.755594: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=386304
              dd-4267  [001] ....  8267.757105: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=388352
       flush-8:0-4272  [003] ....  8267.757895: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=382208
              dd-4267  [001] ....  8267.758599: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=390400
              dd-4267  [001] ....  8267.760122: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=392448
       flush-8:0-4272  [003] ....  8267.760868: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=384256
       flush-8:0-4272  [003] ....  8267.768745: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=386304
       flush-8:0-4272  [003] ....  8267.776742: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=388352
       flush-8:0-4272  [003] ....  8267.783814: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=390400
       flush-8:0-4272  [003] ....  8267.790773: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=392448
       flush-8:0-4272  [003] ....  8267.797382: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=394496
       flush-8:0-4272  [003] ....  8267.803995: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=396544
       flush-8:0-4272  [003] ....  8267.809316: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=398592
       flush-8:0-4272  [003] ....  8267.813952: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=400640
       flush-8:0-4272  [003] ....  8267.818094: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=402688
              dd-4267  [002] ....  8267.905908: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
       flush-8:0-4272  [001] ....  8267.938284: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1792 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=404736
       flush-8:0-4272  [001] ....  8267.943129: writeback_pages_written: 24320
              dd-4267  [001] ....  8268.163764: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8268.173986: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [003] ....  8268.184334: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [001] ....  8268.194576: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8268.198511: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=406528
       flush-8:0-4272  [003] ....  8268.198524: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=406528
              dd-4267  [001] ....  8268.198525: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=406528
       flush-8:0-4272  [003] ....  8268.202131: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1280 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=406528
       flush-8:0-4272  [003] ....  8268.204478: writeback_pages_written: 1280
              dd-4267  [001] ....  8268.204959: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8268.209828: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=407808
       flush-8:0-4272  [003] ....  8268.209837: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=407808
              dd-4267  [001] ....  8268.209837: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=407808
       flush-8:0-4272  [003] ....  8268.212352: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=407808
       flush-8:0-4272  [003] ....  8268.212358: writeback_pages_written: 256
              dd-4267  [001] ....  8268.215084: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [003] ....  8268.227002: writeback_congestion_wait: usec_timeout=100000 usec_delayed=8000
              dd-4267  [001] ....  8268.237303: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8268.247561: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8268.257823: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [003] ....  8268.268081: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8268.278409: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8268.288799: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [001] ....  8268.298937: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [003] ....  8268.309285: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8268.319485: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8268.323556: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=408064
       flush-8:0-4272  [003] ....  8268.323569: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=408064
              dd-4267  [001] ....  8268.323570: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=408064
       flush-8:0-4272  [003] ....  8268.326837: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=408064
       flush-8:0-4272  [003] ....  8268.326843: writeback_pages_written: 256
              dd-4267  [001] ....  8268.329773: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [003] ....  8268.333919: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [001] ....  8268.339844: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8268.345842: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8268.350165: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [003] ....  8268.356091: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8268.358321: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [001] ....  8268.362952: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [001] ....  8268.383814: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [001] ....  8268.437266: writeback_congestion_wait: usec_timeout=100000 usec_delayed=52000
              dd-4267  [001] ....  8268.442101: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [001] ....  8268.444583: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [003] ....  8268.448725: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [003] ....  8268.457661: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [001] ....  8268.467635: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8268.477643: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [001] ....  8268.487599: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8268.497414: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8268.501582: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [001] ....  8268.507385: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [003] ....  8268.511569: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [001] ....  8268.517350: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8268.521564: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [001] ....  8268.527355: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [003] ....  8268.531537: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [001] ....  8268.533925: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [001] ....  8268.539707: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8268.561714: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=408320
       flush-8:0-4272  [003] ....  8268.561729: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=408320
              dd-4267  [001] ....  8268.561736: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=408320
              dd-4267  [001] ....  8268.562225: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [001] ....  8268.564550: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
       flush-8:0-4272  [003] ....  8268.566727: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=408320
       flush-8:0-4272  [003] ....  8268.566734: writeback_pages_written: 256
              dd-4267  [001] ....  8268.567268: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [001] ....  8268.569268: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [003] ....  8268.575561: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [001] ....  8268.579702: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [001] ....  8268.585533: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8268.589691: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [003] ....  8268.595494: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8268.599670: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [001] ....  8268.605495: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8268.609659: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [003] ....  8268.615463: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [003] ....  8268.617783: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=408576
       flush-8:0-4272  [001] ....  8268.617795: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=408576
              dd-4267  [003] ....  8268.617796: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=408576
              dd-4267  [003] ....  8268.619270: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=410624
              dd-4267  [003] ....  8268.622449: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=412672
       flush-8:0-4272  [001] ....  8268.622860: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=408576
              dd-4267  [003] ....  8268.623989: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=414720
       flush-8:0-4272  [001] ....  8268.625976: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=410624
              dd-4267  [003] ....  8268.635422: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
       flush-8:0-4272  [001] ....  8268.840617: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=412672
       flush-8:0-4272  [001] ....  8268.848404: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=414720
              dd-4267  [003] ....  8268.930278: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=419584
              dd-4267  [003] ....  8268.930296: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=419584
              dd-4267  [003] ....  8268.931994: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=421632
              dd-4267  [003] ....  8268.934099: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=423680
              dd-4267  [003] ....  8268.935726: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=425728
              dd-4267  [003] ....  8268.937352: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=427776
              dd-4267  [003] ....  8268.939046: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=429824
              dd-4267  [001] ....  8269.039200: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [003] ....  8269.139135: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [001] ....  8269.239185: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [001] ....  8269.408981: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [001] ....  8269.584400: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=431104
              dd-4267  [001] ....  8269.584469: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=431104
              dd-4267  [001] ....  8269.593152: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=431360
              dd-4267  [001] ....  8269.597285: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=431616
              dd-4267  [001] ....  8269.597492: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=431616
              dd-4267  [001] ....  8269.598967: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=433664
              dd-4267  [001] ....  8269.614402: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=434944
              dd-4267  [001] ....  8269.614421: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=434944
              dd-4267  [001] ....  8269.616055: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=436992
              dd-4267  [002] ....  8269.637753: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [002] ....  8269.680296: writeback_congestion_wait: usec_timeout=100000 usec_delayed=29000
              dd-4267  [002] ....  8269.817372: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
       flush-8:0-4272  [002] ....  8269.864374: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=437248
       flush-8:0-4272  [002] ....  8269.868771: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=437248
              dd-4267  [001] ....  8269.871123: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=443392
       flush-8:0-4272  [002] ....  8269.871919: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=439296
              dd-4267  [001] ....  8269.875734: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=445440
              dd-4267  [001] ....  8269.877352: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=447488
       flush-8:0-4272  [002] ....  8269.878824: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=441344
              dd-4267  [001] ....  8269.881258: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=449536
              dd-4267  [002] ....  8269.885307: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8269.889882: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
       flush-8:0-4272  [002] ....  8269.936593: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=443392
       flush-8:0-4272  [002] ....  8269.942755: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=445440
       flush-8:0-4272  [002] ....  8269.948672: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=447488
       flush-8:0-4272  [002] ....  8269.953747: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=449536
       flush-8:0-4272  [002] ....  8269.956079: writeback_pages_written: 13056
              dd-4267  [001] ....  8270.015147: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=450304
       flush-8:0-4272  [002] ....  8270.015168: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=450304
              dd-4267  [001] ....  8270.015174: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=450304
       flush-8:0-4272  [002] ....  8270.018425: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=450304
       flush-8:0-4272  [002] ....  8270.019354: writeback_pages_written: 768
              dd-4267  [002] ....  8270.064531: writeback_congestion_wait: usec_timeout=100000 usec_delayed=48000
              dd-4267  [002] ....  8270.106521: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=451072
              dd-4267  [002] ....  8270.106542: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=451072
              dd-4267  [002] ....  8270.109830: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=452352
              dd-4267  [002] ....  8270.111279: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=454400
       flush-8:0-4272  [002] ....  8270.274267: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=455424
       flush-8:0-4272  [002] ....  8270.277302: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=455424
       flush-8:0-4272  [002] ....  8270.277308: writeback_pages_written: 256
              dd-4267  [002] ....  8270.281460: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
      flush-0:16-4223  [001] ....  8270.300792: writeback_start: bdi 0:16: sb_dev 0:0 nr_pages=34352 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
      flush-0:16-4223  [001] ....  8270.300795: writeback_queue_io: bdi 0:16: older=4302911993 age=30000 enqueue=1 reason=periodic
      flush-0:16-4223  [001] ....  8270.300805: writeback_single_inode: bdi 0:16: ino=4874359 state= dirtied_when=4302911988 age=65 index=0 to_write=1024 wrote=0
      flush-0:16-4223  [001] ....  8270.300808: writeback_written: bdi 0:16: sb_dev 0:0 nr_pages=34352 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
      flush-0:16-4223  [001] ....  8270.300809: writeback_start: bdi 0:16: sb_dev 0:0 nr_pages=34352 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
      flush-0:16-4223  [001] ....  8270.300809: writeback_queue_io: bdi 0:16: older=4302911993 age=30000 enqueue=0 reason=periodic
      flush-0:16-4223  [001] ....  8270.300809: writeback_written: bdi 0:16: sb_dev 0:0 nr_pages=34352 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
      flush-0:16-4223  [001] ....  8270.300814: writeback_pages_written: 0
              dd-4267  [002] ....  8270.302002: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [002] ....  8270.316621: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8270.343276: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [002] ....  8270.373944: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [002] ....  8270.394471: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [002] ....  8270.415038: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8270.435568: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [002] ....  8270.472530: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [002] ....  8270.513637: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [002] ....  8270.517612: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=455680
              dd-4267  [002] ....  8270.517625: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=455680
              dd-4267  [002] ....  8270.554764: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [002] ....  8270.558840: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=455936
              dd-4267  [002] ....  8270.558852: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=455936
              dd-4267  [002] ....  8270.560339: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=457984
              dd-4267  [002] ....  8270.561810: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=460032
              dd-4267  [002] ....  8270.564781: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8270.575044: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
       flush-8:0-4272  [002] ....  8270.590611: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=461056
       flush-8:0-4272  [002] ....  8270.594701: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=461056
       flush-8:0-4272  [002] ....  8270.598284: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=463104
       flush-8:0-4272  [002] ....  8270.600496: writeback_pages_written: 2304
              dd-4267  [002] ....  8270.601943: writeback_congestion_wait: usec_timeout=100000 usec_delayed=9000
              dd-4267  [002] ....  8270.633486: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
       flush-8:0-4272  [002] ....  8270.657444: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=463360
       flush-8:0-4272  [002] ....  8270.660617: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=463360
       flush-8:0-4272  [002] ....  8270.660624: writeback_pages_written: 256
              dd-4267  [002] ....  8270.673407: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [002] ....  8270.703164: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [002] ....  8270.723126: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [002] ....  8270.743057: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [002] ....  8270.883761: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=463616
              dd-4267  [002] ....  8270.883782: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=463616
              dd-4267  [002] ....  8270.885303: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=465664
              dd-4267  [002] ....  8270.886785: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=467712
              dd-4267  [002] ....  8270.888382: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=469760
              dd-4267  [002] ....  8270.889945: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=471808
       flush-8:0-4272  [001] ....  8271.327896: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=467712
       flush-8:0-4272  [001] ....  8271.334733: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=469760
       flush-8:0-4272  [001] ....  8271.340404: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=471808
       flush-8:0-4272  [001] ....  8271.345681: writeback_pages_written: 10240
              dd-4267  [002] ....  8271.421367: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=473856
       flush-8:0-4272  [001] ....  8271.421387: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=473856
              dd-4267  [002] ....  8271.421390: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=473856
              dd-4267  [002] ....  8271.423065: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=475904
              dd-4267  [002] ....  8271.425186: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=477952
       flush-8:0-4272  [001] ....  8271.425640: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=473856
              dd-4267  [002] ....  8271.426874: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=480000
       flush-8:0-4272  [001] ....  8271.492067: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=475904
       flush-8:0-4272  [001] ....  8271.497902: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=477952
       flush-8:0-4272  [001] ....  8271.502398: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1280 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=480000
       flush-8:0-4272  [001] ....  8271.505244: writeback_pages_written: 7424
              dd-4267  [002] ....  8271.678718: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [001] ....  8271.777600: writeback_congestion_wait: usec_timeout=100000 usec_delayed=98000
              dd-4267  [001] ....  8271.817785: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [001] ....  8271.827867: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8271.837939: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [003] ....  8271.847990: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [001] ....  8271.858059: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8271.868133: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8271.878146: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [003] ....  8271.888271: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8271.898364: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8271.903945: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [001] ....  8271.908242: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [003] ....  8271.910635: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [001] ....  8271.918477: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [001] ....  8271.941108: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [001] ....  8271.945668: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [001] ....  8271.948169: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [001] ....  8271.954621: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [001] ....  8271.958741: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=481280
       flush-8:0-4272  [003] ....  8271.958757: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=481280
              dd-4267  [001] ....  8271.958758: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=481280
              dd-4267  [001] ....  8271.960266: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=483328
              dd-4267  [001] ....  8271.961761: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=485376
       flush-8:0-4272  [003] ....  8271.962732: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=481280
              dd-4267  [001] ....  8271.963264: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=487424
              dd-4267  [001] ....  8271.964441: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [001] ....  8271.974773: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [003] ....  8272.009232: writeback_congestion_wait: usec_timeout=100000 usec_delayed=30000
              dd-4267  [001] ....  8272.029378: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [001] ....  8272.035038: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [003] ....  8272.039234: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [001] ....  8272.045124: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8272.049333: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [001] ....  8272.051819: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [003] ....  8272.057558: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [003] ....  8272.078026: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [001] ....  8272.082691: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [001] ....  8272.090003: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [003] ....  8272.100194: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8272.110246: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8272.120279: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8272.130351: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [003] ....  8272.136822: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [001] ....  8272.142901: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [001] ....  8272.183805: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=488192
              dd-4267  [001] ....  8272.183823: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=488192
              dd-4267  [001] ....  8272.195352: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=489216
              dd-4267  [001] ....  8272.195373: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=489216
              dd-4267  [001] ....  8272.197069: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=491264
              dd-4267  [001] ....  8272.198688: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=493312
              dd-4267  [001] ....  8272.200844: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=495360
              dd-4267  [001] ....  8272.202448: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=497408
              dd-4267  [003] ....  8272.302380: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [001] ....  8272.402340: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
##### CPU 0 buffer started ####
              dd-4267  [000] ....  8272.587442: writeback_congestion_wait: usec_timeout=100000 usec_delayed=95000
              dd-4267  [000] ....  8272.589707: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [000] ....  8272.594405: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8272.603546: writeback_congestion_wait: usec_timeout=100000 usec_delayed=8000
              dd-4267  [000] ....  8272.613529: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8272.617066: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=498176
       flush-8:0-4272  [002] ....  8272.617081: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=498176
              dd-4267  [000] ....  8272.617086: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=498176
              dd-4267  [000] ....  8272.619094: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=500224
       flush-8:0-4272  [002] ....  8272.620566: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=498176
              dd-4267  [000] ....  8272.620634: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=502272
              dd-4267  [000] ....  8272.622174: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=504320
       flush-8:0-4272  [002] ....  8272.623787: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=500224
       flush-8:0-4272  [002] ....  8272.629276: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=502272
       flush-8:0-4272  [002] ....  8272.633956: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1792 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=504320
       flush-8:0-4272  [002] ....  8272.637736: writeback_pages_written: 7936
              dd-4267  [003] ....  8272.728244: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [003] ....  8272.816879: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=506112
       flush-8:0-4272  [002] ....  8272.816896: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=506112
              dd-4267  [003] ....  8272.816897: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=506112
       flush-8:0-4272  [002] ....  8272.820965: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=506112
       flush-8:0-4272  [002] ....  8272.822144: writeback_pages_written: 768
              dd-4267  [000] ....  8272.883681: writeback_congestion_wait: usec_timeout=100000 usec_delayed=66000
              dd-4267  [000] ....  8272.901202: writeback_congestion_wait: usec_timeout=100000 usec_delayed=9000
              dd-4267  [000] ....  8272.911472: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8272.921776: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8272.932042: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [000] ....  8272.942300: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [002] ....  8272.952536: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8272.962856: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8272.972948: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [002] ....  8272.977268: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8272.983253: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8272.989219: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8272.993531: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8272.999473: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8273.003785: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8273.009743: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8273.014080: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8273.020012: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8273.024305: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8273.030315: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8273.033647: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=506880
       flush-8:0-4272  [002] ....  8273.033663: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=506880
              dd-4267  [000] ....  8273.033664: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=506880
              dd-4267  [000] ....  8273.035857: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=508928
       flush-8:0-4272  [002] ....  8273.037953: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=506880
       flush-8:0-4272  [000] ....  8273.118847: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=508928
       flush-8:0-4272  [000] ....  8273.125008: writeback_pages_written: 2816
              dd-4267  [001] ....  8273.201257: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=509696
       flush-8:0-4272  [000] ....  8273.201275: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=509696
              dd-4267  [001] ....  8273.201283: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=509696
              dd-4267  [001] ....  8273.202988: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=511744
              dd-4267  [001] ....  8273.204095: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [001] ....  8273.204351: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
       flush-8:0-4272  [000] ....  8273.207289: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=509696
       flush-8:0-4272  [000] ....  8273.210162: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=511744
              dd-4267  [001] ....  8273.214288: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
       flush-8:0-4272  [000] ....  8273.214741: writeback_pages_written: 2816
              dd-4267  [002] ....  8273.220237: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8273.224624: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8273.230590: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8273.234929: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [002] ....  8273.240847: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8273.245152: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8273.252550: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [000] ....  8273.256650: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8273.262386: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8273.266457: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8273.272133: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8273.274530: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [002] ....  8273.276786: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8273.281514: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [000] ....  8273.303716: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=512512
       flush-8:0-4272  [002] ....  8273.303734: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=512512
              dd-4267  [000] ....  8273.303742: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=512512
              dd-4267  [000] ....  8273.305253: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=514560
              dd-4267  [000] ....  8273.306756: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=516608
       flush-8:0-4272  [002] ....  8273.308082: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=512512
              dd-4267  [000] ....  8273.308310: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=518656
              dd-4267  [001] ....  8273.309955: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=520704
       flush-8:0-4272  [002] ....  8273.310937: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=514560
       flush-8:0-4272  [002] ....  8273.317219: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=516608
       flush-8:0-4272  [002] ....  8273.322499: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=518656
       flush-8:0-4272  [002] ....  8273.327288: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=520704
       flush-8:0-4272  [002] ....  8273.331707: writeback_pages_written: 10240
              dd-4267  [003] ....  8273.411759: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [003] ....  8273.467526: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=522752
       flush-8:0-4272  [002] ....  8273.467545: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=522752
              dd-4267  [003] ....  8273.467547: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=522752
              dd-4267  [003] ....  8273.469243: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=524800
              dd-4267  [003] ....  8273.470853: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=526848
       flush-8:0-4272  [002] ....  8273.471531: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=522752
              dd-4267  [003] ....  8273.472458: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
       flush-8:0-4272  [002] ....  8273.476836: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=524800
       flush-8:0-4272  [002] ....  8273.482005: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1280 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=526848
       flush-8:0-4272  [002] ....  8273.484987: writeback_pages_written: 5376
              dd-4267  [001] ....  8273.576718: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [001] ....  8273.630560: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=528128
       flush-8:0-4272  [002] ....  8273.630579: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=528128
              dd-4267  [001] ....  8273.630580: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=528128
              dd-4267  [001] ....  8273.632915: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=530176
       flush-8:0-4272  [002] ....  8273.634217: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=528128
       flush-8:0-4272  [002] ....  8273.637881: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1280 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=530176
       flush-8:0-4272  [002] ....  8273.641495: writeback_pages_written: 3328
              dd-4267  [003] ....  8273.733643: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [000] ....  8273.803010: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8273.812834: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8273.822636: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [002] ....  8273.832432: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [000] ....  8273.842319: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8273.852182: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8273.861874: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [002] ....  8273.872685: writeback_congestion_wait: usec_timeout=100000 usec_delayed=8000
              dd-4267  [000] ....  8273.877222: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [000] ....  8273.881324: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8273.887043: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [002] ....  8273.891135: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8273.896818: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8273.900890: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8273.906661: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [002] ....  8273.910747: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8273.916474: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8273.920550: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8273.926259: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [002] ....  8273.930361: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8273.936114: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8273.940195: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8273.942458: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=531456
       flush-8:0-4272  [002] ....  8273.942469: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=531456
              dd-4267  [000] ....  8273.942470: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=531456
              dd-4267  [000] ....  8273.943968: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=533504
              dd-4267  [000] ....  8273.945828: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
       flush-8:0-4272  [002] ....  8273.946273: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=531456
       flush-8:0-4272  [002] ....  8273.949248: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1536 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=533504
              dd-4267  [000] ....  8273.949900: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
       flush-8:0-4272  [002] .N..  8273.954420: writeback_pages_written: 3584
              dd-4267  [002] ....  8273.955696: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8273.959774: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8273.965545: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [002] ....  8273.969610: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8273.975340: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8273.979456: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8273.987689: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8273.992204: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8274.026101: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [002] ....  8274.031783: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8274.035939: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8274.040032: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8274.045692: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [002] ....  8274.049762: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8274.055579: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8274.059648: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8274.065335: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [002] ....  8274.069439: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8274.075170: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8274.079279: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8274.086684: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [002] ....  8274.091051: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8274.097019: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8274.102975: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8274.107277: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8274.111257: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=535040
       flush-8:0-4272  [002] ....  8274.111272: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=535040
              dd-4267  [000] ....  8274.111280: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=535040
              dd-4267  [000] ....  8274.114916: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=537088
       flush-8:0-4272  [002] ....  8274.115545: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=535040
       flush-8:0-4272  [002] ....  8274.118567: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1280 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=537088
              dd-4267  [001] ....  8274.120214: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=538112
              dd-4267  [001] ....  8274.123035: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
       flush-8:0-4272  [002] ....  8274.123531: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=538112
       flush-8:0-4272  [002] ....  8274.124428: writeback_pages_written: 3840
              dd-4267  [001] ....  8274.125289: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8274.127722: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8274.130996: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [002] ....  8274.133237: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8274.135492: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [002] ....  8274.175907: writeback_congestion_wait: usec_timeout=100000 usec_delayed=40000
              dd-4267  [000] ....  8274.290285: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [000] ....  8274.336268: writeback_congestion_wait: usec_timeout=100000 usec_delayed=46000
              dd-4267  [002] ....  8274.340912: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [000] ....  8274.347609: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8274.357660: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [002] ....  8274.367884: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8274.373602: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8274.377990: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8274.383934: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8274.386279: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=538880
       flush-8:0-4272  [002] ....  8274.386291: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=538880
              dd-4267  [000] ....  8274.386292: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=538880
              dd-4267  [000] ....  8274.387774: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=540928
       flush-8:0-4272  [002] ....  8274.390547: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=538880
              dd-4267  [000] ....  8274.391322: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=542976
              dd-4267  [000] ....  8274.392891: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=545024
       flush-8:0-4272  [000] ....  8274.536676: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=540928
       flush-8:0-4272  [000] ....  8274.543142: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=542976
       flush-8:0-4272  [000] ....  8274.549982: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1536 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=545024
       flush-8:0-4272  [000] ....  8274.554681: writeback_pages_written: 7680
              dd-4267  [001] ....  8274.657074: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=546560
       flush-8:0-4272  [000] ....  8274.657090: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=546560
              dd-4267  [001] ....  8274.657093: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=546560
              dd-4267  [001] ....  8274.661141: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=548608
       flush-8:0-4272  [000] ....  8274.662544: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=546560
              dd-4267  [001] ....  8274.662787: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=550656
       flush-8:0-4272  [000] ....  8274.666715: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=548608
              dd-4267  [001] ....  8274.666972: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=552704
              dd-4267  [001] ....  8274.670755: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=554752
              dd-4267  [001] ....  8274.673577: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
       flush-8:0-4272  [000] ....  8274.674196: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=550656
              dd-4267  [001] ....  8274.677756: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
       flush-8:0-4272  [000] ....  8274.680418: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=552704
       flush-8:0-4272  [000] ....  8274.685640: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1024 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=554752
       flush-8:0-4272  [000] ....  8274.688771: writeback_pages_written: 9216
              dd-4267  [003] ....  8274.779091: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [002] ....  8274.843553: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [000] ....  8274.853430: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8274.863405: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [000] ....  8274.870132: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [002] ....  8274.874634: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [000] ....  8274.878968: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8274.883432: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8274.887848: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8274.930432: writeback_congestion_wait: usec_timeout=100000 usec_delayed=37000
              dd-4267  [000] ....  8274.932637: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [000] ....  8274.943276: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [000] ....  8274.943281: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=555776
       flush-8:0-4272  [002] ....  8274.943292: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=555776
       flush-8:0-4272  [002] ....  8274.947234: writeback_pages_written: 256
              dd-4267  [000] ....  8274.947540: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=556032
       flush-8:0-4272  [002] ....  8274.947548: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=556032
              dd-4267  [000] ....  8274.947549: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=556032
              dd-4267  [000] ....  8274.949050: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=558080
       flush-8:0-4272  [002] ....  8274.950731: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=556032
       flush-8:0-4272  [002] ....  8274.954025: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1024 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=558080
       flush-8:0-4272  [002] ....  8274.958120: writeback_pages_written: 3072
              dd-4267  [001] ....  8274.962461: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=559104
       flush-8:0-4272  [002] ....  8274.962480: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=559104
              dd-4267  [001] ....  8274.962481: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=559104
       flush-8:0-4272  [002] ....  8274.965051: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1024 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=559104
       flush-8:0-4272  [002] ....  8274.966440: writeback_pages_written: 1024
              dd-4267  [001] ....  8274.971338: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=560128
       flush-8:0-4272  [002] ....  8274.971354: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=560128
              dd-4267  [001] ....  8274.971355: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=560128
       flush-8:0-4272  [002] ....  8274.973875: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=560128
       flush-8:0-4272  [002] ....  8274.974803: writeback_pages_written: 768
              dd-4267  [001] ....  8274.976380: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=560896
       flush-8:0-4272  [002] ....  8274.976392: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=560896
              dd-4267  [001] ....  8274.976393: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=560896
       flush-8:0-4272  [002] ....  8274.980232: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1024 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=560896
              dd-4267  [001] ....  8274.981786: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=561920
       flush-8:0-4272  [002] ....  8274.982195: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=512 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=561920
              dd-4267  [001] ....  8274.982196: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=562176
       flush-8:0-4272  [002] ....  8274.985722: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1536 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=562176
       flush-8:0-4272  [002] ....  8274.988800: writeback_pages_written: 2816
              dd-4267  [003] ....  8275.082867: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [000] ....  8275.128437: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8275.138326: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=563712
       flush-8:0-4272  [002] ....  8275.138340: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=563712
              dd-4267  [000] ....  8275.138341: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=563712
       flush-8:0-4272  [002] ....  8275.141461: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=563712
       flush-8:0-4272  [002] ....  8275.141469: writeback_pages_written: 256
              dd-4267  [002] ....  8275.194016: writeback_congestion_wait: usec_timeout=100000 usec_delayed=55000
              dd-4267  [000] ....  8275.198421: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8275.292553: writeback_congestion_wait: usec_timeout=100000 usec_delayed=69000
              dd-4267  [000] ....  8275.296650: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=563968
       flush-8:0-4272  [002] ....  8275.296664: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=563968
              dd-4267  [000] ....  8275.296665: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=563968
              dd-4267  [000] ....  8275.297205: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
      flush-0:16-4223  [003] ....  8275.297726: writeback_start: bdi 0:16: sb_dev 0:0 nr_pages=35970 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
      flush-0:16-4223  [003] ....  8275.297730: writeback_queue_io: bdi 0:16: older=4302916993 age=30000 enqueue=2 reason=periodic
      flush-0:16-4223  [003] ....  8275.297741: writeback_single_inode: bdi 0:16: ino=1573485 state= dirtied_when=4302912037 age=65 index=0 to_write=1024 wrote=0
      flush-0:16-4223  [003] ....  8275.297765: writeback_single_inode: bdi 0:16: ino=1573501 state= dirtied_when=4302915356 age=61 index=24 to_write=1024 wrote=1
      flush-0:16-4223  [003] ....  8275.297769: writeback_written: bdi 0:16: sb_dev 0:0 nr_pages=35969 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
      flush-0:16-4223  [003] ....  8275.297770: writeback_start: bdi 0:16: sb_dev 0:0 nr_pages=35969 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
      flush-0:16-4223  [003] ....  8275.297770: writeback_queue_io: bdi 0:16: older=4302916993 age=30000 enqueue=0 reason=periodic
      flush-0:16-4223  [003] ....  8275.297771: writeback_written: bdi 0:16: sb_dev 0:0 nr_pages=35969 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
      flush-0:16-4223  [003] ....  8275.297777: writeback_pages_written: 1
       flush-8:0-4272  [002] ....  8275.300356: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=563968
       flush-8:0-4272  [002] ....  8275.300363: writeback_pages_written: 256
              dd-4267  [000] ....  8275.301842: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8275.307614: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [002] ....  8275.312153: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8275.319068: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8275.328375: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [002] ....  8275.338401: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [002] ....  8275.342574: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=564224
       flush-8:0-4272  [000] ....  8275.342588: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=564224
              dd-4267  [002] ....  8275.342589: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=564224
              dd-4267  [002] ....  8275.346229: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=566272
       flush-8:0-4272  [000] ....  8275.346586: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=564224
              dd-4267  [002] ....  8275.347786: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=568320
       flush-8:0-4272  [000] ....  8275.349746: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=566272
              dd-4267  [002] ....  8275.351519: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=570368
              dd-4267  [002] ....  8275.353984: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
       flush-8:0-4272  [000] ....  8275.355780: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=568320
              dd-4267  [002] ....  8275.356369: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
       flush-8:0-4272  [000] ....  8275.361337: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=570368
       flush-8:0-4272  [000] ....  8275.363817: writeback_pages_written: 6912
              dd-4267  [000] ....  8275.455701: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [000] ....  8275.493067: writeback_congestion_wait: usec_timeout=100000 usec_delayed=14000
              dd-4267  [002] ....  8275.513157: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8275.523329: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8275.533356: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8275.536894: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=571136
       flush-8:0-4272  [002] ....  8275.536907: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=571136
              dd-4267  [000] ....  8275.536915: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=571136
       flush-8:0-4272  [002] ....  8275.540007: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=571136
       flush-8:0-4272  [002] ....  8275.540783: writeback_pages_written: 768
              dd-4267  [000] ....  8275.549196: writeback_congestion_wait: usec_timeout=100000 usec_delayed=12000
              dd-4267  [002] ....  8275.559066: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8275.569257: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8275.574835: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8275.576992: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=571904
       flush-8:0-4272  [002] ....  8275.577003: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=571904
              dd-4267  [000] ....  8275.577010: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=571904
              dd-4267  [000] ....  8275.579035: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
       flush-8:0-4272  [002] ....  8275.580062: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=512 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=571904
       flush-8:0-4272  [002] ....  8275.580471: writeback_pages_written: 512
              dd-4267  [002] ....  8275.584884: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8275.590767: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8275.595004: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8275.600852: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8275.607811: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8275.611310: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=572416
       flush-8:0-4272  [002] ....  8275.611324: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=572416
              dd-4267  [000] ....  8275.611327: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=572416
              dd-4267  [000] ....  8275.612088: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
       flush-8:0-4272  [002] ....  8275.614606: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=572416
       flush-8:0-4272  [002] ....  8275.614613: writeback_pages_written: 256
              dd-4267  [000] ....  8275.616597: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [002] ....  8275.620899: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [002] ....  8275.622607: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=572672
       flush-8:0-4272  [000] ....  8275.622616: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=572672
              dd-4267  [002] ....  8275.622617: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=572672
       flush-8:0-4272  [000] ....  8275.626265: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=572672
       flush-8:0-4272  [000] ....  8275.626271: writeback_pages_written: 256
              dd-4267  [000] ....  8275.632606: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8275.642731: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8275.656678: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=572928
       flush-8:0-4272  [002] ....  8275.656693: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=572928
              dd-4267  [000] ....  8275.656694: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=572928
              dd-4267  [000] ....  8275.658185: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=574976
       flush-8:0-4272  [002] ....  8275.660256: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=572928
              dd-4267  [002] ....  8275.663561: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
       flush-8:0-4272  [000] ....  8275.762227: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1280 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=574976
       flush-8:0-4272  [000] ....  8275.768192: writeback_pages_written: 3328
              dd-4267  [001] ....  8275.813954: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=576256
       flush-8:0-4272  [000] ....  8275.813972: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=576256
              dd-4267  [001] ....  8275.813973: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=576256
       flush-8:0-4272  [000] ....  8275.818853: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1280 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=576256
              dd-4267  [001] ....  8275.819749: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=577536
       flush-8:0-4272  [000] ....  8275.821425: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=577536
              dd-4267  [001] ....  8275.821426: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=579328
              dd-4267  [001] ....  8275.825312: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
       flush-8:0-4272  [000] ....  8275.829046: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1024 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=579328
       flush-8:0-4272  [000] ....  8275.830311: writeback_pages_written: 4096
              dd-4267  [000] ....  8275.835347: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [002] ....  8275.841226: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8275.845447: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8276.548367: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=580352
       flush-8:0-4272  [002] ....  8276.548384: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=580352
              dd-4267  [000] ....  8276.548400: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=580352
              dd-4267  [000] ....  8276.550364: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=582400
              dd-4267  [000] ....  8276.551926: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=584448
              dd-4267  [000] ....  8276.553518: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=586496
       flush-8:0-4272  [002] ....  8276.554547: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=580352
              dd-4267  [000] ....  8276.555127: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=588544
              dd-4267  [000] ....  8276.556649: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=590592
       flush-8:0-4272  [002] ....  8276.557449: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=582400
              dd-4267  [001] ....  8276.560430: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=592640
              dd-4267  [001] ....  8276.562084: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=594688
              dd-4267  [001] ....  8276.563704: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=596736
              dd-4267  [001] ....  8276.565324: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=598784
       flush-8:0-4272  [002] ....  8276.565402: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=584448
              dd-4267  [001] ....  8276.566863: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=600832
              dd-4267  [001] ....  8276.568420: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=602880
       flush-8:0-4272  [002] ....  8276.573699: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=586496
       flush-8:0-4272  [002] ....  8276.580114: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=588544
       flush-8:0-4272  [002] ....  8276.585744: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=590592
       flush-8:0-4272  [002] ....  8276.590904: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=592640
       flush-8:0-4272  [002] ....  8276.595589: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=594688
       flush-8:0-4272  [002] ....  8276.599917: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=596736
       flush-8:0-4272  [002] ....  8276.603987: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=598784
              dd-4267  [003] ....  8276.668972: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [001] ....  8276.769045: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
       flush-8:0-4272  [002] ....  8276.890790: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=600832
       flush-8:0-4272  [002] ....  8276.895314: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1792 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=602880
       flush-8:0-4272  [002] ....  8276.898568: writeback_pages_written: 24320
              dd-4267  [003] ....  8276.908887: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [001] ....  8277.050810: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [000] ....  8277.200263: writeback_congestion_wait: usec_timeout=100000 usec_delayed=88000
              dd-4267  [000] ....  8277.258031: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=604672
       flush-8:0-4272  [002] ....  8277.258049: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=604672
              dd-4267  [000] ....  8277.258050: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=604672
              dd-4267  [000] ....  8277.258823: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
       flush-8:0-4272  [002] ....  8277.261572: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=604672
       flush-8:0-4272  [002] ....  8277.261579: writeback_start: bdi 8:0: sb_dev 0:0 nr_pages=35048 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
       flush-8:0-4272  [002] ....  8277.261580: writeback_queue_io: bdi 8:0: older=4302918957 age=30000 enqueue=0 reason=periodic
       flush-8:0-4272  [002] ....  8277.261580: writeback_written: bdi 8:0: sb_dev 0:0 nr_pages=35048 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
       flush-8:0-4272  [002] ....  8277.261582: writeback_pages_written: 256
              dd-4267  [000] ....  8277.262937: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8277.268671: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [002] ....  8277.272757: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8277.278481: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8277.282585: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8277.288315: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [002] ....  8277.292395: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8277.298115: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8277.302215: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8277.307931: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [002] ....  8277.312008: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8277.317735: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8277.324837: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8277.355709: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8277.364584: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8277.374398: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8277.382588: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [002] ....  8277.392402: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [000] ....  8277.397901: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8277.402057: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8277.407739: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [002] ....  8277.411818: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8277.417560: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8277.421658: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8277.427424: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [002] ....  8277.431473: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [000] ....  8277.437191: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8277.441271: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8277.447015: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [002] ....  8277.451087: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8277.456772: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8277.460844: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8277.466614: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [002] ....  8277.470713: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8277.476407: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8277.480519: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8277.486224: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [002] ....  8277.490324: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8277.496067: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8277.500124: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8277.505806: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [002] ....  8277.509968: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8277.515667: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8277.519729: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8277.521881: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [002] ....  8277.524031: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8277.526217: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8277.528385: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8277.530862: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [002] ....  8277.533032: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [000] ....  8277.539390: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [000] ....  8277.546494: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [002] ....  8277.550594: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8277.556326: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8277.560414: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8277.566132: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [002] ....  8277.570213: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8277.575939: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8277.580028: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8277.585719: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [002] ....  8277.589773: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8277.595538: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8277.599643: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8277.605356: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [002] ....  8277.609463: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8277.615151: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8277.619253: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8277.624990: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [002] ....  8277.629080: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8277.630998: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=604928
       flush-8:0-4272  [000] ....  8277.631010: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=604928
              dd-4267  [002] ....  8277.631011: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=604928
              dd-4267  [002] ....  8277.632513: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=606976
              dd-4267  [002] ....  8277.634731: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
       flush-8:0-4272  [000] ....  8277.636659: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=604928
              dd-4267  [002] ....  8277.638506: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=609024
              dd-4267  [002] ....  8277.640040: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=611072
              dd-4267  [002] ....  8277.641550: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=613120
              dd-4267  [002] ....  8277.643094: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=615168
       flush-8:0-4272  [000] ....  8277.715839: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=606976
       flush-8:0-4272  [000] ....  8277.724516: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=609024
       flush-8:0-4272  [000] ....  8277.733064: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=611072
       flush-8:0-4272  [000] ....  8277.740840: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=613120
       flush-8:0-4272  [000] ....  8277.748062: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=615168
       flush-8:0-4272  [000] ....  8277.755291: writeback_pages_written: 12288
              dd-4267  [001] ....  8277.802545: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=617216
       flush-8:0-4272  [000] ....  8277.802565: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=617216
              dd-4267  [001] ....  8277.802566: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=617216
              dd-4267  [001] ....  8277.804416: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=619264
              dd-4267  [001] ....  8277.806516: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=621312
       flush-8:0-4272  [000] ....  8277.808332: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=617216
       flush-8:0-4272  [000] ....  8277.813105: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=619264
              dd-4267  [001] ....  8277.813218: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=623360
              dd-4267  [001] ....  8277.815522: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=625408
       flush-8:0-4272  [000] ....  8277.820794: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=621312
              dd-4267  [001] ....  8277.821744: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=627456
       flush-8:0-4272  [000] ....  8277.826650: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=623360
              dd-4267  [001] ....  8277.826949: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
       flush-8:0-4272  [000] ....  8277.832483: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=625408
       flush-8:0-4272  [000] ....  8277.837046: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1536 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=627456
       flush-8:0-4272  [000] ....  8277.840357: writeback_pages_written: 11776
              dd-4267  [003] ....  8277.931348: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [000] ....  8278.051459: writeback_congestion_wait: usec_timeout=100000 usec_delayed=55000
              dd-4267  [000] ....  8278.116343: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=628992
       flush-8:0-4272  [002] ....  8278.116363: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=628992
              dd-4267  [000] ....  8278.116364: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=628992
              dd-4267  [000] ....  8278.118347: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=631040
              dd-4267  [000] ....  8278.119860: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=633088
       flush-8:0-4272  [001] ....  8278.120392: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=628992
              dd-4267  [000] ....  8278.123157: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
       flush-8:0-4272  [001] ....  8278.123728: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=631040
       flush-8:0-4272  [001] ....  8278.130201: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1280 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=633088
       flush-8:0-4272  [001] ....  8278.133845: writeback_pages_written: 5376
              dd-4267  [002] ....  8278.225775: writeback_congestion_wait: usec_timeout=100000 usec_delayed=99000
              dd-4267  [002] ....  8278.294088: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=634368
       flush-8:0-4272  [001] ....  8278.294107: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=634368
              dd-4267  [002] ....  8278.294109: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=634368
       flush-8:0-4272  [001] ....  8278.297397: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=634368
       flush-8:0-4272  [001] ....  8278.297405: writeback_pages_written: 256
              dd-4267  [002] ....  8278.303627: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8278.308159: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [000] ....  8278.359684: writeback_congestion_wait: usec_timeout=100000 usec_delayed=51000
              dd-4267  [000] ....  8278.362293: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [002] ....  8278.414805: writeback_congestion_wait: usec_timeout=100000 usec_delayed=52000
              dd-4267  [000] ....  8278.417186: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8278.419534: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8278.421935: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8278.424848: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [002] ....  8278.523938: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [000] ....  8278.609350: writeback_congestion_wait: usec_timeout=100000 usec_delayed=85000
              dd-4267  [002] ....  8278.611904: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8278.614392: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8278.617127: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8278.621765: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [002] ....  8278.629512: writeback_congestion_wait: usec_timeout=100000 usec_delayed=8000
              dd-4267  [000] ....  8278.639410: writeback_congestion_wait: usec_timeout=100000 usec_delayed=10000
              dd-4267  [003] ....  8278.649387: writeback_congestion_wait: usec_timeout=100000 usec_delayed=10000
              dd-4267  [001] ....  8278.655016: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8278.659199: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8278.664970: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8278.744840: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8278.748992: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8278.754791: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [003] ....  8278.758983: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8278.764795: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8278.768952: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8278.774753: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [003] ....  8278.778942: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [001] ....  8278.784729: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [001] ....  8278.788914: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [001] ....  8278.794714: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [003] ....  8278.798872: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [001] ....  8278.804677: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [001] ....  8278.808866: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [001] ....  8278.814669: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [003] ....  8278.818829: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [001] ....  8278.824640: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [001] ....  8278.828858: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [001] ....  8278.834601: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [003] ....  8278.838792: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [000] ....  8278.901654: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8278.908163: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=634624
       flush-8:0-4272  [001] ....  8278.908182: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=634624
              dd-4267  [000] ....  8278.908650: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=634624
              dd-4267  [000] ....  8278.911296: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
       flush-8:0-4272  [001] ....  8278.913786: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=634624
       flush-8:0-4272  [001] ....  8278.913794: writeback_pages_written: 256
              dd-4267  [002] ....  8278.921513: writeback_congestion_wait: usec_timeout=100000 usec_delayed=8000
              dd-4267  [000] ....  8278.925513: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [000] ....  8278.931324: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8278.935488: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8278.941321: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8278.945468: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8278.951282: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [002] ....  8278.955458: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8278.961249: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8278.964003: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=634880
       flush-8:0-4272  [001] ....  8278.964020: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=634880
              dd-4267  [000] ....  8278.964035: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=634880
              dd-4267  [000] ....  8278.966502: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=636928
              dd-4267  [000] ....  8278.968844: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=638976
       flush-8:0-4272  [001] ....  8278.969634: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=634880
              dd-4267  [000] ....  8278.971204: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=641024
       flush-8:0-4272  [001] ....  8278.973125: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=636928
              dd-4267  [000] ....  8278.975599: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=643072
              dd-4267  [000] ....  8278.977264: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=645120
       flush-8:0-4272  [001] ....  8278.982123: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=638976
              dd-4267  [000] ....  8278.985235: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=647168
              dd-4267  [000] ....  8278.986899: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=649216
              dd-4267  [000] ....  8278.988478: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=651264
       flush-8:0-4272  [001] ....  8279.107478: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=641024
       flush-8:0-4272  [001] ....  8279.115590: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=643072
       flush-8:0-4272  [001] ....  8279.122629: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=645120
       flush-8:0-4272  [001] ....  8279.128637: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=647168
       flush-8:0-4272  [001] ....  8279.134624: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=649216
       flush-8:0-4272  [001] ....  8279.139908: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1024 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=651264
       flush-8:0-4272  [001] ....  8279.143584: writeback_pages_written: 17408
              dd-4267  [000] ....  8279.191039: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=652288
       flush-8:0-4272  [001] ....  8279.191060: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=652288
              dd-4267  [000] ....  8279.191070: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=652288
              dd-4267  [000] ....  8279.192824: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=654336
       flush-8:0-4272  [001] ....  8279.194897: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=652288
       flush-8:0-4272  [001] ....  8279.198344: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1280 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=654336
              dd-4267  [000] ....  8279.198346: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=655360
              dd-4267  [000] ....  8279.199880: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=657408
       flush-8:0-4272  [001] ....  8279.202409: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=655360
       flush-8:0-4272  [001] ....  8279.205272: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1792 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=657408
       flush-8:0-4272  [001] ....  8279.209400: writeback_pages_written: 6912
              dd-4267  [002] ....  8279.300548: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [003] ....  8279.420643: writeback_congestion_wait: usec_timeout=100000 usec_delayed=59000
              dd-4267  [003] ....  8279.473709: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=659200
       flush-8:0-4272  [001] ....  8279.473725: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=659200
              dd-4267  [003] ....  8279.473735: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=659200
       flush-8:0-4272  [001] ....  8279.477129: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=512 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=659200
       flush-8:0-4272  [001] ....  8279.477537: writeback_pages_written: 512
              dd-4267  [001] ....  8279.567996: writeback_congestion_wait: usec_timeout=100000 usec_delayed=94000
              dd-4267  [001] ....  8279.570412: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [001] ....  8279.575103: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [001] ....  8279.583971: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [001] ....  8279.594089: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [003] ....  8279.605732: writeback_congestion_wait: usec_timeout=100000 usec_delayed=8000
              dd-4267  [001] ....  8279.615825: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8279.625850: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8279.635943: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [003] ....  8279.646070: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8279.656125: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8279.666165: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8279.676215: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [003] ....  8279.686308: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [001] ....  8279.696389: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [001] ....  8279.706454: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8279.716539: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8279.720112: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=659712
       flush-8:0-4272  [003] ....  8279.720126: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=659712
              dd-4267  [001] ....  8279.720127: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=659712
              dd-4267  [001] ....  8279.721617: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=661760
              dd-4267  [001] ....  8279.723124: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=663808
       flush-8:0-4272  [003] ....  8279.724157: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=659712
              dd-4267  [003] ....  8279.728278: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
       flush-8:0-4272  [000] ....  8279.728765: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=661760
       flush-8:0-4272  [000] ....  8279.736675: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1024 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=663808
              dd-4267  [001] ....  8279.738442: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
       flush-8:0-4272  [000] ....  8279.740568: writeback_pages_written: 5120
              dd-4267  [001] ....  8279.748446: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [001] ....  8279.758431: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [003] ....  8279.768498: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [001] ....  8279.778569: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8279.788654: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [001] ....  8279.798693: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [003] ....  8279.808736: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [001] ....  8279.818831: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8279.828949: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8279.839007: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8279.842997: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=664832
       flush-8:0-4272  [000] ....  8279.843014: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=664832
              dd-4267  [001] ....  8279.843016: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=664832
       flush-8:0-4272  [000] ....  8279.847182: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=664832
       flush-8:0-4272  [000] ....  8279.847191: writeback_pages_written: 256
              dd-4267  [003] ....  8279.849076: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8279.853108: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [001] ....  8279.863346: writeback_congestion_wait: usec_timeout=100000 usec_delayed=9000
              dd-4267  [001] ....  8279.874975: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [003] ....  8279.880557: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [001] ....  8279.884813: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [001] ....  8279.890544: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8279.894781: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [003] ....  8279.900525: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8279.904786: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [001] ....  8279.910495: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8279.914742: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [003] ....  8279.920501: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8279.924716: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [001] ....  8279.930462: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8279.934704: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [003] ....  8279.940466: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8279.944675: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [001] ....  8279.950443: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8279.954671: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [003] ....  8279.960415: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8279.964640: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [001] ....  8279.970407: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8279.974635: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [003] ....  8279.980386: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8279.984609: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [001] ....  8279.990382: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8279.992689: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=665088
       flush-8:0-4272  [000] ....  8279.992705: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=665088
              dd-4267  [001] ....  8279.992706: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=665088
              dd-4267  [001] ....  8279.996636: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=667136
       flush-8:0-4272  [000] ....  8279.998271: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=665088
              dd-4267  [001] ....  8279.998348: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=669184
              dd-4267  [001] ....  8279.999931: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=671232
       flush-8:0-4272  [000] ....  8280.002961: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=667136
              dd-4267  [001] ....  8280.004506: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
       flush-8:0-4272  [000] ....  8280.010048: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=669184
              dd-4267  [001] ....  8280.010313: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8280.012905: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=673280
       flush-8:0-4272  [000] ....  8280.016172: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=671232
              dd-4267  [001] ....  8280.017137: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=675328
              dd-4267  [001] ....  8280.018724: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [002] ....  8280.020975: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8280.023205: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [003] ....  8280.025762: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
       flush-8:0-4272  [002] ....  8280.166717: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=673280
       flush-8:0-4272  [002] ....  8280.172594: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1280 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=675328
       flush-8:0-4272  [002] ....  8280.176266: writeback_pages_written: 11520
              dd-4267  [003] ....  8280.266612: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=676608
       flush-8:0-4272  [002] ....  8280.266632: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=676608
              dd-4267  [003] ....  8280.266635: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=676608
              dd-4267  [003] ....  8280.268438: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=678656
       flush-8:0-4272  [002] ....  8280.270275: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=676608
              dd-4267  [003] ....  8280.272860: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=680704
       flush-8:0-4272  [002] ....  8280.273667: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=678656
              dd-4267  [003] ....  8280.277004: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=682752
       flush-8:0-4272  [002] ....  8280.279511: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=680704
       flush-8:0-4272  [002] ....  8280.284580: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=682752
       flush-8:0-4272  [002] ....  8280.286725: writeback_pages_written: 6912
      flush-0:16-4223  [001] ....  8280.294981: writeback_start: bdi 0:16: sb_dev 0:0 nr_pages=29073 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
      flush-0:16-4223  [001] ....  8280.294985: writeback_queue_io: bdi 0:16: older=4302921993 age=30000 enqueue=3 reason=periodic
      flush-0:16-4223  [001] ....  8280.294994: writeback_single_inode: bdi 0:16: ino=3080195 state= dirtied_when=4302920855 age=56 index=0 to_write=1024 wrote=0
      flush-0:16-4223  [001] ....  8280.295020: writeback_single_inode: bdi 0:16: ino=1573416 state= dirtied_when=4302921098 age=56 index=0 to_write=1024 wrote=1
      flush-0:16-4223  [001] ....  8280.295032: writeback_single_inode: bdi 0:16: ino=1573486 state= dirtied_when=4302921098 age=56 index=52481 to_write=1024 wrote=1
      flush-0:16-4223  [001] ....  8280.295035: writeback_written: bdi 0:16: sb_dev 0:0 nr_pages=29071 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
      flush-0:16-4223  [001] ....  8280.295036: writeback_start: bdi 0:16: sb_dev 0:0 nr_pages=29071 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
      flush-0:16-4223  [001] ....  8280.295037: writeback_queue_io: bdi 0:16: older=4302921993 age=30000 enqueue=0 reason=periodic
      flush-0:16-4223  [001] ....  8280.295037: writeback_written: bdi 0:16: sb_dev 0:0 nr_pages=29071 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
      flush-0:16-4223  [001] ....  8280.295042: writeback_pages_written: 2
              dd-4267  [001] ....  8280.297036: writeback_congestion_wait: usec_timeout=100000 usec_delayed=20000
              dd-4267  [003] ....  8280.408987: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [002] ....  8280.467718: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8280.477905: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [000] ....  8280.488203: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8280.492193: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=683520
       flush-8:0-4272  [002] ....  8280.492204: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=683520
              dd-4267  [000] ....  8280.492205: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=683520
              dd-4267  [000] ....  8280.494904: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
       flush-8:0-4272  [002] ....  8280.495455: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=683520
       flush-8:0-4272  [002] ....  8280.495461: writeback_pages_written: 256
              dd-4267  [002] ....  8280.499328: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8280.503823: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8280.508200: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8280.516395: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=683776
       flush-8:0-4272  [002] ....  8280.516410: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=683776
              dd-4267  [000] ....  8280.516410: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=683776
              dd-4267  [000] ....  8280.516691: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [000] ....  8280.516877: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
       flush-8:0-4272  [002] ....  8280.520252: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=683776
       flush-8:0-4272  [002] ....  8280.520259: writeback_pages_written: 256
              dd-4267  [000] ....  8280.521328: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8280.525593: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8280.613120: writeback_congestion_wait: usec_timeout=100000 usec_delayed=84000
              dd-4267  [000] ....  8280.621118: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8280.628140: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [002] ....  8280.636293: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8280.640774: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8280.645305: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [000] ....  8280.649723: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [002] ....  8280.657246: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8280.661750: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8280.670022: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=684032
       flush-8:0-4272  [002] ....  8280.670036: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=684032
              dd-4267  [000] ....  8280.670040: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=684032
              dd-4267  [000] ....  8280.671552: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=686080
              dd-4267  [000] ....  8280.673073: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=688128
       flush-8:0-4272  [002] ....  8280.674106: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=684032
       flush-8:0-4272  [001] ....  8280.677134: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=686080
              dd-4267  [000] ....  8280.680892: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=690176
       flush-8:0-4272  [001] ....  8280.685604: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=688128
              dd-4267  [000] ....  8280.686261: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
       flush-8:0-4272  [001] ....  8280.691619: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=690176
       flush-8:0-4272  [001] ....  8280.693880: writeback_pages_written: 6400
              dd-4267  [000] ....  8280.699811: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8280.725768: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [002] ....  8280.735583: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8280.745458: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8280.749433: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=690432
       flush-8:0-4272  [001] ....  8280.749453: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=690432
              dd-4267  [000] ....  8280.749455: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=690432
       flush-8:0-4272  [001] ....  8280.753038: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=690432
       flush-8:0-4272  [001] ....  8280.753044: writeback_pages_written: 256
              dd-4267  [000] ....  8280.755184: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8280.763233: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [002] ....  8280.767808: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8280.772434: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8280.776929: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8280.781392: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [002] ....  8280.785741: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8280.790431: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8280.805974: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=690688
       flush-8:0-4272  [001] ....  8280.805990: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=690688
              dd-4267  [000] ....  8280.805995: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=690688
       flush-8:0-4272  [001] ....  8280.810334: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=690688
       flush-8:0-4272  [001] ....  8280.810343: writeback_pages_written: 256
              dd-4267  [002] ....  8280.835906: writeback_congestion_wait: usec_timeout=100000 usec_delayed=30000
              dd-4267  [002] ....  8280.942587: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [000] ....  8280.999893: writeback_congestion_wait: usec_timeout=100000 usec_delayed=57000
              dd-4267  [000] ....  8281.002359: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [000] ....  8281.004620: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [002] ....  8281.007144: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8281.011505: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8281.015104: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=690944
       flush-8:0-4272  [001] ....  8281.015120: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=690944
              dd-4267  [000] ....  8281.015121: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=690944
              dd-4267  [000] ....  8281.018879: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
       flush-8:0-4272  [001] ....  8281.019801: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=690944
       flush-8:0-4272  [001] ....  8281.021129: writeback_pages_written: 768
              dd-4267  [000] ....  8281.023390: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=691712
       flush-8:0-4272  [001] ....  8281.023405: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=691712
              dd-4267  [000] ....  8281.023406: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=691712
              dd-4267  [000] ....  8281.025078: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=693760
              dd-4267  [000] ....  8281.026726: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=695808
       flush-8:0-4272  [001] ....  8281.027870: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=691712
              dd-4267  [000] ....  8281.028680: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
       flush-8:0-4272  [001] ....  8281.032562: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=693760
              dd-4267  [000] ....  8281.034082: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=697856
              dd-4267  [002] ....  8281.038551: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
       flush-8:0-4272  [001] ....  8281.039182: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=695808
              dd-4267  [002] ....  8281.043151: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=699904
       flush-8:0-4272  [001] ....  8281.045416: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=697856
       flush-8:0-4272  [001] ....  8281.050590: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1280 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=699904
       flush-8:0-4272  [001] ....  8281.054085: writeback_pages_written: 9472
              dd-4267  [000] ....  8281.146586: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [000] ....  8281.267657: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=701184
       flush-8:0-4272  [001] ....  8281.267676: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=701184
              dd-4267  [000] ....  8281.267682: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=701184
              dd-4267  [000] ....  8281.269420: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=703232
       flush-8:0-4272  [001] ....  8281.273221: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=701184
              dd-4267  [000] ....  8281.275742: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=705280
       flush-8:0-4272  [001] ....  8281.276549: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=703232
              dd-4267  [000] ....  8281.277912: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=707328
              dd-4267  [003] ....  8281.280371: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
       flush-8:0-4272  [001] ....  8281.318655: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=705280
       flush-8:0-4272  [001] ....  8281.324616: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=707328
       flush-8:0-4272  [001] ....  8281.328030: writeback_pages_written: 6912
              dd-4267  [002] ....  8281.450334: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [003] ....  8281.461260: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [003] ....  8281.483632: writeback_congestion_wait: usec_timeout=100000 usec_delayed=20000
              dd-4267  [003] ....  8281.486146: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [001] ....  8281.488493: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [003] ....  8281.491058: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [003] ....  8281.495253: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [003] ....  8281.503364: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [003] ....  8281.513429: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [003] ....  8281.522995: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [003] ....  8281.532819: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [003] ....  8281.542599: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [003] ....  8281.546558: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=708096
       flush-8:0-4272  [001] ....  8281.546572: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=708096
              dd-4267  [003] ....  8281.546572: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=708096
              dd-4267  [001] ....  8281.556507: writeback_congestion_wait: usec_timeout=100000 usec_delayed=10000
       flush-8:0-4272  [003] ....  8282.192050: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=708096
       flush-8:0-4272  [003] ....  8282.192057: writeback_pages_written: 256
              dd-4267  [003] ....  8282.363940: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=708352
       flush-8:0-4272  [001] ....  8282.363957: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=708352
              dd-4267  [003] ....  8282.363965: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=708352
              dd-4267  [003] ....  8282.365488: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=710400
              dd-4267  [003] ....  8282.366985: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=712448
              dd-4267  [003] ....  8282.368510: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=714496
       flush-8:0-4272  [001] ....  8282.369497: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=708352
              dd-4267  [003] ....  8282.370048: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=716544
              dd-4267  [003] ....  8282.371633: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=718592
       flush-8:0-4272  [000] ....  8282.374078: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=710400
              dd-4267  [003] ....  8282.378185: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=720640
       flush-8:0-4272  [000] ....  8282.385965: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=712448
              dd-4267  [003] ....  8282.389259: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=722688
              dd-4267  [003] ....  8282.390925: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=724736
       flush-8:0-4272  [000] ....  8282.394370: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=714496
       flush-8:0-4272  [000] ....  8282.403366: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=716544
       flush-8:0-4272  [000] ....  8282.411168: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=718592
       flush-8:0-4272  [000] ....  8282.418447: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=720640
              dd-4267  [003] ....  8282.419012: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=726784
              dd-4267  [003] ....  8282.420711: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=728832
       flush-8:0-4272  [000] ....  8282.425497: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=722688
       flush-8:0-4272  [000] ....  8282.432174: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=724736
              dd-4267  [003] ....  8282.435607: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=730880
       flush-8:0-4272  [000] ....  8282.437550: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=726784
       flush-8:0-4272  [000] ....  8282.443269: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=728832
       flush-8:0-4272  [000] ....  8282.447516: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1792 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=730880
              dd-4267  [001] ....  8282.536821: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
       flush-8:0-4272  [002] ....  8282.594711: writeback_start: bdi 8:0: sb_dev 0:0 nr_pages=28847 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
       flush-8:0-4272  [002] ....  8282.594714: writeback_queue_io: bdi 8:0: older=4302924294 age=30000 enqueue=0 reason=periodic
       flush-8:0-4272  [002] ....  8282.594714: writeback_written: bdi 8:0: sb_dev 0:0 nr_pages=28847 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
       flush-8:0-4272  [002] ....  8282.594717: writeback_pages_written: 24320
              dd-4267  [003] ....  8282.702707: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [003] ....  8282.766257: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=732672
       flush-8:0-4272  [002] ....  8282.766275: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=732672
              dd-4267  [003] ....  8282.766276: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=732672
              dd-4267  [000] ....  8282.769626: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
       flush-8:0-4272  [002] ....  8282.770439: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=732672
       flush-8:0-4272  [002] ....  8282.770448: writeback_pages_written: 256
              dd-4267  [000] ....  8282.792970: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=732928
       flush-8:0-4272  [002] ....  8282.792985: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=732928
              dd-4267  [000] ....  8282.792991: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=732928
       flush-8:0-4272  [002] ....  8282.796612: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=512 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=732928
       flush-8:0-4272  [002] ....  8282.797095: writeback_pages_written: 512
              dd-4267  [001] ....  8282.803815: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=733440
       flush-8:0-4272  [002] ....  8282.803828: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=733440
              dd-4267  [000] ....  8282.805987: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
       flush-8:0-4272  [002] ....  8282.807590: writeback_pages_written: 256
              dd-4267  [002] ....  8282.816053: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8282.826012: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [000] ....  8282.836053: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8282.840123: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [002] ....  8282.862607: writeback_congestion_wait: usec_timeout=100000 usec_delayed=21000
              dd-4267  [000] ....  8282.866842: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8282.872692: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8282.876853: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [002] ....  8282.882737: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8282.887007: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8282.892840: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8282.897051: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [002] ....  8282.902856: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8282.907142: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8282.912979: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8282.917185: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [002] ....  8282.923050: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8282.927261: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8282.933120: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8282.937351: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [002] ....  8282.943169: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8282.947400: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8282.953258: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8282.957497: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8282.963315: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8282.965832: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8282.967943: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8282.976215: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=733696
       flush-8:0-4272  [002] ....  8282.976230: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=733696
              dd-4267  [000] ....  8282.976230: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=733696
              dd-4267  [000] ....  8282.977340: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
       flush-8:0-4272  [002] ....  8282.980514: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=733696
              dd-4267  [000] ....  8282.981192: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=734464
       flush-8:0-4272  [002] ....  8282.981312: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=734464
              dd-4267  [000] ....  8282.981745: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8282.981778: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=734464
       flush-8:0-4272  [002] ....  8282.985084: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=734464
       flush-8:0-4272  [002] ....  8282.985091: writeback_pages_written: 1024
              dd-4267  [000] ....  8282.985665: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=734720
       flush-8:0-4272  [002] ....  8282.985674: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=734720
              dd-4267  [000] ....  8282.985675: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=734720
       flush-8:0-4272  [002] ....  8282.989807: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=512 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=734720
       flush-8:0-4272  [002] ....  8282.990230: writeback_pages_written: 512
              dd-4267  [003] ....  8283.089416: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [000] ....  8283.122929: writeback_congestion_wait: usec_timeout=100000 usec_delayed=33000
              dd-4267  [000] ....  8283.135611: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8283.145666: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8283.155723: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [002] ....  8283.165791: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8283.176075: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [000] ....  8283.181528: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8283.185688: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8283.187835: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=735232
       flush-8:0-4272  [002] ....  8283.187848: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=735232
              dd-4267  [000] ....  8283.187849: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=735232
              dd-4267  [000] ....  8283.191527: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
       flush-8:0-4272  [002] ....  8283.192426: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=512 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=735232
       flush-8:0-4272  [002] ....  8283.192847: writeback_pages_written: 512
              dd-4267  [000] ....  8283.195835: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8283.201634: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8283.205915: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8283.211685: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8283.215994: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8283.221809: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8283.226052: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [002] ....  8283.231889: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8283.236089: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8283.241969: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8283.246186: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [002] ....  8283.252004: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8283.256226: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8283.262106: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8283.266341: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8283.273554: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [000] ....  8283.277824: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8283.283655: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8283.287868: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8283.293646: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8283.297947: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8283.303809: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8283.312730: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8283.328557: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=735744
       flush-8:0-4272  [002] ....  8283.328574: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=735744
              dd-4267  [000] ....  8283.328582: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=735744
              dd-4267  [000] ....  8283.330129: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=737792
              dd-4267  [000] ....  8283.331848: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=739840
              dd-4267  [000] ....  8283.333393: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=741888
       flush-8:0-4272  [002] ....  8283.333676: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=735744
              dd-4267  [000] ....  8283.334935: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=743936
              dd-4267  [000] ....  8283.337887: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8283.338115: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
       flush-8:0-4272  [002] ....  8283.431725: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=737792
       flush-8:0-4272  [002] ....  8283.439317: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=739840
       flush-8:0-4272  [002] ....  8283.447089: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=741888
       flush-8:0-4272  [002] ....  8283.454001: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1792 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=743936
       flush-8:0-4272  [002] ....  8283.459772: writeback_pages_written: 9984
              dd-4267  [001] ....  8283.537146: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=745728
       flush-8:0-4272  [002] ....  8283.537162: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=745728
              dd-4267  [001] ....  8283.537171: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=745728
              dd-4267  [001] ....  8283.539419: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=747776
              dd-4267  [001] ....  8283.541208: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=749824
       flush-8:0-4272  [002] ....  8283.541833: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=745728
       flush-8:0-4272  [002] ....  8283.545118: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=747776
              dd-4267  [001] ....  8283.545748: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=751872
              dd-4267  [001] ....  8283.547488: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=753920
       flush-8:0-4272  [002] ....  8283.552959: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=749824
              dd-4267  [001] ....  8283.558579: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=755968
       flush-8:0-4272  [002] ....  8283.559396: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=751872
              dd-4267  [001] ....  8283.561724: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
       flush-8:0-4272  [002] ....  8283.565992: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=753920
       flush-8:0-4272  [002] ....  8283.570817: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1792 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=755968
       flush-8:0-4272  [002] ....  8283.574693: writeback_pages_written: 12032
              dd-4267  [003] ....  8283.668117: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [000] ....  8283.743847: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8283.747107: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=757760
       flush-8:0-4272  [002] ....  8283.747123: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=757760
              dd-4267  [000] ....  8283.747124: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=757760
       flush-8:0-4272  [002] ....  8283.751308: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1792 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=757760
              dd-4267  [000] ....  8283.751309: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=759296
              dd-4267  [000] ....  8283.752829: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=761344
       flush-8:0-4272  [002] ....  8283.753579: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=759296
              dd-4267  [001] ....  8283.756564: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=763392
       flush-8:0-4272  [002] ....  8283.756906: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=761344
       flush-8:0-4272  [002] ....  8283.762110: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=763392
       flush-8:0-4272  [002] ....  8283.764271: writeback_pages_written: 6400
              dd-4267  [003] ....  8283.857011: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [002] ....  8283.973375: writeback_congestion_wait: usec_timeout=100000 usec_delayed=55000
              dd-4267  [002] ....  8284.011625: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=764160
       flush-8:0-4272  [000] ....  8284.011640: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=764160
              dd-4267  [002] ....  8284.011645: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=764160
              dd-4267  [002] ....  8284.013208: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
       flush-8:0-4272  [000] ....  8284.015143: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=764160
       flush-8:0-4272  [000] ....  8284.015149: writeback_pages_written: 256
              dd-4267  [000] ....  8284.019069: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8284.023215: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8284.030639: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [002] ....  8284.034941: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8284.040920: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [000] ....  8284.051372: writeback_congestion_wait: usec_timeout=100000 usec_delayed=8000
              dd-4267  [000] ....  8284.061660: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [002] ....  8284.071910: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [000] ....  8284.082199: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [000] ....  8284.092449: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8284.102756: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8284.106747: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=764416
       flush-8:0-4272  [002] ....  8284.106761: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=764416
              dd-4267  [000] ....  8284.106772: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=764416
       flush-8:0-4272  [002] ....  8284.110412: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=764416
       flush-8:0-4272  [002] ....  8284.110420: writeback_pages_written: 256
              dd-4267  [002] ....  8284.113017: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [002] ....  8284.117024: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=764672
       flush-8:0-4272  [000] ....  8284.117035: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=764672
              dd-4267  [002] ....  8284.117041: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=764672
       flush-8:0-4272  [000] ....  8284.120264: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=764672
       flush-8:0-4272  [000] ....  8284.120271: writeback_pages_written: 256
              dd-4267  [000] ....  8284.123300: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8284.133583: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [000] ....  8284.145288: writeback_congestion_wait: usec_timeout=100000 usec_delayed=8000
              dd-4267  [002] ....  8284.149617: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [000] ....  8284.153959: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8284.159928: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [000] ....  8284.164244: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [002] ....  8284.170143: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8284.174494: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8284.180462: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8284.184792: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8284.190718: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8284.195060: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8284.201037: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8284.205349: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8284.211282: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8284.215607: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8284.217861: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=764928
       flush-8:0-4272  [002] ....  8284.217873: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=764928
              dd-4267  [000] ....  8284.217883: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=764928
              dd-4267  [000] ....  8284.219412: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=766976
       flush-8:0-4272  [002] ....  8284.222662: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=764928
              dd-4267  [000] ....  8284.224852: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=769024
       flush-8:0-4272  [001] ....  8284.226978: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=766976
              dd-4267  [000] ....  8284.231789: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
       flush-8:0-4272  [001] ....  8284.234737: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1280 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=769024
              dd-4267  [000] ....  8284.236127: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8284.242083: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
       flush-8:0-4272  [001] ....  8284.537975: writeback_pages_written: 5376
              dd-4267  [002] ....  8284.712091: writeback_congestion_wait: usec_timeout=100000 usec_delayed=97000
              dd-4267  [002] ....  8284.714592: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [002] ....  8284.718902: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [002] ....  8284.721113: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [002] ....  8284.723558: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8284.729274: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8284.730891: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=770304
       flush-8:0-4272  [001] ....  8284.730909: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=770304
              dd-4267  [000] ....  8284.731346: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=770304
              dd-4267  [000] ....  8284.733058: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=772352
       flush-8:0-4272  [001] ....  8284.735492: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=770304
              dd-4267  [000] ....  8284.736899: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=774400
              dd-4267  [000] ....  8284.738556: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=776448
       flush-8:0-4272  [001] ....  8284.738766: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=772352
              dd-4267  [000] ....  8284.743348: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
       flush-8:0-4272  [001] ....  8284.745968: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=774400
              dd-4267  [000] ....  8284.749665: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=778496
       flush-8:0-4272  [001] ....  8284.751572: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=776448
              dd-4267  [000] ....  8284.753153: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
       flush-8:0-4272  [001] ....  8284.757398: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=778496
              dd-4267  [000] ....  8284.757400: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=779008
       flush-8:0-4272  [001] ....  8284.760124: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1024 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=779008
       flush-8:0-4272  [001] ....  8284.761353: writeback_pages_written: 9728
              dd-4267  [002] ....  8284.857465: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [002] ....  8284.912787: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=780032
       flush-8:0-4272  [001] ....  8284.912808: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=780032
              dd-4267  [002] ....  8284.912815: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=780032
       flush-8:0-4272  [001] ....  8284.916025: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=780032
       flush-8:0-4272  [001] ....  8284.918844: writeback_pages_written: 2048
              dd-4267  [003] ....  8284.940686: writeback_congestion_wait: usec_timeout=100000 usec_delayed=26000
              dd-4267  [003] ....  8284.978581: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=782080
       flush-8:0-4272  [001] ....  8284.978599: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=782080
              dd-4267  [003] ....  8284.978602: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=782080
       flush-8:0-4272  [001] ....  8284.982521: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=512 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=782080
       flush-8:0-4272  [001] ....  8284.982946: writeback_pages_written: 512
              dd-4267  [003] ....  8284.983056: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=782592
       flush-8:0-4272  [001] ....  8284.983063: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=782592
              dd-4267  [003] ....  8284.983064: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=782592
              dd-4267  [003] ....  8284.984577: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=784640
       flush-8:0-4272  [001] ....  8284.985859: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=782592
       flush-8:0-4272  [000] ....  8284.989292: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1280 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=784640
              dd-4267  [003] ....  8284.990536: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=785664
              dd-4267  [003] ....  8284.992120: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=787712
       flush-8:0-4272  [000] ....  8284.994372: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=785664
       flush-8:0-4272  [000] ....  8284.997795: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=787712
       flush-8:0-4272  [000] ....  8285.000496: writeback_pages_written: 5888
              dd-4267  [001] ....  8285.092449: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [001] ....  8285.177444: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=788480
       flush-8:0-4272  [000] ....  8285.177462: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=788480
              dd-4267  [001] ....  8285.177462: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=788480
              dd-4267  [001] ....  8285.179141: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
       flush-8:0-4272  [000] ....  8285.181333: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=788480
       flush-8:0-4272  [000] ....  8285.181342: writeback_pages_written: 256
              dd-4267  [001] ....  8285.183347: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=788736
       flush-8:0-4272  [000] ....  8285.183364: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=788736
              dd-4267  [001] ....  8285.183373: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=788736
       flush-8:0-4272  [000] ....  8285.185778: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=788736
       flush-8:0-4272  [000] ....  8285.185784: writeback_pages_written: 256
              dd-4267  [001] ....  8285.188015: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8285.197816: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [003] ....  8285.207603: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8285.217253: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8285.219617: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [003] ....  8285.224224: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [003] ....  8285.260785: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=788992
       flush-8:0-4272  [000] ....  8285.260803: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=788992
              dd-4267  [003] ....  8285.260804: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=788992
              dd-4267  [003] ....  8285.262513: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=791040
       flush-8:0-4272  [000] ....  8285.265222: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=788992
       flush-8:0-4272  [000] ....  8285.269537: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1792 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=791040
              dd-4267  [001] ....  8285.270972: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
       flush-8:0-4272  [000] ....  8285.275330: writeback_pages_written: 3840
              dd-4267  [003] ....  8285.324595: writeback_congestion_wait: usec_timeout=100000 usec_delayed=46000
              dd-4267  [001] ....  8285.428150: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [001] ....  8285.428951: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [001] ....  8285.433539: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [003] ....  8285.440157: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [001] ....  8285.449269: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [001] ....  8285.455291: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=792832
       flush-8:0-4272  [000] ....  8285.455308: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=792832
              dd-4267  [001] ....  8285.455310: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=792832
              dd-4267  [001] ....  8285.457006: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=794880
              dd-4267  [001] ....  8285.458912: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
       flush-8:0-4272  [000] ....  8285.459455: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=792832
       flush-8:0-4272  [000] ....  8285.462753: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=794880
              dd-4267  [003] ....  8285.462940: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
       flush-8:0-4272  [000] ....  8285.465164: writeback_pages_written: 2304
              dd-4267  [003] ....  8285.465311: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=795136
       flush-8:0-4272  [000] ....  8285.465328: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=795136
              dd-4267  [003] ....  8285.465330: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=795136
       flush-8:0-4272  [000] ....  8285.467314: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=795136
       flush-8:0-4272  [000] ....  8285.467319: writeback_pages_written: 256
              dd-4267  [001] ....  8285.478696: writeback_congestion_wait: usec_timeout=100000 usec_delayed=13000
              dd-4267  [002] ....  8285.511501: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8285.519624: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [002] ....  8285.529305: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [002] ....  8285.529483: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [000] ....  8285.541032: writeback_congestion_wait: usec_timeout=100000 usec_delayed=9000
              dd-4267  [000] ....  8285.577136: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=795392
       flush-8:0-4272  [002] ....  8285.577156: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=795392
              dd-4267  [000] ....  8285.577638: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=795392
       flush-8:0-4272  [002] ....  8285.581251: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=512 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=795392
       flush-8:0-4272  [002] ....  8285.581647: writeback_pages_written: 512
              dd-4267  [002] ....  8285.676988: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [000] ....  8285.698727: writeback_congestion_wait: usec_timeout=100000 usec_delayed=21000
              dd-4267  [000] ....  8285.701218: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8285.710431: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8285.718230: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8285.722253: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=795904
       flush-8:0-4272  [002] ....  8285.722266: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=795904
              dd-4267  [000] ....  8285.722267: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=795904
              dd-4267  [000] ....  8285.723781: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=797952
              dd-4267  [000] ....  8285.725289: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=800000
       flush-8:0-4272  [002] ....  8285.726260: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=795904
              dd-4267  [000] ....  8285.728248: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
       flush-8:0-4272  [002] ....  8285.729667: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=797952
       flush-8:0-4272  [002] ....  8285.735947: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1536 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=800000
       flush-8:0-4272  [002] ....  8285.740515: writeback_pages_written: 5632
              dd-4267  [000] ....  8285.743317: writeback_congestion_wait: usec_timeout=100000 usec_delayed=11000
              dd-4267  [000] ....  8285.748737: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=801536
       flush-8:0-4272  [002] ....  8285.748749: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=801536
              dd-4267  [000] ....  8285.748750: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=801536
              dd-4267  [000] ....  8285.750301: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=803584
       flush-8:0-4272  [002] ....  8285.751451: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=801536
              dd-4267  [000] ....  8285.789981: writeback_congestion_wait: usec_timeout=100000 usec_delayed=39000
       flush-8:0-4272  [000] ....  8285.820051: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1024 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=803584
       flush-8:0-4272  [000] ....  8285.824204: writeback_pages_written: 3072
              dd-4267  [002] ....  8285.877041: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=804608
       flush-8:0-4272  [000] ....  8285.877059: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=804608
              dd-4267  [002] ....  8285.877059: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=804608
       flush-8:0-4272  [000] ....  8285.880219: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=804608
              dd-4267  [002] ....  8285.881976: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
       flush-8:0-4272  [000] ....  8285.883304: writeback_pages_written: 2048
              dd-4267  [000] ....  8285.913064: writeback_congestion_wait: usec_timeout=100000 usec_delayed=27000
              dd-4267  [000] ....  8285.953294: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8285.993092: writeback_congestion_wait: usec_timeout=100000 usec_delayed=40000
              dd-4267  [000] ....  8286.017921: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=806656
       flush-8:0-4272  [002] ....  8286.017941: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=806656
              dd-4267  [000] ....  8286.017942: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=806656
       flush-8:0-4272  [002] ....  8286.022344: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=806656
       flush-8:0-4272  [002] ....  8286.022351: writeback_pages_written: 256
              dd-4267  [000] ....  8286.022457: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=806912
       flush-8:0-4272  [002] ....  8286.022463: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=806912
              dd-4267  [000] ....  8286.022464: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=806912
       flush-8:0-4272  [002] ....  8286.025804: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1024 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=806912
              dd-4267  [001] ....  8286.025808: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=807680
       flush-8:0-4272  [002] ....  8286.027152: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1792 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=807680
              dd-4267  [001] ....  8286.027154: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=809216
       flush-8:0-4272  [002] ....  8286.030162: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1536 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=809216
              dd-4267  [001] ....  8286.030934: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=810752
       flush-8:0-4272  [002] ....  8286.032432: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1792 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=810752
              dd-4267  [001] ....  8286.032436: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=812288
       flush-8:0-4272  [002] ....  8286.037770: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=812288
       flush-8:0-4272  [002] ....  8286.038525: writeback_pages_written: 6144
              dd-4267  [003] ....  8286.131784: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [003] ....  8286.193153: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=813056
       flush-8:0-4272  [002] ....  8286.193171: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=813056
              dd-4267  [003] ....  8286.193182: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=813056
              dd-4267  [000] ....  8286.194538: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
       flush-8:0-4272  [002] ....  8286.197515: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=512 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=813056
       flush-8:0-4272  [002] ....  8286.198149: writeback_pages_written: 512
              dd-4267  [000] ....  8286.204432: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [002] ....  8286.208413: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [000] ....  8286.214233: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8286.218379: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8286.224189: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [002] ....  8286.228384: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8286.234169: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8286.238355: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8286.244159: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [002] ....  8286.248349: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8286.254131: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8286.258310: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8286.268537: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [002] ....  8286.278455: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8286.288438: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8286.298422: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [002] ....  8286.308378: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [002] ....  8286.311951: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=813568
       flush-8:0-4272  [000] ....  8286.311964: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=813568
              dd-4267  [002] ....  8286.311974: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=813568
              dd-4267  [002] ....  8286.313470: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=815616
       flush-8:0-4272  [000] ....  8286.316042: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=813568
              dd-4267  [002] ....  8286.318069: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
       flush-8:0-4272  [000] ....  8286.319085: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1536 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=815616
              dd-4267  [001] ....  8286.323896: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
       flush-8:0-4272  [000] ....  8286.324156: writeback_pages_written: 3584
              dd-4267  [002] ....  8286.328179: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8286.333974: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8286.338144: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8286.343939: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [002] ....  8286.348136: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8286.353914: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8286.358096: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8286.363902: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [002] ....  8286.368092: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8286.375470: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8286.379691: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8286.390024: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [000] ....  8286.394213: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=817152
       flush-8:0-4272  [002] ....  8286.394227: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=817152
              dd-4267  [000] ....  8286.394228: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=817152
              dd-4267  [000] ....  8286.395740: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=819200
       flush-8:0-4272  [002] ....  8286.397700: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=817152
              dd-4267  [000] ....  8286.399729: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8286.399908: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
       flush-8:0-4272  [002] ....  8286.400793: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=819200
              dd-4267  [000] ....  8286.403696: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=821248
       flush-8:0-4272  [002] ....  8286.406096: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=512 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=821248
       flush-8:0-4272  [002] ....  8286.408455: writeback_pages_written: 4608
              dd-4267  [000] ....  8286.490429: writeback_congestion_wait: usec_timeout=100000 usec_delayed=86000
              dd-4267  [000] ....  8287.253038: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=821760
       flush-8:0-4272  [002] ....  8287.253056: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=821760
              dd-4267  [000] ....  8287.253111: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=821760
              dd-4267  [000] ....  8287.256585: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=823808
              dd-4267  [000] ....  8287.258183: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=825856
       flush-8:0-4272  [002] ....  8287.259493: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=821760
              dd-4267  [001] ....  8287.259950: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=827904
              dd-4267  [001] ....  8287.261742: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=829952
       flush-8:0-4272  [002] ....  8287.262677: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=823808
              dd-4267  [001] ....  8287.264192: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=832000
              dd-4267  [001] ....  8287.265825: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=834048
              dd-4267  [001] ....  8287.267436: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=836096
              dd-4267  [001] ....  8287.269029: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=838144
              dd-4267  [001] ....  8287.270652: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=840192
       flush-8:0-4272  [002] ....  8287.271798: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=825856
              dd-4267  [001] ....  8287.272253: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=842240
              dd-4267  [001] ....  8287.273814: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=844288
       flush-8:0-4272  [002] ....  8287.280257: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=827904
       flush-8:0-4272  [002] ....  8287.286640: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=829952
       flush-8:0-4272  [002] ....  8287.292224: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=832000
       flush-8:0-4272  [002] ....  8287.297298: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=834048
              dd-4267  [003] ....  8287.375066: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
       flush-8:0-4272  [000] ....  8287.433845: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=836096
       flush-8:0-4272  [000] ....  8287.439490: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=838144
       flush-8:0-4272  [000] ....  8287.444495: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=840192
       flush-8:0-4272  [000] ....  8287.449427: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=842240
              dd-4267  [001] ....  8287.568053: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
       flush-8:0-4272  [002] ....  8287.589239: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=844288
       flush-8:0-4272  [002] ....  8287.594059: writeback_start: bdi 8:0: sb_dev 0:0 nr_pages=25531 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
       flush-8:0-4272  [002] ....  8287.594061: writeback_queue_io: bdi 8:0: older=4302929296 age=30000 enqueue=1 reason=periodic
       flush-8:0-4272  [002] ....  8287.594217: writeback_single_inode: bdi 8:0: ino=0 state= dirtied_when=4302926220 age=51 index=62423040 to_write=1024 wrote=60
       flush-8:0-4272  [002] ....  8287.594220: writeback_written: bdi 8:0: sb_dev 0:0 nr_pages=25471 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
       flush-8:0-4272  [002] ....  8287.594220: writeback_start: bdi 8:0: sb_dev 0:0 nr_pages=25471 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
       flush-8:0-4272  [002] ....  8287.594220: writeback_queue_io: bdi 8:0: older=4302929296 age=30000 enqueue=0 reason=periodic
       flush-8:0-4272  [002] ....  8287.594221: writeback_written: bdi 8:0: sb_dev 0:0 nr_pages=25471 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
       flush-8:0-4272  [002] ....  8287.594223: writeback_pages_written: 24636
              dd-4267  [003] ....  8287.725915: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [000] ....  8287.867240: writeback_congestion_wait: usec_timeout=100000 usec_delayed=68000
              dd-4267  [000] ....  8287.929273: writeback_congestion_wait: usec_timeout=100000 usec_delayed=29000
              dd-4267  [000] ....  8287.933932: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [002] ....  8287.943436: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8287.953437: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8287.963311: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8287.968957: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8287.971328: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [000] ....  8287.975932: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [000] ....  8287.996004: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8288.000541: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [002] ....  8288.007408: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8288.017420: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8288.024017: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8288.029668: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [002] ....  8288.034148: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8288.038464: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [000] ....  8288.042966: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [002] ....  8288.145630: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [000] ....  8288.198092: writeback_congestion_wait: usec_timeout=100000 usec_delayed=52000
              dd-4267  [000] ....  8288.214718: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8288.224982: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [002] ....  8288.235294: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8288.245537: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [000] ....  8288.255814: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [000] ....  8288.262402: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8288.270398: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [002] ....  8288.281573: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8288.285848: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8288.293826: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [002] ....  8288.298364: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [000] ....  8288.306692: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8288.316956: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [002] ....  8288.327285: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8288.337529: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [000] ....  8288.343307: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [000] ....  8288.350208: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8288.360343: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8288.364844: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8288.368427: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=846336
       flush-8:0-4272  [002] ....  8288.368440: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=846336
              dd-4267  [000] ....  8288.368442: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=846336
              dd-4267  [000] ....  8288.374199: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=848384
       flush-8:0-4272  [002] ....  8288.374913: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=846336
              dd-4267  [001] ....  8288.376663: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=850432
       flush-8:0-4272  [002] ....  8288.377993: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=848384
              dd-4267  [001] ....  8288.378387: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=852480
              dd-4267  [001] ....  8288.380037: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=854528
              dd-4267  [001] ....  8288.381692: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=856576
       flush-8:0-4272  [002] ....  8288.387743: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=850432
              dd-4267  [001] ....  8288.390190: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=858624
       flush-8:0-4272  [002] ....  8288.395961: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=852480
       flush-8:0-4272  [002] ....  8288.403948: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=854528
              dd-4267  [001] ....  8288.404992: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=860672
              dd-4267  [001] ....  8288.406709: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=862720
              dd-4267  [001] ....  8288.408423: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=864768
              dd-4267  [001] ....  8288.410722: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=866816
       flush-8:0-4272  [002] ....  8288.410728: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=856576
              dd-4267  [001] ....  8288.412372: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=868864
       flush-8:0-4272  [002] ....  8288.417731: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=858624
       flush-8:0-4272  [002] ....  8288.422959: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=860672
       flush-8:0-4272  [002] ....  8288.427513: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=862720
       flush-8:0-4272  [002] ....  8288.431706: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=864768
       flush-8:0-4272  [002] ....  8288.435613: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=866816
              dd-4267  [003] ....  8288.513425: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [001] ....  8288.613432: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
       flush-8:0-4272  [002] ....  8288.807055: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1792 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=868864
       flush-8:0-4272  [002] ....  8288.813588: writeback_pages_written: 24320
              dd-4267  [003] ....  8288.964246: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [003] ....  8289.018315: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=870656
       flush-8:0-4272  [002] ....  8289.018333: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=870656
              dd-4267  [003] ....  8289.018334: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=870656
       flush-8:0-4272  [002] ....  8289.022448: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=512 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=870656
       flush-8:0-4272  [002] ....  8289.023068: writeback_pages_written: 512
              dd-4267  [000] .N..  8289.043118: writeback_congestion_wait: usec_timeout=100000 usec_delayed=24000
              dd-4267  [000] ....  8289.062610: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8289.072452: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8289.082237: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [002] ....  8289.092057: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8289.101888: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8289.113107: writeback_congestion_wait: usec_timeout=100000 usec_delayed=8000
              dd-4267  [000] ....  8289.117139: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=871168
       flush-8:0-4272  [002] ....  8289.117151: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=871168
              dd-4267  [000] ....  8289.117152: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=871168
       flush-8:0-4272  [002] ....  8289.120609: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=871168
       flush-8:0-4272  [002] ....  8289.121378: writeback_pages_written: 768
              dd-4267  [000] ....  8289.122889: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [002] ....  8289.132666: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8289.140943: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8289.145446: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8289.149985: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [002] ....  8289.154423: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8289.169836: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8289.174289: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [002] ....  8289.239886: writeback_congestion_wait: usec_timeout=100000 usec_delayed=62000
              dd-4267  [000] ....  8289.244347: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8289.248919: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8289.258780: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8289.268549: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8289.278371: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8289.288216: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [002] ....  8289.297985: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [000] ....  8289.307821: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8289.317661: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8289.327410: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [002] ....  8289.337217: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [000] ....  8289.347038: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8289.353471: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8289.359251: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [002] ....  8289.363735: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [000] ....  8289.368137: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8289.372635: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [000] ....  8289.377206: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8289.480585: writeback_congestion_wait: usec_timeout=100000 usec_delayed=81000
              dd-4267  [000] ....  8289.482791: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [000] ....  8289.485066: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [002] ....  8289.487555: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8289.495298: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8289.505137: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [000] ....  8289.514910: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [002] ....  8289.520479: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8289.527108: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [000] ....  8289.544021: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=871936
       flush-8:0-4272  [002] ....  8289.544039: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=871936
              dd-4267  [000] ....  8289.544046: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=871936
              dd-4267  [000] ....  8289.545565: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=873984
              dd-4267  [000] ....  8289.547139: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=876032
              dd-4267  [000] ....  8289.548677: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=878080
       flush-8:0-4272  [002] ....  8289.549822: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=871936
              dd-4267  [000] ....  8289.553126: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=880128
       flush-8:0-4272  [001] ....  8289.554779: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=873984
       flush-8:0-4272  [001] ....  8289.567484: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=876032
              dd-4267  [000] ....  8289.568912: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=882176
              dd-4267  [000] ....  8289.570643: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=884224
              dd-4267  [000] ....  8289.572336: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=886272
              dd-4267  [000] ....  8289.574000: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=888320
       flush-8:0-4272  [001] ....  8289.575450: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=878080
              dd-4267  [000] ....  8289.575687: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=890368
              dd-4267  [000] ....  8289.577346: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=892416
              dd-4267  [000] ....  8289.579606: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=894464
       flush-8:0-4272  [001] ....  8289.583703: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=880128
       flush-8:0-4272  [001] ....  8289.590178: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=882176
       flush-8:0-4272  [001] ....  8289.595612: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=884224
       flush-8:0-4272  [001] ....  8289.600583: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=886272
       flush-8:0-4272  [001] ....  8289.605480: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=888320
       flush-8:0-4272  [001] ....  8289.610298: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=890368
       flush-8:0-4272  [001] ....  8289.614567: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=892416
              dd-4267  [002] ....  8289.679818: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
       flush-8:0-4272  [001] ....  8289.790935: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1280 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=894464
       flush-8:0-4272  [001] ....  8289.794239: writeback_pages_written: 23808
              dd-4267  [000] ....  8289.830712: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [000] ....  8289.979274: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=895744
       flush-8:0-4272  [001] ....  8289.979295: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=895744
              dd-4267  [000] ....  8289.979299: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=895744
       flush-8:0-4272  [001] ....  8289.984149: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1024 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=895744
       flush-8:0-4272  [001] ....  8289.985910: writeback_pages_written: 1024
              dd-4267  [003] ....  8289.986879: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [001] ....  8289.989278: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [001] ....  8289.993746: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [001] ....  8290.022891: writeback_congestion_wait: usec_timeout=100000 usec_delayed=29000
              dd-4267  [003] ....  8290.138516: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [001] ....  8290.238484: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [001] ....  8290.258044: writeback_congestion_wait: usec_timeout=100000 usec_delayed=19000
              dd-4267  [001] ....  8290.272386: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [003] ....  8290.282389: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8290.292359: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8290.302381: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [003] ....  8290.312326: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8290.330620: writeback_congestion_wait: usec_timeout=100000 usec_delayed=15000
              dd-4267  [001] ....  8290.340609: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [001] ....  8290.350579: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [003] ....  8290.360570: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [001] ....  8290.370533: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8290.380548: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [001] ....  8290.390506: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [003] ....  8290.400468: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8290.410469: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [001] ....  8290.420238: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [003] ....  8290.424433: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [001] ....  8290.430244: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [001] ....  8290.434405: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [001] ....  8290.440201: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [003] ....  8290.444394: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [001] ....  8290.450177: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [001] ....  8290.454365: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [001] ....  8290.460164: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [003] ....  8290.464360: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [001] ....  8290.466690: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [001] ....  8290.472687: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8290.494518: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [001] ....  8290.499431: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [001] ....  8290.506101: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [001] ....  8290.514874: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [001] ....  8290.525026: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [003] ....  8290.535090: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8290.545126: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8290.550820: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [001] ....  8290.555010: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [003] ....  8290.560856: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8290.573413: writeback_congestion_wait: usec_timeout=100000 usec_delayed=11000
              dd-4267  [001] ....  8290.579244: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8290.585075: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [003] ....  8290.589299: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [003] ....  8290.591103: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=896768
       flush-8:0-4272  [001] ....  8290.591114: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=896768
              dd-4267  [003] ....  8290.591115: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=896768
              dd-4267  [003] ....  8290.592629: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=898816
              dd-4267  [003] ....  8290.594151: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=900864
              dd-4267  [003] ....  8290.595687: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=902912
       flush-8:0-4272  [001] ....  8290.596742: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=896768
       flush-8:0-4272  [001] ....  8290.599560: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=898816
              dd-4267  [003] ....  8290.599589: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=904960
              dd-4267  [000] ....  8290.601759: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=907008
              dd-4267  [000] ....  8290.604180: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=909056
              dd-4267  [000] ....  8290.606776: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8290.606967: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
       flush-8:0-4272  [001] ....  8290.608163: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=900864
              dd-4267  [003] ....  8290.616932: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
       flush-8:0-4272  [001] ....  8290.807466: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=902912
       flush-8:0-4272  [001] ....  8290.815087: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=904960
       flush-8:0-4272  [000] ....  8290.825087: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=907008
       flush-8:0-4272  [000] ....  8290.833291: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=909056
       flush-8:0-4272  [000] ....  8290.837230: writeback_pages_written: 13056
              dd-4267  [003] ....  8290.898320: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=909824
       flush-8:0-4272  [000] ....  8290.898341: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=909824
              dd-4267  [003] ....  8290.898342: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=909824
              dd-4267  [003] ....  8290.900086: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=911872
              dd-4267  [003] ....  8290.902263: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=913920
       flush-8:0-4272  [000] ....  8290.903241: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=909824
              dd-4267  [003] ....  8290.903966: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=915968
              dd-4267  [003] ....  8290.905605: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=918016
       flush-8:0-4272  [000] ....  8290.907371: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=911872
       flush-8:0-4272  [000] ....  8290.914048: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=913920
       flush-8:0-4272  [000] ....  8290.919004: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=915968
       flush-8:0-4272  [000] ....  8290.923475: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=918016
       flush-8:0-4272  [000] ....  8290.927621: writeback_pages_written: 10240
              dd-4267  [001] ....  8291.007124: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [001] ....  8291.055558: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=920064
       flush-8:0-4272  [000] ....  8291.055576: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=920064
              dd-4267  [001] ....  8291.055577: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=920064
       flush-8:0-4272  [000] ....  8291.058663: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=920064
       flush-8:0-4272  [000] ....  8291.058672: writeback_pages_written: 256
              dd-4267  [000] ....  8291.120020: writeback_congestion_wait: usec_timeout=100000 usec_delayed=65000
              dd-4267  [000] ....  8291.228552: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=920320
       flush-8:0-4272  [002] ....  8291.228569: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=920320
              dd-4267  [000] ....  8291.228570: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=920320
              dd-4267  [000] ....  8291.230580: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=922368
              dd-4267  [000] ....  8291.232100: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=924416
       flush-8:0-4272  [002] ....  8291.232583: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=920320
       flush-8:0-4272  [002] ....  8291.235968: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=922368
       flush-8:0-4272  [002] ....  8291.241987: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=924416
              dd-4267  [000] ....  8291.242861: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [000] ....  8291.246767: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=926464
       flush-8:0-4272  [002] ....  8291.247352: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=926464
       flush-8:0-4272  [002] ....  8291.249590: writeback_pages_written: 6400
              dd-4267  [000] ....  8291.303192: writeback_congestion_wait: usec_timeout=100000 usec_delayed=57000
              dd-4267  [002] ....  8291.418747: writeback_congestion_wait: usec_timeout=100000 usec_delayed=70000
              dd-4267  [000] ....  8291.421290: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [002] ....  8291.430170: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8291.438818: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [000] ....  8291.448613: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8291.452546: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [002] ....  8291.458291: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8291.463059: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8291.501991: writeback_congestion_wait: usec_timeout=100000 usec_delayed=8000
              dd-4267  [002] ....  8291.511822: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [000] ....  8291.521615: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [000] ....  8291.531477: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8291.541259: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8291.545313: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=926720
       flush-8:0-4272  [002] ....  8291.545327: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=926720
              dd-4267  [000] ....  8291.545329: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=926720
              dd-4267  [000] ....  8291.546859: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=928768
       flush-8:0-4272  [002] ....  8291.549677: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=926720
              dd-4267  [000] ....  8291.550805: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [000] ....  8291.550981: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
       flush-8:0-4272  [002] ....  8291.553082: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1024 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=928768
              dd-4267  [000] ....  8291.554588: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=929536
              dd-4267  [001] ....  8291.556446: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=931584
       flush-8:0-4272  [002] ....  8291.558013: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=929536
              dd-4267  [001] ....  8291.558925: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=933632
       flush-8:0-4272  [002] ....  8291.560978: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=931584
              dd-4267  [001] ....  8291.561309: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=935680
              dd-4267  [000] ....  8291.566775: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
       flush-8:0-4272  [002] ....  8291.567625: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=933632
       flush-8:0-4272  [000] ....  8291.691938: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1024 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=935680
       flush-8:0-4272  [000] ....  8291.696544: writeback_pages_written: 9984
              dd-4267  [000] ....  8291.755429: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8291.765681: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [000] ....  8291.769854: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=936704
       flush-8:0-4272  [002] ....  8291.769869: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=936704
              dd-4267  [000] ....  8291.769870: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=936704
              dd-4267  [000] ....  8291.771371: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
       flush-8:0-4272  [002] ....  8291.773234: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=936704
       flush-8:0-4272  [002] ....  8291.773241: writeback_pages_written: 256
              dd-4267  [002] ....  8291.775760: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8291.781707: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [000] ....  8291.784087: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8291.788791: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [002] ....  8291.809228: writeback_congestion_wait: usec_timeout=100000 usec_delayed=12000
              dd-4267  [000] ....  8291.811942: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8291.819305: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=936960
       flush-8:0-4272  [002] ....  8291.819318: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=936960
              dd-4267  [000] ....  8291.819319: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=936960
              dd-4267  [000] ....  8291.820882: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=939008
              dd-4267  [000] ....  8291.822428: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=941056
       flush-8:0-4272  [002] ....  8291.822757: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=936960
              dd-4267  [000] ....  8291.824434: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=943104
       flush-8:0-4272  [002] ....  8291.825686: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=939008
              dd-4267  [000] ....  8291.827048: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8291.827226: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
       flush-8:0-4272  [002] ....  8291.831471: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=941056
       flush-8:0-4272  [002] ....  8291.836478: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1536 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=943104
       flush-8:0-4272  [002] ....  8291.839973: writeback_pages_written: 7680
              dd-4267  [002] ....  8291.930645: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [002] ....  8291.992029: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=944640
       flush-8:0-4272  [000] ....  8291.992046: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=944640
              dd-4267  [002] ....  8291.992056: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=944640
       flush-8:0-4272  [000] ....  8291.994832: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=944640
       flush-8:0-4272  [000] ....  8291.994838: writeback_pages_written: 256
              dd-4267  [002] ....  8292.030123: writeback_congestion_wait: usec_timeout=100000 usec_delayed=38000
              dd-4267  [002] ....  8292.050503: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [002] ....  8292.054831: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [000] ....  8292.060738: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [002] ....  8292.065103: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8292.071061: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [002] ....  8292.075356: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8292.081306: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [002] ....  8292.087263: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [002] ....  8292.091595: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8292.097536: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [002] ....  8292.122806: writeback_congestion_wait: usec_timeout=100000 usec_delayed=24000
              dd-4267  [002] ....  8292.127164: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8292.168759: writeback_congestion_wait: usec_timeout=100000 usec_delayed=38000
              dd-4267  [000] ....  8292.173114: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8292.235518: writeback_congestion_wait: usec_timeout=100000 usec_delayed=59000
              dd-4267  [002] ....  8292.240176: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8292.247496: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8292.257469: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [002] ....  8292.267454: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [002] ....  8292.273257: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [000] ....  8292.277430: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [002] ....  8292.283231: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [002] ....  8292.287413: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [002] ....  8292.293190: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [002] ....  8292.297383: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8292.303189: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [002] ....  8292.305193: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=944896
       flush-8:0-4272  [000] ....  8292.305204: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=944896
              dd-4267  [002] ....  8292.305205: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=944896
              dd-4267  [002] ....  8292.306738: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=946944
              dd-4267  [002] ....  8292.309833: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=948992
              dd-4267  [000] ....  8292.313157: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
       flush-8:0-4272  [000] ....  8292.735920: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=944896
       flush-8:0-4272  [000] ....  8292.740409: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=946944
       flush-8:0-4272  [001] ....  8292.748894: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=948992
       flush-8:0-4272  [001] ....  8292.752966: writeback_start: bdi 8:0: sb_dev 0:0 nr_pages=33828 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
       flush-8:0-4272  [001] ....  8292.752968: writeback_queue_io: bdi 8:0: older=4302934457 age=30000 enqueue=0 reason=periodic
       flush-8:0-4272  [001] ....  8292.752968: writeback_written: bdi 8:0: sb_dev 0:0 nr_pages=33828 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
       flush-8:0-4272  [001] ....  8292.752971: writeback_pages_written: 4864
              dd-4267  [002] ....  8292.865367: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=949760
       flush-8:0-4272  [001] ....  8292.865385: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=949760
              dd-4267  [002] ....  8292.865386: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=949760
              dd-4267  [002] ....  8292.867161: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=951808
              dd-4267  [002] ....  8292.868811: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=953856
              dd-4267  [002] ....  8292.870466: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=955904
       flush-8:0-4272  [001] ....  8292.871705: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=949760
              dd-4267  [002] ....  8292.873150: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=957952
              dd-4267  [002] ....  8292.875731: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=960000
       flush-8:0-4272  [001] ....  8292.876282: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=951808
       flush-8:0-4272  [001] ....  8292.883855: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=953856
       flush-8:0-4272  [001] ....  8292.889576: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=955904
       flush-8:0-4272  [001] ....  8292.894750: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=957952
       flush-8:0-4272  [001] ....  8292.899538: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=960000
       flush-8:0-4272  [001] ....  8292.901892: writeback_pages_written: 11008
              dd-4267  [000] ....  8292.976950: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [001] ....  8293.020761: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=960768
       flush-8:0-4272  [003] ....  8293.020778: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=960768
              dd-4267  [001] ....  8293.021248: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=960768
       flush-8:0-4272  [003] ....  8293.023781: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=960768
       flush-8:0-4272  [003] ....  8293.023789: writeback_pages_written: 256
              dd-4267  [001] ....  8293.083843: writeback_congestion_wait: usec_timeout=100000 usec_delayed=62000
              dd-4267  [003] ....  8293.097927: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8293.108016: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [001] ....  8293.126388: writeback_congestion_wait: usec_timeout=100000 usec_delayed=14000
              dd-4267  [001] ....  8293.136458: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [003] ....  8293.146577: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8293.156636: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8293.166659: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8293.170791: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=961024
       flush-8:0-4272  [003] ....  8293.170804: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=961024
              dd-4267  [001] ....  8293.170805: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=961024
              dd-4267  [001] ....  8293.172326: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=963072
              dd-4267  [001] ....  8293.173830: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=965120
       flush-8:0-4272  [003] ....  8293.174168: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=961024
              dd-4267  [001] ....  8293.175352: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=967168
       flush-8:0-4272  [003] ....  8293.177053: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=963072
       flush-8:0-4272  [003] ....  8293.182445: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=965120
              dd-4267  [002] ....  8293.282859: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
       flush-8:0-4272  [003] ....  8293.321004: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=967168
       flush-8:0-4272  [003] ....  8293.325879: writeback_pages_written: 8192
              dd-4267  [002] ....  8293.424762: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [002] ....  8293.428997: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8293.434825: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [002] ....  8293.439025: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [002] ....  8293.444901: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [002] ....  8293.449071: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [002] ....  8293.454973: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [002] ....  8293.459202: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8293.465002: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [002] ....  8293.469260: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8293.475132: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [002] ....  8293.479335: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8293.485174: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [002] ....  8293.489424: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8293.495268: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [002] ....  8293.499474: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8293.505310: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [002] ....  8293.509565: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8293.515405: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [002] ....  8293.519611: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [003] ....  8293.529871: writeback_congestion_wait: usec_timeout=100000 usec_delayed=8000
              dd-4267  [001] ....  8293.539968: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [001] ....  8293.550034: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8293.560097: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [003] ....  8293.570116: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [003] ....  8293.574101: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=969216
       flush-8:0-4272  [001] ....  8293.574114: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=969216
              dd-4267  [003] ....  8293.574119: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=969216
       flush-8:0-4272  [001] ....  8293.578134: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=969216
       flush-8:0-4272  [001] ....  8293.578142: writeback_pages_written: 256
              dd-4267  [001] ....  8293.580245: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8293.583993: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=969472
       flush-8:0-4272  [003] ....  8293.584005: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=969472
              dd-4267  [001] ....  8293.584006: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=969472
              dd-4267  [001] ....  8293.585670: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=971520
       flush-8:0-4272  [003] ....  8293.587582: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=969472
              dd-4267  [001] ....  8293.590259: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
       flush-8:0-4272  [003] ....  8293.590489: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=512 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=971520
       flush-8:0-4272  [003] ....  8293.593743: writeback_pages_written: 2560
              dd-4267  [001] ....  8293.594087: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=972032
       flush-8:0-4272  [003] ....  8293.594095: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=972032
              dd-4267  [001] ....  8293.594104: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=972032
              dd-4267  [001] ....  8293.595677: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=974080
       flush-8:0-4272  [003] ....  8293.596789: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=972032
       flush-8:0-4272  [003] ....  8293.599663: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=974080
       flush-8:0-4272  [003] ....  8293.601695: writeback_pages_written: 2304
              dd-4267  [001] ....  8293.620507: writeback_congestion_wait: usec_timeout=100000 usec_delayed=24000
              dd-4267  [001] ....  8293.640663: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [003] ....  8293.648906: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [001] ....  8293.657732: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [001] ....  8293.662163: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [003] ....  8293.666667: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [001] ....  8293.719584: writeback_congestion_wait: usec_timeout=100000 usec_delayed=48000
              dd-4267  [001] ....  8293.721790: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [001] ....  8293.724056: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [001] ....  8293.731220: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [001] ....  8293.741252: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8293.748399: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [001] ....  8293.752893: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [001] ....  8293.757387: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [003] ....  8293.761705: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [001] ....  8293.768565: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8293.773111: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [003] ....  8293.784513: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [001] ....  8293.794486: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [001] ....  8293.804558: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8293.816096: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [000] ....  8293.820338: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [002] ....  8293.822648: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [002] ....  8293.824862: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [003] ....  8293.827337: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [001] ....  8293.831982: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [001] ....  8293.850276: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=974336
       flush-8:0-4272  [003] ....  8293.850291: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=974336
              dd-4267  [001] ....  8293.850292: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=974336
       flush-8:0-4272  [003] ....  8293.854749: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=974336
       flush-8:0-4272  [003] ....  8293.854756: writeback_pages_written: 256
              dd-4267  [003] ....  8293.949419: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [001] ....  8293.950099: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [001] ....  8293.954661: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [003] ....  8293.957135: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [003] ....  8293.959495: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=974592
       flush-8:0-4272  [001] ....  8293.959505: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=974592
              dd-4267  [003] ....  8293.959506: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=974592
       flush-8:0-4272  [001] ....  8293.964429: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=974592
              dd-4267  [003] ....  8293.964831: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=976640
              dd-4267  [003] ....  8293.966420: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=978688
       flush-8:0-4272  [001] ....  8293.967125: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=976640
              dd-4267  [003] ....  8293.967957: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=980736
       flush-8:0-4272  [001] ....  8293.974557: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=978688
              dd-4267  [000] ....  8293.976671: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=982784
              dd-4267  [000] ....  8293.979998: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=984832
              dd-4267  [000] ....  8293.980544: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
       flush-8:0-4272  [001] ....  8293.981418: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=980736
       flush-8:0-4272  [001] ....  8293.987940: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=982784
              dd-4267  [000] ....  8293.988730: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
       flush-8:0-4272  [001] ....  8293.994540: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=984832
       flush-8:0-4272  [001] ....  8293.996489: writeback_pages_written: 10496
              dd-4267  [001] ....  8294.029115: writeback_congestion_wait: usec_timeout=100000 usec_delayed=27000
              dd-4267  [001] ....  8294.064816: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=985088
       flush-8:0-4272  [003] ....  8294.064833: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=985088
              dd-4267  [001] ....  8294.064834: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=985088
       flush-8:0-4272  [003] ....  8294.067992: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=985088
       flush-8:0-4272  [003] ....  8294.068000: writeback_pages_written: 256
              dd-4267  [001] ....  8294.068875: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8294.078863: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8294.088870: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [003] ....  8294.098835: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8294.108811: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8294.118806: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8294.122874: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=985344
       flush-8:0-4272  [003] ....  8294.122890: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=985344
              dd-4267  [001] ....  8294.122895: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=985344
              dd-4267  [001] ....  8294.124491: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=987392
              dd-4267  [001] ....  8294.126057: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=989440
       flush-8:0-4272  [003] .N..  8294.126285: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=985344
              dd-4267  [001] ....  8294.127632: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=991488
       flush-8:0-4272  [003] ....  8294.129080: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=987392
       flush-8:0-4272  [003] ....  8294.134718: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=989440
       flush-8:0-4272  [003] ....  8294.139698: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1792 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=991488
       flush-8:0-4272  [003] ....  8294.143930: writeback_pages_written: 7936
              dd-4267  [000] ....  8294.154721: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=993280
       flush-8:0-4272  [003] ....  8294.154740: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=993280
              dd-4267  [000] ....  8294.154741: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=993280
       flush-8:0-4272  [003] ....  8294.156714: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=993280
       flush-8:0-4272  [003] ....  8294.156722: writeback_pages_written: 256
              dd-4267  [002] ....  8294.254308: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [002] ....  8294.351041: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=993536
       flush-8:0-4272  [003] ....  8294.351061: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=993536
              dd-4267  [002] ....  8294.351067: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=993536
       flush-8:0-4272  [003] ....  8294.355385: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=993536
       flush-8:0-4272  [003] ....  8294.355392: writeback_pages_written: 256
              dd-4267  [003] ....  8294.429885: writeback_congestion_wait: usec_timeout=100000 usec_delayed=74000
              dd-4267  [001] ....  8294.432726: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [001] ....  8294.435228: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [001] ....  8294.449908: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [001] ....  8294.455841: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [003] ....  8294.461793: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8294.466140: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [001] .N..  8294.472105: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8294.476408: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [003] ....  8294.482332: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8294.486666: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8294.492631: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8294.496951: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [003] ....  8294.502897: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8294.507247: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [001] ....  8294.513196: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8294.517488: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [003] ....  8294.523451: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8294.527785: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8294.533730: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8294.538058: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [003] ....  8294.544007: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [001] ....  8294.548348: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [001] ....  8294.613136: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [001] ....  8294.617232: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8294.622950: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [003] ....  8294.627046: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [001] ....  8294.632738: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [003] ....  8294.646693: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [001] ....  8294.656647: writeback_congestion_wait: usec_timeout=100000 usec_delayed=10000
              dd-4267  [001] ....  8294.662184: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8294.676087: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [001] ....  8294.680177: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8294.685900: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [003] ....  8294.690000: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [001] ....  8294.695720: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [001] ....  8294.699792: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8294.705508: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [003] ....  8294.709607: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8294.715336: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8294.719415: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8294.725125: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [003] ....  8294.729222: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8294.734952: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8294.739050: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8294.744735: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [003] ....  8294.748838: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8294.754564: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8294.758690: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8294.768628: writeback_congestion_wait: usec_timeout=100000 usec_delayed=10000
              dd-4267  [001] ....  8294.827256: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [000] ....  8294.829463: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [000] ....  8294.831574: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8294.833749: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [002] ....  8294.835917: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [003] ....  8294.849642: writeback_congestion_wait: usec_timeout=100000 usec_delayed=13000
              dd-4267  [001] ....  8294.851883: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [001] ....  8294.926921: writeback_congestion_wait: usec_timeout=100000 usec_delayed=71000
              dd-4267  [003] ....  8294.929117: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [003] ....  8294.931419: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
              dd-4267  [003] ....  8294.936759: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=993792
       flush-8:0-4272  [001] ....  8294.936771: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=993792
              dd-4267  [003] ....  8294.936776: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=993792
              dd-4267  [003] ....  8294.938346: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=995840
              dd-4267  [003] ....  8294.939895: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=997888
              dd-4267  [003] ....  8294.941461: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=999936
       flush-8:0-4272  [001] ....  8294.942432: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=993792
              dd-4267  [003] ....  8294.943044: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1001984
              dd-4267  [003] ....  8294.944594: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1004032
       flush-8:0-4272  [001] ....  8294.945567: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=995840
              dd-4267  [003] ....  8294.948509: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1006080
              dd-4267  [000] ....  8294.950142: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1008128
       flush-8:0-4272  [000] ....  8294.954509: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=997888
       flush-8:0-4272  [000] ....  8295.060073: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=999936
       flush-8:0-4272  [001] ....  8295.070258: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1001984
       flush-8:0-4272  [001] ....  8295.079359: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1004032
       flush-8:0-4272  [001] ....  8295.086159: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1006080
       flush-8:0-4272  [001] ....  8295.092211: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1024 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1008128
       flush-8:0-4272  [001] ....  8295.095943: writeback_pages_written: 15360
              dd-4267  [002] ....  8295.149629: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1009152
       flush-8:0-4272  [001] ....  8295.149649: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1009152
              dd-4267  [002] ....  8295.149661: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1009152
       flush-8:0-4272  [001] ....  8295.154294: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1009152
              dd-4267  [002] ....  8295.154608: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1009920
       flush-8:0-4272  [001] ....  8295.155299: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1024 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1009920
              dd-4267  [002] ....  8295.155309: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1010688
              dd-4267  [002] ....  8295.156976: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1012736
              dd-4267  [002] ....  8295.158592: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1014784
       flush-8:0-4272  [001] ....  8295.159917: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1010688
       flush-8:0-4272  [001] ....  8295.163902: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1012736
              dd-4267  [002] ....  8295.164768: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1016832
       flush-8:0-4272  [001] ....  8295.170302: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1014784
       flush-8:0-4272  [001] ....  8295.175152: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1536 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1016832
       flush-8:0-4272  [001] ....  8295.178552: writeback_pages_written: 9216
              dd-4267  [000] ....  8295.265787: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [003] ....  8295.368201: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8295.377759: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [001] ....  8295.381826: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [003] ....  8295.387549: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [001] ....  8295.391664: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [001] ....  8295.397376: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8295.401466: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [003] ....  8295.407182: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [001] ....  8295.411274: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [001] ....  8295.417013: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8295.421116: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [003] ....  8295.426809: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [001] ....  8295.430867: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [001] ....  8295.436645: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8295.446602: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
              dd-4267  [001] ....  8295.450132: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1018368
       flush-8:0-4272  [003] ....  8295.450147: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1018368
              dd-4267  [001] ....  8295.450152: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1018368
              dd-4267  [001] ....  8295.454012: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1020416
       flush-8:0-4272  [003] ....  8295.454533: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1018368
              dd-4267  [001] ....  8295.455564: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1022464
       flush-8:0-4272  [003] ....  8295.457413: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1020416
              dd-4267  [000] ....  8295.459310: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1024512
              dd-4267  [001] ....  8295.460273: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
       flush-8:0-4272  [003] ....  8295.463336: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1022464
       flush-8:0-4272  [003] ....  8295.468614: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1024512
       flush-8:0-4272  [003] ....  8295.470473: writeback_pages_written: 6400
              dd-4267  [003] ....  8295.561536: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
              dd-4267  [000] ....  8295.619605: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1024768
       flush-8:0-4272  [003] ....  8295.619628: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1024768
              dd-4267  [000] ....  8295.619629: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1024768
       flush-8:0-4272  [003] ....  8295.624325: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1024768
       flush-8:0-4272  [003] ....  8295.624334: writeback_pages_written: 256
              dd-4267  [001] ....  8295.626436: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [001] ....  8295.630738: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [001] ....  8295.636693: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [003] ....  8295.641010: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [001] ....  8295.646977: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8295.651282: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [001] ....  8295.657228: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [003] ....  8295.661569: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [001] ....  8295.667523: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [001] ....  8295.671798: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [001] ....  8295.677742: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [003] ....  8295.682120: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [001] ....  8295.688077: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [001] ....  8295.698626: writeback_congestion_wait: usec_timeout=100000 usec_delayed=9000
              dd-4267  [001] ....  8295.708811: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
              dd-4267  [003] ....  8295.719075: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [001] ....  8295.724796: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [001] ....  8295.729170: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [001] ....  8295.735113: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [003] ....  8295.739452: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [003] ....  8295.741771: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1025024
       flush-8:0-4272  [001] ....  8295.741782: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1025024
              dd-4267  [003] ....  8295.741783: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1025024
              dd-4267  [003] ....  8295.745338: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
       flush-8:0-4272  [001] ....  8295.745929: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1025024
       flush-8:0-4272  [001] ....  8295.745935: writeback_pages_written: 256
              dd-4267  [001] ....  8295.749693: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [001] ....  8295.755677: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [003] ....  8295.760008: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [001] ....  8295.765967: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [001] ....  8295.770295: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
              dd-4267  [001] ....  8295.776226: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [003] ....  8295.780561: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [001] ....  8295.786491: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8295.790761: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
              dd-4267  [001] ....  8295.796727: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [003] ....  8295.801103: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
              dd-4267  [001] ....  8295.807061: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
              dd-4267  [001] ....  8295.808918: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1025280
       flush-8:0-4272  [003] ....  8295.808931: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1025280
              dd-4267  [001] ....  8295.808942: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1025280
              dd-4267  [001] ....  8295.810446: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1027328
              dd-4267  [001] ....  8295.811934: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1029376
       flush-8:0-4272  [003] ....  8295.813114: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1025280
              dd-4267  [001] ....  8295.815817: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1031424
       flush-8:0-4272  [003] ....  8295.816182: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1027328
              dd-4267  [000] ....  8295.821315: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1033472
       flush-8:0-4272  [003] ....  8295.822702: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1029376
              dd-4267  [000] ....  8295.827544: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
              dd-4267  [000] ....  8295.827734: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
       flush-8:0-4272  [003] ....  8295.828932: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1031424
              dd-4267  [001] ....  8295.864365: writeback_congestion_wait: usec_timeout=100000 usec_delayed=33000
       flush-8:0-4272  [001] ....  8296.051145: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=512 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1033472
       flush-8:0-4272  [001] ....  8296.054689: writeback_pages_written: 8704
       flush-8:0-4272  [003] ....  8301.051487: writeback_start: bdi 8:0: sb_dev 0:0 nr_pages=21838 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
       flush-8:0-4272  [003] ....  8301.051491: writeback_queue_io: bdi 8:0: older=4302942761 age=30000 enqueue=1 reason=periodic
       flush-8:0-4272  [003] ....  8301.051498: writeback_single_inode: bdi 8:0: ino=13 state= dirtied_when=4302938315 age=39 index=0 to_write=1024 wrote=0
       flush-8:0-4272  [003] ....  8301.051501: writeback_written: bdi 8:0: sb_dev 0:0 nr_pages=21838 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
       flush-8:0-4272  [003] ....  8301.051502: writeback_start: bdi 8:0: sb_dev 0:0 nr_pages=21838 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
       flush-8:0-4272  [003] ....  8301.051502: writeback_queue_io: bdi 8:0: older=4302942761 age=30000 enqueue=0 reason=periodic
       flush-8:0-4272  [003] ....  8301.051502: writeback_written: bdi 8:0: sb_dev 0:0 nr_pages=21838 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
       flush-8:0-4272  [003] ....  8301.051505: writeback_pages_written: 0

[-- Attachment #2: balance_dirty_pages-task-bw-300.png --]
[-- Type: image/png, Size: 41820 bytes --]

[-- Attachment #3: balance_dirty_pages-task-bw.png --]
[-- Type: image/png, Size: 34173 bytes --]

Re: reclaim the LRU lists full of dirty/writeback pages

[Jan Kara]

On Sun 12-02-12 11:10:29, Wu Fengguang wrote:
> On Sat, Feb 11, 2012 at 09:55:38AM -0500, Rik van Riel wrote:
> > On 02/11/2012 07:44 AM, Wu Fengguang wrote:
> > 
> > >Note that it's data for XFS. ext4 seems to have some problem with the
> > >workload: the majority pages are found to be writeback pages, and the
> > >flusher ends up blocking on the unconditional wait_on_page_writeback()
> > >in write_cache_pages_da() from time to time...
> 
> Sorry I overlooked the WB_SYNC_NONE test before the wait_on_page_writeback()
> call! And the issue can no longer be reproduce anyway. ext4 performs pretty
> good now, here is the result for one single memcg dd:
> 
>         dd if=/dev/zero of=/fs/f$i bs=4k count=1M
> 
>         4294967296 bytes (4.3 GB) copied, 44.5759 s, 96.4 MB/s
> 
> iostat -kx 3
> 
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>            0.25    0.00   11.03   28.54    0.00   60.19
>            0.25    0.00   13.71   16.65    0.00   69.39
>            0.17    0.00    8.41   24.81    0.00   66.61
>            0.25    0.00   15.00   19.63    0.00   65.12
> 
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
> sda               0.00    17.00    0.00  178.33     0.00 90694.67  1017.14   111.34  520.88   5.45  97.23
> sda               0.00     0.00    0.00  193.67     0.00 98816.00  1020.48    86.22  496.81   4.81  93.07
> sda               0.00     3.33    0.00  182.33     0.00 92345.33  1012.93   101.14  623.98   5.49 100.03
> sda               0.00     3.00    0.00  187.00     0.00 95586.67  1022.32    89.36  441.70   4.96  92.70
> 
> > >XXX: commit NFS unstable pages via write_inode()
> > >XXX: the added congestion_wait() may be undesirable in some situations
> > 
> > Even with these caveats, this seems to be the right way forward.
> 
> > Acked-by: Rik van Riel <riel@redhat.com>
> 
> Thank you!
>  
> Here is the updated patch.
> - ~10ms write around chunk size, adaptive to the bdi bandwith 
> - cleanup flush_inode_page()
> 
> Thanks,
> Fengguang
> ---
> Subject: writeback: introduce the pageout work
> Date: Thu Jul 29 14:41:19 CST 2010
> 
> This relays file pageout IOs to the flusher threads.
> 
> The ultimate target is to gracefully handle the LRU lists full of
> dirty/writeback pages.
> 
> 1) I/O efficiency
> 
> The flusher will piggy back the nearby ~10ms worth of dirty pages for I/O.
> 
> This takes advantage of the time/spacial locality in most workloads: the
> nearby pages of one file are typically populated into the LRU at the same
> time, hence will likely be close to each other in the LRU list. Writing
> them in one shot helps clean more pages effectively for page reclaim.
> 
> 2) OOM avoidance and scan rate control
> 
> Typically we do LRU scan w/o rate control and quickly get enough clean
> pages for the LRU lists not full of dirty pages.
> 
> Or we can still get a number of freshly cleaned pages (moved to LRU tail
> by end_page_writeback()) when the queued pageout I/O is completed within
> tens of milli-seconds.
> 
> However if the LRU list is small and full of dirty pages, it can be
> quickly fully scanned and go OOM before the flusher manages to clean
> enough pages.
> 
> A simple yet reliable scheme is employed to avoid OOM and keep scan rate
> in sync with the I/O rate:
> 
> 	if (PageReclaim(page))
> 		congestion_wait(HZ/10);
> 
> PG_reclaim plays the key role. When dirty pages are encountered, we
> queue I/O for it, set PG_reclaim and put it back to the LRU head.
> So if PG_reclaim pages are encountered again, it means the dirty page
> has not yet been cleaned by the flusher after a full zone scan. It
> indicates we are scanning more fast than I/O and shall take a snap.
> 
> The runtime behavior on a fully dirtied small LRU list would be:
> It will start with a quick scan of the list, queuing all pages for I/O.
> Then the scan will be slowed down by the PG_reclaim pages *adaptively*
> to match the I/O bandwidth.
> 
> 3) writeback work coordinations
> 
> To avoid memory allocations at page reclaim, a mempool for struct
> wb_writeback_work is created.
> 
> wakeup_flusher_threads() is removed because it can easily delay the
> more oriented pageout works and even exhaust the mempool reservations.
> It's also found to not I/O efficient by frequently submitting writeback
> works with small ->nr_pages.
> 
> Background/periodic works will quit automatically, so as to clean the
> pages under reclaim ASAP. However for now the sync work can still block
> us for long time.
> 
> Jan Kara: limit the search scope. Note that the limited search and work
> pool is not a big problem: 1000 IOs under flight are typically more than
> enough to saturate the disk. And the overheads of searching in the work
> list didn't even show up in the perf report.
> 
> 4) test case
> 
> Run 2 dd tasks in a 100MB memcg (a very handy test case from Greg Thelen):
> 
> 	mkdir /cgroup/x
> 	echo 100M > /cgroup/x/memory.limit_in_bytes
> 	echo $$ > /cgroup/x/tasks
> 
> 	for i in `seq 2`
> 	do
> 		dd if=/dev/zero of=/fs/f$i bs=1k count=1M &
> 	done
> 
> Before patch, the dd tasks are quickly OOM killed.
> After patch, they run well with reasonably good performance and overheads:
> 
> 1073741824 bytes (1.1 GB) copied, 22.2196 s, 48.3 MB/s
> 1073741824 bytes (1.1 GB) copied, 22.4675 s, 47.8 MB/s
  I wonder what happens if you run:
       mkdir /cgroup/x
       echo 100M > /cgroup/x/memory.limit_in_bytes
       echo $$ > /cgroup/x/tasks

       for (( i = 0; i < 2; i++ )); do
         mkdir /fs/d$i
         for (( j = 0; j < 5000; j++ )); do 
           dd if=/dev/zero of=/fs/d$i/f$j bs=1k count=50
         done &
       done
  Because for small files the writearound logic won't help much... Also the
number of work items queued might become interesting.

Another common case to test - run 'slapadd' command in each cgroup to
create big LDAP database. That does pretty much random IO on a big mmaped
DB file.

> +/*
> + * schedule writeback on a range of inode pages.
> + */
> +static struct wb_writeback_work *
> +bdi_flush_inode_range(struct backing_dev_info *bdi,
> +		      struct inode *inode,
> +		      pgoff_t offset,
> +		      pgoff_t len,
> +		      bool wait)
> +{
> +	struct wb_writeback_work *work;
> +
> +	if (!igrab(inode))
> +		return ERR_PTR(-ENOENT);
  One technical note here: If the inode is deleted while it is queued, this
reference will keep it living until flusher thread gets to it. Then when
flusher thread puts its reference, the inode will get deleted in flusher
thread context. I don't see an immediate problem in that but it might be
surprising sometimes. Another problem I see is that if you try to
unmount the filesystem while the work item is queued, you'll get EBUSY for
no apparent reason (for userspace).

									Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: reclaim the LRU lists full of dirty/writeback pages

[Wu Fengguang]

[-- Attachment #1: Type: text/plain, Size: 32079 bytes --]

On Mon, Feb 13, 2012 at 04:43:13PM +0100, Jan Kara wrote:
> On Sun 12-02-12 11:10:29, Wu Fengguang wrote:

> > 4) test case
> > 
> > Run 2 dd tasks in a 100MB memcg (a very handy test case from Greg Thelen):
> > 
> > 	mkdir /cgroup/x
> > 	echo 100M > /cgroup/x/memory.limit_in_bytes
> > 	echo $$ > /cgroup/x/tasks
> > 
> > 	for i in `seq 2`
> > 	do
> > 		dd if=/dev/zero of=/fs/f$i bs=1k count=1M &
> > 	done
> > 
> > Before patch, the dd tasks are quickly OOM killed.
> > After patch, they run well with reasonably good performance and overheads:
> > 
> > 1073741824 bytes (1.1 GB) copied, 22.2196 s, 48.3 MB/s
> > 1073741824 bytes (1.1 GB) copied, 22.4675 s, 47.8 MB/s
>   I wonder what happens if you run:
>        mkdir /cgroup/x
>        echo 100M > /cgroup/x/memory.limit_in_bytes
>        echo $$ > /cgroup/x/tasks
> 
>        for (( i = 0; i < 2; i++ )); do
>          mkdir /fs/d$i
>          for (( j = 0; j < 5000; j++ )); do 
>            dd if=/dev/zero of=/fs/d$i/f$j bs=1k count=50
>          done &
>        done

That's a very good case, thanks!
 
>   Because for small files the writearound logic won't help much...

Right, it also means the native background work cannot be more I/O
efficient than the pageout works, except for the overheads of more
work items..

>   Also the number of work items queued might become interesting.

It turns out that the 1024 mempool reservations are not exhausted at
all (the below patch as a trace_printk on alloc failure and it didn't
trigger at all).

Here is the representative iostat lines on XFS (full "iostat -kx 1 20" log attached):

avg-cpu:  %user   %nice %system %iowait  %steal   %idle                                                                     
           0.80    0.00    6.03    0.03    0.00   93.14                                                                     
                                                                                                                            
Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util                   
sda               0.00   205.00    0.00  163.00     0.00 16900.00   207.36     4.09   21.63   1.88  30.70                   

The attached dirtied/written progress graph looks interesting.
Although the iostat disk utilization is low, the "dirtied" progress
line is pretty straight and there is no single congestion_wait event
in the trace log. Which makes me wonder if there are some unknown
blocking issues in the way.

> Another common case to test - run 'slapadd' command in each cgroup to
> create big LDAP database. That does pretty much random IO on a big mmaped
> DB file.

I've not used this. Will it need some configuration and data feed?
fio looks more handy to me for emulating mmap random IO.

> > +/*
> > + * schedule writeback on a range of inode pages.
> > + */
> > +static struct wb_writeback_work *
> > +bdi_flush_inode_range(struct backing_dev_info *bdi,
> > +		      struct inode *inode,
> > +		      pgoff_t offset,
> > +		      pgoff_t len,
> > +		      bool wait)
> > +{
> > +	struct wb_writeback_work *work;
> > +
> > +	if (!igrab(inode))
> > +		return ERR_PTR(-ENOENT);
>   One technical note here: If the inode is deleted while it is queued, this
> reference will keep it living until flusher thread gets to it. Then when
> flusher thread puts its reference, the inode will get deleted in flusher
> thread context. I don't see an immediate problem in that but it might be
> surprising sometimes. Another problem I see is that if you try to
> unmount the filesystem while the work item is queued, you'll get EBUSY for
> no apparent reason (for userspace).

Yeah, we need to make umount work.

And I find the pageout works seem to have some problems with ext4.
For example, this can be easily triggered with 10 dd tasks running
inside the 100MB limited memcg:

[18006.858109] INFO: task jbd2/sda1-8:51294 blocked for more than 120 seconds.
[18006.866425] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[18006.876096] jbd2/sda1-8     D 0000000000000000  5464 51294      2 0x00000000
[18006.884729]  ffff88040b097c70 0000000000000046 ffff880823032310 ffff88040b096000
[18006.894356]  00000000001d2f00 00000000001d2f00 ffff8808230322a0 00000000001d2f00
[18006.904000]  ffff88040b097fd8 00000000001d2f00 ffff88040b097fd8 00000000001d2f00
[18006.913652] Call Trace:
[18006.916901]  [<ffffffff8103d4af>] ? native_sched_clock+0x29/0x70
[18006.924134]  [<ffffffff81232aab>] ? jbd2_journal_commit_transaction+0x1d0/0x1281
[18006.933324]  [<ffffffff8109660d>] ? local_clock+0x41/0x5a
[18006.939879]  [<ffffffff810b0ddd>] ? lock_release_holdtime+0xa3/0xac
[18006.947410]  [<ffffffff81232aab>] ? jbd2_journal_commit_transaction+0x1d0/0x1281
[18006.956607]  [<ffffffff81a57904>] schedule+0x5a/0x5c
[18006.962677]  [<ffffffff81232ab0>] jbd2_journal_commit_transaction+0x1d5/0x1281
[18006.971683]  [<ffffffff8103d4af>] ? native_sched_clock+0x29/0x70
[18006.978933]  [<ffffffff810738ce>] ? try_to_del_timer_sync+0xba/0xc8
[18006.986452]  [<ffffffff8109660d>] ? local_clock+0x41/0x5a
[18006.992999]  [<ffffffff8108683a>] ? wake_up_bit+0x2a/0x2a
[18006.999542]  [<ffffffff810738ce>] ? try_to_del_timer_sync+0xba/0xc8
[18007.007062]  [<ffffffff81073a6f>] ? del_timer_sync+0xbb/0xce
[18007.013898]  [<ffffffff810739b4>] ? process_timeout+0x10/0x10
[18007.020835]  [<ffffffff81237bc1>] kjournald2+0xcf/0x242
[18007.027187]  [<ffffffff8108683a>] ? wake_up_bit+0x2a/0x2a
[18007.033733]  [<ffffffff81237af2>] ? commit_timeout+0x10/0x10
[18007.040574]  [<ffffffff81086384>] kthread+0x95/0x9d
[18007.046542]  [<ffffffff81a61134>] kernel_thread_helper+0x4/0x10
[18007.053675]  [<ffffffff81a591b4>] ? retint_restore_args+0x13/0x13
[18007.061003]  [<ffffffff810862ef>] ? __init_kthread_worker+0x5b/0x5b
[18007.068521]  [<ffffffff81a61130>] ? gs_change+0x13/0x13
[18007.074878] no locks held by jbd2/sda1-8/51294.

Sometimes I also catch dd/ext4lazyinit/flush all stalling in start_this_handle:

[17985.439567] dd              D 0000000000000007  3616 61440      1 0x00000004
[17985.448088]  ffff88080d71b9b8 0000000000000046 ffff88081ec80070 ffff88080d71a000
[17985.457545]  00000000001d2f00 00000000001d2f00 ffff88081ec80000 00000000001d2f00
[17985.467168]  ffff88080d71bfd8 00000000001d2f00 ffff88080d71bfd8 00000000001d2f00
[17985.476647] Call Trace:
[17985.479843]  [<ffffffff8103d4af>] ? native_sched_clock+0x29/0x70
[17985.487025]  [<ffffffff81230b9d>] ? start_this_handle+0x357/0x4ed
[17985.494313]  [<ffffffff8109660d>] ? local_clock+0x41/0x5a
[17985.500815]  [<ffffffff810b0ddd>] ? lock_release_holdtime+0xa3/0xac
[17985.508287]  [<ffffffff81230b9d>] ? start_this_handle+0x357/0x4ed
[17985.515575]  [<ffffffff81a57904>] schedule+0x5a/0x5c
[17985.521588]  [<ffffffff81230c39>] start_this_handle+0x3f3/0x4ed
[17985.528669]  [<ffffffff81147820>] ? kmem_cache_free+0xfa/0x13a
[17985.545142]  [<ffffffff8108683a>] ? wake_up_bit+0x2a/0x2a
[17985.551650]  [<ffffffff81230f0e>] jbd2__journal_start+0xb0/0xf6
[17985.558732]  [<ffffffff811f7ad7>] ? ext4_dirty_inode+0x1d/0x4c
[17985.565716]  [<ffffffff81230f67>] jbd2_journal_start+0x13/0x15
[17985.572703]  [<ffffffff8120e3e9>] ext4_journal_start_sb+0x13f/0x157
[17985.580172]  [<ffffffff8109660d>] ? local_clock+0x41/0x5a
[17985.586680]  [<ffffffff811f7ad7>] ext4_dirty_inode+0x1d/0x4c
[17985.593472]  [<ffffffff81176827>] __mark_inode_dirty+0x2e/0x1cc
[17985.600552]  [<ffffffff81168e84>] file_update_time+0xe4/0x106
[17985.607441]  [<ffffffff811079f6>] __generic_file_aio_write+0x254/0x364
[17985.615202]  [<ffffffff81a565da>] ? mutex_lock_nested+0x2e4/0x2f3
[17985.622488]  [<ffffffff81107b50>] ? generic_file_aio_write+0x4a/0xc1
[17985.630057]  [<ffffffff81107b6c>] generic_file_aio_write+0x66/0xc1
[17985.637442]  [<ffffffff811ef72b>] ext4_file_write+0x1f9/0x251
[17985.644330]  [<ffffffff8109660d>] ? local_clock+0x41/0x5a
[17985.650835]  [<ffffffff8118809e>] ? fsnotify+0x222/0x27b
[17985.657238]  [<ffffffff81153612>] do_sync_write+0xce/0x10b
[17985.663844]  [<ffffffff8118809e>] ? fsnotify+0x222/0x27b
[17985.670243]  [<ffffffff81187ef8>] ? fsnotify+0x7c/0x27b
[17985.676561]  [<ffffffff81153dbe>] vfs_write+0xb8/0x157
[17985.682767]  [<ffffffff81154075>] sys_write+0x4d/0x77
[17985.688878]  [<ffffffff81a5fce9>] system_call_fastpath+0x16/0x1b

and jbd2 in

[17983.623657] jbd2/sda1-8     D 0000000000000000  5464 51294      2 0x00000000
[17983.632173]  ffff88040b097c70 0000000000000046 ffff880823032310 ffff88040b096000
[17983.641640]  00000000001d2f00 00000000001d2f00 ffff8808230322a0 00000000001d2f00
[17983.651119]  ffff88040b097fd8 00000000001d2f00 ffff88040b097fd8 00000000001d2f00
[17983.660603] Call Trace:
[17983.663808]  [<ffffffff8103d4af>] ? native_sched_clock+0x29/0x70
[17983.670997]  [<ffffffff81232aab>] ? jbd2_journal_commit_transaction+0x1d0/0x1281
[17983.680124]  [<ffffffff8109660d>] ? local_clock+0x41/0x5a
[17983.686638]  [<ffffffff810b0ddd>] ? lock_release_holdtime+0xa3/0xac
[17983.694108]  [<ffffffff81232aab>] ? jbd2_journal_commit_transaction+0x1d0/0x1281
[17983.703243]  [<ffffffff81a57904>] schedule+0x5a/0x5c
[17983.709262]  [<ffffffff81232ab0>] jbd2_journal_commit_transaction+0x1d5/0x1281
[17983.718195]  [<ffffffff8103d4af>] ? native_sched_clock+0x29/0x70
[17983.725392]  [<ffffffff810738ce>] ? try_to_del_timer_sync+0xba/0xc8
[17983.732867]  [<ffffffff8109660d>] ? local_clock+0x41/0x5a
[17983.739374]  [<ffffffff8108683a>] ? wake_up_bit+0x2a/0x2a
[17983.745864]  [<ffffffff810738ce>] ? try_to_del_timer_sync+0xba/0xc8
[17983.753343]  [<ffffffff81073a6f>] ? del_timer_sync+0xbb/0xce
[17983.760137]  [<ffffffff810739b4>] ? process_timeout+0x10/0x10
[17983.767041]  [<ffffffff81237bc1>] kjournald2+0xcf/0x242
[17983.773361]  [<ffffffff8108683a>] ? wake_up_bit+0x2a/0x2a
[17983.779863]  [<ffffffff81237af2>] ? commit_timeout+0x10/0x10
[17983.786665]  [<ffffffff81086384>] kthread+0x95/0x9d
[17983.792585]  [<ffffffff81a61134>] kernel_thread_helper+0x4/0x10
[17983.799670]  [<ffffffff81a591b4>] ? retint_restore_args+0x13/0x13
[17983.806948]  [<ffffffff810862ef>] ? __init_kthread_worker+0x5b/0x5b

Here is the updated patch used in the new tests. It moves
congestion_wait() out of the page lock and make flush_inode_page() no
longer wait for memory allocation (looks unnecessary).

Thanks,
Fengguang
---
Subject: writeback: introduce the pageout work
Date: Thu Jul 29 14:41:19 CST 2010

This relays file pageout IOs to the flusher threads.

The ultimate target is to gracefully handle the LRU lists full of
dirty/writeback pages.

1) I/O efficiency

The flusher will piggy back the nearby ~10ms worth of dirty pages for I/O.

This takes advantage of the time/spacial locality in most workloads: the
nearby pages of one file are typically populated into the LRU at the same
time, hence will likely be close to each other in the LRU list. Writing
them in one shot helps clean more pages effectively for page reclaim.

2) OOM avoidance and scan rate control

Typically we do LRU scan w/o rate control and quickly get enough clean
pages for the LRU lists not full of dirty pages.

Or we can still get a number of freshly cleaned pages (moved to LRU tail
by end_page_writeback()) when the queued pageout I/O is completed within
tens of milli-seconds.

However if the LRU list is small and full of dirty pages, it can be
quickly fully scanned and go OOM before the flusher manages to clean
enough pages.

A simple yet reliable scheme is employed to avoid OOM and keep scan rate
in sync with the I/O rate:

	if (PageReclaim(page))
		congestion_wait(HZ/10);

PG_reclaim plays the key role. When dirty pages are encountered, we
queue I/O for it, set PG_reclaim and put it back to the LRU head.
So if PG_reclaim pages are encountered again, it means the dirty page
has not yet been cleaned by the flusher after a full zone scan. It
indicates we are scanning more fast than I/O and shall take a snap.

The runtime behavior on a fully dirtied small LRU list would be:
It will start with a quick scan of the list, queuing all pages for I/O.
Then the scan will be slowed down by the PG_reclaim pages *adaptively*
to match the I/O bandwidth.

3) writeback work coordinations

To avoid memory allocations at page reclaim, a mempool for struct
wb_writeback_work is created.

wakeup_flusher_threads() is removed because it can easily delay the
more oriented pageout works and even exhaust the mempool reservations.
It's also found to not I/O efficient by frequently submitting writeback
works with small ->nr_pages.

Background/periodic works will quit automatically, so as to clean the
pages under reclaim ASAP. However for now the sync work can still block
us for long time.

Jan Kara: limit the search scope. Note that the limited search and work
pool is not a big problem: 1000 IOs under flight are typically more than
enough to saturate the disk. And the overheads of searching in the work
list didn't even show up in the perf report.

4) test case

Run 2 dd tasks in a 100MB memcg (a very handy test case from Greg Thelen):

	mkdir /cgroup/x
	echo 100M > /cgroup/x/memory.limit_in_bytes
	echo $$ > /cgroup/x/tasks

	for i in `seq 2`
	do
		dd if=/dev/zero of=/fs/f$i bs=1k count=1M &
	done

Before patch, the dd tasks are quickly OOM killed.
After patch, they run well with reasonably good performance and overheads:

1073741824 bytes (1.1 GB) copied, 22.2196 s, 48.3 MB/s
1073741824 bytes (1.1 GB) copied, 22.4675 s, 47.8 MB/s

iostat -kx 1

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00    0.00  178.00     0.00 89568.00  1006.38    74.35  417.71   4.80  85.40
sda               0.00     2.00    0.00  191.00     0.00 94428.00   988.77    53.34  219.03   4.34  82.90
sda               0.00    20.00    0.00  196.00     0.00 97712.00   997.06    71.11  337.45   4.77  93.50
sda               0.00     5.00    0.00  175.00     0.00 84648.00   967.41    54.03  316.44   5.06  88.60
sda               0.00     0.00    0.00  186.00     0.00 92432.00   993.89    56.22  267.54   5.38 100.00
sda               0.00     1.00    0.00  183.00     0.00 90156.00   985.31    37.99  325.55   4.33  79.20
sda               0.00     0.00    0.00  175.00     0.00 88692.00  1013.62    48.70  218.43   4.69  82.10
sda               0.00     0.00    0.00  196.00     0.00 97528.00   995.18    43.38  236.87   5.10 100.00
sda               0.00     0.00    0.00  179.00     0.00 88648.00   990.48    45.83  285.43   5.59 100.00
sda               0.00     0.00    0.00  178.00     0.00 88500.00   994.38    28.28  158.89   4.99  88.80
sda               0.00     0.00    0.00  194.00     0.00 95852.00   988.16    32.58  167.39   5.15 100.00
sda               0.00     2.00    0.00  215.00     0.00 105996.00   986.01    41.72  201.43   4.65 100.00
sda               0.00     4.00    0.00  173.00     0.00 84332.00   974.94    50.48  260.23   5.76  99.60
sda               0.00     0.00    0.00  182.00     0.00 90312.00   992.44    36.83  212.07   5.49 100.00
sda               0.00     8.00    0.00  195.00     0.00 95940.50   984.01    50.18  221.06   5.13 100.00
sda               0.00     1.00    0.00  220.00     0.00 108852.00   989.56    40.99  202.68   4.55 100.00
sda               0.00     2.00    0.00  161.00     0.00 80384.00   998.56    37.19  268.49   6.21 100.00
sda               0.00     4.00    0.00  182.00     0.00 90830.00   998.13    50.58  239.77   5.49 100.00
sda               0.00     0.00    0.00  197.00     0.00 94877.00   963.22    36.68  196.79   5.08 100.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.25    0.00   15.08   33.92    0.00   50.75
           0.25    0.00   14.54   35.09    0.00   50.13
           0.50    0.00   13.57   32.41    0.00   53.52
           0.50    0.00   11.28   36.84    0.00   51.38
           0.50    0.00   15.75   32.00    0.00   51.75
           0.50    0.00   10.50   34.00    0.00   55.00
           0.50    0.00   17.63   27.46    0.00   54.41
           0.50    0.00   15.08   30.90    0.00   53.52
           0.50    0.00   11.28   32.83    0.00   55.39
           0.75    0.00   16.79   26.82    0.00   55.64
           0.50    0.00   16.08   29.15    0.00   54.27
           0.50    0.00   13.50   30.50    0.00   55.50
           0.50    0.00   14.32   35.18    0.00   50.00
           0.50    0.00   12.06   33.92    0.00   53.52
           0.50    0.00   17.29   30.58    0.00   51.63
           0.50    0.00   15.08   29.65    0.00   54.77
           0.50    0.00   12.53   29.32    0.00   57.64
           0.50    0.00   15.29   31.83    0.00   52.38

The global dd numbers for comparison:

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00    0.00  189.00     0.00 95752.00  1013.25   143.09  684.48   5.29 100.00
sda               0.00     0.00    0.00  208.00     0.00 105480.00  1014.23   143.06  733.29   4.81 100.00
sda               0.00     0.00    0.00  161.00     0.00 81924.00  1017.69   141.71  757.79   6.21 100.00
sda               0.00     0.00    0.00  217.00     0.00 109580.00  1009.95   143.09  749.55   4.61 100.10
sda               0.00     0.00    0.00  187.00     0.00 94728.00  1013.13   144.31  773.67   5.35 100.00
sda               0.00     0.00    0.00  189.00     0.00 95752.00  1013.25   144.14  742.00   5.29 100.00
sda               0.00     0.00    0.00  177.00     0.00 90032.00  1017.31   143.32  656.59   5.65 100.00
sda               0.00     0.00    0.00  215.00     0.00 108640.00  1010.60   142.90  817.54   4.65 100.00
sda               0.00     2.00    0.00  166.00     0.00 83858.00  1010.34   143.64  808.61   6.02 100.00
sda               0.00     0.00    0.00  186.00     0.00 92813.00   997.99   141.18  736.95   5.38 100.00
sda               0.00     0.00    0.00  206.00     0.00 104456.00  1014.14   146.27  729.33   4.85 100.00
sda               0.00     0.00    0.00  213.00     0.00 107024.00  1004.92   143.25  705.70   4.69 100.00
sda               0.00     0.00    0.00  188.00     0.00 95748.00  1018.60   141.82  764.78   5.32 100.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.51    0.00   11.22   52.30    0.00   35.97
           0.25    0.00   10.15   52.54    0.00   37.06
           0.25    0.00    5.01   56.64    0.00   38.10
           0.51    0.00   15.15   43.94    0.00   40.40
           0.25    0.00   12.12   48.23    0.00   39.39
           0.51    0.00   11.20   53.94    0.00   34.35
           0.26    0.00    9.72   51.41    0.00   38.62
           0.76    0.00    9.62   50.63    0.00   38.99
           0.51    0.00   10.46   53.32    0.00   35.71
           0.51    0.00    9.41   51.91    0.00   38.17
           0.25    0.00   10.69   49.62    0.00   39.44
           0.51    0.00   12.21   52.67    0.00   34.61
           0.51    0.00   11.45   53.18    0.00   34.86

XXX: commit NFS unstable pages via write_inode()
XXX: the added congestion_wait() may be undesirable in some situations

CC: Jan Kara <jack@suse.cz>
CC: Mel Gorman <mgorman@suse.de>
Acked-by: Rik van Riel <riel@redhat.com>
CC: Greg Thelen <gthelen@google.com>
CC: Minchan Kim <minchan.kim@gmail.com>
Signed-off-by: Wu Fengguang <fengguang.wu@intel.com>
---
 fs/fs-writeback.c                |  169 ++++++++++++++++++++++++++++-
 include/linux/writeback.h        |    4 
 include/trace/events/writeback.h |   12 +-
 mm/vmscan.c                      |   35 ++++--
 4 files changed, 202 insertions(+), 18 deletions(-)

- move congestion_wait() out of the page lock: it's blocking btrfs lock_delalloc_pages()

--- linux.orig/mm/vmscan.c	2012-02-12 21:27:28.000000000 +0800
+++ linux/mm/vmscan.c	2012-02-13 12:14:20.000000000 +0800
@@ -767,7 +767,8 @@ static unsigned long shrink_page_list(st
 				      struct scan_control *sc,
 				      int priority,
 				      unsigned long *ret_nr_dirty,
-				      unsigned long *ret_nr_writeback)
+				      unsigned long *ret_nr_writeback,
+				      unsigned long *ret_nr_pgreclaim)
 {
 	LIST_HEAD(ret_pages);
 	LIST_HEAD(free_pages);
@@ -776,6 +777,7 @@ static unsigned long shrink_page_list(st
 	unsigned long nr_congested = 0;
 	unsigned long nr_reclaimed = 0;
 	unsigned long nr_writeback = 0;
+	unsigned long nr_pgreclaim = 0;
 
 	cond_resched();
 
@@ -813,6 +815,10 @@ static unsigned long shrink_page_list(st
 
 		if (PageWriteback(page)) {
 			nr_writeback++;
+			if (PageReclaim(page))
+				nr_pgreclaim++;
+			else
+				SetPageReclaim(page);
 			/*
 			 * Synchronous reclaim cannot queue pages for
 			 * writeback due to the possibility of stack overflow
@@ -874,12 +880,15 @@ static unsigned long shrink_page_list(st
 			nr_dirty++;
 
 			/*
-			 * Only kswapd can writeback filesystem pages to
-			 * avoid risk of stack overflow but do not writeback
-			 * unless under significant pressure.
+			 * run into the visited page again: we are scanning
+			 * faster than the flusher can writeout dirty pages
 			 */
-			if (page_is_file_cache(page) &&
-					(!current_is_kswapd() || priority >= DEF_PRIORITY - 2)) {
+			if (page_is_file_cache(page) && PageReclaim(page)) {
+				nr_pgreclaim++;
+				goto keep_locked;
+			}
+			if (page_is_file_cache(page) && mapping &&
+			    flush_inode_page(mapping, page, false) >= 0) {
 				/*
 				 * Immediately reclaim when written back.
 				 * Similar in principal to deactivate_page()
@@ -1028,6 +1037,7 @@ keep_lumpy:
 	count_vm_events(PGACTIVATE, pgactivate);
 	*ret_nr_dirty += nr_dirty;
 	*ret_nr_writeback += nr_writeback;
+	*ret_nr_pgreclaim += nr_pgreclaim;
 	return nr_reclaimed;
 }
 
@@ -1087,8 +1097,10 @@ int __isolate_lru_page(struct page *page
 	 */
 	if (mode & (ISOLATE_CLEAN|ISOLATE_ASYNC_MIGRATE)) {
 		/* All the caller can do on PageWriteback is block */
-		if (PageWriteback(page))
+		if (PageWriteback(page)) {
+			SetPageReclaim(page);
 			return ret;
+		}
 
 		if (PageDirty(page)) {
 			struct address_space *mapping;
@@ -1509,6 +1521,7 @@ shrink_inactive_list(unsigned long nr_to
 	unsigned long nr_file;
 	unsigned long nr_dirty = 0;
 	unsigned long nr_writeback = 0;
+	unsigned long nr_pgreclaim = 0;
 	isolate_mode_t reclaim_mode = ISOLATE_INACTIVE;
 	struct zone *zone = mz->zone;
 
@@ -1559,13 +1572,13 @@ shrink_inactive_list(unsigned long nr_to
 	spin_unlock_irq(&zone->lru_lock);
 
 	nr_reclaimed = shrink_page_list(&page_list, mz, sc, priority,
-						&nr_dirty, &nr_writeback);
+				&nr_dirty, &nr_writeback, &nr_pgreclaim);
 
 	/* Check if we should syncronously wait for writeback */
 	if (should_reclaim_stall(nr_taken, nr_reclaimed, priority, sc)) {
 		set_reclaim_mode(priority, sc, true);
 		nr_reclaimed += shrink_page_list(&page_list, mz, sc,
-					priority, &nr_dirty, &nr_writeback);
+			priority, &nr_dirty, &nr_writeback, &nr_pgreclaim);
 	}
 
 	spin_lock_irq(&zone->lru_lock);
@@ -1608,6 +1621,8 @@ shrink_inactive_list(unsigned long nr_to
 	 */
 	if (nr_writeback && nr_writeback >= (nr_taken >> (DEF_PRIORITY-priority)))
 		wait_iff_congested(zone, BLK_RW_ASYNC, HZ/10);
+	if (nr_pgreclaim)
+		congestion_wait(BLK_RW_ASYNC, HZ/10);
 
 	trace_mm_vmscan_lru_shrink_inactive(zone->zone_pgdat->node_id,
 		zone_idx(zone),
@@ -2382,8 +2397,6 @@ static unsigned long do_try_to_free_page
 		 */
 		writeback_threshold = sc->nr_to_reclaim + sc->nr_to_reclaim / 2;
 		if (total_scanned > writeback_threshold) {
-			wakeup_flusher_threads(laptop_mode ? 0 : total_scanned,
-						WB_REASON_TRY_TO_FREE_PAGES);
 			sc->may_writepage = 1;
 		}
 
--- linux.orig/fs/fs-writeback.c	2012-02-12 21:27:28.000000000 +0800
+++ linux/fs/fs-writeback.c	2012-02-13 12:15:50.000000000 +0800
@@ -41,6 +41,8 @@ struct wb_writeback_work {
 	long nr_pages;
 	struct super_block *sb;
 	unsigned long *older_than_this;
+	struct inode *inode;
+	pgoff_t offset;
 	enum writeback_sync_modes sync_mode;
 	unsigned int tagged_writepages:1;
 	unsigned int for_kupdate:1;
@@ -65,6 +67,27 @@ struct wb_writeback_work {
  */
 int nr_pdflush_threads;
 
+static mempool_t *wb_work_mempool;
+
+static void *wb_work_alloc(gfp_t gfp_mask, void *pool_data)
+{
+	/*
+	 * bdi_flush_inode_range() may be called on page reclaim
+	 */
+	if (current->flags & PF_MEMALLOC)
+		return NULL;
+
+	return kmalloc(sizeof(struct wb_writeback_work), gfp_mask);
+}
+
+static __init int wb_work_init(void)
+{
+	wb_work_mempool = mempool_create(1024,
+					 wb_work_alloc, mempool_kfree, NULL);
+	return wb_work_mempool ? 0 : -ENOMEM;
+}
+fs_initcall(wb_work_init);
+
 /**
  * writeback_in_progress - determine whether there is writeback in progress
  * @bdi: the device's backing_dev_info structure.
@@ -129,7 +152,7 @@ __bdi_start_writeback(struct backing_dev
 	 * This is WB_SYNC_NONE writeback, so if allocation fails just
 	 * wakeup the thread for old dirty data writeback
 	 */
-	work = kzalloc(sizeof(*work), GFP_ATOMIC);
+	work = mempool_alloc(wb_work_mempool, GFP_NOWAIT);
 	if (!work) {
 		if (bdi->wb.task) {
 			trace_writeback_nowork(bdi);
@@ -138,6 +161,7 @@ __bdi_start_writeback(struct backing_dev
 		return;
 	}
 
+	memset(work, 0, sizeof(*work));
 	work->sync_mode	= WB_SYNC_NONE;
 	work->nr_pages	= nr_pages;
 	work->range_cyclic = range_cyclic;
@@ -186,6 +210,125 @@ void bdi_start_background_writeback(stru
 	spin_unlock_bh(&bdi->wb_lock);
 }
 
+static bool extend_writeback_range(struct wb_writeback_work *work,
+				   pgoff_t offset,
+				   unsigned long write_around_pages)
+{
+	pgoff_t end = work->offset + work->nr_pages;
+
+	if (offset >= work->offset && offset < end)
+		return true;
+
+	/*
+	 * for sequential workloads with good locality, include up to 8 times
+	 * more data in one chunk
+	 */
+	if (work->nr_pages >= 8 * write_around_pages)
+		return false;
+
+	/* the unsigned comparison helps eliminate one compare */
+	if (work->offset - offset < write_around_pages) {
+		work->nr_pages += write_around_pages;
+		work->offset -= write_around_pages;
+		return true;
+	}
+
+	if (offset - end < write_around_pages) {
+		work->nr_pages += write_around_pages;
+		return true;
+	}
+
+	return false;
+}
+
+/*
+ * schedule writeback on a range of inode pages.
+ */
+static struct wb_writeback_work *
+bdi_flush_inode_range(struct backing_dev_info *bdi,
+		      struct inode *inode,
+		      pgoff_t offset,
+		      pgoff_t len,
+		      bool wait)
+{
+	struct wb_writeback_work *work;
+
+	if (!igrab(inode))
+		return ERR_PTR(-ENOENT);
+
+	work = mempool_alloc(wb_work_mempool, wait ? GFP_NOIO : GFP_NOWAIT);
+	if (!work) {
+		trace_printk("wb_work_mempool alloc fail\n");
+		return ERR_PTR(-ENOMEM);
+	}
+
+	memset(work, 0, sizeof(*work));
+	work->sync_mode		= WB_SYNC_NONE;
+	work->inode		= inode;
+	work->offset		= offset;
+	work->nr_pages		= len;
+	work->reason		= WB_REASON_PAGEOUT;
+
+	bdi_queue_work(bdi, work);
+
+	return work;
+}
+
+/*
+ * Called by page reclaim code to flush the dirty page ASAP. Do write-around to
+ * improve IO throughput. The nearby pages will have good chance to reside in
+ * the same LRU list that vmscan is working on, and even close to each other
+ * inside the LRU list in the common case of sequential read/write.
+ *
+ * ret > 0: success, found/reused a previous writeback work
+ * ret = 0: success, allocated/queued a new writeback work
+ * ret < 0: failed
+ */
+long flush_inode_page(struct address_space *mapping,
+		      struct page *page,
+		      bool wait)
+{
+	struct backing_dev_info *bdi = mapping->backing_dev_info;
+	struct inode *inode = mapping->host;
+	struct wb_writeback_work *work;
+	unsigned long write_around_pages;
+	pgoff_t offset = page->index;
+	int i;
+	long ret = 0;
+
+	if (unlikely(!inode))
+		return -ENOENT;
+
+	/*
+	 * piggy back 8-15ms worth of data
+	 */
+	write_around_pages = bdi->avg_write_bandwidth + MIN_WRITEBACK_PAGES;
+	write_around_pages = rounddown_pow_of_two(write_around_pages) >> 6;
+
+	i = 1;
+	spin_lock_bh(&bdi->wb_lock);
+	list_for_each_entry_reverse(work, &bdi->work_list, list) {
+		if (work->inode != inode)
+			continue;
+		if (extend_writeback_range(work, offset, write_around_pages)) {
+			ret = i;
+			break;
+		}
+		if (i++ > 100)	/* limit search depth */
+			break;
+	}
+	spin_unlock_bh(&bdi->wb_lock);
+
+	if (!ret) {
+		offset = round_down(offset, write_around_pages);
+		work = bdi_flush_inode_range(bdi, inode,
+					     offset, write_around_pages, wait);
+		if (IS_ERR(work))
+			ret = PTR_ERR(work);
+	}
+	return ret;
+}
+
 /*
  * Remove the inode from the writeback list it is on.
  */
@@ -833,6 +976,23 @@ static unsigned long get_nr_dirty_pages(
 		get_nr_dirty_inodes();
 }
 
+static long wb_flush_inode(struct bdi_writeback *wb,
+			   struct wb_writeback_work *work)
+{
+	struct writeback_control wbc = {
+		.sync_mode = WB_SYNC_NONE,
+		.nr_to_write = LONG_MAX,
+		.range_start = work->offset << PAGE_CACHE_SHIFT,
+		.range_end = (work->offset + work->nr_pages - 1)
+						<< PAGE_CACHE_SHIFT,
+	};
+
+	do_writepages(work->inode->i_mapping, &wbc);
+	iput(work->inode);
+
+	return LONG_MAX - wbc.nr_to_write;
+}
+
 static long wb_check_background_flush(struct bdi_writeback *wb)
 {
 	if (over_bground_thresh(wb->bdi)) {
@@ -905,7 +1065,10 @@ long wb_do_writeback(struct bdi_writebac
 
 		trace_writeback_exec(bdi, work);
 
-		wrote += wb_writeback(wb, work);
+		if (work->inode)
+			wrote += wb_flush_inode(wb, work);
+		else
+			wrote += wb_writeback(wb, work);
 
 		/*
 		 * Notify the caller of completion if this is a synchronous
@@ -914,7 +1077,7 @@ long wb_do_writeback(struct bdi_writebac
 		if (work->done)
 			complete(work->done);
 		else
-			kfree(work);
+			mempool_free(work, wb_work_mempool);
 	}
 
 	/*
--- linux.orig/include/trace/events/writeback.h	2012-02-12 21:27:33.000000000 +0800
+++ linux/include/trace/events/writeback.h	2012-02-12 21:27:34.000000000 +0800
@@ -23,7 +23,7 @@
 
 #define WB_WORK_REASON							\
 		{WB_REASON_BACKGROUND,		"background"},		\
-		{WB_REASON_TRY_TO_FREE_PAGES,	"try_to_free_pages"},	\
+		{WB_REASON_PAGEOUT,		"pageout"},		\
 		{WB_REASON_SYNC,		"sync"},		\
 		{WB_REASON_PERIODIC,		"periodic"},		\
 		{WB_REASON_LAPTOP_TIMER,	"laptop_timer"},	\
@@ -45,6 +45,8 @@ DECLARE_EVENT_CLASS(writeback_work_class
 		__field(int, range_cyclic)
 		__field(int, for_background)
 		__field(int, reason)
+		__field(unsigned long, ino)
+		__field(unsigned long, offset)
 	),
 	TP_fast_assign(
 		strncpy(__entry->name, dev_name(bdi->dev), 32);
@@ -55,9 +57,11 @@ DECLARE_EVENT_CLASS(writeback_work_class
 		__entry->range_cyclic = work->range_cyclic;
 		__entry->for_background	= work->for_background;
 		__entry->reason = work->reason;
+		__entry->ino = work->inode ? work->inode->i_ino : 0;
+		__entry->offset = work->offset;
 	),
 	TP_printk("bdi %s: sb_dev %d:%d nr_pages=%ld sync_mode=%d "
-		  "kupdate=%d range_cyclic=%d background=%d reason=%s",
+		  "kupdate=%d range_cyclic=%d background=%d reason=%s ino=%lu offset=%lu",
 		  __entry->name,
 		  MAJOR(__entry->sb_dev), MINOR(__entry->sb_dev),
 		  __entry->nr_pages,
@@ -65,7 +69,9 @@ DECLARE_EVENT_CLASS(writeback_work_class
 		  __entry->for_kupdate,
 		  __entry->range_cyclic,
 		  __entry->for_background,
-		  __print_symbolic(__entry->reason, WB_WORK_REASON)
+		  __print_symbolic(__entry->reason, WB_WORK_REASON),
+		  __entry->ino,
+		  __entry->offset
 	)
 );
 #define DEFINE_WRITEBACK_WORK_EVENT(name) \
--- linux.orig/include/linux/writeback.h	2012-02-12 21:27:28.000000000 +0800
+++ linux/include/linux/writeback.h	2012-02-12 21:27:34.000000000 +0800
@@ -40,7 +40,7 @@ enum writeback_sync_modes {
  */
 enum wb_reason {
 	WB_REASON_BACKGROUND,
-	WB_REASON_TRY_TO_FREE_PAGES,
+	WB_REASON_PAGEOUT,
 	WB_REASON_SYNC,
 	WB_REASON_PERIODIC,
 	WB_REASON_LAPTOP_TIMER,
@@ -94,6 +94,8 @@ long writeback_inodes_wb(struct bdi_writ
 				enum wb_reason reason);
 long wb_do_writeback(struct bdi_writeback *wb, int force_wait);
 void wakeup_flusher_threads(long nr_pages, enum wb_reason reason);
+long flush_inode_page(struct address_space *mapping, struct page *page,
+		      bool wait);
 
 /* writeback.h requires fs.h; it, too, is not included from here. */
 static inline void wait_on_inode(struct inode *inode)

[-- Attachment #2: global_dirtied_written.png --]
[-- Type: image/png, Size: 37386 bytes --]

[-- Attachment #3: iostat --]
[-- Type: text/plain, Size: 6586 bytes --]

Linux 3.3.0-rc3-flush-page+ (snb) 	02/14/2012 	_x86_64_	(32 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.03    0.00    0.05    0.38    0.00   99.54

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.20    33.00    0.05    2.91     0.16  1025.66   694.73     1.50  508.37   6.10   1.80

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.90    0.00    5.72    0.00    0.00   93.38

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00    1.00    0.00     8.00     0.00    16.00     0.00    1.00   1.00   0.10

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.90    0.00    5.82    0.00    0.00   93.28

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.90    0.00    5.81    0.00    0.00   93.29

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.83    0.00    5.84    0.00    0.00   93.33

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.83    0.00    5.94    0.00    0.00   93.23

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00   370.00    0.00   71.00     0.00 12480.00   351.55    11.17   71.76   3.56  25.30

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.80    0.00    5.84    0.00    0.00   93.35

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00    65.00    0.00   51.00     0.00 13104.00   513.88     2.85  170.22   4.00  20.40

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.87    0.00    5.93    0.00    0.00   93.20

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00   379.00    0.00  134.00     0.00 30004.00   447.82    15.43  116.99   4.11  55.10

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.90    0.00    5.91    0.00    0.00   93.19

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00   133.00    0.00  114.00     0.00 12792.00   224.42     3.20   28.12   2.30  26.20

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.83    0.00    5.96    0.00    0.00   93.21

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00   367.00    0.00  114.00     0.00 24960.00   437.89    12.55  110.12   4.35  49.60

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.80    0.00    6.03    0.03    0.00   93.14

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00   205.00    0.00  163.00     0.00 16900.00   207.36     4.09   21.63   1.88  30.70

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.87    0.00    5.98    0.00    0.00   93.16

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00   219.00    0.00  182.00     0.00 19292.00   212.00     4.42   21.41   1.97  35.80

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.93    0.00    5.87    0.00    0.00   93.20

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00   201.00    0.00  135.00     0.00 21216.00   314.31     6.39   55.44   2.97  40.10

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.93    0.00    5.99    0.00    0.00   93.08

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00   329.00    0.00  166.00     0.00 24336.00   293.20    10.42   58.39   3.19  53.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.80    0.00    5.96    0.00    0.00   93.24

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00   258.00    0.00  181.00     0.00 16848.00   186.17     4.01   17.64   1.36  24.70

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.90    0.00    5.92    0.00    0.00   93.18

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00   166.00    0.00   88.00     0.00 20488.00   465.64     6.09   86.72   4.68  41.20

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.86    0.00    6.05    0.00    0.00   93.08

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00   566.00    0.00  194.00     0.00 34528.00   355.96    20.50   78.32   3.16  61.30

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.86    0.00    5.79    0.00    0.00   93.34

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00    0.00   12.00     0.00  5616.00   936.00     0.77  506.67   9.58  11.50

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.90    0.00    5.93    0.00    0.00   93.18

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00   259.00    0.00  173.00     0.00 22464.00   259.70     4.79   27.67   2.09  36.10

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.74    0.00    6.16    0.00    0.00   93.10

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00   190.00    0.00  163.00     0.00 18304.00   224.59     3.67   22.50   1.78  29.00

Re: reclaim the LRU lists full of dirty/writeback pages

[Jan Kara]

On Tue 14-02-12 18:03:48, Wu Fengguang wrote:
> On Mon, Feb 13, 2012 at 04:43:13PM +0100, Jan Kara wrote:
> > On Sun 12-02-12 11:10:29, Wu Fengguang wrote:
> 
> > > 4) test case
> > > 
> > > Run 2 dd tasks in a 100MB memcg (a very handy test case from Greg Thelen):
> > > 
> > > 	mkdir /cgroup/x
> > > 	echo 100M > /cgroup/x/memory.limit_in_bytes
> > > 	echo $$ > /cgroup/x/tasks
> > > 
> > > 	for i in `seq 2`
> > > 	do
> > > 		dd if=/dev/zero of=/fs/f$i bs=1k count=1M &
> > > 	done
> > > 
> > > Before patch, the dd tasks are quickly OOM killed.
> > > After patch, they run well with reasonably good performance and overheads:
> > > 
> > > 1073741824 bytes (1.1 GB) copied, 22.2196 s, 48.3 MB/s
> > > 1073741824 bytes (1.1 GB) copied, 22.4675 s, 47.8 MB/s
> >   I wonder what happens if you run:
> >        mkdir /cgroup/x
> >        echo 100M > /cgroup/x/memory.limit_in_bytes
> >        echo $$ > /cgroup/x/tasks
> > 
> >        for (( i = 0; i < 2; i++ )); do
> >          mkdir /fs/d$i
> >          for (( j = 0; j < 5000; j++ )); do 
> >            dd if=/dev/zero of=/fs/d$i/f$j bs=1k count=50
> >          done &
> >        done
> 
> That's a very good case, thanks!
>  
> >   Because for small files the writearound logic won't help much...
> 
> Right, it also means the native background work cannot be more I/O
> efficient than the pageout works, except for the overheads of more
> work items..
  Yes, that's true.

> >   Also the number of work items queued might become interesting.
> 
> It turns out that the 1024 mempool reservations are not exhausted at
> all (the below patch as a trace_printk on alloc failure and it didn't
> trigger at all).
> 
> Here is the representative iostat lines on XFS (full "iostat -kx 1 20" log attached):
> 
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle                                                                     
>            0.80    0.00    6.03    0.03    0.00   93.14                                                                     
>                                                                                                                             
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util                   
> sda               0.00   205.00    0.00  163.00     0.00 16900.00   207.36     4.09   21.63   1.88  30.70                   
> 
> The attached dirtied/written progress graph looks interesting.
> Although the iostat disk utilization is low, the "dirtied" progress
> line is pretty straight and there is no single congestion_wait event
> in the trace log. Which makes me wonder if there are some unknown
> blocking issues in the way.
  Interesting. I'd also expect we should block in reclaim path. How fast
can dd threads progress when there is no cgroup involved?
 
> > Another common case to test - run 'slapadd' command in each cgroup to
> > create big LDAP database. That does pretty much random IO on a big mmaped
> > DB file.
> 
> I've not used this. Will it need some configuration and data feed?
> fio looks more handy to me for emulating mmap random IO.
  Yes, fio can generate random mmap IO. It's just that this is a real life
workload. So it is not completely random, it happens on several files and
is also interleaved with other memory allocations from DB. I can send you
the config files and data feed if you are interested.

> > > +/*
> > > + * schedule writeback on a range of inode pages.
> > > + */
> > > +static struct wb_writeback_work *
> > > +bdi_flush_inode_range(struct backing_dev_info *bdi,
> > > +		      struct inode *inode,
> > > +		      pgoff_t offset,
> > > +		      pgoff_t len,
> > > +		      bool wait)
> > > +{
> > > +	struct wb_writeback_work *work;
> > > +
> > > +	if (!igrab(inode))
> > > +		return ERR_PTR(-ENOENT);
> >   One technical note here: If the inode is deleted while it is queued, this
> > reference will keep it living until flusher thread gets to it. Then when
> > flusher thread puts its reference, the inode will get deleted in flusher
> > thread context. I don't see an immediate problem in that but it might be
> > surprising sometimes. Another problem I see is that if you try to
> > unmount the filesystem while the work item is queued, you'll get EBUSY for
> > no apparent reason (for userspace).
> 
> Yeah, we need to make umount work.
  The positive thing is that if the inode is reaped while the work item is
queue, we know all that needed to be done is done. So we don't really need
to pin the inode.

> And I find the pageout works seem to have some problems with ext4.
> For example, this can be easily triggered with 10 dd tasks running
> inside the 100MB limited memcg:
  So journal thread is getting stuck while committing transaction. Most
likely waiting for some dd thread to stop a transaction so that commit can
proceed. The processes waiting in start_this_handle() are just secondary
effect resulting from the first problem. It might be interesting to get
stack traces of all bloked processes when the journal thread is stuck.


> [18006.858109] INFO: task jbd2/sda1-8:51294 blocked for more than 120 seconds.
> [18006.866425] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [18006.876096] jbd2/sda1-8     D 0000000000000000  5464 51294      2 0x00000000
> [18006.884729]  ffff88040b097c70 0000000000000046 ffff880823032310 ffff88040b096000
> [18006.894356]  00000000001d2f00 00000000001d2f00 ffff8808230322a0 00000000001d2f00
> [18006.904000]  ffff88040b097fd8 00000000001d2f00 ffff88040b097fd8 00000000001d2f00
> [18006.913652] Call Trace:
> [18006.916901]  [<ffffffff8103d4af>] ? native_sched_clock+0x29/0x70
> [18006.924134]  [<ffffffff81232aab>] ? jbd2_journal_commit_transaction+0x1d0/0x1281
> [18006.933324]  [<ffffffff8109660d>] ? local_clock+0x41/0x5a
> [18006.939879]  [<ffffffff810b0ddd>] ? lock_release_holdtime+0xa3/0xac
> [18006.947410]  [<ffffffff81232aab>] ? jbd2_journal_commit_transaction+0x1d0/0x1281
> [18006.956607]  [<ffffffff81a57904>] schedule+0x5a/0x5c
> [18006.962677]  [<ffffffff81232ab0>] jbd2_journal_commit_transaction+0x1d5/0x1281
> [18006.971683]  [<ffffffff8103d4af>] ? native_sched_clock+0x29/0x70
> [18006.978933]  [<ffffffff810738ce>] ? try_to_del_timer_sync+0xba/0xc8
> [18006.986452]  [<ffffffff8109660d>] ? local_clock+0x41/0x5a
> [18006.992999]  [<ffffffff8108683a>] ? wake_up_bit+0x2a/0x2a
> [18006.999542]  [<ffffffff810738ce>] ? try_to_del_timer_sync+0xba/0xc8
> [18007.007062]  [<ffffffff81073a6f>] ? del_timer_sync+0xbb/0xce
> [18007.013898]  [<ffffffff810739b4>] ? process_timeout+0x10/0x10
> [18007.020835]  [<ffffffff81237bc1>] kjournald2+0xcf/0x242
> [18007.027187]  [<ffffffff8108683a>] ? wake_up_bit+0x2a/0x2a
> [18007.033733]  [<ffffffff81237af2>] ? commit_timeout+0x10/0x10
> [18007.040574]  [<ffffffff81086384>] kthread+0x95/0x9d
> [18007.046542]  [<ffffffff81a61134>] kernel_thread_helper+0x4/0x10
> [18007.053675]  [<ffffffff81a591b4>] ? retint_restore_args+0x13/0x13
> [18007.061003]  [<ffffffff810862ef>] ? __init_kthread_worker+0x5b/0x5b
> [18007.068521]  [<ffffffff81a61130>] ? gs_change+0x13/0x13
> [18007.074878] no locks held by jbd2/sda1-8/51294.
> 
> Sometimes I also catch dd/ext4lazyinit/flush all stalling in start_this_handle:
> 
> [17985.439567] dd              D 0000000000000007  3616 61440      1 0x00000004
> [17985.448088]  ffff88080d71b9b8 0000000000000046 ffff88081ec80070 ffff88080d71a000
> [17985.457545]  00000000001d2f00 00000000001d2f00 ffff88081ec80000 00000000001d2f00
> [17985.467168]  ffff88080d71bfd8 00000000001d2f00 ffff88080d71bfd8 00000000001d2f00
> [17985.476647] Call Trace:
> [17985.479843]  [<ffffffff8103d4af>] ? native_sched_clock+0x29/0x70
> [17985.487025]  [<ffffffff81230b9d>] ? start_this_handle+0x357/0x4ed
> [17985.494313]  [<ffffffff8109660d>] ? local_clock+0x41/0x5a
> [17985.500815]  [<ffffffff810b0ddd>] ? lock_release_holdtime+0xa3/0xac
> [17985.508287]  [<ffffffff81230b9d>] ? start_this_handle+0x357/0x4ed
> [17985.515575]  [<ffffffff81a57904>] schedule+0x5a/0x5c
> [17985.521588]  [<ffffffff81230c39>] start_this_handle+0x3f3/0x4ed
> [17985.528669]  [<ffffffff81147820>] ? kmem_cache_free+0xfa/0x13a
> [17985.545142]  [<ffffffff8108683a>] ? wake_up_bit+0x2a/0x2a
> [17985.551650]  [<ffffffff81230f0e>] jbd2__journal_start+0xb0/0xf6
> [17985.558732]  [<ffffffff811f7ad7>] ? ext4_dirty_inode+0x1d/0x4c
> [17985.565716]  [<ffffffff81230f67>] jbd2_journal_start+0x13/0x15
> [17985.572703]  [<ffffffff8120e3e9>] ext4_journal_start_sb+0x13f/0x157
> [17985.580172]  [<ffffffff8109660d>] ? local_clock+0x41/0x5a
> [17985.586680]  [<ffffffff811f7ad7>] ext4_dirty_inode+0x1d/0x4c
> [17985.593472]  [<ffffffff81176827>] __mark_inode_dirty+0x2e/0x1cc
> [17985.600552]  [<ffffffff81168e84>] file_update_time+0xe4/0x106
> [17985.607441]  [<ffffffff811079f6>] __generic_file_aio_write+0x254/0x364
> [17985.615202]  [<ffffffff81a565da>] ? mutex_lock_nested+0x2e4/0x2f3
> [17985.622488]  [<ffffffff81107b50>] ? generic_file_aio_write+0x4a/0xc1
> [17985.630057]  [<ffffffff81107b6c>] generic_file_aio_write+0x66/0xc1
> [17985.637442]  [<ffffffff811ef72b>] ext4_file_write+0x1f9/0x251
> [17985.644330]  [<ffffffff8109660d>] ? local_clock+0x41/0x5a
> [17985.650835]  [<ffffffff8118809e>] ? fsnotify+0x222/0x27b
> [17985.657238]  [<ffffffff81153612>] do_sync_write+0xce/0x10b
> [17985.663844]  [<ffffffff8118809e>] ? fsnotify+0x222/0x27b
> [17985.670243]  [<ffffffff81187ef8>] ? fsnotify+0x7c/0x27b
> [17985.676561]  [<ffffffff81153dbe>] vfs_write+0xb8/0x157
> [17985.682767]  [<ffffffff81154075>] sys_write+0x4d/0x77
> [17985.688878]  [<ffffffff81a5fce9>] system_call_fastpath+0x16/0x1b
> 
> and jbd2 in
> 
> [17983.623657] jbd2/sda1-8     D 0000000000000000  5464 51294      2 0x00000000
> [17983.632173]  ffff88040b097c70 0000000000000046 ffff880823032310 ffff88040b096000
> [17983.641640]  00000000001d2f00 00000000001d2f00 ffff8808230322a0 00000000001d2f00
> [17983.651119]  ffff88040b097fd8 00000000001d2f00 ffff88040b097fd8 00000000001d2f00
> [17983.660603] Call Trace:
> [17983.663808]  [<ffffffff8103d4af>] ? native_sched_clock+0x29/0x70
> [17983.670997]  [<ffffffff81232aab>] ? jbd2_journal_commit_transaction+0x1d0/0x1281
> [17983.680124]  [<ffffffff8109660d>] ? local_clock+0x41/0x5a
> [17983.686638]  [<ffffffff810b0ddd>] ? lock_release_holdtime+0xa3/0xac
> [17983.694108]  [<ffffffff81232aab>] ? jbd2_journal_commit_transaction+0x1d0/0x1281
> [17983.703243]  [<ffffffff81a57904>] schedule+0x5a/0x5c
> [17983.709262]  [<ffffffff81232ab0>] jbd2_journal_commit_transaction+0x1d5/0x1281
> [17983.718195]  [<ffffffff8103d4af>] ? native_sched_clock+0x29/0x70
> [17983.725392]  [<ffffffff810738ce>] ? try_to_del_timer_sync+0xba/0xc8
> [17983.732867]  [<ffffffff8109660d>] ? local_clock+0x41/0x5a
> [17983.739374]  [<ffffffff8108683a>] ? wake_up_bit+0x2a/0x2a
> [17983.745864]  [<ffffffff810738ce>] ? try_to_del_timer_sync+0xba/0xc8
> [17983.753343]  [<ffffffff81073a6f>] ? del_timer_sync+0xbb/0xce
> [17983.760137]  [<ffffffff810739b4>] ? process_timeout+0x10/0x10
> [17983.767041]  [<ffffffff81237bc1>] kjournald2+0xcf/0x242
> [17983.773361]  [<ffffffff8108683a>] ? wake_up_bit+0x2a/0x2a
> [17983.779863]  [<ffffffff81237af2>] ? commit_timeout+0x10/0x10
> [17983.786665]  [<ffffffff81086384>] kthread+0x95/0x9d
> [17983.792585]  [<ffffffff81a61134>] kernel_thread_helper+0x4/0x10
> [17983.799670]  [<ffffffff81a591b4>] ? retint_restore_args+0x13/0x13
> [17983.806948]  [<ffffffff810862ef>] ? __init_kthread_worker+0x5b/0x5b
> 
> Here is the updated patch used in the new tests. It moves
> congestion_wait() out of the page lock and make flush_inode_page() no
> longer wait for memory allocation (looks unnecessary).

								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: reclaim the LRU lists full of dirty/writeback pages

[Wu Fengguang]

On Tue, Feb 14, 2012 at 02:29:50PM +0100, Jan Kara wrote:

> > >   I wonder what happens if you run:
> > >        mkdir /cgroup/x
> > >        echo 100M > /cgroup/x/memory.limit_in_bytes
> > >        echo $$ > /cgroup/x/tasks
> > > 
> > >        for (( i = 0; i < 2; i++ )); do
> > >          mkdir /fs/d$i
> > >          for (( j = 0; j < 5000; j++ )); do 
> > >            dd if=/dev/zero of=/fs/d$i/f$j bs=1k count=50
> > >          done &
> > >        done
> > 
> > That's a very good case, thanks!
> >  
> > >   Because for small files the writearound logic won't help much...
> > 
> > Right, it also means the native background work cannot be more I/O
> > efficient than the pageout works, except for the overheads of more
> > work items..
>   Yes, that's true.
> 
> > >   Also the number of work items queued might become interesting.
> > 
> > It turns out that the 1024 mempool reservations are not exhausted at
> > all (the below patch as a trace_printk on alloc failure and it didn't
> > trigger at all).
> > 
> > Here is the representative iostat lines on XFS (full "iostat -kx 1 20" log attached):
> > 
> > avg-cpu:  %user   %nice %system %iowait  %steal   %idle                                                                     
> >            0.80    0.00    6.03    0.03    0.00   93.14                                                                     
> >                                                                                                                             
> > Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util                   
> > sda               0.00   205.00    0.00  163.00     0.00 16900.00   207.36     4.09   21.63   1.88  30.70                   
> > 
> > The attached dirtied/written progress graph looks interesting.
> > Although the iostat disk utilization is low, the "dirtied" progress
> > line is pretty straight and there is no single congestion_wait event
> > in the trace log. Which makes me wonder if there are some unknown
> > blocking issues in the way.
>   Interesting. I'd also expect we should block in reclaim path. How fast
> can dd threads progress when there is no cgroup involved?

I tried running the dd tasks in global context with

        echo $((100<<20)) > /proc/sys/vm/dirty_bytes

and got mostly the same results on XFS:

        avg-cpu:  %user   %nice %system %iowait  %steal   %idle
                   0.85    0.00    8.88    0.00    0.00   90.26

        Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
        sda               0.00     0.00    0.00   50.00     0.00 23036.00   921.44     9.59  738.02   7.38  36.90

        avg-cpu:  %user   %nice %system %iowait  %steal   %idle
                   0.95    0.00    8.95    0.00    0.00   90.11

        Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
        sda               0.00   854.00    0.00   99.00     0.00 19552.00   394.99    34.14   87.98   3.82  37.80

Interestingly, ext4 shows comparable throughput, however is reporting
near 100% disk utilization:

        avg-cpu:  %user   %nice %system %iowait  %steal   %idle
                   0.76    0.00    9.02    0.00    0.00   90.23

        Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
        sda               0.00     0.00    0.00  317.00     0.00 20956.00   132.21    28.57   82.71   3.16 100.10

        avg-cpu:  %user   %nice %system %iowait  %steal   %idle
                   0.82    0.00    8.95    0.00    0.00   90.23

        Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
        sda               0.00     0.00    0.00  402.00     0.00 24388.00   121.33    21.09   58.55   2.42  97.40

        avg-cpu:  %user   %nice %system %iowait  %steal   %idle
                   0.82    0.00    8.99    0.00    0.00   90.19

        Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
        sda               0.00     0.00    0.00  409.00     0.00 21996.00   107.56    15.25   36.74   2.30  94.10

And btrfs shows

        avg-cpu:  %user   %nice %system %iowait  %steal   %idle
                   0.76    0.00   23.59    0.00    0.00   75.65

        Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
        sda               0.00   801.00    0.00  141.00     0.00 48984.00   694.81    41.08  291.36   6.11  86.20

        avg-cpu:  %user   %nice %system %iowait  %steal   %idle
                   0.72    0.00   12.65    0.00    0.00   86.62

        Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
        sda               0.00   792.00    0.00   69.00     0.00 15288.00   443.13    22.74   69.35   4.09  28.20

        avg-cpu:  %user   %nice %system %iowait  %steal   %idle
                   0.83    0.00   23.11    0.00    0.00   76.06

        Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
        sda               0.00     0.00    0.00   73.00     0.00 33280.00   911.78    22.09  548.58   8.10  59.10

> > > Another common case to test - run 'slapadd' command in each cgroup to
> > > create big LDAP database. That does pretty much random IO on a big mmaped
> > > DB file.
> > 
> > I've not used this. Will it need some configuration and data feed?
> > fio looks more handy to me for emulating mmap random IO.
>   Yes, fio can generate random mmap IO. It's just that this is a real life
> workload. So it is not completely random, it happens on several files and
> is also interleaved with other memory allocations from DB. I can send you
> the config files and data feed if you are interested.

I'm very interested, thank you!

> > > > +/*
> > > > + * schedule writeback on a range of inode pages.
> > > > + */
> > > > +static struct wb_writeback_work *
> > > > +bdi_flush_inode_range(struct backing_dev_info *bdi,
> > > > +		      struct inode *inode,
> > > > +		      pgoff_t offset,
> > > > +		      pgoff_t len,
> > > > +		      bool wait)
> > > > +{
> > > > +	struct wb_writeback_work *work;
> > > > +
> > > > +	if (!igrab(inode))
> > > > +		return ERR_PTR(-ENOENT);
> > >   One technical note here: If the inode is deleted while it is queued, this
> > > reference will keep it living until flusher thread gets to it. Then when
> > > flusher thread puts its reference, the inode will get deleted in flusher
> > > thread context. I don't see an immediate problem in that but it might be
> > > surprising sometimes. Another problem I see is that if you try to
> > > unmount the filesystem while the work item is queued, you'll get EBUSY for
> > > no apparent reason (for userspace).
> > 
> > Yeah, we need to make umount work.
>   The positive thing is that if the inode is reaped while the work item is
> queue, we know all that needed to be done is done. So we don't really need
> to pin the inode.

But I do need to make sure the *inode pointer does not point to some
invalid memory at work exec time. Is this possible without raising
->i_count?

> > And I find the pageout works seem to have some problems with ext4.
> > For example, this can be easily triggered with 10 dd tasks running
> > inside the 100MB limited memcg:
>   So journal thread is getting stuck while committing transaction. Most
> likely waiting for some dd thread to stop a transaction so that commit can
> proceed. The processes waiting in start_this_handle() are just secondary
> effect resulting from the first problem. It might be interesting to get
> stack traces of all bloked processes when the journal thread is stuck.

For completeness of discussion, citing your conclusion on my private
data feed:

: We enter memcg reclaim from grab_cache_page_write_begin() and are
: waiting in congestion_wait(). Because grab_cache_page_write_begin() is
: called with transaction started, this blocks transaction from
: committing and subsequently blocks all other activity on the
: filesystem. The fact is this isn't new with your patches, just your
: changes or the fact that we are running in a memory constrained cgroup
: make this more visible.

Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: reclaim the LRU lists full of dirty/writeback pages

[Jan Kara]

On Thu 16-02-12 12:00:19, Wu Fengguang wrote:
> On Tue, Feb 14, 2012 at 02:29:50PM +0100, Jan Kara wrote:
> > > >   I wonder what happens if you run:
> > > >        mkdir /cgroup/x
> > > >        echo 100M > /cgroup/x/memory.limit_in_bytes
> > > >        echo $$ > /cgroup/x/tasks
> > > > 
> > > >        for (( i = 0; i < 2; i++ )); do
> > > >          mkdir /fs/d$i
> > > >          for (( j = 0; j < 5000; j++ )); do 
> > > >            dd if=/dev/zero of=/fs/d$i/f$j bs=1k count=50
> > > >          done &
> > > >        done
> > > 
> > > That's a very good case, thanks!
> > >  
> > > >   Because for small files the writearound logic won't help much...
> > > 
> > > Right, it also means the native background work cannot be more I/O
> > > efficient than the pageout works, except for the overheads of more
> > > work items..
> >   Yes, that's true.
> > 
> > > >   Also the number of work items queued might become interesting.
> > > 
> > > It turns out that the 1024 mempool reservations are not exhausted at
> > > all (the below patch as a trace_printk on alloc failure and it didn't
> > > trigger at all).
> > > 
> > > Here is the representative iostat lines on XFS (full "iostat -kx 1 20" log attached):
> > > 
> > > avg-cpu:  %user   %nice %system %iowait  %steal   %idle                                                                     
> > >            0.80    0.00    6.03    0.03    0.00   93.14                                                                     
> > >                                                                                                                             
> > > Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util                   
> > > sda               0.00   205.00    0.00  163.00     0.00 16900.00   207.36     4.09   21.63   1.88  30.70                   
> > > 
> > > The attached dirtied/written progress graph looks interesting.
> > > Although the iostat disk utilization is low, the "dirtied" progress
> > > line is pretty straight and there is no single congestion_wait event
> > > in the trace log. Which makes me wonder if there are some unknown
> > > blocking issues in the way.
> >   Interesting. I'd also expect we should block in reclaim path. How fast
> > can dd threads progress when there is no cgroup involved?
> 
> I tried running the dd tasks in global context with
> 
>         echo $((100<<20)) > /proc/sys/vm/dirty_bytes
> 
> and got mostly the same results on XFS:
> 
>         avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>                    0.85    0.00    8.88    0.00    0.00   90.26
> 
>         Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
>         sda               0.00     0.00    0.00   50.00     0.00 23036.00   921.44     9.59  738.02   7.38  36.90
> 
>         avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>                    0.95    0.00    8.95    0.00    0.00   90.11
> 
>         Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
>         sda               0.00   854.00    0.00   99.00     0.00 19552.00   394.99    34.14   87.98   3.82  37.80
  OK, so it seems that reclaiming pages in memcg reclaim acted as a natural
throttling similar to what balance_dirty_pages() does in the global case.


> Interestingly, ext4 shows comparable throughput, however is reporting
> near 100% disk utilization:
> 
>         avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>                    0.76    0.00    9.02    0.00    0.00   90.23
> 
>         Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
>         sda               0.00     0.00    0.00  317.00     0.00 20956.00   132.21    28.57   82.71   3.16 100.10
> 
>         avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>                    0.82    0.00    8.95    0.00    0.00   90.23
> 
>         Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
>         sda               0.00     0.00    0.00  402.00     0.00 24388.00   121.33    21.09   58.55   2.42  97.40
> 
>         avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>                    0.82    0.00    8.99    0.00    0.00   90.19
> 
>         Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
>         sda               0.00     0.00    0.00  409.00     0.00 21996.00   107.56    15.25   36.74   2.30  94.10
  Average request size is smaller so maybe ext4 does more seeking.

> > > > Another common case to test - run 'slapadd' command in each cgroup to
> > > > create big LDAP database. That does pretty much random IO on a big mmaped
> > > > DB file.
> > > 
> > > I've not used this. Will it need some configuration and data feed?
> > > fio looks more handy to me for emulating mmap random IO.
> >   Yes, fio can generate random mmap IO. It's just that this is a real life
> > workload. So it is not completely random, it happens on several files and
> > is also interleaved with other memory allocations from DB. I can send you
> > the config files and data feed if you are interested.
> 
> I'm very interested, thank you!
  OK, I'll send it in private email...

> > > > > +/*
> > > > > + * schedule writeback on a range of inode pages.
> > > > > + */
> > > > > +static struct wb_writeback_work *
> > > > > +bdi_flush_inode_range(struct backing_dev_info *bdi,
> > > > > +		      struct inode *inode,
> > > > > +		      pgoff_t offset,
> > > > > +		      pgoff_t len,
> > > > > +		      bool wait)
> > > > > +{
> > > > > +	struct wb_writeback_work *work;
> > > > > +
> > > > > +	if (!igrab(inode))
> > > > > +		return ERR_PTR(-ENOENT);
> > > >   One technical note here: If the inode is deleted while it is queued, this
> > > > reference will keep it living until flusher thread gets to it. Then when
> > > > flusher thread puts its reference, the inode will get deleted in flusher
> > > > thread context. I don't see an immediate problem in that but it might be
> > > > surprising sometimes. Another problem I see is that if you try to
> > > > unmount the filesystem while the work item is queued, you'll get EBUSY for
> > > > no apparent reason (for userspace).
> > > 
> > > Yeah, we need to make umount work.
> >   The positive thing is that if the inode is reaped while the work item is
> > queue, we know all that needed to be done is done. So we don't really need
> > to pin the inode.
> 
> But I do need to make sure the *inode pointer does not point to some
> invalid memory at work exec time. Is this possible without raising
> ->i_count?
  I was thinking about it and what should work is that we have inode
reference in work item but in generic_shutdown_super() we go through
the worklist and drop all work items for superblock before calling
evict_inodes()...

								Honza

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: reclaim the LRU lists full of dirty/writeback pages

[Wu Fengguang]

On Thu, Feb 16, 2012 at 01:44:45PM +0100, Jan Kara wrote:
> On Thu 16-02-12 12:00:19, Wu Fengguang wrote:
> > On Tue, Feb 14, 2012 at 02:29:50PM +0100, Jan Kara wrote:

> > > > > > +/*
> > > > > > + * schedule writeback on a range of inode pages.
> > > > > > + */
> > > > > > +static struct wb_writeback_work *
> > > > > > +bdi_flush_inode_range(struct backing_dev_info *bdi,
> > > > > > +		      struct inode *inode,
> > > > > > +		      pgoff_t offset,
> > > > > > +		      pgoff_t len,
> > > > > > +		      bool wait)
> > > > > > +{
> > > > > > +	struct wb_writeback_work *work;
> > > > > > +
> > > > > > +	if (!igrab(inode))
> > > > > > +		return ERR_PTR(-ENOENT);
> > > > >   One technical note here: If the inode is deleted while it is queued, this
> > > > > reference will keep it living until flusher thread gets to it. Then when
> > > > > flusher thread puts its reference, the inode will get deleted in flusher
> > > > > thread context. I don't see an immediate problem in that but it might be
> > > > > surprising sometimes. Another problem I see is that if you try to
> > > > > unmount the filesystem while the work item is queued, you'll get EBUSY for
> > > > > no apparent reason (for userspace).
> > > > 
> > > > Yeah, we need to make umount work.
> > >   The positive thing is that if the inode is reaped while the work item is
> > > queue, we know all that needed to be done is done. So we don't really need
> > > to pin the inode.
> > 
> > But I do need to make sure the *inode pointer does not point to some
> > invalid memory at work exec time. Is this possible without raising
> > ->i_count?
>   I was thinking about it and what should work is that we have inode
> reference in work item but in generic_shutdown_super() we go through
> the worklist and drop all work items for superblock before calling
> evict_inodes()...

Good point!

This diff removes the works after the sync_filesystem(sb) call.  After
which, no more dirty pages are expected on that sb (otherwise the
umount will fail anyway), hence no more pageout works will be queued
for that sb.

+static void wb_free_work(struct wb_writeback_work *work)
+{
+	/*
+	 * Notify the caller of completion if this is a synchronous
+	 * work item, otherwise just free it.
+	 */
+	if (work->done)
+		complete(work->done);
+	else
+		mempool_free(work, wb_work_mempool);
+}
+
+/*
+ * Remove works for @sb; or if (@sb == NULL), remove all works on @bdi.
+ */
+void bdi_remove_works(struct backing_dev_info *bdi, struct super_block *sb)
+{
+	struct inode *inode = mapping->host;
+	struct wb_writeback_work *work;
+
+	spin_lock_bh(&bdi->wb_lock);
+	list_for_each_entry_safe(work, &bdi->work_list, list) {
+		if (work->inode && work->inode->i_sb == sb) {
+			iput(inode);
+		} else if (sb && work->sb != sb)
+			continue;
+
+		list_del_init(&work->list);
+		wb_free_work(work);
+	}
+	spin_unlock_bh(&bdi->wb_lock);
+}

--- linux.orig/fs/super.c	2012-02-16 21:08:09.000000000 +0800
+++ linux/fs/super.c	2012-02-16 21:22:19.000000000 +0800
@@ -389,6 +389,7 @@ void generic_shutdown_super(struct super
 
 		fsnotify_unmount_inodes(&sb->s_inodes);
 
+		bdi_remove_works(sb->s_bdi, sb);
 		evict_inodes(sb);
 
 		if (sop->put_super)

Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: reclaim the LRU lists full of dirty/writeback pages

[Wu Fengguang]

On Thu, Feb 16, 2012 at 09:32:33PM +0800, Wu Fengguang wrote:
> On Thu, Feb 16, 2012 at 01:44:45PM +0100, Jan Kara wrote:
> > On Thu 16-02-12 12:00:19, Wu Fengguang wrote:
> > > On Tue, Feb 14, 2012 at 02:29:50PM +0100, Jan Kara wrote:
> 
> > > > > > > +/*
> > > > > > > + * schedule writeback on a range of inode pages.
> > > > > > > + */
> > > > > > > +static struct wb_writeback_work *
> > > > > > > +bdi_flush_inode_range(struct backing_dev_info *bdi,
> > > > > > > +		      struct inode *inode,
> > > > > > > +		      pgoff_t offset,
> > > > > > > +		      pgoff_t len,
> > > > > > > +		      bool wait)
> > > > > > > +{
> > > > > > > +	struct wb_writeback_work *work;
> > > > > > > +
> > > > > > > +	if (!igrab(inode))
> > > > > > > +		return ERR_PTR(-ENOENT);
> > > > > >   One technical note here: If the inode is deleted while it is queued, this
> > > > > > reference will keep it living until flusher thread gets to it. Then when
> > > > > > flusher thread puts its reference, the inode will get deleted in flusher
> > > > > > thread context. I don't see an immediate problem in that but it might be
> > > > > > surprising sometimes. Another problem I see is that if you try to
> > > > > > unmount the filesystem while the work item is queued, you'll get EBUSY for
> > > > > > no apparent reason (for userspace).
> > > > > 
> > > > > Yeah, we need to make umount work.
> > > >   The positive thing is that if the inode is reaped while the work item is
> > > > queue, we know all that needed to be done is done. So we don't really need
> > > > to pin the inode.
> > > 
> > > But I do need to make sure the *inode pointer does not point to some
> > > invalid memory at work exec time. Is this possible without raising
> > > ->i_count?
> >   I was thinking about it and what should work is that we have inode
> > reference in work item but in generic_shutdown_super() we go through
> > the worklist and drop all work items for superblock before calling
> > evict_inodes()...
> 
> Good point!
> 
> This diff removes the works after the sync_filesystem(sb) call.  After
> which, no more dirty pages are expected on that sb (otherwise the
> umount will fail anyway), hence no more pageout works will be queued
> for that sb.
> 
> +static void wb_free_work(struct wb_writeback_work *work)
> +{
> +	/*
> +	 * Notify the caller of completion if this is a synchronous
> +	 * work item, otherwise just free it.
> +	 */
> +	if (work->done)
> +		complete(work->done);
> +	else
> +		mempool_free(work, wb_work_mempool);
> +}
> +
> +/*
> + * Remove works for @sb; or if (@sb == NULL), remove all works on @bdi.
> + */
> +void bdi_remove_works(struct backing_dev_info *bdi, struct super_block *sb)
> +{
> +	struct inode *inode = mapping->host;
> +	struct wb_writeback_work *work;
> +
> +	spin_lock_bh(&bdi->wb_lock);
> +	list_for_each_entry_safe(work, &bdi->work_list, list) {
> +		if (work->inode && work->inode->i_sb == sb) {
> +			iput(inode);
> +		} else if (sb && work->sb != sb)
> +			continue;
> +
> +		list_del_init(&work->list);
> +		wb_free_work(work);
> +	}
> +	spin_unlock_bh(&bdi->wb_lock);
> +}

Sorry, this corrected function actually compiles:

+/* 
+ * Remove works for @sb; or if (@sb == NULL), remove all works on @bdi. 
+ */ 
+void bdi_remove_works(struct backing_dev_info *bdi, struct super_block *sb) 
+{ 
+       struct wb_writeback_work *work, *tmp; 
+       LIST_HEAD(works); 
+ 
+       spin_lock_bh(&bdi->wb_lock); 
+       list_for_each_entry_safe(work, tmp, &bdi->work_list, list) { 
+               if (sb) { 
+                       if (work->sb && work->sb != sb) 
+                               continue; 
+                       if (work->inode && work->inode->i_sb != sb) 
+                               continue; 
+               } 
+               list_move(&work->list, &works); 
+       } 
+       spin_unlock_bh(&bdi->wb_lock); 
+ 
+       while (!list_empty(&works)) { 
+               work = list_entry(works.next, 
+                                 struct wb_writeback_work, list); 
+               list_del_init(&work->list); 
+               if (work->inode) 
+                       iput(work->inode); 
+               wb_free_work(work); 
+       } 
+} 

> --- linux.orig/fs/super.c	2012-02-16 21:08:09.000000000 +0800
> +++ linux/fs/super.c	2012-02-16 21:22:19.000000000 +0800
> @@ -389,6 +389,7 @@ void generic_shutdown_super(struct super
>  
>  		fsnotify_unmount_inodes(&sb->s_inodes);
>  
> +		bdi_remove_works(sb->s_bdi, sb);
>  		evict_inodes(sb);
>  
>  		if (sop->put_super)
> 
> Thanks,
> Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: reclaim the LRU lists full of dirty/writeback pages

[Wu Fengguang]

> > > And I find the pageout works seem to have some problems with ext4.
> > > For example, this can be easily triggered with 10 dd tasks running
> > > inside the 100MB limited memcg:
> >   So journal thread is getting stuck while committing transaction. Most
> > likely waiting for some dd thread to stop a transaction so that commit can
> > proceed. The processes waiting in start_this_handle() are just secondary
> > effect resulting from the first problem. It might be interesting to get
> > stack traces of all bloked processes when the journal thread is stuck.
> 
> For completeness of discussion, citing your conclusion on my private
> data feed:
> 
> : We enter memcg reclaim from grab_cache_page_write_begin() and are
> : waiting in congestion_wait(). Because grab_cache_page_write_begin() is
> : called with transaction started, this blocks transaction from
> : committing and subsequently blocks all other activity on the
> : filesystem. The fact is this isn't new with your patches, just your
> : changes or the fact that we are running in a memory constrained cgroup
> : make this more visible.

Maybe I'm missing some deep FS restrictions, but can this page
allocation (and the one in ext4_write_begin) be moved before
ext4_journal_start()? So that the page reclaim can throttle the
__GFP_WRITE allocations at will.

---
 fs/ext4/inode.c |   22 +++++++++++-----------
 1 file changed, 11 insertions(+), 11 deletions(-)

--- linux.orig/fs/ext4/inode.c	2012-02-18 00:10:27.000000000 +0800
+++ linux/fs/ext4/inode.c	2012-02-18 00:31:19.000000000 +0800
@@ -2398,38 +2398,38 @@ static int ext4_da_write_begin(struct fi
 	if (ext4_nonda_switch(inode->i_sb)) {
 		*fsdata = (void *)FALL_BACK_TO_NONDELALLOC;
 		return ext4_write_begin(file, mapping, pos,
 					len, flags, pagep, fsdata);
 	}
 	*fsdata = (void *)0;
 	trace_ext4_da_write_begin(inode, pos, len, flags);
 retry:
+	page = grab_cache_page_write_begin(mapping, index, flags);
+	if (!page) {
+		ret = -ENOMEM;
+		goto out;
+	}
+	*pagep = page;
+
 	/*
 	 * With delayed allocation, we don't log the i_disksize update
 	 * if there is delayed block allocation. But we still need
 	 * to journalling the i_disksize update if writes to the end
 	 * of file which has an already mapped buffer.
 	 */
 	handle = ext4_journal_start(inode, 1);
 	if (IS_ERR(handle)) {
 		ret = PTR_ERR(handle);
+		unlock_page(page);
+		page_cache_release(page);
+		if (pos + len > inode->i_size)
+			truncate_inode_pages(inode->i_mapping, inode->i_size);
 		goto out;
 	}
-	/* We cannot recurse into the filesystem as the transaction is already
-	 * started */
-	flags |= AOP_FLAG_NOFS;
-
-	page = grab_cache_page_write_begin(mapping, index, flags);
-	if (!page) {
-		ext4_journal_stop(handle);
-		ret = -ENOMEM;
-		goto out;
-	}
-	*pagep = page;
 
 	ret = __block_write_begin(page, pos, len, ext4_da_get_block_prep);
 	if (ret < 0) {
 		unlock_page(page);
 		ext4_journal_stop(handle);
 		page_cache_release(page);
 		/*
 		 * block_write_begin may have instantiated a few blocks

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: reclaim the LRU lists full of dirty/writeback pages

[Jan Kara]

On Sat 18-02-12 00:41:33, Wu Fengguang wrote:
> > > > And I find the pageout works seem to have some problems with ext4.
> > > > For example, this can be easily triggered with 10 dd tasks running
> > > > inside the 100MB limited memcg:
> > >   So journal thread is getting stuck while committing transaction. Most
> > > likely waiting for some dd thread to stop a transaction so that commit can
> > > proceed. The processes waiting in start_this_handle() are just secondary
> > > effect resulting from the first problem. It might be interesting to get
> > > stack traces of all bloked processes when the journal thread is stuck.
> > 
> > For completeness of discussion, citing your conclusion on my private
> > data feed:
> > 
> > : We enter memcg reclaim from grab_cache_page_write_begin() and are
> > : waiting in congestion_wait(). Because grab_cache_page_write_begin() is
> > : called with transaction started, this blocks transaction from
> > : committing and subsequently blocks all other activity on the
> > : filesystem. The fact is this isn't new with your patches, just your
> > : changes or the fact that we are running in a memory constrained cgroup
> > : make this more visible.
> 
> Maybe I'm missing some deep FS restrictions, but can this page
> allocation (and the one in ext4_write_begin) be moved before
> ext4_journal_start()? So that the page reclaim can throttle the
> __GFP_WRITE allocations at will.
  You are missing the fact that in this way, things would deadlock quickly.
Lock ordering of ext4 is 'transaction start' -> 'page lock' (so that
writepages may be efficient) and so you cannot really have it differently
here - you can think of transaction start - transaction end pair as a
lock - unlock for lock ordering purposes. You could play some tricks like
allocating the page without locking, starting a transaction, locking a page
and checking that everything is OK but it isn't really nice and when memory
pressure is big, the gain is questionable.

								Honza
> ---
>  fs/ext4/inode.c |   22 +++++++++++-----------
>  1 file changed, 11 insertions(+), 11 deletions(-)
> 
> --- linux.orig/fs/ext4/inode.c	2012-02-18 00:10:27.000000000 +0800
> +++ linux/fs/ext4/inode.c	2012-02-18 00:31:19.000000000 +0800
> @@ -2398,38 +2398,38 @@ static int ext4_da_write_begin(struct fi
>  	if (ext4_nonda_switch(inode->i_sb)) {
>  		*fsdata = (void *)FALL_BACK_TO_NONDELALLOC;
>  		return ext4_write_begin(file, mapping, pos,
>  					len, flags, pagep, fsdata);
>  	}
>  	*fsdata = (void *)0;
>  	trace_ext4_da_write_begin(inode, pos, len, flags);
>  retry:
> +	page = grab_cache_page_write_begin(mapping, index, flags);
> +	if (!page) {
> +		ret = -ENOMEM;
> +		goto out;
> +	}
> +	*pagep = page;
> +
>  	/*
>  	 * With delayed allocation, we don't log the i_disksize update
>  	 * if there is delayed block allocation. But we still need
>  	 * to journalling the i_disksize update if writes to the end
>  	 * of file which has an already mapped buffer.
>  	 */
>  	handle = ext4_journal_start(inode, 1);
>  	if (IS_ERR(handle)) {
>  		ret = PTR_ERR(handle);
> +		unlock_page(page);
> +		page_cache_release(page);
> +		if (pos + len > inode->i_size)
> +			truncate_inode_pages(inode->i_mapping, inode->i_size);
>  		goto out;
>  	}
> -	/* We cannot recurse into the filesystem as the transaction is already
> -	 * started */
> -	flags |= AOP_FLAG_NOFS;
> -
> -	page = grab_cache_page_write_begin(mapping, index, flags);
> -	if (!page) {
> -		ext4_journal_stop(handle);
> -		ret = -ENOMEM;
> -		goto out;
> -	}
> -	*pagep = page;
>  
>  	ret = __block_write_begin(page, pos, len, ext4_da_get_block_prep);
>  	if (ret < 0) {
>  		unlock_page(page);
>  		ext4_journal_stop(handle);
>  		page_cache_release(page);
>  		/*
>  		 * block_write_begin may have instantiated a few blocks
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: reclaim the LRU lists full of dirty/writeback pages

[Mel Gorman]

On Sat, Feb 11, 2012 at 08:44:45PM +0800, Wu Fengguang wrote:
> <SNIP>
> --- linux.orig/mm/vmscan.c	2012-02-03 21:42:21.000000000 +0800
> +++ linux/mm/vmscan.c	2012-02-11 17:28:54.000000000 +0800
> @@ -813,6 +813,8 @@ static unsigned long shrink_page_list(st
>  
>  		if (PageWriteback(page)) {
>  			nr_writeback++;
> +			if (PageReclaim(page))
> +				congestion_wait(BLK_RW_ASYNC, HZ/10);
>  			/*
>  			 * Synchronous reclaim cannot queue pages for
>  			 * writeback due to the possibility of stack overflow

I didn't look closely at the rest of the patch, I'm just focusing on the
congestion_wait part. You called this out yourself but this is in fact
really really bad. If this is in place and a user copies a large amount of
data to slow storage like a USB stick, the system will stall severely. A
parallel streaming reader will certainly have major issues as it will enter
page reclaim, find a bunch of dirty USB-backed pages at the end of the LRU
(20% of memory potentially) and stall for HZ/10 on each one of them. How
badly each process is affected will vary.

For the OOM problem, a more reasonable stopgap might be to identify when
a process is scanning a memcg at high priority and encountered all
PageReclaim with no forward progress and to congestion_wait() if that
situation occurs. A preferable way would be to wait until the flusher
wakes up a waiter on PageReclaim pages to be written out because we want
to keep moving way from congestion_wait() if at all possible.

Another possibility would be to relook at LRU_IMMEDIATE but right now it
requires a page flag and I haven't devised a way around that. Besides,
it would only address the problem of PageREclaim pages being encountered,
it would not handle the case where a memcg was filled with PageReclaim pages.


-- 
Mel Gorman
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: reclaim the LRU lists full of dirty/writeback pages

[Wu Fengguang]

On Tue, Feb 14, 2012 at 10:19:31AM +0000, Mel Gorman wrote:
> On Sat, Feb 11, 2012 at 08:44:45PM +0800, Wu Fengguang wrote:
> > <SNIP>
> > --- linux.orig/mm/vmscan.c	2012-02-03 21:42:21.000000000 +0800
> > +++ linux/mm/vmscan.c	2012-02-11 17:28:54.000000000 +0800
> > @@ -813,6 +813,8 @@ static unsigned long shrink_page_list(st
> >  
> >  		if (PageWriteback(page)) {
> >  			nr_writeback++;
> > +			if (PageReclaim(page))
> > +				congestion_wait(BLK_RW_ASYNC, HZ/10);
> >  			/*
> >  			 * Synchronous reclaim cannot queue pages for
> >  			 * writeback due to the possibility of stack overflow
> 
> I didn't look closely at the rest of the patch, I'm just focusing on the
> congestion_wait part. You called this out yourself but this is in fact
> really really bad. If this is in place and a user copies a large amount of
> data to slow storage like a USB stick, the system will stall severely. A
> parallel streaming reader will certainly have major issues as it will enter
> page reclaim, find a bunch of dirty USB-backed pages at the end of the LRU
> (20% of memory potentially) and stall for HZ/10 on each one of them. How
> badly each process is affected will vary.

Cannot agree any more the principle...me just want to demonstrate the
idea first :-)
 
> For the OOM problem, a more reasonable stopgap might be to identify when
> a process is scanning a memcg at high priority and encountered all
> PageReclaim with no forward progress and to congestion_wait() if that
> situation occurs. A preferable way would be to wait until the flusher
> wakes up a waiter on PageReclaim pages to be written out because we want
> to keep moving way from congestion_wait() if at all possible.

Good points! Below is the more serious page reclaim changes.

The dirty/writeback pages may often come close to each other in the
LRU list, so the local test during a 32-page scan may still trigger
reclaim waits unnecessarily. Some global information on the percent
of dirty/writeback pages in the LRU list may help. Anyway the added
tests should still be much better than no protection.

A global wait queue and reclaim_wait() is introduced. The waiters will
be wakeup when pages are rotated by end_page_writeback() or lru drain.

I have to say its effectiveness depends on the filesystem... ext4
and btrfs do fluent IO completions, so reclaim_wait() works pretty
well:
              dd-14560 [017] ....  1360.894605: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=10000
              dd-14560 [017] ....  1360.904456: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=8000
              dd-14560 [017] ....  1360.908293: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
              dd-14560 [017] ....  1360.923960: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=15000
              dd-14560 [017] ....  1360.927810: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
              dd-14560 [017] ....  1360.931656: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
              dd-14560 [017] ....  1360.943503: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=10000
              dd-14560 [017] ....  1360.953289: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=7000
              dd-14560 [017] ....  1360.957177: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
              dd-14560 [017] ....  1360.972949: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=15000

However XFS does IO completions in very large batches (there may be
only several big IO completions in one second). So reclaim_wait()
mostly end up waiting to the full HZ/10 timeout:

              dd-4177  [008] ....   866.367661: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
              dd-4177  [010] ....   866.567583: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
              dd-4177  [012] ....   866.767458: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
              dd-4177  [013] ....   866.867419: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
              dd-4177  [008] ....   867.167266: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
              dd-4177  [010] ....   867.367168: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
              dd-4177  [012] ....   867.818950: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
              dd-4177  [013] ....   867.918905: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
              dd-4177  [013] ....   867.971657: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=52000
              dd-4177  [013] ....   867.971812: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=0
              dd-4177  [008] ....   868.355700: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
              dd-4177  [010] ....   868.700515: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000

> Another possibility would be to relook at LRU_IMMEDIATE but right now it
> requires a page flag and I haven't devised a way around that. Besides,
> it would only address the problem of PageREclaim pages being encountered,
> it would not handle the case where a memcg was filled with PageReclaim pages.

I also considered things like LRU_IMMEDIATE, however got no clear idea yet.
Since the simple "wait on PG_reclaim" approach appears to work for this
memcg dd case, it effectively disables me to think any further ;-)

For the single dd inside memcg, ext4 is now working pretty well, with
least CPU overheads:

(running from another test box, so not directly comparable with old tests)

        avg-cpu:  %user   %nice %system %iowait  %steal   %idle
                   0.03    0.00    0.85    5.35    0.00   93.77

        Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
        sda               0.00     0.00    0.00  112.00     0.00 57348.00  1024.07    81.66 1045.21   8.93 100.00

        avg-cpu:  %user   %nice %system %iowait  %steal   %idle
                   0.00    0.00    0.69    4.07    0.00   95.24

        Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
        sda               0.00   142.00    0.00  112.00     0.00 56832.00  1014.86   127.94  790.04   8.93 100.00

And xfs a bit less fluent:

        avg-cpu:  %user   %nice %system %iowait  %steal   %idle
                   0.00    0.00    3.79    2.54    0.00   93.68

        Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
        sda               0.00     0.00    0.00  108.00     0.00 54644.00  1011.93    48.13 1044.83   8.44  91.20

        avg-cpu:  %user   %nice %system %iowait  %steal   %idle
                   0.00    0.00    3.38    3.88    0.00   92.74

        Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
        sda               0.00     0.00    0.00  105.00     0.00 53156.00  1012.50   128.50  451.90   9.25  97.10

btrfs also looks good:

        avg-cpu:  %user   %nice %system %iowait  %steal   %idle
                   0.00    0.00    8.05    3.85    0.00   88.10

        Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
        sda               0.00     0.00    0.00  108.00     0.00 53248.00   986.07    88.11  643.99   9.26 100.00

        avg-cpu:  %user   %nice %system %iowait  %steal   %idle
                   0.00    0.00    4.04    2.51    0.00   93.45

        Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
        sda               0.00     0.00    0.00  112.00     0.00 57344.00  1024.00    91.58  998.41   8.93 100.00


Thanks,
Fengguang
---

--- linux.orig/include/linux/backing-dev.h	2012-02-14 19:43:06.000000000 +0800
+++ linux/include/linux/backing-dev.h	2012-02-14 19:49:26.000000000 +0800
@@ -304,6 +304,8 @@ void clear_bdi_congested(struct backing_
 void set_bdi_congested(struct backing_dev_info *bdi, int sync);
 long congestion_wait(int sync, long timeout);
 long wait_iff_congested(struct zone *zone, int sync, long timeout);
+long reclaim_wait(long timeout);
+void reclaim_rotated(void);
 
 static inline bool bdi_cap_writeback_dirty(struct backing_dev_info *bdi)
 {
--- linux.orig/mm/backing-dev.c	2012-02-14 19:26:15.000000000 +0800
+++ linux/mm/backing-dev.c	2012-02-14 20:09:45.000000000 +0800
@@ -873,3 +873,38 @@ out:
 	return ret;
 }
 EXPORT_SYMBOL(wait_iff_congested);
+
+static DECLARE_WAIT_QUEUE_HEAD(reclaim_wqh);
+
+/**
+ * reclaim_wait - wait for some pages being rotated to the LRU tail
+ * @timeout: timeout in jiffies
+ *
+ * Wait until @timeout, or when some (typically PG_reclaim under writeback)
+ * pages rotated to the LRU so that page reclaim can make progress.
+ */
+long reclaim_wait(long timeout)
+{
+	long ret;
+	unsigned long start = jiffies;
+	DEFINE_WAIT(wait);
+
+	prepare_to_wait(&reclaim_wqh, &wait, TASK_KILLABLE);
+	ret = io_schedule_timeout(timeout);
+	finish_wait(&reclaim_wqh, &wait);
+
+	trace_writeback_reclaim_wait(jiffies_to_usecs(timeout),
+				     jiffies_to_usecs(jiffies - start));
+
+	return ret;
+}
+EXPORT_SYMBOL(reclaim_wait);
+
+void reclaim_rotated()
+{
+	wait_queue_head_t *wqh = &reclaim_wqh;
+
+	if (waitqueue_active(wqh))
+		wake_up(wqh);
+}
+
--- linux.orig/mm/swap.c	2012-02-14 19:40:10.000000000 +0800
+++ linux/mm/swap.c	2012-02-14 19:45:13.000000000 +0800
@@ -253,6 +253,7 @@ static void pagevec_move_tail(struct pag
 
 	pagevec_lru_move_fn(pvec, pagevec_move_tail_fn, &pgmoved);
 	__count_vm_events(PGROTATED, pgmoved);
+	reclaim_rotated();
 }
 
 /*
--- linux.orig/mm/vmscan.c	2012-02-14 17:53:27.000000000 +0800
+++ linux/mm/vmscan.c	2012-02-14 19:44:11.000000000 +0800
@@ -767,7 +767,8 @@ static unsigned long shrink_page_list(st
 				      struct scan_control *sc,
 				      int priority,
 				      unsigned long *ret_nr_dirty,
-				      unsigned long *ret_nr_writeback)
+				      unsigned long *ret_nr_writeback,
+				      unsigned long *ret_nr_pgreclaim)
 {
 	LIST_HEAD(ret_pages);
 	LIST_HEAD(free_pages);
@@ -776,6 +777,7 @@ static unsigned long shrink_page_list(st
 	unsigned long nr_congested = 0;
 	unsigned long nr_reclaimed = 0;
 	unsigned long nr_writeback = 0;
+	unsigned long nr_pgreclaim = 0;
 
 	cond_resched();
 
@@ -813,6 +815,10 @@ static unsigned long shrink_page_list(st
 
 		if (PageWriteback(page)) {
 			nr_writeback++;
+			if (PageReclaim(page))
+				nr_pgreclaim++;
+			else
+				SetPageReclaim(page);
 			/*
 			 * Synchronous reclaim cannot queue pages for
 			 * writeback due to the possibility of stack overflow
@@ -874,12 +880,15 @@ static unsigned long shrink_page_list(st
 			nr_dirty++;
 
 			/*
-			 * Only kswapd can writeback filesystem pages to
-			 * avoid risk of stack overflow but do not writeback
-			 * unless under significant pressure.
+			 * run into the visited page again: we are scanning
+			 * faster than the flusher can writeout dirty pages
 			 */
-			if (page_is_file_cache(page) &&
-					(!current_is_kswapd() || priority >= DEF_PRIORITY - 2)) {
+			if (page_is_file_cache(page) && PageReclaim(page)) {
+				nr_pgreclaim++;
+				goto keep_locked;
+			}
+			if (page_is_file_cache(page) && mapping &&
+			    flush_inode_page(mapping, page, false) >= 0) {
 				/*
 				 * Immediately reclaim when written back.
 				 * Similar in principal to deactivate_page()
@@ -1028,6 +1037,7 @@ keep_lumpy:
 	count_vm_events(PGACTIVATE, pgactivate);
 	*ret_nr_dirty += nr_dirty;
 	*ret_nr_writeback += nr_writeback;
+	*ret_nr_pgreclaim += nr_pgreclaim;
 	return nr_reclaimed;
 }
 
@@ -1087,8 +1097,10 @@ int __isolate_lru_page(struct page *page
 	 */
 	if (mode & (ISOLATE_CLEAN|ISOLATE_ASYNC_MIGRATE)) {
 		/* All the caller can do on PageWriteback is block */
-		if (PageWriteback(page))
+		if (PageWriteback(page)) {
+			SetPageReclaim(page);
 			return ret;
+		}
 
 		if (PageDirty(page)) {
 			struct address_space *mapping;
@@ -1509,6 +1521,7 @@ shrink_inactive_list(unsigned long nr_to
 	unsigned long nr_file;
 	unsigned long nr_dirty = 0;
 	unsigned long nr_writeback = 0;
+	unsigned long nr_pgreclaim = 0;
 	isolate_mode_t reclaim_mode = ISOLATE_INACTIVE;
 	struct zone *zone = mz->zone;
 
@@ -1559,13 +1572,13 @@ shrink_inactive_list(unsigned long nr_to
 	spin_unlock_irq(&zone->lru_lock);
 
 	nr_reclaimed = shrink_page_list(&page_list, mz, sc, priority,
-						&nr_dirty, &nr_writeback);
+				&nr_dirty, &nr_writeback, &nr_pgreclaim);
 
 	/* Check if we should syncronously wait for writeback */
 	if (should_reclaim_stall(nr_taken, nr_reclaimed, priority, sc)) {
 		set_reclaim_mode(priority, sc, true);
 		nr_reclaimed += shrink_page_list(&page_list, mz, sc,
-					priority, &nr_dirty, &nr_writeback);
+			priority, &nr_dirty, &nr_writeback, &nr_pgreclaim);
 	}
 
 	spin_lock_irq(&zone->lru_lock);
@@ -1608,6 +1621,8 @@ shrink_inactive_list(unsigned long nr_to
 	 */
 	if (nr_writeback && nr_writeback >= (nr_taken >> (DEF_PRIORITY-priority)))
 		wait_iff_congested(zone, BLK_RW_ASYNC, HZ/10);
+	if (nr_pgreclaim && nr_pgreclaim >= (nr_taken >> (DEF_PRIORITY-priority)))
+		reclaim_wait(HZ/10);
 
 	trace_mm_vmscan_lru_shrink_inactive(zone->zone_pgdat->node_id,
 		zone_idx(zone),
@@ -2382,8 +2397,6 @@ static unsigned long do_try_to_free_page
 		 */
 		writeback_threshold = sc->nr_to_reclaim + sc->nr_to_reclaim / 2;
 		if (total_scanned > writeback_threshold) {
-			wakeup_flusher_threads(laptop_mode ? 0 : total_scanned,
-						WB_REASON_TRY_TO_FREE_PAGES);
 			sc->may_writepage = 1;
 		}
 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: reclaim the LRU lists full of dirty/writeback pages

[Wu Fengguang]

[-- Attachment #1: Type: text/plain, Size: 432 bytes --]

> For the single dd inside memcg, ext4 is now working pretty well, with
> least CPU overheads:

Attached are the dd progresses for btrfs, ext4 and xfs in turn, with
btrfs performing the most fluent and fast.

Also attached the "10 memcg each running 1 dd" graph for btrfs. Well
it's unacceptably bumpy because there are no any efforts to make a
round-robin walk between the works from different memcg LRU lists.

Thanks,
Fengguang


[-- Attachment #2: balance_dirty_pages-task-bw.png --]
[-- Type: image/png, Size: 36661 bytes --]

[-- Attachment #3: balance_dirty_pages-task-bw.png --]
[-- Type: image/png, Size: 37858 bytes --]

[-- Attachment #4: balance_dirty_pages-task-bw.png --]
[-- Type: image/png, Size: 38995 bytes --]

[-- Attachment #5: balance_dirty_pages-task-bw.png --]
[-- Type: image/png, Size: 45482 bytes --]

Re: reclaim the LRU lists full of dirty/writeback pages

[Mel Gorman]

On Tue, Feb 14, 2012 at 09:18:12PM +0800, Wu Fengguang wrote:
> > For the OOM problem, a more reasonable stopgap might be to identify when
> > a process is scanning a memcg at high priority and encountered all
> > PageReclaim with no forward progress and to congestion_wait() if that
> > situation occurs. A preferable way would be to wait until the flusher
> > wakes up a waiter on PageReclaim pages to be written out because we want
> > to keep moving way from congestion_wait() if at all possible.
> 
> Good points! Below is the more serious page reclaim changes.
> 
> The dirty/writeback pages may often come close to each other in the
> LRU list, so the local test during a 32-page scan may still trigger
> reclaim waits unnecessarily.

Yes, this is particularly the case when writing back to USB. It is not
unusual that all dirty pages under writeback are backed by USB and at the
end of the LRU. Right now what happens is that reclaimers see higher CPU
usage as they scan over these pages uselessly. If the wrong choice is
made on how to throttle, we'll see yet more variants of the "system
responsiveness drops when writing to USB".

> Some global information on the percent
> of dirty/writeback pages in the LRU list may help. Anyway the added
> tests should still be much better than no protection.
> 

You can tell how many dirty pages and writeback pages are in the zone
already.

> A global wait queue and reclaim_wait() is introduced. The waiters will
> be wakeup when pages are rotated by end_page_writeback() or lru drain.
> 
> I have to say its effectiveness depends on the filesystem... ext4
> and btrfs do fluent IO completions, so reclaim_wait() works pretty
> well:
>               dd-14560 [017] ....  1360.894605: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=10000
>               dd-14560 [017] ....  1360.904456: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=8000
>               dd-14560 [017] ....  1360.908293: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
>               dd-14560 [017] ....  1360.923960: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=15000
>               dd-14560 [017] ....  1360.927810: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
>               dd-14560 [017] ....  1360.931656: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
>               dd-14560 [017] ....  1360.943503: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=10000
>               dd-14560 [017] ....  1360.953289: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=7000
>               dd-14560 [017] ....  1360.957177: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
>               dd-14560 [017] ....  1360.972949: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=15000
> 
> However XFS does IO completions in very large batches (there may be
> only several big IO completions in one second). So reclaim_wait()
> mostly end up waiting to the full HZ/10 timeout:
> 
>               dd-4177  [008] ....   866.367661: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
>               dd-4177  [010] ....   866.567583: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
>               dd-4177  [012] ....   866.767458: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
>               dd-4177  [013] ....   866.867419: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
>               dd-4177  [008] ....   867.167266: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
>               dd-4177  [010] ....   867.367168: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
>               dd-4177  [012] ....   867.818950: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
>               dd-4177  [013] ....   867.918905: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
>               dd-4177  [013] ....   867.971657: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=52000
>               dd-4177  [013] ....   867.971812: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=0
>               dd-4177  [008] ....   868.355700: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
>               dd-4177  [010] ....   868.700515: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> 

And where people will get hit by regressions in this area is writing to
vfat and in more rare cases ntfs on USB stick.

> > Another possibility would be to relook at LRU_IMMEDIATE but right now it
> > requires a page flag and I haven't devised a way around that. Besides,
> > it would only address the problem of PageREclaim pages being encountered,
> > it would not handle the case where a memcg was filled with PageReclaim pages.
> 
> I also considered things like LRU_IMMEDIATE, however got no clear idea yet.
> Since the simple "wait on PG_reclaim" approach appears to work for this
> memcg dd case, it effectively disables me to think any further ;-)
> 

Test with interactive use while writing heavily to a USB stick.

> For the single dd inside memcg, ext4 is now working pretty well, with
> least CPU overheads:
> 
> (running from another test box, so not directly comparable with old tests)
> 
>         avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>                    0.03    0.00    0.85    5.35    0.00   93.77
> 
>         Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
>         sda               0.00     0.00    0.00  112.00     0.00 57348.00  1024.07    81.66 1045.21   8.93 100.00
> 
>         avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>                    0.00    0.00    0.69    4.07    0.00   95.24
> 
>         Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
>         sda               0.00   142.00    0.00  112.00     0.00 56832.00  1014.86   127.94  790.04   8.93 100.00
> 
> And xfs a bit less fluent:
> 
>         avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>                    0.00    0.00    3.79    2.54    0.00   93.68
> 
>         Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
>         sda               0.00     0.00    0.00  108.00     0.00 54644.00  1011.93    48.13 1044.83   8.44  91.20
> 
>         avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>                    0.00    0.00    3.38    3.88    0.00   92.74
> 
>         Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
>         sda               0.00     0.00    0.00  105.00     0.00 53156.00  1012.50   128.50  451.90   9.25  97.10
> 
> btrfs also looks good:
> 
>         avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>                    0.00    0.00    8.05    3.85    0.00   88.10
> 
>         Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
>         sda               0.00     0.00    0.00  108.00     0.00 53248.00   986.07    88.11  643.99   9.26 100.00
> 
>         avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>                    0.00    0.00    4.04    2.51    0.00   93.45
> 
>         Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
>         sda               0.00     0.00    0.00  112.00     0.00 57344.00  1024.00    91.58  998.41   8.93 100.00
> 
> ---
> 
> --- linux.orig/include/linux/backing-dev.h	2012-02-14 19:43:06.000000000 +0800
> +++ linux/include/linux/backing-dev.h	2012-02-14 19:49:26.000000000 +0800
> @@ -304,6 +304,8 @@ void clear_bdi_congested(struct backing_
>  void set_bdi_congested(struct backing_dev_info *bdi, int sync);
>  long congestion_wait(int sync, long timeout);
>  long wait_iff_congested(struct zone *zone, int sync, long timeout);
> +long reclaim_wait(long timeout);
> +void reclaim_rotated(void);
>  
>  static inline bool bdi_cap_writeback_dirty(struct backing_dev_info *bdi)
>  {
> --- linux.orig/mm/backing-dev.c	2012-02-14 19:26:15.000000000 +0800
> +++ linux/mm/backing-dev.c	2012-02-14 20:09:45.000000000 +0800
> @@ -873,3 +873,38 @@ out:
>  	return ret;
>  }
>  EXPORT_SYMBOL(wait_iff_congested);
> +
> +static DECLARE_WAIT_QUEUE_HEAD(reclaim_wqh);
> +
> +/**
> + * reclaim_wait - wait for some pages being rotated to the LRU tail
> + * @timeout: timeout in jiffies
> + *
> + * Wait until @timeout, or when some (typically PG_reclaim under writeback)
> + * pages rotated to the LRU so that page reclaim can make progress.
> + */
> +long reclaim_wait(long timeout)
> +{
> +	long ret;
> +	unsigned long start = jiffies;
> +	DEFINE_WAIT(wait);
> +
> +	prepare_to_wait(&reclaim_wqh, &wait, TASK_KILLABLE);
> +	ret = io_schedule_timeout(timeout);
> +	finish_wait(&reclaim_wqh, &wait);
> +
> +	trace_writeback_reclaim_wait(jiffies_to_usecs(timeout),
> +				     jiffies_to_usecs(jiffies - start));
> +
> +	return ret;
> +}
> +EXPORT_SYMBOL(reclaim_wait);
> +
> +void reclaim_rotated()
> +{
> +	wait_queue_head_t *wqh = &reclaim_wqh;
> +
> +	if (waitqueue_active(wqh))
> +		wake_up(wqh);
> +}
> +
> --- linux.orig/mm/swap.c	2012-02-14 19:40:10.000000000 +0800
> +++ linux/mm/swap.c	2012-02-14 19:45:13.000000000 +0800
> @@ -253,6 +253,7 @@ static void pagevec_move_tail(struct pag
>  
>  	pagevec_lru_move_fn(pvec, pagevec_move_tail_fn, &pgmoved);
>  	__count_vm_events(PGROTATED, pgmoved);
> +	reclaim_rotated();
>  }
>  
>  /*
> --- linux.orig/mm/vmscan.c	2012-02-14 17:53:27.000000000 +0800
> +++ linux/mm/vmscan.c	2012-02-14 19:44:11.000000000 +0800
> @@ -767,7 +767,8 @@ static unsigned long shrink_page_list(st
>  				      struct scan_control *sc,
>  				      int priority,
>  				      unsigned long *ret_nr_dirty,
> -				      unsigned long *ret_nr_writeback)
> +				      unsigned long *ret_nr_writeback,
> +				      unsigned long *ret_nr_pgreclaim)
>  {
>  	LIST_HEAD(ret_pages);
>  	LIST_HEAD(free_pages);
> @@ -776,6 +777,7 @@ static unsigned long shrink_page_list(st
>  	unsigned long nr_congested = 0;
>  	unsigned long nr_reclaimed = 0;
>  	unsigned long nr_writeback = 0;
> +	unsigned long nr_pgreclaim = 0;
>  
>  	cond_resched();
>  
> @@ -813,6 +815,10 @@ static unsigned long shrink_page_list(st
>  
>  		if (PageWriteback(page)) {
>  			nr_writeback++;
> +			if (PageReclaim(page))
> +				nr_pgreclaim++;
> +			else
> +				SetPageReclaim(page);
>  			/*

This check is unexpected. We already SetPageReclaim when queuing pages for
IO from reclaim context and if dirty pages are encountered during the LRU
scan that cannot be queued for IO. How often is it that nr_pgreclaim !=
nr_writeback and by how much do they differ?

>  			 * Synchronous reclaim cannot queue pages for
>  			 * writeback due to the possibility of stack overflow
> @@ -874,12 +880,15 @@ static unsigned long shrink_page_list(st
>  			nr_dirty++;
>  
>  			/*
> -			 * Only kswapd can writeback filesystem pages to
> -			 * avoid risk of stack overflow but do not writeback
> -			 * unless under significant pressure.
> +			 * run into the visited page again: we are scanning
> +			 * faster than the flusher can writeout dirty pages
>  			 */

which in itself is not an abnormal condition. We get into this situation
when writing to USB. Dirty throttling stops too much memory getting dirtied
but that does not mean we should throttle instead of reclaiming clean pages.

That's why I worry that if this is aimed at fixing a memcg problem, it
will have the impact of making interactive performance on normal systems
worse.

> -			if (page_is_file_cache(page) &&
> -					(!current_is_kswapd() || priority >= DEF_PRIORITY - 2)) {
> +			if (page_is_file_cache(page) && PageReclaim(page)) {
> +				nr_pgreclaim++;
> +				goto keep_locked;
> +			}
> +			if (page_is_file_cache(page) && mapping &&
> +			    flush_inode_page(mapping, page, false) >= 0) {
>  				/*
>  				 * Immediately reclaim when written back.
>  				 * Similar in principal to deactivate_page()
> @@ -1028,6 +1037,7 @@ keep_lumpy:
>  	count_vm_events(PGACTIVATE, pgactivate);
>  	*ret_nr_dirty += nr_dirty;
>  	*ret_nr_writeback += nr_writeback;
> +	*ret_nr_pgreclaim += nr_pgreclaim;
>  	return nr_reclaimed;
>  }
>  
> @@ -1087,8 +1097,10 @@ int __isolate_lru_page(struct page *page
>  	 */
>  	if (mode & (ISOLATE_CLEAN|ISOLATE_ASYNC_MIGRATE)) {
>  		/* All the caller can do on PageWriteback is block */
> -		if (PageWriteback(page))
> +		if (PageWriteback(page)) {
> +			SetPageReclaim(page);
>  			return ret;
> +		}
>  

This hunk means that if async compaction (common for THP) encounters a page
under writeback, it will still skip it but mark it for immediate reclaim
after IO completes. This will have the impact that compaction causes an
abnormally high number of pages to be reclaimed.

>  		if (PageDirty(page)) {
>  			struct address_space *mapping;
> @@ -1509,6 +1521,7 @@ shrink_inactive_list(unsigned long nr_to
>  	unsigned long nr_file;
>  	unsigned long nr_dirty = 0;
>  	unsigned long nr_writeback = 0;
> +	unsigned long nr_pgreclaim = 0;
>  	isolate_mode_t reclaim_mode = ISOLATE_INACTIVE;
>  	struct zone *zone = mz->zone;
>  
> @@ -1559,13 +1572,13 @@ shrink_inactive_list(unsigned long nr_to
>  	spin_unlock_irq(&zone->lru_lock);
>  
>  	nr_reclaimed = shrink_page_list(&page_list, mz, sc, priority,
> -						&nr_dirty, &nr_writeback);
> +				&nr_dirty, &nr_writeback, &nr_pgreclaim);
>  
>  	/* Check if we should syncronously wait for writeback */
>  	if (should_reclaim_stall(nr_taken, nr_reclaimed, priority, sc)) {
>  		set_reclaim_mode(priority, sc, true);
>  		nr_reclaimed += shrink_page_list(&page_list, mz, sc,
> -					priority, &nr_dirty, &nr_writeback);
> +			priority, &nr_dirty, &nr_writeback, &nr_pgreclaim);
>  	}
>  
>  	spin_lock_irq(&zone->lru_lock);
> @@ -1608,6 +1621,8 @@ shrink_inactive_list(unsigned long nr_to
>  	 */
>  	if (nr_writeback && nr_writeback >= (nr_taken >> (DEF_PRIORITY-priority)))
>  		wait_iff_congested(zone, BLK_RW_ASYNC, HZ/10);
> +	if (nr_pgreclaim && nr_pgreclaim >= (nr_taken >> (DEF_PRIORITY-priority)))
> +		reclaim_wait(HZ/10);
>  

We risk going to sleep too easily when USB-backed pages are at the end of
the LRU list. Note that the nr_writeback check only goes to sleep if it
detects that the underlying storage is also congested. In contrast, it
will take very few PageReclaim pages at teh end of the LRU to cause the
process to sleep when it instead should find clean pages to discard.

If the intention is to avoid memcg going OOM prematurely, the
nr_pgreclaim value needs to be treated at a higher level that records
how many PageReclaim pages were encountered. If no progress was made
because all the pages were PageReclaim, then throttle and return 1 to
the page allocator where it will retry the allocation without going OOM
after some pages have been cleaned and reclaimed.

>  	trace_mm_vmscan_lru_shrink_inactive(zone->zone_pgdat->node_id,
>  		zone_idx(zone),
> @@ -2382,8 +2397,6 @@ static unsigned long do_try_to_free_page
>  		 */
>  		writeback_threshold = sc->nr_to_reclaim + sc->nr_to_reclaim / 2;
>  		if (total_scanned > writeback_threshold) {
> -			wakeup_flusher_threads(laptop_mode ? 0 : total_scanned,
> -						WB_REASON_TRY_TO_FREE_PAGES);
>  			sc->may_writepage = 1;
>  		}
>  

-- 
Mel Gorman
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: reclaim the LRU lists full of dirty/writeback pages

[Wu Fengguang]

On Tue, Feb 14, 2012 at 03:51:24PM +0000, Mel Gorman wrote:
> On Tue, Feb 14, 2012 at 09:18:12PM +0800, Wu Fengguang wrote:
> > > For the OOM problem, a more reasonable stopgap might be to identify when
> > > a process is scanning a memcg at high priority and encountered all
> > > PageReclaim with no forward progress and to congestion_wait() if that
> > > situation occurs. A preferable way would be to wait until the flusher
> > > wakes up a waiter on PageReclaim pages to be written out because we want
> > > to keep moving way from congestion_wait() if at all possible.
> > 
> > Good points! Below is the more serious page reclaim changes.
> > 
> > The dirty/writeback pages may often come close to each other in the
> > LRU list, so the local test during a 32-page scan may still trigger
> > reclaim waits unnecessarily.
> 
> Yes, this is particularly the case when writing back to USB. It is not
> unusual that all dirty pages under writeback are backed by USB and at the
> end of the LRU. Right now what happens is that reclaimers see higher CPU
> usage as they scan over these pages uselessly. If the wrong choice is
> made on how to throttle, we'll see yet more variants of the "system
> responsiveness drops when writing to USB".

Yes, USB is an important case to support.  I'd imagine the heavy USB
writes typically happen in desktops and run *outside* of any memcg.
So they'll typically take <= 20% memory in the zone. As long as we
start the PG_reclaim throttling only when above the 20% dirty
threshold (ie. on zone_dirty_ok()), the USB case should be safe.

> > Some global information on the percent
> > of dirty/writeback pages in the LRU list may help. Anyway the added
> > tests should still be much better than no protection.
> > 
> 
> You can tell how many dirty pages and writeback pages are in the zone
> already.

Right. I changed the test to

+       if (nr_pgreclaim && nr_pgreclaim >= (nr_taken >> (DEF_PRIORITY-priority)) &&
+           (!global_reclaim(sc) || !zone_dirty_ok(zone)))
+               reclaim_wait(HZ/10);

And I'd prefer to use a higher threshold than the default 20% for the
above zone_dirty_ok() test, so that when Johannes' zone dirty
balancing does the job fine, PG_reclaim based page reclaim throttling
won't happen at all.

> > A global wait queue and reclaim_wait() is introduced. The waiters will
> > be wakeup when pages are rotated by end_page_writeback() or lru drain.
> > 
> > I have to say its effectiveness depends on the filesystem... ext4
> > and btrfs do fluent IO completions, so reclaim_wait() works pretty
> > well:
> >               dd-14560 [017] ....  1360.894605: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=10000
> >               dd-14560 [017] ....  1360.904456: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=8000
> >               dd-14560 [017] ....  1360.908293: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
> >               dd-14560 [017] ....  1360.923960: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=15000
> >               dd-14560 [017] ....  1360.927810: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
> >               dd-14560 [017] ....  1360.931656: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
> >               dd-14560 [017] ....  1360.943503: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=10000
> >               dd-14560 [017] ....  1360.953289: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=7000
> >               dd-14560 [017] ....  1360.957177: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
> >               dd-14560 [017] ....  1360.972949: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=15000
> > 
> > However XFS does IO completions in very large batches (there may be
> > only several big IO completions in one second). So reclaim_wait()
> > mostly end up waiting to the full HZ/10 timeout:
> > 
> >               dd-4177  [008] ....   866.367661: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> >               dd-4177  [010] ....   866.567583: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> >               dd-4177  [012] ....   866.767458: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> >               dd-4177  [013] ....   866.867419: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> >               dd-4177  [008] ....   867.167266: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> >               dd-4177  [010] ....   867.367168: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> >               dd-4177  [012] ....   867.818950: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> >               dd-4177  [013] ....   867.918905: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> >               dd-4177  [013] ....   867.971657: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=52000
> >               dd-4177  [013] ....   867.971812: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=0
> >               dd-4177  [008] ....   868.355700: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> >               dd-4177  [010] ....   868.700515: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > 
> 
> And where people will get hit by regressions in this area is writing to
> vfat and in more rare cases ntfs on USB stick.

vfat IO completions seem to lie somewhere between ext4 and xfs:

           <...>-46385 [010] .... 143570.714470: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
           <...>-46385 [008] .... 143570.752391: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=12000
           <...>-46385 [008] .... 143570.937327: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=52000
           <...>-46385 [010] .... 143571.160252: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
           <...>-46385 [011] .... 143571.286197: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
           <...>-46385 [008] .... 143571.329644: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=15000
           <...>-46385 [008] .... 143571.475433: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=18000
           <...>-46385 [008] .... 143571.653461: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=52000
           <...>-46385 [008] .... 143571.839949: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=56000
           <...>-46385 [010] .... 143572.060816: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
           <...>-46385 [011] .... 143572.185754: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
           <...>-46385 [008] .... 143572.212522: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=1000
           <...>-46385 [008] .... 143572.217825: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
           <...>-46385 [008] .... 143572.312395: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=91000
           <...>-46385 [008] .... 143572.315122: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=1000
           <...>-46385 [009] .... 143572.433630: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
           <...>-46385 [010] .... 143572.534569: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
 
and has lower throughput

        avg-cpu:  %user   %nice %system %iowait  %steal   %idle
                   0.03    0.00    3.88    2.22    0.00   93.86

        Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
        sda               0.00    34.00   20.00   82.00    10.00 34137.50   669.56     8.09   79.34   5.97  60.90

I'm yet to get a USB stick for the vfat-on-USB test.

> > > Another possibility would be to relook at LRU_IMMEDIATE but right now it
> > > requires a page flag and I haven't devised a way around that. Besides,
> > > it would only address the problem of PageREclaim pages being encountered,
> > > it would not handle the case where a memcg was filled with PageReclaim pages.
> > 
> > I also considered things like LRU_IMMEDIATE, however got no clear idea yet.
> > Since the simple "wait on PG_reclaim" approach appears to work for this
> > memcg dd case, it effectively disables me to think any further ;-)
> > 
> 
> Test with interactive use while writing heavily to a USB stick.

Sure.

> > @@ -813,6 +815,10 @@ static unsigned long shrink_page_list(st
> >  
> >  		if (PageWriteback(page)) {
> >  			nr_writeback++;
> > +			if (PageReclaim(page))
> > +				nr_pgreclaim++;
> > +			else
> > +				SetPageReclaim(page);
> >  			/*
> 
> This check is unexpected. We already SetPageReclaim when queuing pages for
> IO from reclaim context and if dirty pages are encountered during the LRU
> scan that cannot be queued for IO. How often is it that nr_pgreclaim !=
> nr_writeback and by how much do they differ?

Quite often, I suspect. The pageout writeback works do 1-8MB write
around which may start I/O a bit earlier than the covered pages are
encountered by page reclaim. ext4 forces 128MB write chunk size, which
further increases the opportunities.

> >  			 * Synchronous reclaim cannot queue pages for
> >  			 * writeback due to the possibility of stack overflow
> > @@ -874,12 +880,15 @@ static unsigned long shrink_page_list(st
> >  			nr_dirty++;
> >  
> >  			/*
> > -			 * Only kswapd can writeback filesystem pages to
> > -			 * avoid risk of stack overflow but do not writeback
> > -			 * unless under significant pressure.
> > +			 * run into the visited page again: we are scanning
> > +			 * faster than the flusher can writeout dirty pages
> >  			 */
> 
> which in itself is not an abnormal condition. We get into this situation
> when writing to USB. Dirty throttling stops too much memory getting dirtied
> but that does not mean we should throttle instead of reclaiming clean pages.
> 
> That's why I worry that if this is aimed at fixing a memcg problem, it
> will have the impact of making interactive performance on normal systems
> worse.

You are right. This patch only addresses the pageout I/O efficiency
and dirty throttling problems for a fully dirtied LRU. Next step, I'll
think about the interactive performance problem for a less dirtied LRU.

> > @@ -1087,8 +1097,10 @@ int __isolate_lru_page(struct page *page
> >  	 */
> >  	if (mode & (ISOLATE_CLEAN|ISOLATE_ASYNC_MIGRATE)) {
> >  		/* All the caller can do on PageWriteback is block */
> > -		if (PageWriteback(page))
> > +		if (PageWriteback(page)) {
> > +			SetPageReclaim(page);
> >  			return ret;
> > +		}
> >  
> 
> This hunk means that if async compaction (common for THP) encounters a page
> under writeback, it will still skip it but mark it for immediate reclaim
> after IO completes. This will have the impact that compaction causes an
> abnormally high number of pages to be reclaimed.

Sorry I overlooked that, will drop it.  isolate_migratepages() walks
by PFN and the opportunistic peak at the writeback pages should not
make it rotated (and disturb its LRU order) on I/O completion.

> > @@ -1608,6 +1621,8 @@ shrink_inactive_list(unsigned long nr_to
> >  	 */
> >  	if (nr_writeback && nr_writeback >= (nr_taken >> (DEF_PRIORITY-priority)))
> >  		wait_iff_congested(zone, BLK_RW_ASYNC, HZ/10);
> > +	if (nr_pgreclaim && nr_pgreclaim >= (nr_taken >> (DEF_PRIORITY-priority)))
> > +		reclaim_wait(HZ/10);
> >  
> 
> We risk going to sleep too easily when USB-backed pages are at the end of
> the LRU list. Note that the nr_writeback check only goes to sleep if it
> detects that the underlying storage is also congested. In contrast, it
> will take very few PageReclaim pages at teh end of the LRU to cause the
> process to sleep when it instead should find clean pages to discard.

Right.

> If the intention is to avoid memcg going OOM prematurely, the
> nr_pgreclaim value needs to be treated at a higher level that records
> how many PageReclaim pages were encountered. If no progress was made
> because all the pages were PageReclaim, then throttle and return 1 to
> the page allocator where it will retry the allocation without going OOM
> after some pages have been cleaned and reclaimed.
 
Agreed in general, but changed to this test for now, which is made a
bit more global wise with the use of zone_dirty_ok().

memcg is ignored due to no dirty accounting (Greg has the patch though).
And even zone_dirty_ok() may be inaccurate for the global reclaim, if
some memcgs are skipped by the global reclaim by the memcg soft limit.

But anyway, it's a handy hack for now. I'm looking into some more
radical changes to put most dirty/writeback pages into a standalone
LRU list (in addition to your LRU_IMMEDIATE, which I think is a good
idea) for addressing the clustered way they tend to lie in the
inactive LRU list.

+       if (nr_pgreclaim && nr_pgreclaim >= (nr_taken >> (DEF_PRIORITY-priority)) &&
+           (!global_reclaim(sc) || !zone_dirty_ok(zone)))
+               reclaim_wait(HZ/10);

Thanks,
Fengguang
---
Subject: writeback: introduce the pageout work
Date: Thu Jul 29 14:41:19 CST 2010

This relays file pageout IOs to the flusher threads.

The ultimate target is to gracefully handle the LRU lists full of
dirty/writeback pages.

1) I/O efficiency

The flusher will piggy back the nearby ~10ms worth of dirty pages for I/O.

This takes advantage of the time/spacial locality in most workloads: the
nearby pages of one file are typically populated into the LRU at the same
time, hence will likely be close to each other in the LRU list. Writing
them in one shot helps clean more pages effectively for page reclaim.

2) OOM avoidance and scan rate control

Typically we do LRU scan w/o rate control and quickly get enough clean
pages for the LRU lists not full of dirty pages.

Or we can still get a number of freshly cleaned pages (moved to LRU tail
by end_page_writeback()) when the queued pageout I/O is completed within
tens of milli-seconds.

However if the LRU list is small and full of dirty pages, it can be
quickly fully scanned and go OOM before the flusher manages to clean
enough pages.

A simple yet reliable scheme is employed to avoid OOM and keep scan rate
in sync with the I/O rate:

	if (PageReclaim(page))
		congestion_wait(HZ/10);

PG_reclaim plays the key role. When dirty pages are encountered, we
queue I/O for it, set PG_reclaim and put it back to the LRU head.
So if PG_reclaim pages are encountered again, it means the dirty page
has not yet been cleaned by the flusher after a full zone scan. It
indicates we are scanning more fast than I/O and shall take a snap.

The runtime behavior on a fully dirtied small LRU list would be:
It will start with a quick scan of the list, queuing all pages for I/O.
Then the scan will be slowed down by the PG_reclaim pages *adaptively*
to match the I/O bandwidth.

3) writeback work coordinations

To avoid memory allocations at page reclaim, a mempool for struct
wb_writeback_work is created.

wakeup_flusher_threads() is removed because it can easily delay the
more oriented pageout works and even exhaust the mempool reservations.
It's also found to not I/O efficient by frequently submitting writeback
works with small ->nr_pages.

Background/periodic works will quit automatically, so as to clean the
pages under reclaim ASAP. However for now the sync work can still block
us for long time.

Jan Kara: limit the search scope. Note that the limited search and work
pool is not a big problem: 1000 IOs under flight are typically more than
enough to saturate the disk. And the overheads of searching in the work
list didn't even show up in the perf report.

4) test case

Run 2 dd tasks in a 100MB memcg (a very handy test case from Greg Thelen):

	mkdir /cgroup/x
	echo 100M > /cgroup/x/memory.limit_in_bytes
	echo $$ > /cgroup/x/tasks

	for i in `seq 2`
	do
		dd if=/dev/zero of=/fs/f$i bs=1k count=1M &
	done

Before patch, the dd tasks are quickly OOM killed.
After patch, they run well with reasonably good performance and overheads:

1073741824 bytes (1.1 GB) copied, 22.2196 s, 48.3 MB/s
1073741824 bytes (1.1 GB) copied, 22.4675 s, 47.8 MB/s

iostat -kx 1

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00    0.00  178.00     0.00 89568.00  1006.38    74.35  417.71   4.80  85.40
sda               0.00     2.00    0.00  191.00     0.00 94428.00   988.77    53.34  219.03   4.34  82.90
sda               0.00    20.00    0.00  196.00     0.00 97712.00   997.06    71.11  337.45   4.77  93.50
sda               0.00     5.00    0.00  175.00     0.00 84648.00   967.41    54.03  316.44   5.06  88.60
sda               0.00     0.00    0.00  186.00     0.00 92432.00   993.89    56.22  267.54   5.38 100.00
sda               0.00     1.00    0.00  183.00     0.00 90156.00   985.31    37.99  325.55   4.33  79.20
sda               0.00     0.00    0.00  175.00     0.00 88692.00  1013.62    48.70  218.43   4.69  82.10
sda               0.00     0.00    0.00  196.00     0.00 97528.00   995.18    43.38  236.87   5.10 100.00
sda               0.00     0.00    0.00  179.00     0.00 88648.00   990.48    45.83  285.43   5.59 100.00
sda               0.00     0.00    0.00  178.00     0.00 88500.00   994.38    28.28  158.89   4.99  88.80
sda               0.00     0.00    0.00  194.00     0.00 95852.00   988.16    32.58  167.39   5.15 100.00
sda               0.00     2.00    0.00  215.00     0.00 105996.00   986.01    41.72  201.43   4.65 100.00
sda               0.00     4.00    0.00  173.00     0.00 84332.00   974.94    50.48  260.23   5.76  99.60
sda               0.00     0.00    0.00  182.00     0.00 90312.00   992.44    36.83  212.07   5.49 100.00
sda               0.00     8.00    0.00  195.00     0.00 95940.50   984.01    50.18  221.06   5.13 100.00
sda               0.00     1.00    0.00  220.00     0.00 108852.00   989.56    40.99  202.68   4.55 100.00
sda               0.00     2.00    0.00  161.00     0.00 80384.00   998.56    37.19  268.49   6.21 100.00
sda               0.00     4.00    0.00  182.00     0.00 90830.00   998.13    50.58  239.77   5.49 100.00
sda               0.00     0.00    0.00  197.00     0.00 94877.00   963.22    36.68  196.79   5.08 100.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.25    0.00   15.08   33.92    0.00   50.75
           0.25    0.00   14.54   35.09    0.00   50.13
           0.50    0.00   13.57   32.41    0.00   53.52
           0.50    0.00   11.28   36.84    0.00   51.38
           0.50    0.00   15.75   32.00    0.00   51.75
           0.50    0.00   10.50   34.00    0.00   55.00
           0.50    0.00   17.63   27.46    0.00   54.41
           0.50    0.00   15.08   30.90    0.00   53.52
           0.50    0.00   11.28   32.83    0.00   55.39
           0.75    0.00   16.79   26.82    0.00   55.64
           0.50    0.00   16.08   29.15    0.00   54.27
           0.50    0.00   13.50   30.50    0.00   55.50
           0.50    0.00   14.32   35.18    0.00   50.00
           0.50    0.00   12.06   33.92    0.00   53.52
           0.50    0.00   17.29   30.58    0.00   51.63
           0.50    0.00   15.08   29.65    0.00   54.77
           0.50    0.00   12.53   29.32    0.00   57.64
           0.50    0.00   15.29   31.83    0.00   52.38

The global dd numbers for comparison:

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00    0.00  189.00     0.00 95752.00  1013.25   143.09  684.48   5.29 100.00
sda               0.00     0.00    0.00  208.00     0.00 105480.00  1014.23   143.06  733.29   4.81 100.00
sda               0.00     0.00    0.00  161.00     0.00 81924.00  1017.69   141.71  757.79   6.21 100.00
sda               0.00     0.00    0.00  217.00     0.00 109580.00  1009.95   143.09  749.55   4.61 100.10
sda               0.00     0.00    0.00  187.00     0.00 94728.00  1013.13   144.31  773.67   5.35 100.00
sda               0.00     0.00    0.00  189.00     0.00 95752.00  1013.25   144.14  742.00   5.29 100.00
sda               0.00     0.00    0.00  177.00     0.00 90032.00  1017.31   143.32  656.59   5.65 100.00
sda               0.00     0.00    0.00  215.00     0.00 108640.00  1010.60   142.90  817.54   4.65 100.00
sda               0.00     2.00    0.00  166.00     0.00 83858.00  1010.34   143.64  808.61   6.02 100.00
sda               0.00     0.00    0.00  186.00     0.00 92813.00   997.99   141.18  736.95   5.38 100.00
sda               0.00     0.00    0.00  206.00     0.00 104456.00  1014.14   146.27  729.33   4.85 100.00
sda               0.00     0.00    0.00  213.00     0.00 107024.00  1004.92   143.25  705.70   4.69 100.00
sda               0.00     0.00    0.00  188.00     0.00 95748.00  1018.60   141.82  764.78   5.32 100.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.51    0.00   11.22   52.30    0.00   35.97
           0.25    0.00   10.15   52.54    0.00   37.06
           0.25    0.00    5.01   56.64    0.00   38.10
           0.51    0.00   15.15   43.94    0.00   40.40
           0.25    0.00   12.12   48.23    0.00   39.39
           0.51    0.00   11.20   53.94    0.00   34.35
           0.26    0.00    9.72   51.41    0.00   38.62
           0.76    0.00    9.62   50.63    0.00   38.99
           0.51    0.00   10.46   53.32    0.00   35.71
           0.51    0.00    9.41   51.91    0.00   38.17
           0.25    0.00   10.69   49.62    0.00   39.44
           0.51    0.00   12.21   52.67    0.00   34.61
           0.51    0.00   11.45   53.18    0.00   34.86

XXX: commit NFS unstable pages via write_inode()
XXX: the added congestion_wait() may be undesirable in some situations

CC: Jan Kara <jack@suse.cz>
CC: Mel Gorman <mgorman@suse.de>
Acked-by: Rik van Riel <riel@redhat.com>
CC: Greg Thelen <gthelen@google.com>
CC: Minchan Kim <minchan.kim@gmail.com>
Signed-off-by: Wu Fengguang <fengguang.wu@intel.com>
---
 fs/fs-writeback.c                |  169 ++++++++++++++++++++++++++++-
 include/linux/backing-dev.h      |    2 
 include/linux/writeback.h        |    4 
 include/trace/events/writeback.h |   19 ++-
 mm/backing-dev.c                 |   35 ++++++
 mm/swap.c                        |    1 
 mm/vmscan.c                      |   32 +++--
 7 files changed, 245 insertions(+), 17 deletions(-)

- move congestion_wait() out of the page lock: it's blocking btrfs lock_delalloc_pages()

--- linux.orig/include/linux/backing-dev.h	2012-02-14 20:11:21.000000000 +0800
+++ linux/include/linux/backing-dev.h	2012-02-15 12:34:24.000000000 +0800
@@ -304,6 +304,8 @@ void clear_bdi_congested(struct backing_
 void set_bdi_congested(struct backing_dev_info *bdi, int sync);
 long congestion_wait(int sync, long timeout);
 long wait_iff_congested(struct zone *zone, int sync, long timeout);
+long reclaim_wait(long timeout);
+void reclaim_rotated(void);
 
 static inline bool bdi_cap_writeback_dirty(struct backing_dev_info *bdi)
 {
--- linux.orig/mm/backing-dev.c	2012-02-14 20:11:21.000000000 +0800
+++ linux/mm/backing-dev.c	2012-02-15 12:34:19.000000000 +0800
@@ -873,3 +873,38 @@ out:
 	return ret;
 }
 EXPORT_SYMBOL(wait_iff_congested);
+
+static DECLARE_WAIT_QUEUE_HEAD(reclaim_wqh);
+
+/**
+ * reclaim_wait - wait for some pages being rotated to the LRU tail
+ * @timeout: timeout in jiffies
+ *
+ * Wait until @timeout, or when some (typically PG_reclaim under writeback)
+ * pages rotated to the LRU so that page reclaim can make progress.
+ */
+long reclaim_wait(long timeout)
+{
+	long ret;
+	unsigned long start = jiffies;
+	DEFINE_WAIT(wait);
+
+	prepare_to_wait(&reclaim_wqh, &wait, TASK_KILLABLE);
+	ret = io_schedule_timeout(timeout);
+	finish_wait(&reclaim_wqh, &wait);
+
+	trace_writeback_reclaim_wait(jiffies_to_usecs(timeout),
+				     jiffies_to_usecs(jiffies - start));
+
+	return ret;
+}
+EXPORT_SYMBOL(reclaim_wait);
+
+void reclaim_rotated()
+{
+	wait_queue_head_t *wqh = &reclaim_wqh;
+
+	if (waitqueue_active(wqh))
+		wake_up(wqh);
+}
+
--- linux.orig/mm/swap.c	2012-02-14 20:11:21.000000000 +0800
+++ linux/mm/swap.c	2012-02-15 12:27:35.000000000 +0800
@@ -253,6 +253,7 @@ static void pagevec_move_tail(struct pag
 
 	pagevec_lru_move_fn(pvec, pagevec_move_tail_fn, &pgmoved);
 	__count_vm_events(PGROTATED, pgmoved);
+	reclaim_rotated();
 }
 
 /*
--- linux.orig/mm/vmscan.c	2012-02-14 20:11:21.000000000 +0800
+++ linux/mm/vmscan.c	2012-02-16 17:23:17.000000000 +0800
@@ -767,7 +767,8 @@ static unsigned long shrink_page_list(st
 				      struct scan_control *sc,
 				      int priority,
 				      unsigned long *ret_nr_dirty,
-				      unsigned long *ret_nr_writeback)
+				      unsigned long *ret_nr_writeback,
+				      unsigned long *ret_nr_pgreclaim)
 {
 	LIST_HEAD(ret_pages);
 	LIST_HEAD(free_pages);
@@ -776,6 +777,7 @@ static unsigned long shrink_page_list(st
 	unsigned long nr_congested = 0;
 	unsigned long nr_reclaimed = 0;
 	unsigned long nr_writeback = 0;
+	unsigned long nr_pgreclaim = 0;
 
 	cond_resched();
 
@@ -813,6 +815,10 @@ static unsigned long shrink_page_list(st
 
 		if (PageWriteback(page)) {
 			nr_writeback++;
+			if (PageReclaim(page))
+				nr_pgreclaim++;
+			else
+				SetPageReclaim(page);
 			/*
 			 * Synchronous reclaim cannot queue pages for
 			 * writeback due to the possibility of stack overflow
@@ -874,12 +880,15 @@ static unsigned long shrink_page_list(st
 			nr_dirty++;
 
 			/*
-			 * Only kswapd can writeback filesystem pages to
-			 * avoid risk of stack overflow but do not writeback
-			 * unless under significant pressure.
+			 * run into the visited page again: we are scanning
+			 * faster than the flusher can writeout dirty pages
 			 */
-			if (page_is_file_cache(page) &&
-					(!current_is_kswapd() || priority >= DEF_PRIORITY - 2)) {
+			if (page_is_file_cache(page) && PageReclaim(page)) {
+				nr_pgreclaim++;
+				goto keep_locked;
+			}
+			if (page_is_file_cache(page) && mapping &&
+			    flush_inode_page(mapping, page, false) >= 0) {
 				/*
 				 * Immediately reclaim when written back.
 				 * Similar in principal to deactivate_page()
@@ -1028,6 +1037,7 @@ keep_lumpy:
 	count_vm_events(PGACTIVATE, pgactivate);
 	*ret_nr_dirty += nr_dirty;
 	*ret_nr_writeback += nr_writeback;
+	*ret_nr_pgreclaim += nr_pgreclaim;
 	return nr_reclaimed;
 }
 
@@ -1509,6 +1519,7 @@ shrink_inactive_list(unsigned long nr_to
 	unsigned long nr_file;
 	unsigned long nr_dirty = 0;
 	unsigned long nr_writeback = 0;
+	unsigned long nr_pgreclaim = 0;
 	isolate_mode_t reclaim_mode = ISOLATE_INACTIVE;
 	struct zone *zone = mz->zone;
 
@@ -1559,13 +1570,13 @@ shrink_inactive_list(unsigned long nr_to
 	spin_unlock_irq(&zone->lru_lock);
 
 	nr_reclaimed = shrink_page_list(&page_list, mz, sc, priority,
-						&nr_dirty, &nr_writeback);
+				&nr_dirty, &nr_writeback, &nr_pgreclaim);
 
 	/* Check if we should syncronously wait for writeback */
 	if (should_reclaim_stall(nr_taken, nr_reclaimed, priority, sc)) {
 		set_reclaim_mode(priority, sc, true);
 		nr_reclaimed += shrink_page_list(&page_list, mz, sc,
-					priority, &nr_dirty, &nr_writeback);
+			priority, &nr_dirty, &nr_writeback, &nr_pgreclaim);
 	}
 
 	spin_lock_irq(&zone->lru_lock);
@@ -1608,6 +1619,9 @@ shrink_inactive_list(unsigned long nr_to
 	 */
 	if (nr_writeback && nr_writeback >= (nr_taken >> (DEF_PRIORITY-priority)))
 		wait_iff_congested(zone, BLK_RW_ASYNC, HZ/10);
+	if (nr_pgreclaim && nr_pgreclaim >= (nr_taken >> (DEF_PRIORITY-priority)) &&
+	    (!global_reclaim(sc) || !zone_dirty_ok(zone)))
+		reclaim_wait(HZ/10);
 
 	trace_mm_vmscan_lru_shrink_inactive(zone->zone_pgdat->node_id,
 		zone_idx(zone),
@@ -2382,8 +2396,6 @@ static unsigned long do_try_to_free_page
 		 */
 		writeback_threshold = sc->nr_to_reclaim + sc->nr_to_reclaim / 2;
 		if (total_scanned > writeback_threshold) {
-			wakeup_flusher_threads(laptop_mode ? 0 : total_scanned,
-						WB_REASON_TRY_TO_FREE_PAGES);
 			sc->may_writepage = 1;
 		}
 
--- linux.orig/fs/fs-writeback.c	2012-02-14 20:11:21.000000000 +0800
+++ linux/fs/fs-writeback.c	2012-02-15 12:27:35.000000000 +0800
@@ -41,6 +41,8 @@ struct wb_writeback_work {
 	long nr_pages;
 	struct super_block *sb;
 	unsigned long *older_than_this;
+	struct inode *inode;
+	pgoff_t offset;
 	enum writeback_sync_modes sync_mode;
 	unsigned int tagged_writepages:1;
 	unsigned int for_kupdate:1;
@@ -65,6 +67,27 @@ struct wb_writeback_work {
  */
 int nr_pdflush_threads;
 
+static mempool_t *wb_work_mempool;
+
+static void *wb_work_alloc(gfp_t gfp_mask, void *pool_data)
+{
+	/*
+	 * bdi_flush_inode_range() may be called on page reclaim
+	 */
+	if (current->flags & PF_MEMALLOC)
+		return NULL;
+
+	return kmalloc(sizeof(struct wb_writeback_work), gfp_mask);
+}
+
+static __init int wb_work_init(void)
+{
+	wb_work_mempool = mempool_create(1024,
+					 wb_work_alloc, mempool_kfree, NULL);
+	return wb_work_mempool ? 0 : -ENOMEM;
+}
+fs_initcall(wb_work_init);
+
 /**
  * writeback_in_progress - determine whether there is writeback in progress
  * @bdi: the device's backing_dev_info structure.
@@ -129,7 +152,7 @@ __bdi_start_writeback(struct backing_dev
 	 * This is WB_SYNC_NONE writeback, so if allocation fails just
 	 * wakeup the thread for old dirty data writeback
 	 */
-	work = kzalloc(sizeof(*work), GFP_ATOMIC);
+	work = mempool_alloc(wb_work_mempool, GFP_NOWAIT);
 	if (!work) {
 		if (bdi->wb.task) {
 			trace_writeback_nowork(bdi);
@@ -138,6 +161,7 @@ __bdi_start_writeback(struct backing_dev
 		return;
 	}
 
+	memset(work, 0, sizeof(*work));
 	work->sync_mode	= WB_SYNC_NONE;
 	work->nr_pages	= nr_pages;
 	work->range_cyclic = range_cyclic;
@@ -186,6 +210,125 @@ void bdi_start_background_writeback(stru
 	spin_unlock_bh(&bdi->wb_lock);
 }
 
+static bool extend_writeback_range(struct wb_writeback_work *work,
+				   pgoff_t offset,
+				   unsigned long write_around_pages)
+{
+	pgoff_t end = work->offset + work->nr_pages;
+
+	if (offset >= work->offset && offset < end)
+		return true;
+
+	/*
+	 * for sequential workloads with good locality, include up to 8 times
+	 * more data in one chunk
+	 */
+	if (work->nr_pages >= 8 * write_around_pages)
+		return false;
+
+	/* the unsigned comparison helps eliminate one compare */
+	if (work->offset - offset < write_around_pages) {
+		work->nr_pages += write_around_pages;
+		work->offset -= write_around_pages;
+		return true;
+	}
+
+	if (offset - end < write_around_pages) {
+		work->nr_pages += write_around_pages;
+		return true;
+	}
+
+	return false;
+}
+
+/*
+ * schedule writeback on a range of inode pages.
+ */
+static struct wb_writeback_work *
+bdi_flush_inode_range(struct backing_dev_info *bdi,
+		      struct inode *inode,
+		      pgoff_t offset,
+		      pgoff_t len,
+		      bool wait)
+{
+	struct wb_writeback_work *work;
+
+	if (!igrab(inode))
+		return ERR_PTR(-ENOENT);
+
+	work = mempool_alloc(wb_work_mempool, wait ? GFP_NOIO : GFP_NOWAIT);
+	if (!work) {
+		trace_printk("wb_work_mempool alloc fail\n");
+		return ERR_PTR(-ENOMEM);
+	}
+
+	memset(work, 0, sizeof(*work));
+	work->sync_mode		= WB_SYNC_NONE;
+	work->inode		= inode;
+	work->offset		= offset;
+	work->nr_pages		= len;
+	work->reason		= WB_REASON_PAGEOUT;
+
+	bdi_queue_work(bdi, work);
+
+	return work;
+}
+
+/*
+ * Called by page reclaim code to flush the dirty page ASAP. Do write-around to
+ * improve IO throughput. The nearby pages will have good chance to reside in
+ * the same LRU list that vmscan is working on, and even close to each other
+ * inside the LRU list in the common case of sequential read/write.
+ *
+ * ret > 0: success, found/reused a previous writeback work
+ * ret = 0: success, allocated/queued a new writeback work
+ * ret < 0: failed
+ */
+long flush_inode_page(struct address_space *mapping,
+		      struct page *page,
+		      bool wait)
+{
+	struct backing_dev_info *bdi = mapping->backing_dev_info;
+	struct inode *inode = mapping->host;
+	struct wb_writeback_work *work;
+	unsigned long write_around_pages;
+	pgoff_t offset = page->index;
+	int i;
+	long ret = 0;
+
+	if (unlikely(!inode))
+		return -ENOENT;
+
+	/*
+	 * piggy back 8-15ms worth of data
+	 */
+	write_around_pages = bdi->avg_write_bandwidth + MIN_WRITEBACK_PAGES;
+	write_around_pages = rounddown_pow_of_two(write_around_pages) >> 6;
+
+	i = 1;
+	spin_lock_bh(&bdi->wb_lock);
+	list_for_each_entry_reverse(work, &bdi->work_list, list) {
+		if (work->inode != inode)
+			continue;
+		if (extend_writeback_range(work, offset, write_around_pages)) {
+			ret = i;
+			break;
+		}
+		if (i++ > 100)	/* limit search depth */
+			break;
+	}
+	spin_unlock_bh(&bdi->wb_lock);
+
+	if (!ret) {
+		offset = round_down(offset, write_around_pages);
+		work = bdi_flush_inode_range(bdi, inode,
+					     offset, write_around_pages, wait);
+		if (IS_ERR(work))
+			ret = PTR_ERR(work);
+	}
+	return ret;
+}
+
 /*
  * Remove the inode from the writeback list it is on.
  */
@@ -833,6 +976,23 @@ static unsigned long get_nr_dirty_pages(
 		get_nr_dirty_inodes();
 }
 
+static long wb_flush_inode(struct bdi_writeback *wb,
+			   struct wb_writeback_work *work)
+{
+	struct writeback_control wbc = {
+		.sync_mode = WB_SYNC_NONE,
+		.nr_to_write = LONG_MAX,
+		.range_start = work->offset << PAGE_CACHE_SHIFT,
+		.range_end = (work->offset + work->nr_pages - 1)
+						<< PAGE_CACHE_SHIFT,
+	};
+
+	do_writepages(work->inode->i_mapping, &wbc);
+	iput(work->inode);
+
+	return LONG_MAX - wbc.nr_to_write;
+}
+
 static long wb_check_background_flush(struct bdi_writeback *wb)
 {
 	if (over_bground_thresh(wb->bdi)) {
@@ -905,7 +1065,10 @@ long wb_do_writeback(struct bdi_writebac
 
 		trace_writeback_exec(bdi, work);
 
-		wrote += wb_writeback(wb, work);
+		if (work->inode)
+			wrote += wb_flush_inode(wb, work);
+		else
+			wrote += wb_writeback(wb, work);
 
 		/*
 		 * Notify the caller of completion if this is a synchronous
@@ -914,7 +1077,7 @@ long wb_do_writeback(struct bdi_writebac
 		if (work->done)
 			complete(work->done);
 		else
-			kfree(work);
+			mempool_free(work, wb_work_mempool);
 	}
 
 	/*
--- linux.orig/include/trace/events/writeback.h	2012-02-14 20:11:22.000000000 +0800
+++ linux/include/trace/events/writeback.h	2012-02-15 12:27:35.000000000 +0800
@@ -23,7 +23,7 @@
 
 #define WB_WORK_REASON							\
 		{WB_REASON_BACKGROUND,		"background"},		\
-		{WB_REASON_TRY_TO_FREE_PAGES,	"try_to_free_pages"},	\
+		{WB_REASON_PAGEOUT,		"pageout"},		\
 		{WB_REASON_SYNC,		"sync"},		\
 		{WB_REASON_PERIODIC,		"periodic"},		\
 		{WB_REASON_LAPTOP_TIMER,	"laptop_timer"},	\
@@ -45,6 +45,8 @@ DECLARE_EVENT_CLASS(writeback_work_class
 		__field(int, range_cyclic)
 		__field(int, for_background)
 		__field(int, reason)
+		__field(unsigned long, ino)
+		__field(unsigned long, offset)
 	),
 	TP_fast_assign(
 		strncpy(__entry->name, dev_name(bdi->dev), 32);
@@ -55,9 +57,11 @@ DECLARE_EVENT_CLASS(writeback_work_class
 		__entry->range_cyclic = work->range_cyclic;
 		__entry->for_background	= work->for_background;
 		__entry->reason = work->reason;
+		__entry->ino = work->inode ? work->inode->i_ino : 0;
+		__entry->offset = work->offset;
 	),
 	TP_printk("bdi %s: sb_dev %d:%d nr_pages=%ld sync_mode=%d "
-		  "kupdate=%d range_cyclic=%d background=%d reason=%s",
+		  "kupdate=%d range_cyclic=%d background=%d reason=%s ino=%lu offset=%lu",
 		  __entry->name,
 		  MAJOR(__entry->sb_dev), MINOR(__entry->sb_dev),
 		  __entry->nr_pages,
@@ -65,7 +69,9 @@ DECLARE_EVENT_CLASS(writeback_work_class
 		  __entry->for_kupdate,
 		  __entry->range_cyclic,
 		  __entry->for_background,
-		  __print_symbolic(__entry->reason, WB_WORK_REASON)
+		  __print_symbolic(__entry->reason, WB_WORK_REASON),
+		  __entry->ino,
+		  __entry->offset
 	)
 );
 #define DEFINE_WRITEBACK_WORK_EVENT(name) \
@@ -437,6 +443,13 @@ DEFINE_EVENT(writeback_congest_waited_te
 	TP_ARGS(usec_timeout, usec_delayed)
 );
 
+DEFINE_EVENT(writeback_congest_waited_template, writeback_reclaim_wait,
+
+	TP_PROTO(unsigned int usec_timeout, unsigned int usec_delayed),
+
+	TP_ARGS(usec_timeout, usec_delayed)
+);
+
 DECLARE_EVENT_CLASS(writeback_single_inode_template,
 
 	TP_PROTO(struct inode *inode,
--- linux.orig/include/linux/writeback.h	2012-02-14 20:11:21.000000000 +0800
+++ linux/include/linux/writeback.h	2012-02-15 12:27:35.000000000 +0800
@@ -40,7 +40,7 @@ enum writeback_sync_modes {
  */
 enum wb_reason {
 	WB_REASON_BACKGROUND,
-	WB_REASON_TRY_TO_FREE_PAGES,
+	WB_REASON_PAGEOUT,
 	WB_REASON_SYNC,
 	WB_REASON_PERIODIC,
 	WB_REASON_LAPTOP_TIMER,
@@ -94,6 +94,8 @@ long writeback_inodes_wb(struct bdi_writ
 				enum wb_reason reason);
 long wb_do_writeback(struct bdi_writeback *wb, int force_wait);
 void wakeup_flusher_threads(long nr_pages, enum wb_reason reason);
+long flush_inode_page(struct address_space *mapping, struct page *page,
+		      bool wait);
 
 /* writeback.h requires fs.h; it, too, is not included from here. */
 static inline void wait_on_inode(struct inode *inode)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: reclaim the LRU lists full of dirty/writeback pages

[Mel Gorman]

On Thu, Feb 16, 2012 at 05:50:42PM +0800, Wu Fengguang wrote:
> On Tue, Feb 14, 2012 at 03:51:24PM +0000, Mel Gorman wrote:
> > On Tue, Feb 14, 2012 at 09:18:12PM +0800, Wu Fengguang wrote:
> > > > For the OOM problem, a more reasonable stopgap might be to identify when
> > > > a process is scanning a memcg at high priority and encountered all
> > > > PageReclaim with no forward progress and to congestion_wait() if that
> > > > situation occurs. A preferable way would be to wait until the flusher
> > > > wakes up a waiter on PageReclaim pages to be written out because we want
> > > > to keep moving way from congestion_wait() if at all possible.
> > > 
> > > Good points! Below is the more serious page reclaim changes.
> > > 
> > > The dirty/writeback pages may often come close to each other in the
> > > LRU list, so the local test during a 32-page scan may still trigger
> > > reclaim waits unnecessarily.
> > 
> > Yes, this is particularly the case when writing back to USB. It is not
> > unusual that all dirty pages under writeback are backed by USB and at the
> > end of the LRU. Right now what happens is that reclaimers see higher CPU
> > usage as they scan over these pages uselessly. If the wrong choice is
> > made on how to throttle, we'll see yet more variants of the "system
> > responsiveness drops when writing to USB".
> 
> Yes, USB is an important case to support.  I'd imagine the heavy USB
> writes typically happen in desktops and run *outside* of any memcg.

I would expect it's common that USB writes are outside a memcg.

> So they'll typically take <= 20% memory in the zone. As long as we
> start the PG_reclaim throttling only when above the 20% dirty
> threshold (ie. on zone_dirty_ok()), the USB case should be safe.
> 

It's not just the USB writer, it's unrelated process that are allocating
memory at the same time the writing happens. What we want to avoid is a
situation where something like firefox or evolution or even gnome-terminal
is performing a small read and gets either

a) started for IO bandwidth and stalls (not the focus here obviously)
b) enter page reclaim, finds PG_reclaim pages from the USB write and stalls

It's (b) we need to watch out for. I accept that this patch is heading
in the right direction and that the tracepoint can be used to identify
processes get throttled unfairly. Before merging, it'd be nice to hear of
such a test and include details in the changelog similar to the test case
in https://bugzilla.kernel.org/show_bug.cgi?id=31142 (a bug that lasted a
*long* time as it turned out, fixes merged for 3.3 with sync-light migration).

> > > Some global information on the percent
> > > of dirty/writeback pages in the LRU list may help. Anyway the added
> > > tests should still be much better than no protection.
> > > 
> > 
> > You can tell how many dirty pages and writeback pages are in the zone
> > already.
> 
> Right. I changed the test to
> 
> +       if (nr_pgreclaim && nr_pgreclaim >= (nr_taken >> (DEF_PRIORITY-priority)) &&
> +           (!global_reclaim(sc) || !zone_dirty_ok(zone)))
> +               reclaim_wait(HZ/10);
> 
> And I'd prefer to use a higher threshold than the default 20% for the
> above zone_dirty_ok() test, so that when Johannes' zone dirty
> balancing does the job fine, PG_reclaim based page reclaim throttling
> won't happen at all.
> 

We'd also need to watch that we do not get throttled on small zones
like ZONE_DMA (which shouldn't happen but still). To detect this if
it happens, please consider including node and zone information in the
writeback_reclaim_wait tracepoint. The memcg people might want to be
able to see the memcg which I guess could be available

node=[NID|memcg]
NID if zone >=0
memcg if zone == -1

Which is hacky but avoids creating two tracepoints.

> > > A global wait queue and reclaim_wait() is introduced. The waiters will
> > > be wakeup when pages are rotated by end_page_writeback() or lru drain.
> > > 
> > > I have to say its effectiveness depends on the filesystem... ext4
> > > and btrfs do fluent IO completions, so reclaim_wait() works pretty
> > > well:
> > >               dd-14560 [017] ....  1360.894605: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=10000
> > >               dd-14560 [017] ....  1360.904456: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=8000
> > >               dd-14560 [017] ....  1360.908293: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
> > >               dd-14560 [017] ....  1360.923960: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=15000
> > >               dd-14560 [017] ....  1360.927810: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
> > >               dd-14560 [017] ....  1360.931656: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
> > >               dd-14560 [017] ....  1360.943503: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=10000
> > >               dd-14560 [017] ....  1360.953289: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=7000
> > >               dd-14560 [017] ....  1360.957177: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
> > >               dd-14560 [017] ....  1360.972949: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=15000
> > > 
> > > However XFS does IO completions in very large batches (there may be
> > > only several big IO completions in one second). So reclaim_wait()
> > > mostly end up waiting to the full HZ/10 timeout:
> > > 
> > >               dd-4177  [008] ....   866.367661: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > >               dd-4177  [010] ....   866.567583: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > >               dd-4177  [012] ....   866.767458: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > >               dd-4177  [013] ....   866.867419: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > >               dd-4177  [008] ....   867.167266: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > >               dd-4177  [010] ....   867.367168: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > >               dd-4177  [012] ....   867.818950: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > >               dd-4177  [013] ....   867.918905: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > >               dd-4177  [013] ....   867.971657: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=52000
> > >               dd-4177  [013] ....   867.971812: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=0
> > >               dd-4177  [008] ....   868.355700: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > >               dd-4177  [010] ....   868.700515: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > > 
> > 
> > And where people will get hit by regressions in this area is writing to
> > vfat and in more rare cases ntfs on USB stick.
> 
> vfat IO completions seem to lie somewhere between ext4 and xfs:
> 
>            <...>-46385 [010] .... 143570.714470: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
>            <...>-46385 [008] .... 143570.752391: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=12000
>            <...>-46385 [008] .... 143570.937327: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=52000
>            <...>-46385 [010] .... 143571.160252: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
>            <...>-46385 [011] .... 143571.286197: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
>            <...>-46385 [008] .... 143571.329644: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=15000
>            <...>-46385 [008] .... 143571.475433: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=18000
>            <...>-46385 [008] .... 143571.653461: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=52000
>            <...>-46385 [008] .... 143571.839949: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=56000
>            <...>-46385 [010] .... 143572.060816: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
>            <...>-46385 [011] .... 143572.185754: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
>            <...>-46385 [008] .... 143572.212522: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=1000
>            <...>-46385 [008] .... 143572.217825: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
>            <...>-46385 [008] .... 143572.312395: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=91000
>            <...>-46385 [008] .... 143572.315122: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=1000
>            <...>-46385 [009] .... 143572.433630: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
>            <...>-46385 [010] .... 143572.534569: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
>  

Ok. It's interesting to note that we are stalling a lot there - roughly
30ms every second. As long as it's the writer, that's fine. If it's
firefox, it will create bug reports :)

> > > <SNIP>
> > > @@ -813,6 +815,10 @@ static unsigned long shrink_page_list(st
> > >  
> > >  		if (PageWriteback(page)) {
> > >  			nr_writeback++;
> > > +			if (PageReclaim(page))
> > > +				nr_pgreclaim++;
> > > +			else
> > > +				SetPageReclaim(page);
> > >  			/*
> > 
> > This check is unexpected. We already SetPageReclaim when queuing pages for
> > IO from reclaim context and if dirty pages are encountered during the LRU
> > scan that cannot be queued for IO. How often is it that nr_pgreclaim !=
> > nr_writeback and by how much do they differ?
> 
> Quite often, I suspect. The pageout writeback works do 1-8MB write
> around which may start I/O a bit earlier than the covered pages are
> encountered by page reclaim. ext4 forces 128MB write chunk size, which
> further increases the opportunities.
> 

Ok, thanks for the clarification. Stick a wee comment on it please.

> > >  			 * Synchronous reclaim cannot queue pages for
> > >  			 * writeback due to the possibility of stack overflow
> > > @@ -874,12 +880,15 @@ static unsigned long shrink_page_list(st
> > >  			nr_dirty++;
> > >  
> > >  			/*
> > > -			 * Only kswapd can writeback filesystem pages to
> > > -			 * avoid risk of stack overflow but do not writeback
> > > -			 * unless under significant pressure.
> > > +			 * run into the visited page again: we are scanning
> > > +			 * faster than the flusher can writeout dirty pages
> > >  			 */
> > 
> > which in itself is not an abnormal condition. We get into this situation
> > when writing to USB. Dirty throttling stops too much memory getting dirtied
> > but that does not mean we should throttle instead of reclaiming clean pages.
> > 
> > That's why I worry that if this is aimed at fixing a memcg problem, it
> > will have the impact of making interactive performance on normal systems
> > worse.
> 
> You are right. This patch only addresses the pageout I/O efficiency
> and dirty throttling problems for a fully dirtied LRU. Next step, I'll
> think about the interactive performance problem for a less dirtied LRU.
> 

Ok, thanks.

> > <SNIP>
> >
> > If the intention is to avoid memcg going OOM prematurely, the
> > nr_pgreclaim value needs to be treated at a higher level that records
> > how many PageReclaim pages were encountered. If no progress was made
> > because all the pages were PageReclaim, then throttle and return 1 to
> > the page allocator where it will retry the allocation without going OOM
> > after some pages have been cleaned and reclaimed.
>  
> Agreed in general, but changed to this test for now, which is made a
> bit more global wise with the use of zone_dirty_ok().
> 

Ok, sure. We know what to look out for and where unrelated regressions
might get introduced.

> memcg is ignored due to no dirty accounting (Greg has the patch though).
> And even zone_dirty_ok() may be inaccurate for the global reclaim, if
> some memcgs are skipped by the global reclaim by the memcg soft limit.
> 
> But anyway, it's a handy hack for now. I'm looking into some more
> radical changes to put most dirty/writeback pages into a standalone
> LRU list (in addition to your LRU_IMMEDIATE, which I think is a good
> idea) for addressing the clustered way they tend to lie in the
> inactive LRU list.
> 
> +       if (nr_pgreclaim && nr_pgreclaim >= (nr_taken >> (DEF_PRIORITY-priority)) &&
> +           (!global_reclaim(sc) || !zone_dirty_ok(zone)))
> +               reclaim_wait(HZ/10);
> 

This should make it harder to get stalled. Your tracepoint should help
us catch if it happens unnecessarily.

> ---
> Subject: writeback: introduce the pageout work
> Date: Thu Jul 29 14:41:19 CST 2010
> 
> This relays file pageout IOs to the flusher threads.
> 
> The ultimate target is to gracefully handle the LRU lists full of
> dirty/writeback pages.
> 

It would be worth mentioning in the changelog that this is much more
important now that page reclaim generally does not writeout filesystem-backed
pages.

> 1) I/O efficiency
> 
> The flusher will piggy back the nearby ~10ms worth of dirty pages for I/O.
> 
> This takes advantage of the time/spacial locality in most workloads: the
> nearby pages of one file are typically populated into the LRU at the same
> time, hence will likely be close to each other in the LRU list. Writing
> them in one shot helps clean more pages effectively for page reclaim.
> 
> 2) OOM avoidance and scan rate control
> 
> Typically we do LRU scan w/o rate control and quickly get enough clean
> pages for the LRU lists not full of dirty pages.
> 
> Or we can still get a number of freshly cleaned pages (moved to LRU tail
> by end_page_writeback()) when the queued pageout I/O is completed within
> tens of milli-seconds.
> 
> However if the LRU list is small and full of dirty pages, it can be
> quickly fully scanned and go OOM before the flusher manages to clean
> enough pages.
> 

It's worth pointing out here that generally this does not happen for global
reclaim which does dirty throttling but happens easily with memcg LRUs.

> A simple yet reliable scheme is employed to avoid OOM and keep scan rate
> in sync with the I/O rate:
> 
> 	if (PageReclaim(page))
> 		congestion_wait(HZ/10);
> 

This comment is stale now.

> PG_reclaim plays the key role. When dirty pages are encountered, we
> queue I/O for it,

This is misleading. The process that encounters the dirty page does
not queue the page for IO unless it is kswapd scanning at high priority
(currently anyway, you patch changes it). The process that finds the page
queues work for flusher threads that will queue the actual I/O for it at
some unknown time in the future.

> set PG_reclaim and put it back to the LRU head.
> So if PG_reclaim pages are encountered again, it means the dirty page
> has not yet been cleaned by the flusher after a full zone scan. It
> indicates we are scanning more fast than I/O and shall take a snap.
> 

This is also slightly misleading because the page can be encountered
after rescanning the inactive list, not necessarily a full zone scan but
it's a minor point.

> The runtime behavior on a fully dirtied small LRU list would be:
> It will start with a quick scan of the list, queuing all pages for I/O.
> Then the scan will be slowed down by the PG_reclaim pages *adaptively*
> to match the I/O bandwidth.
> 
> 3) writeback work coordinations
> 
> To avoid memory allocations at page reclaim, a mempool for struct
> wb_writeback_work is created.
> 
> wakeup_flusher_threads() is removed because it can easily delay the
> more oriented pageout works and even exhaust the mempool reservations.
> It's also found to not I/O efficient by frequently submitting writeback
> works with small ->nr_pages.
> 
> Background/periodic works will quit automatically, so as to clean the
> pages under reclaim ASAP. However for now the sync work can still block
> us for long time.
> 
> Jan Kara: limit the search scope. Note that the limited search and work
> pool is not a big problem: 1000 IOs under flight are typically more than
> enough to saturate the disk. And the overheads of searching in the work
> list didn't even show up in the perf report.
> 
> 4) test case
> 
> Run 2 dd tasks in a 100MB memcg (a very handy test case from Greg Thelen):
> 
> 	mkdir /cgroup/x
> 	echo 100M > /cgroup/x/memory.limit_in_bytes
> 	echo $$ > /cgroup/x/tasks
> 
> 	for i in `seq 2`
> 	do
> 		dd if=/dev/zero of=/fs/f$i bs=1k count=1M &
> 	done
> 
> Before patch, the dd tasks are quickly OOM killed.
> After patch, they run well with reasonably good performance and overheads:
> 
> 1073741824 bytes (1.1 GB) copied, 22.2196 s, 48.3 MB/s
> 1073741824 bytes (1.1 GB) copied, 22.4675 s, 47.8 MB/s
> 
> iostat -kx 1
> 
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
> sda               0.00     0.00    0.00  178.00     0.00 89568.00  1006.38    74.35  417.71   4.80  85.40
> sda               0.00     2.00    0.00  191.00     0.00 94428.00   988.77    53.34  219.03   4.34  82.90
> sda               0.00    20.00    0.00  196.00     0.00 97712.00   997.06    71.11  337.45   4.77  93.50
> sda               0.00     5.00    0.00  175.00     0.00 84648.00   967.41    54.03  316.44   5.06  88.60
> sda               0.00     0.00    0.00  186.00     0.00 92432.00   993.89    56.22  267.54   5.38 100.00
> sda               0.00     1.00    0.00  183.00     0.00 90156.00   985.31    37.99  325.55   4.33  79.20
> sda               0.00     0.00    0.00  175.00     0.00 88692.00  1013.62    48.70  218.43   4.69  82.10
> sda               0.00     0.00    0.00  196.00     0.00 97528.00   995.18    43.38  236.87   5.10 100.00
> sda               0.00     0.00    0.00  179.00     0.00 88648.00   990.48    45.83  285.43   5.59 100.00
> sda               0.00     0.00    0.00  178.00     0.00 88500.00   994.38    28.28  158.89   4.99  88.80
> sda               0.00     0.00    0.00  194.00     0.00 95852.00   988.16    32.58  167.39   5.15 100.00
> sda               0.00     2.00    0.00  215.00     0.00 105996.00   986.01    41.72  201.43   4.65 100.00
> sda               0.00     4.00    0.00  173.00     0.00 84332.00   974.94    50.48  260.23   5.76  99.60
> sda               0.00     0.00    0.00  182.00     0.00 90312.00   992.44    36.83  212.07   5.49 100.00
> sda               0.00     8.00    0.00  195.00     0.00 95940.50   984.01    50.18  221.06   5.13 100.00
> sda               0.00     1.00    0.00  220.00     0.00 108852.00   989.56    40.99  202.68   4.55 100.00
> sda               0.00     2.00    0.00  161.00     0.00 80384.00   998.56    37.19  268.49   6.21 100.00
> sda               0.00     4.00    0.00  182.00     0.00 90830.00   998.13    50.58  239.77   5.49 100.00
> sda               0.00     0.00    0.00  197.00     0.00 94877.00   963.22    36.68  196.79   5.08 100.00
> 
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>            0.25    0.00   15.08   33.92    0.00   50.75
>            0.25    0.00   14.54   35.09    0.00   50.13
>            0.50    0.00   13.57   32.41    0.00   53.52
>            0.50    0.00   11.28   36.84    0.00   51.38
>            0.50    0.00   15.75   32.00    0.00   51.75
>            0.50    0.00   10.50   34.00    0.00   55.00
>            0.50    0.00   17.63   27.46    0.00   54.41
>            0.50    0.00   15.08   30.90    0.00   53.52
>            0.50    0.00   11.28   32.83    0.00   55.39
>            0.75    0.00   16.79   26.82    0.00   55.64
>            0.50    0.00   16.08   29.15    0.00   54.27
>            0.50    0.00   13.50   30.50    0.00   55.50
>            0.50    0.00   14.32   35.18    0.00   50.00
>            0.50    0.00   12.06   33.92    0.00   53.52
>            0.50    0.00   17.29   30.58    0.00   51.63
>            0.50    0.00   15.08   29.65    0.00   54.77
>            0.50    0.00   12.53   29.32    0.00   57.64
>            0.50    0.00   15.29   31.83    0.00   52.38
> 
> The global dd numbers for comparison:
> 
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
> sda               0.00     0.00    0.00  189.00     0.00 95752.00  1013.25   143.09  684.48   5.29 100.00
> sda               0.00     0.00    0.00  208.00     0.00 105480.00  1014.23   143.06  733.29   4.81 100.00
> sda               0.00     0.00    0.00  161.00     0.00 81924.00  1017.69   141.71  757.79   6.21 100.00
> sda               0.00     0.00    0.00  217.00     0.00 109580.00  1009.95   143.09  749.55   4.61 100.10
> sda               0.00     0.00    0.00  187.00     0.00 94728.00  1013.13   144.31  773.67   5.35 100.00
> sda               0.00     0.00    0.00  189.00     0.00 95752.00  1013.25   144.14  742.00   5.29 100.00
> sda               0.00     0.00    0.00  177.00     0.00 90032.00  1017.31   143.32  656.59   5.65 100.00
> sda               0.00     0.00    0.00  215.00     0.00 108640.00  1010.60   142.90  817.54   4.65 100.00
> sda               0.00     2.00    0.00  166.00     0.00 83858.00  1010.34   143.64  808.61   6.02 100.00
> sda               0.00     0.00    0.00  186.00     0.00 92813.00   997.99   141.18  736.95   5.38 100.00
> sda               0.00     0.00    0.00  206.00     0.00 104456.00  1014.14   146.27  729.33   4.85 100.00
> sda               0.00     0.00    0.00  213.00     0.00 107024.00  1004.92   143.25  705.70   4.69 100.00
> sda               0.00     0.00    0.00  188.00     0.00 95748.00  1018.60   141.82  764.78   5.32 100.00
> 
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>            0.51    0.00   11.22   52.30    0.00   35.97
>            0.25    0.00   10.15   52.54    0.00   37.06
>            0.25    0.00    5.01   56.64    0.00   38.10
>            0.51    0.00   15.15   43.94    0.00   40.40
>            0.25    0.00   12.12   48.23    0.00   39.39
>            0.51    0.00   11.20   53.94    0.00   34.35
>            0.26    0.00    9.72   51.41    0.00   38.62
>            0.76    0.00    9.62   50.63    0.00   38.99
>            0.51    0.00   10.46   53.32    0.00   35.71
>            0.51    0.00    9.41   51.91    0.00   38.17
>            0.25    0.00   10.69   49.62    0.00   39.44
>            0.51    0.00   12.21   52.67    0.00   34.61
>            0.51    0.00   11.45   53.18    0.00   34.86
> 
> XXX: commit NFS unstable pages via write_inode()
> XXX: the added congestion_wait() may be undesirable in some situations
> 

This second XXX may also not be redundant.

> CC: Jan Kara <jack@suse.cz>
> CC: Mel Gorman <mgorman@suse.de>
> Acked-by: Rik van Riel <riel@redhat.com>
> CC: Greg Thelen <gthelen@google.com>
> CC: Minchan Kim <minchan.kim@gmail.com>
> Signed-off-by: Wu Fengguang <fengguang.wu@intel.com>
> ---
>  fs/fs-writeback.c                |  169 ++++++++++++++++++++++++++++-
>  include/linux/backing-dev.h      |    2 
>  include/linux/writeback.h        |    4 
>  include/trace/events/writeback.h |   19 ++-
>  mm/backing-dev.c                 |   35 ++++++
>  mm/swap.c                        |    1 
>  mm/vmscan.c                      |   32 +++--
>  7 files changed, 245 insertions(+), 17 deletions(-)
> 
> - move congestion_wait() out of the page lock: it's blocking btrfs lock_delalloc_pages()
> 
> --- linux.orig/include/linux/backing-dev.h	2012-02-14 20:11:21.000000000 +0800
> +++ linux/include/linux/backing-dev.h	2012-02-15 12:34:24.000000000 +0800
> @@ -304,6 +304,8 @@ void clear_bdi_congested(struct backing_
>  void set_bdi_congested(struct backing_dev_info *bdi, int sync);
>  long congestion_wait(int sync, long timeout);
>  long wait_iff_congested(struct zone *zone, int sync, long timeout);
> +long reclaim_wait(long timeout);
> +void reclaim_rotated(void);
>  
>  static inline bool bdi_cap_writeback_dirty(struct backing_dev_info *bdi)
>  {
> --- linux.orig/mm/backing-dev.c	2012-02-14 20:11:21.000000000 +0800
> +++ linux/mm/backing-dev.c	2012-02-15 12:34:19.000000000 +0800
> @@ -873,3 +873,38 @@ out:
>  	return ret;
>  }
>  EXPORT_SYMBOL(wait_iff_congested);
> +
> +static DECLARE_WAIT_QUEUE_HEAD(reclaim_wqh);
> +

Should this be declared on a per-NUMA node basis to avoid throttling on one
node being woken up by activity on an unrelated node?  reclaim_rorated()
is called from a context that has a page so looking up the waitqueue would
be easy. Grep for place that initialise kswapd_wait and the initialisation
code will be easier although watch that if a node is hot-removed that
the queue is woken.

> +/**
> + * reclaim_wait - wait for some pages being rotated to the LRU tail
> + * @timeout: timeout in jiffies
> + *
> + * Wait until @timeout, or when some (typically PG_reclaim under writeback)
> + * pages rotated to the LRU so that page reclaim can make progress.
> + */
> +long reclaim_wait(long timeout)
> +{
> +	long ret;
> +	unsigned long start = jiffies;
> +	DEFINE_WAIT(wait);
> +
> +	prepare_to_wait(&reclaim_wqh, &wait, TASK_KILLABLE);
> +	ret = io_schedule_timeout(timeout);
> +	finish_wait(&reclaim_wqh, &wait);
> +
> +	trace_writeback_reclaim_wait(jiffies_to_usecs(timeout),
> +				     jiffies_to_usecs(jiffies - start));
> +
> +	return ret;
> +}
> +EXPORT_SYMBOL(reclaim_wait);
> +

Why do we export this? Only vmscan.c is calling it and I'm scratching my
head trying to figure out why a kernel module would want to call it.

> +void reclaim_rotated()
> +{

style nit

void reclaim_rotated(void)

> +	wait_queue_head_t *wqh = &reclaim_wqh;
> +
> +	if (waitqueue_active(wqh))
> +		wake_up(wqh);
> +}
> +
> --- linux.orig/mm/swap.c	2012-02-14 20:11:21.000000000 +0800
> +++ linux/mm/swap.c	2012-02-15 12:27:35.000000000 +0800
> @@ -253,6 +253,7 @@ static void pagevec_move_tail(struct pag
>  
>  	pagevec_lru_move_fn(pvec, pagevec_move_tail_fn, &pgmoved);
>  	__count_vm_events(PGROTATED, pgmoved);
> +	reclaim_rotated();
>  }
>  
>  /*
> --- linux.orig/mm/vmscan.c	2012-02-14 20:11:21.000000000 +0800
> +++ linux/mm/vmscan.c	2012-02-16 17:23:17.000000000 +0800
> @@ -767,7 +767,8 @@ static unsigned long shrink_page_list(st
>  				      struct scan_control *sc,
>  				      int priority,
>  				      unsigned long *ret_nr_dirty,
> -				      unsigned long *ret_nr_writeback)
> +				      unsigned long *ret_nr_writeback,
> +				      unsigned long *ret_nr_pgreclaim)
>  {
>  	LIST_HEAD(ret_pages);
>  	LIST_HEAD(free_pages);
> @@ -776,6 +777,7 @@ static unsigned long shrink_page_list(st
>  	unsigned long nr_congested = 0;
>  	unsigned long nr_reclaimed = 0;
>  	unsigned long nr_writeback = 0;
> +	unsigned long nr_pgreclaim = 0;
>  
>  	cond_resched();
>  
> @@ -813,6 +815,10 @@ static unsigned long shrink_page_list(st
>  
>  		if (PageWriteback(page)) {
>  			nr_writeback++;
> +			if (PageReclaim(page))
> +				nr_pgreclaim++;
> +			else
> +				SetPageReclaim(page);
>  			/*
>  			 * Synchronous reclaim cannot queue pages for
>  			 * writeback due to the possibility of stack overflow
> @@ -874,12 +880,15 @@ static unsigned long shrink_page_list(st
>  			nr_dirty++;
>  
>  			/*
> -			 * Only kswapd can writeback filesystem pages to
> -			 * avoid risk of stack overflow but do not writeback
> -			 * unless under significant pressure.
> +			 * run into the visited page again: we are scanning
> +			 * faster than the flusher can writeout dirty pages
>  			 */
> -			if (page_is_file_cache(page) &&
> -					(!current_is_kswapd() || priority >= DEF_PRIORITY - 2)) {
> +			if (page_is_file_cache(page) && PageReclaim(page)) {
> +				nr_pgreclaim++;
> +				goto keep_locked;
> +			}

This change means that kswapd is no longer doing any writeback from page
reclaim. Was that intended because it's not discussed in the changelog. I
know writeback from kswapd is poor in terms of IO performance but it's a
last resort for freeing a page when reclaim is in trouble. If we are to
disable it and depend 100% on the flusher threads, it should be in its
own patch for bisection reasons if nothing else.

> +			if (page_is_file_cache(page) && mapping &&
> +			    flush_inode_page(mapping, page, false) >= 0) {
>  				/*
>  				 * Immediately reclaim when written back.
>  				 * Similar in principal to deactivate_page()
> @@ -1028,6 +1037,7 @@ keep_lumpy:
>  	count_vm_events(PGACTIVATE, pgactivate);
>  	*ret_nr_dirty += nr_dirty;
>  	*ret_nr_writeback += nr_writeback;
> +	*ret_nr_pgreclaim += nr_pgreclaim;
>  	return nr_reclaimed;
>  }
>  
> @@ -1509,6 +1519,7 @@ shrink_inactive_list(unsigned long nr_to
>  	unsigned long nr_file;
>  	unsigned long nr_dirty = 0;
>  	unsigned long nr_writeback = 0;
> +	unsigned long nr_pgreclaim = 0;
>  	isolate_mode_t reclaim_mode = ISOLATE_INACTIVE;
>  	struct zone *zone = mz->zone;
>  
> @@ -1559,13 +1570,13 @@ shrink_inactive_list(unsigned long nr_to
>  	spin_unlock_irq(&zone->lru_lock);
>  
>  	nr_reclaimed = shrink_page_list(&page_list, mz, sc, priority,
> -						&nr_dirty, &nr_writeback);
> +				&nr_dirty, &nr_writeback, &nr_pgreclaim);
>  
>  	/* Check if we should syncronously wait for writeback */
>  	if (should_reclaim_stall(nr_taken, nr_reclaimed, priority, sc)) {
>  		set_reclaim_mode(priority, sc, true);
>  		nr_reclaimed += shrink_page_list(&page_list, mz, sc,
> -					priority, &nr_dirty, &nr_writeback);
> +			priority, &nr_dirty, &nr_writeback, &nr_pgreclaim);
>  	}
>  
>  	spin_lock_irq(&zone->lru_lock);
> @@ -1608,6 +1619,9 @@ shrink_inactive_list(unsigned long nr_to
>  	 */
>  	if (nr_writeback && nr_writeback >= (nr_taken >> (DEF_PRIORITY-priority)))
>  		wait_iff_congested(zone, BLK_RW_ASYNC, HZ/10);
> +	if (nr_pgreclaim && nr_pgreclaim >= (nr_taken >> (DEF_PRIORITY-priority)) &&
> +	    (!global_reclaim(sc) || !zone_dirty_ok(zone)))
> +		reclaim_wait(HZ/10);
>  

I prefer this but it would be nice if there was a comment explaining it
or at least expand the comment explaining how nr_writeback can lead to
wait_iff_congested() being called.

>  	trace_mm_vmscan_lru_shrink_inactive(zone->zone_pgdat->node_id,
>  		zone_idx(zone),
> @@ -2382,8 +2396,6 @@ static unsigned long do_try_to_free_page
>  		 */
>  		writeback_threshold = sc->nr_to_reclaim + sc->nr_to_reclaim / 2;
>  		if (total_scanned > writeback_threshold) {
> -			wakeup_flusher_threads(laptop_mode ? 0 : total_scanned,
> -						WB_REASON_TRY_TO_FREE_PAGES);
>  			sc->may_writepage = 1;
>  		}
>  
> --- linux.orig/fs/fs-writeback.c	2012-02-14 20:11:21.000000000 +0800
> +++ linux/fs/fs-writeback.c	2012-02-15 12:27:35.000000000 +0800
> @@ -41,6 +41,8 @@ struct wb_writeback_work {
>  	long nr_pages;
>  	struct super_block *sb;
>  	unsigned long *older_than_this;
> +	struct inode *inode;
> +	pgoff_t offset;
>  	enum writeback_sync_modes sync_mode;
>  	unsigned int tagged_writepages:1;
>  	unsigned int for_kupdate:1;
> @@ -65,6 +67,27 @@ struct wb_writeback_work {
>   */
>  int nr_pdflush_threads;
>  
> +static mempool_t *wb_work_mempool;
> +
> +static void *wb_work_alloc(gfp_t gfp_mask, void *pool_data)
> +{
> +	/*
> +	 * bdi_flush_inode_range() may be called on page reclaim
> +	 */
> +	if (current->flags & PF_MEMALLOC)
> +		return NULL;
> +

This check is why I worry about kswapd being unable to write pages at
all. If the mempool is depleted for whatever reason, reclaim has no way
of telling the flushers what work is needed or waking them. Potentially,
we could be waiting a long time for pending flusher work to complete to
free up a slot.  I recognise it may not be bad in practice because the
pool is large and other work will be completing but it's why kswapd not
writing back pages should be in its own patch.

> +	return kmalloc(sizeof(struct wb_writeback_work), gfp_mask);
> +}
> +
> +static __init int wb_work_init(void)
> +{
> +	wb_work_mempool = mempool_create(1024,
> +					 wb_work_alloc, mempool_kfree, NULL);
> +	return wb_work_mempool ? 0 : -ENOMEM;
> +}
> +fs_initcall(wb_work_init);
> +
>  /**
>   * writeback_in_progress - determine whether there is writeback in progress
>   * @bdi: the device's backing_dev_info structure.
> @@ -129,7 +152,7 @@ __bdi_start_writeback(struct backing_dev
>  	 * This is WB_SYNC_NONE writeback, so if allocation fails just
>  	 * wakeup the thread for old dirty data writeback
>  	 */
> -	work = kzalloc(sizeof(*work), GFP_ATOMIC);
> +	work = mempool_alloc(wb_work_mempool, GFP_NOWAIT);
>  	if (!work) {
>  		if (bdi->wb.task) {
>  			trace_writeback_nowork(bdi);
> @@ -138,6 +161,7 @@ __bdi_start_writeback(struct backing_dev
>  		return;
>  	}
>  
> +	memset(work, 0, sizeof(*work));
>  	work->sync_mode	= WB_SYNC_NONE;
>  	work->nr_pages	= nr_pages;
>  	work->range_cyclic = range_cyclic;
> @@ -186,6 +210,125 @@ void bdi_start_background_writeback(stru
>  	spin_unlock_bh(&bdi->wb_lock);
>  }
>  
> +static bool extend_writeback_range(struct wb_writeback_work *work,
> +				   pgoff_t offset,
> +				   unsigned long write_around_pages)
> +{

comment on what this function is for and what the return values mean.

"returns true if the wb_writeback_work now encompasses the request"

or something

> +	pgoff_t end = work->offset + work->nr_pages;
> +
> +	if (offset >= work->offset && offset < end)
> +		return true;
> +

This does not ensure that the full span of
offset -> offset+write_around_pages is encompassed by work. All it
checks is that the start of the requested range is going to be handled.

I guess it's ok because the page reclaims cares about is covered and
avoids a situation where too much IO is being queued. It's unclear if
this is what you intended though because you check for too much IO being
queued in the next block.

> +	/*
> +	 * for sequential workloads with good locality, include up to 8 times
> +	 * more data in one chunk
> +	 */
> +	if (work->nr_pages >= 8 * write_around_pages)
> +		return false;
> +
> +	/* the unsigned comparison helps eliminate one compare */
> +	if (work->offset - offset < write_around_pages) {
> +		work->nr_pages += write_around_pages;
> +		work->offset -= write_around_pages;
> +		return true;
> +	}
> +
> +	if (offset - end < write_around_pages) {
> +		work->nr_pages += write_around_pages;
> +		return true;
> +	}
> +
> +	return false;
> +}
> +
> +/*
> + * schedule writeback on a range of inode pages.
> + */
> +static struct wb_writeback_work *
> +bdi_flush_inode_range(struct backing_dev_info *bdi,
> +		      struct inode *inode,
> +		      pgoff_t offset,
> +		      pgoff_t len,
> +		      bool wait)
> +{
> +	struct wb_writeback_work *work;
> +
> +	if (!igrab(inode))
> +		return ERR_PTR(-ENOENT);
> +

Explain why the igrab is necessary. I think it's because we are calling
this from page reclaim context and the only thing pinning the
address_space is the page lock . If I'm right, it should be made clear
in the comment for bdi_flush_inode_range that this should only be called
from page reclaim context. Maybe even VM_BUG_ON if
!(current->flags & PF_MEMALLOC)?

> +	work = mempool_alloc(wb_work_mempool, wait ? GFP_NOIO : GFP_NOWAIT);
> +	if (!work) {
> +		trace_printk("wb_work_mempool alloc fail\n");
> +		return ERR_PTR(-ENOMEM);
> +	}
> +
> +	memset(work, 0, sizeof(*work));
> +	work->sync_mode		= WB_SYNC_NONE;
> +	work->inode		= inode;
> +	work->offset		= offset;
> +	work->nr_pages		= len;
> +	work->reason		= WB_REASON_PAGEOUT;
> +
> +	bdi_queue_work(bdi, work);
> +
> +	return work;
> +}
> +
> +/*
> + * Called by page reclaim code to flush the dirty page ASAP. Do write-around to
> + * improve IO throughput. The nearby pages will have good chance to reside in
> + * the same LRU list that vmscan is working on, and even close to each other
> + * inside the LRU list in the common case of sequential read/write.
> + *
> + * ret > 0: success, found/reused a previous writeback work
> + * ret = 0: success, allocated/queued a new writeback work
> + * ret < 0: failed
> + */
> +long flush_inode_page(struct address_space *mapping,
> +		      struct page *page,
> +		      bool wait)
> +{
> +	struct backing_dev_info *bdi = mapping->backing_dev_info;
> +	struct inode *inode = mapping->host;
> +	struct wb_writeback_work *work;
> +	unsigned long write_around_pages;
> +	pgoff_t offset = page->index;
> +	int i;
> +	long ret = 0;
> +
> +	if (unlikely(!inode))
> +		return -ENOENT;
> +
> +	/*
> +	 * piggy back 8-15ms worth of data
> +	 */
> +	write_around_pages = bdi->avg_write_bandwidth + MIN_WRITEBACK_PAGES;
> +	write_around_pages = rounddown_pow_of_two(write_around_pages) >> 6;
> +
> +	i = 1;
> +	spin_lock_bh(&bdi->wb_lock);
> +	list_for_each_entry_reverse(work, &bdi->work_list, list) {
> +		if (work->inode != inode)
> +			continue;
> +		if (extend_writeback_range(work, offset, write_around_pages)) {
> +			ret = i;
> +			break;
> +		}
> +		if (i++ > 100)	/* limit search depth */
> +			break;

No harm to move Jan's comment on depth limit search to here adding why
100 is as good as number as any to use.

> +	}
> +	spin_unlock_bh(&bdi->wb_lock);
> +
> +	if (!ret) {
> +		offset = round_down(offset, write_around_pages);
> +		work = bdi_flush_inode_range(bdi, inode,
> +					     offset, write_around_pages, wait);
> +		if (IS_ERR(work))
> +			ret = PTR_ERR(work);
> +	}
> +	return ret;
> +}
> +
>  /*
>   * Remove the inode from the writeback list it is on.
>   */
> @@ -833,6 +976,23 @@ static unsigned long get_nr_dirty_pages(
>  		get_nr_dirty_inodes();
>  }
>  
> +static long wb_flush_inode(struct bdi_writeback *wb,
> +			   struct wb_writeback_work *work)
> +{
> +	struct writeback_control wbc = {
> +		.sync_mode = WB_SYNC_NONE,
> +		.nr_to_write = LONG_MAX,
> +		.range_start = work->offset << PAGE_CACHE_SHIFT,
> +		.range_end = (work->offset + work->nr_pages - 1)
> +						<< PAGE_CACHE_SHIFT,
> +	};
> +
> +	do_writepages(work->inode->i_mapping, &wbc);
> +	iput(work->inode);
> +
> +	return LONG_MAX - wbc.nr_to_write;
> +}
> +
>  static long wb_check_background_flush(struct bdi_writeback *wb)
>  {
>  	if (over_bground_thresh(wb->bdi)) {
> @@ -905,7 +1065,10 @@ long wb_do_writeback(struct bdi_writebac
>  
>  		trace_writeback_exec(bdi, work);
>  
> -		wrote += wb_writeback(wb, work);
> +		if (work->inode)
> +			wrote += wb_flush_inode(wb, work);
> +		else
> +			wrote += wb_writeback(wb, work);
>  
>  		/*
>  		 * Notify the caller of completion if this is a synchronous
> @@ -914,7 +1077,7 @@ long wb_do_writeback(struct bdi_writebac
>  		if (work->done)
>  			complete(work->done);
>  		else
> -			kfree(work);
> +			mempool_free(work, wb_work_mempool);
>  	}
>  
>  	/*
> --- linux.orig/include/trace/events/writeback.h	2012-02-14 20:11:22.000000000 +0800
> +++ linux/include/trace/events/writeback.h	2012-02-15 12:27:35.000000000 +0800
> @@ -23,7 +23,7 @@
>  
>  #define WB_WORK_REASON							\
>  		{WB_REASON_BACKGROUND,		"background"},		\
> -		{WB_REASON_TRY_TO_FREE_PAGES,	"try_to_free_pages"},	\
> +		{WB_REASON_PAGEOUT,		"pageout"},		\
>  		{WB_REASON_SYNC,		"sync"},		\
>  		{WB_REASON_PERIODIC,		"periodic"},		\
>  		{WB_REASON_LAPTOP_TIMER,	"laptop_timer"},	\
> @@ -45,6 +45,8 @@ DECLARE_EVENT_CLASS(writeback_work_class
>  		__field(int, range_cyclic)
>  		__field(int, for_background)
>  		__field(int, reason)
> +		__field(unsigned long, ino)
> +		__field(unsigned long, offset)
>  	),
>  	TP_fast_assign(
>  		strncpy(__entry->name, dev_name(bdi->dev), 32);
> @@ -55,9 +57,11 @@ DECLARE_EVENT_CLASS(writeback_work_class
>  		__entry->range_cyclic = work->range_cyclic;
>  		__entry->for_background	= work->for_background;
>  		__entry->reason = work->reason;
> +		__entry->ino = work->inode ? work->inode->i_ino : 0;
> +		__entry->offset = work->offset;
>  	),
>  	TP_printk("bdi %s: sb_dev %d:%d nr_pages=%ld sync_mode=%d "
> -		  "kupdate=%d range_cyclic=%d background=%d reason=%s",
> +		  "kupdate=%d range_cyclic=%d background=%d reason=%s ino=%lu offset=%lu",
>  		  __entry->name,
>  		  MAJOR(__entry->sb_dev), MINOR(__entry->sb_dev),
>  		  __entry->nr_pages,
> @@ -65,7 +69,9 @@ DECLARE_EVENT_CLASS(writeback_work_class
>  		  __entry->for_kupdate,
>  		  __entry->range_cyclic,
>  		  __entry->for_background,
> -		  __print_symbolic(__entry->reason, WB_WORK_REASON)
> +		  __print_symbolic(__entry->reason, WB_WORK_REASON),
> +		  __entry->ino,
> +		  __entry->offset
>  	)
>  );
>  #define DEFINE_WRITEBACK_WORK_EVENT(name) \
> @@ -437,6 +443,13 @@ DEFINE_EVENT(writeback_congest_waited_te
>  	TP_ARGS(usec_timeout, usec_delayed)
>  );
>  
> +DEFINE_EVENT(writeback_congest_waited_template, writeback_reclaim_wait,
> +
> +	TP_PROTO(unsigned int usec_timeout, unsigned int usec_delayed),
> +
> +	TP_ARGS(usec_timeout, usec_delayed)
> +);
> +
>  DECLARE_EVENT_CLASS(writeback_single_inode_template,
>  
>  	TP_PROTO(struct inode *inode,
> --- linux.orig/include/linux/writeback.h	2012-02-14 20:11:21.000000000 +0800
> +++ linux/include/linux/writeback.h	2012-02-15 12:27:35.000000000 +0800
> @@ -40,7 +40,7 @@ enum writeback_sync_modes {
>   */
>  enum wb_reason {
>  	WB_REASON_BACKGROUND,
> -	WB_REASON_TRY_TO_FREE_PAGES,
> +	WB_REASON_PAGEOUT,
>  	WB_REASON_SYNC,
>  	WB_REASON_PERIODIC,
>  	WB_REASON_LAPTOP_TIMER,
> @@ -94,6 +94,8 @@ long writeback_inodes_wb(struct bdi_writ
>  				enum wb_reason reason);
>  long wb_do_writeback(struct bdi_writeback *wb, int force_wait);
>  void wakeup_flusher_threads(long nr_pages, enum wb_reason reason);
> +long flush_inode_page(struct address_space *mapping, struct page *page,
> +		      bool wait);
>  
>  /* writeback.h requires fs.h; it, too, is not included from here. */
>  static inline void wait_on_inode(struct inode *inode)

-- 
Mel Gorman
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: reclaim the LRU lists full of dirty/writeback pages

[Fengguang Wu]

Mel,

Thanks a lot for the in depth review!

On Thu, Feb 16, 2012 at 05:31:11PM +0000, Mel Gorman wrote:
> On Thu, Feb 16, 2012 at 05:50:42PM +0800, Wu Fengguang wrote:
> > On Tue, Feb 14, 2012 at 03:51:24PM +0000, Mel Gorman wrote:
> > > On Tue, Feb 14, 2012 at 09:18:12PM +0800, Wu Fengguang wrote:
> > > > > For the OOM problem, a more reasonable stopgap might be to identify when
> > > > > a process is scanning a memcg at high priority and encountered all
> > > > > PageReclaim with no forward progress and to congestion_wait() if that
> > > > > situation occurs. A preferable way would be to wait until the flusher
> > > > > wakes up a waiter on PageReclaim pages to be written out because we want
> > > > > to keep moving way from congestion_wait() if at all possible.
> > > > 
> > > > Good points! Below is the more serious page reclaim changes.
> > > > 
> > > > The dirty/writeback pages may often come close to each other in the
> > > > LRU list, so the local test during a 32-page scan may still trigger
> > > > reclaim waits unnecessarily.
> > > 
> > > Yes, this is particularly the case when writing back to USB. It is not
> > > unusual that all dirty pages under writeback are backed by USB and at the
> > > end of the LRU. Right now what happens is that reclaimers see higher CPU
> > > usage as they scan over these pages uselessly. If the wrong choice is
> > > made on how to throttle, we'll see yet more variants of the "system
> > > responsiveness drops when writing to USB".
> > 
> > Yes, USB is an important case to support.  I'd imagine the heavy USB
> > writes typically happen in desktops and run *outside* of any memcg.
> 
> I would expect it's common that USB writes are outside a memcg.
> 
> > So they'll typically take <= 20% memory in the zone. As long as we
> > start the PG_reclaim throttling only when above the 20% dirty
> > threshold (ie. on zone_dirty_ok()), the USB case should be safe.
> > 
> 
> It's not just the USB writer, it's unrelated process that are allocating
> memory at the same time the writing happens. What we want to avoid is a
> situation where something like firefox or evolution or even gnome-terminal
> is performing a small read and gets either
> 
> a) started for IO bandwidth and stalls (not the focus here obviously)
> b) enter page reclaim, finds PG_reclaim pages from the USB write and stalls
> 
> It's (b) we need to watch out for. I accept that this patch is heading
> in the right direction and that the tracepoint can be used to identify
> processes get throttled unfairly. Before merging, it'd be nice to hear of
> such a test and include details in the changelog similar to the test case
> in https://bugzilla.kernel.org/show_bug.cgi?id=31142 (a bug that lasted a
> *long* time as it turned out, fixes merged for 3.3 with sync-light migration).

To avoid PG_reclaim waiting for unrelated page allocators, I improved
the patch to

a) distinguish dirtier tasks and unrelated clean tasks by testing
   1) whether __GFP_WRITE is set
   2) whether current->nr_dirtied changed recently

b) put dirtier tasks to wait at lower dirty fill levels (~50%) and
   clean tasks at much higher threshold (80%).

To test the "desktop responsiveness on dd to USB" case, I run one dd
to USB (6-7MB/s) and another dd from a huge sparse file (857 MB/s) on
kernel 3.3.0-rc4 plus the to-be-posted dirty reclaim patch. The result
is perfect, I see no single waits of any kind:

$ grep -r . /debug/vm
/debug/vm/nr_reclaim_throttle_clean:0
/debug/vm/nr_reclaim_throttle_kswapd:0
/debug/vm/nr_reclaim_throttle_recent_write:0
/debug/vm/nr_reclaim_throttle_write:0
/debug/vm/nr_congestion_wait:0
/debug/vm/nr_reclaim_wait_congested:0
/debug/vm/nr_reclaim_wait_writeback:0
/debug/vm/nr_migrate_wait_writeback:0

The above numbers are collected by a debug patch that accounts the
wait_on_page_writeback(), congestion_wait(), wait_iff_congested(), and
reclaim_wait() sleeps.

I went further to currently do
- startx
- start a number of applications and switching through all windows in a loop
- run "usemem 6G" (it's a 8G memory box)
- swapon

All goes pretty smooth until swap is turned on. Upon swapon, the
system immediately becomes unusably slow. The wait numbers go up to

/debug/vm/nr_reclaim_throttle_clean:0
/debug/vm/nr_reclaim_throttle_kswapd:1771
/debug/vm/nr_reclaim_throttle_recent_write:0
/debug/vm/nr_reclaim_throttle_write:0
/debug/vm/nr_congestion_wait:33
/debug/vm/nr_reclaim_wait_congested:8626
/debug/vm/nr_reclaim_wait_writeback:0
/debug/vm/nr_migrate_wait_writeback:0

There are over 8k wait_iff_congested() sleeps and 1.7k reclaim_wait() sleeps
for kswapd. I'm a bit surprised to see nr_reclaim_throttle_write still
remains 0 while nr_reclaim_throttle_kswapd goes up..

The congestion_wait traces indicate kswapd run into trouble around some time
and triggered a series of congestion_wait()s. This should in turn decrease its
scan priority and triggered the reclaim_wait()s for kswapd.

           <...>-903   [011] ....  1230.617682: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
           <...>-903   [010] ....  1230.905387: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
           <...>-903   [000] ....  1231.016542: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
           <...>-903   [010] ....  1231.042926: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
           <...>-903   [010] ....  1231.131773: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
           <...>-903   [000] ....  1231.143408: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
           <...>-903   [010] ....  1231.165119: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
           <...>-903   [010] ....  1231.170291: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
           <...>-903   [010] ....  1231.273022: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
           <...>-903   [004] ....  1231.359771: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
           <...>-903   [005] ....  1231.457118: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
           <...>-903   [011] ....  1231.514525: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
           <...>-903   [011] ....  1231.515736: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
           <...>-903   [002] ....  1231.556657: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
           <...>-903   [002] ....  1231.591155: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
           <...>-903   [002] ....  1231.592753: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
           <...>-903   [000] ....  1231.597581: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
           <...>-903   [002] ....  1231.614586: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
           <...>-903   [000] ....  1231.639371: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
           <...>-903   [003] ....  1231.653001: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
           <...>-903   [003] ....  1231.666135: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
           <...>-903   [003] ....  1231.676705: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
           <...>-903   [004] ....  1231.715875: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
           <...>-903   [004] ....  1231.717754: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
           <...>-903   [004] ....  1231.718269: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
           <...>-903   [004] ....  1231.796658: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322

The wait_iff_congested() sleeps w/ swap may be explained by clustered
PG_writeback pages and clustered PG_dirty pages. When there are 32 such pages
come back-to-back in the LRU, wait_iff_congested() can easily be
triggered into sleeps. IMHO the conditions wait_iff_congested() based
on is still kind of "local information" which is not very reliable.
The sleeps are not triggered w/o swap because there are now almost no
pageout() calls for file pages, hence no chances to increase
nr_congested and set ZONE_CONGESTED.

> > > > Some global information on the percent
> > > > of dirty/writeback pages in the LRU list may help. Anyway the added
> > > > tests should still be much better than no protection.
> > > > 
> > > 
> > > You can tell how many dirty pages and writeback pages are in the zone
> > > already.
> > 
> > Right. I changed the test to
> > 
> > +       if (nr_pgreclaim && nr_pgreclaim >= (nr_taken >> (DEF_PRIORITY-priority)) &&
> > +           (!global_reclaim(sc) || !zone_dirty_ok(zone)))
> > +               reclaim_wait(HZ/10);
> > 
> > And I'd prefer to use a higher threshold than the default 20% for the
> > above zone_dirty_ok() test, so that when Johannes' zone dirty
> > balancing does the job fine, PG_reclaim based page reclaim throttling
> > won't happen at all.
> > 
> 
> We'd also need to watch that we do not get throttled on small zones
> like ZONE_DMA (which shouldn't happen but still). To detect this if
> it happens, please consider including node and zone information in the
> writeback_reclaim_wait tracepoint. The memcg people might want to be
> able to see the memcg which I guess could be available
> 
> node=[NID|memcg]
> NID if zone >=0
> memcg if zone == -1
> 
> Which is hacky but avoids creating two tracepoints.

I changed it to mm_vmscan_reclaim_wait w/ output format

+       TP_printk("usec_timeout=%u usec_delayed=%u memcg=%u node=%u zone=%u",

Since it's kind of rate limited code path, there is no need to compact
the output. Actually memcg reclaims do have the node/zone ids which
might be helpful information.

> > > > A global wait queue and reclaim_wait() is introduced. The waiters will
> > > > be wakeup when pages are rotated by end_page_writeback() or lru drain.
> > > > 
> > > > I have to say its effectiveness depends on the filesystem... ext4
> > > > and btrfs do fluent IO completions, so reclaim_wait() works pretty
> > > > well:
> > > >               dd-14560 [017] ....  1360.894605: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=10000
> > > >               dd-14560 [017] ....  1360.904456: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=8000
> > > >               dd-14560 [017] ....  1360.908293: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
> > > >               dd-14560 [017] ....  1360.923960: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=15000
> > > >               dd-14560 [017] ....  1360.927810: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
> > > >               dd-14560 [017] ....  1360.931656: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
> > > >               dd-14560 [017] ....  1360.943503: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=10000
> > > >               dd-14560 [017] ....  1360.953289: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=7000
> > > >               dd-14560 [017] ....  1360.957177: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
> > > >               dd-14560 [017] ....  1360.972949: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=15000
> > > > 
> > > > However XFS does IO completions in very large batches (there may be
> > > > only several big IO completions in one second). So reclaim_wait()
> > > > mostly end up waiting to the full HZ/10 timeout:
> > > > 
> > > >               dd-4177  [008] ....   866.367661: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > > >               dd-4177  [010] ....   866.567583: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > > >               dd-4177  [012] ....   866.767458: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > > >               dd-4177  [013] ....   866.867419: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > > >               dd-4177  [008] ....   867.167266: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > > >               dd-4177  [010] ....   867.367168: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > > >               dd-4177  [012] ....   867.818950: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > > >               dd-4177  [013] ....   867.918905: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > > >               dd-4177  [013] ....   867.971657: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=52000
> > > >               dd-4177  [013] ....   867.971812: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=0
> > > >               dd-4177  [008] ....   868.355700: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > > >               dd-4177  [010] ....   868.700515: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > > > 
> > > 
> > > And where people will get hit by regressions in this area is writing to
> > > vfat and in more rare cases ntfs on USB stick.
> > 
> > vfat IO completions seem to lie somewhere between ext4 and xfs:
> > 
> >            <...>-46385 [010] .... 143570.714470: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> >            <...>-46385 [008] .... 143570.752391: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=12000
> >            <...>-46385 [008] .... 143570.937327: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=52000
> >            <...>-46385 [010] .... 143571.160252: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> >            <...>-46385 [011] .... 143571.286197: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> >            <...>-46385 [008] .... 143571.329644: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=15000
> >            <...>-46385 [008] .... 143571.475433: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=18000
> >            <...>-46385 [008] .... 143571.653461: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=52000
> >            <...>-46385 [008] .... 143571.839949: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=56000
> >            <...>-46385 [010] .... 143572.060816: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> >            <...>-46385 [011] .... 143572.185754: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> >            <...>-46385 [008] .... 143572.212522: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=1000
> >            <...>-46385 [008] .... 143572.217825: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
> >            <...>-46385 [008] .... 143572.312395: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=91000
> >            <...>-46385 [008] .... 143572.315122: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=1000
> >            <...>-46385 [009] .... 143572.433630: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> >            <...>-46385 [010] .... 143572.534569: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> >  
> 
> Ok. It's interesting to note that we are stalling a lot there - roughly
> 30ms every second. As long as it's the writer, that's fine. If it's
> firefox, it will create bug reports :)

Good news is, we can reasonably tell firefox from heavy writers with
the help of __GFP_WRITE/->nr_dirtied indicators  :-)

> > > > <SNIP>
> > > > @@ -813,6 +815,10 @@ static unsigned long shrink_page_list(st
> > > >  
> > > >  		if (PageWriteback(page)) {
> > > >  			nr_writeback++;
> > > > +			if (PageReclaim(page))
> > > > +				nr_pgreclaim++;
> > > > +			else
> > > > +				SetPageReclaim(page);
> > > >  			/*
> > > 
> > > This check is unexpected. We already SetPageReclaim when queuing pages for
> > > IO from reclaim context and if dirty pages are encountered during the LRU
> > > scan that cannot be queued for IO. How often is it that nr_pgreclaim !=
> > > nr_writeback and by how much do they differ?
> > 
> > Quite often, I suspect. The pageout writeback works do 1-8MB write
> > around which may start I/O a bit earlier than the covered pages are
> > encountered by page reclaim. ext4 forces 128MB write chunk size, which
> > further increases the opportunities.
> > 
> 
> Ok, thanks for the clarification. Stick a wee comment on it please.

OK, added a simple comment:

                if (PageWriteback(page)) {
                        nr_writeback++;
                        /*
+                        * The pageout works do write around which may put
+                        * close-to-LRU-tail pages to writeback a bit earlier.
+                        */
+                       if (PageReclaim(page))
+                               nr_pgreclaim++;
+                       else
+                               SetPageReclaim(page);

> > > >  			 * Synchronous reclaim cannot queue pages for
> > > >  			 * writeback due to the possibility of stack overflow
> > > > @@ -874,12 +880,15 @@ static unsigned long shrink_page_list(st
> > > >  			nr_dirty++;
> > > >  
> > > >  			/*
> > > > -			 * Only kswapd can writeback filesystem pages to
> > > > -			 * avoid risk of stack overflow but do not writeback
> > > > -			 * unless under significant pressure.
> > > > +			 * run into the visited page again: we are scanning
> > > > +			 * faster than the flusher can writeout dirty pages
> > > >  			 */
> > > 
> > > which in itself is not an abnormal condition. We get into this situation
> > > when writing to USB. Dirty throttling stops too much memory getting dirtied
> > > but that does not mean we should throttle instead of reclaiming clean pages.
> > > 
> > > That's why I worry that if this is aimed at fixing a memcg problem, it
> > > will have the impact of making interactive performance on normal systems
> > > worse.
> > 
> > You are right. This patch only addresses the pageout I/O efficiency
> > and dirty throttling problems for a fully dirtied LRU. Next step, I'll
> > think about the interactive performance problem for a less dirtied LRU.
> > 
> 
> Ok, thanks.
> 
> > > <SNIP>
> > >
> > > If the intention is to avoid memcg going OOM prematurely, the
> > > nr_pgreclaim value needs to be treated at a higher level that records
> > > how many PageReclaim pages were encountered. If no progress was made
> > > because all the pages were PageReclaim, then throttle and return 1 to
> > > the page allocator where it will retry the allocation without going OOM
> > > after some pages have been cleaned and reclaimed.
> >  
> > Agreed in general, but changed to this test for now, which is made a
> > bit more global wise with the use of zone_dirty_ok().
> > 
> 
> Ok, sure. We know what to look out for and where unrelated regressions
> might get introduced.
> 
> > memcg is ignored due to no dirty accounting (Greg has the patch though).
> > And even zone_dirty_ok() may be inaccurate for the global reclaim, if
> > some memcgs are skipped by the global reclaim by the memcg soft limit.
> > 
> > But anyway, it's a handy hack for now. I'm looking into some more
> > radical changes to put most dirty/writeback pages into a standalone
> > LRU list (in addition to your LRU_IMMEDIATE, which I think is a good
> > idea) for addressing the clustered way they tend to lie in the
> > inactive LRU list.
> > 
> > +       if (nr_pgreclaim && nr_pgreclaim >= (nr_taken >> (DEF_PRIORITY-priority)) &&
> > +           (!global_reclaim(sc) || !zone_dirty_ok(zone)))
> > +               reclaim_wait(HZ/10);
> > 
> 
> This should make it harder to get stalled. Your tracepoint should help
> us catch if it happens unnecessarily.
 
Yeah ;) FYI the above zone_dirty_ok() etc. tests are further expanded to  

+       /*
+        * If reclaimed any pages, it's safe from busy scanning. Otherwise when
+        * encountered PG_reclaim pages or writeback work queue congested,
+        * consider I/O throttling. Try to throttle only the dirtier tasks by
+        * honouring higher throttle thresholds to kswapd and other clean tasks.
+        */
+       if (!nr_reclaimed && nr_pgreclaim &&
+           should_throttle_dirty(mz, sc, priority))
+               reclaim_wait(mz, HZ/10);

I choose to not trust the exact number of nr_pgreclaim because it's
kind of *unreliable* local information. Instead, the reclaim priority
looks a better indicator of a long contiguous run of hard-to-reclaim
pages which will be taken into account in should_throttle_dirty().

> > ---
> > Subject: writeback: introduce the pageout work
> > Date: Thu Jul 29 14:41:19 CST 2010
> > 
> > This relays file pageout IOs to the flusher threads.
> > 
> > The ultimate target is to gracefully handle the LRU lists full of
> > dirty/writeback pages.
> > 
> 
> It would be worth mentioning in the changelog that this is much more
> important now that page reclaim generally does not writeout filesystem-backed
> pages.

OK. Good point.

> > 1) I/O efficiency
> > 
> > The flusher will piggy back the nearby ~10ms worth of dirty pages for I/O.
> > 
> > This takes advantage of the time/spacial locality in most workloads: the
> > nearby pages of one file are typically populated into the LRU at the same
> > time, hence will likely be close to each other in the LRU list. Writing
> > them in one shot helps clean more pages effectively for page reclaim.
> > 
> > 2) OOM avoidance and scan rate control
> > 
> > Typically we do LRU scan w/o rate control and quickly get enough clean
> > pages for the LRU lists not full of dirty pages.
> > 
> > Or we can still get a number of freshly cleaned pages (moved to LRU tail
> > by end_page_writeback()) when the queued pageout I/O is completed within
> > tens of milli-seconds.
> > 
> > However if the LRU list is small and full of dirty pages, it can be
> > quickly fully scanned and go OOM before the flusher manages to clean
> > enough pages.
> > 
> 
> It's worth pointing out here that generally this does not happen for global
> reclaim which does dirty throttling but happens easily with memcg LRUs.

Done.

> > A simple yet reliable scheme is employed to avoid OOM and keep scan rate
> > in sync with the I/O rate:
> > 
> > 	if (PageReclaim(page))
> > 		congestion_wait(HZ/10);
> > 
> 
> This comment is stale now.

Yeah, will update to

        if (encountered PG_reclaim pages)
                do some throttle wait

followed by introduction of the basic idea as well as further
considerations for interactive performance.

> > PG_reclaim plays the key role. When dirty pages are encountered, we
> > queue I/O for it,
> 
> This is misleading. The process that encounters the dirty page does
> not queue the page for IO unless it is kswapd scanning at high priority
> (currently anyway, you patch changes it). The process that finds the page
> queues work for flusher threads that will queue the actual I/O for it at
> some unknown time in the future.

You are right. Will change to "queue pageout writeback work for it".

> > set PG_reclaim and put it back to the LRU head.
> > So if PG_reclaim pages are encountered again, it means the dirty page
> > has not yet been cleaned by the flusher after a full zone scan. It
> > indicates we are scanning more fast than I/O and shall take a snap.
> > 
> 
> This is also slightly misleading because the page can be encountered
> after rescanning the inactive list, not necessarily a full zone scan but
> it's a minor point.

Yeah, "inactive list" is more precise. Updated.

> > The runtime behavior on a fully dirtied small LRU list would be:
> > It will start with a quick scan of the list, queuing all pages for I/O.
> > Then the scan will be slowed down by the PG_reclaim pages *adaptively*
> > to match the I/O bandwidth.
> > 
> > 3) writeback work coordinations
> > 
> > To avoid memory allocations at page reclaim, a mempool for struct
> > wb_writeback_work is created.
> > 
> > wakeup_flusher_threads() is removed because it can easily delay the
> > more oriented pageout works and even exhaust the mempool reservations.
> > It's also found to not I/O efficient by frequently submitting writeback
> > works with small ->nr_pages.
> > 
> > Background/periodic works will quit automatically, so as to clean the
> > pages under reclaim ASAP. However for now the sync work can still block
> > us for long time.
> > 
> > Jan Kara: limit the search scope. Note that the limited search and work
> > pool is not a big problem: 1000 IOs under flight are typically more than
> > enough to saturate the disk. And the overheads of searching in the work
> > list didn't even show up in the perf report.
> > 
> > 4) test case
> > 
> > Run 2 dd tasks in a 100MB memcg (a very handy test case from Greg Thelen):
> > 
> > 	mkdir /cgroup/x
> > 	echo 100M > /cgroup/x/memory.limit_in_bytes
> > 	echo $$ > /cgroup/x/tasks
> > 
> > 	for i in `seq 2`
> > 	do
> > 		dd if=/dev/zero of=/fs/f$i bs=1k count=1M &
> > 	done
> > 
> > Before patch, the dd tasks are quickly OOM killed.
> > After patch, they run well with reasonably good performance and overheads:
> > 
> > 1073741824 bytes (1.1 GB) copied, 22.2196 s, 48.3 MB/s
> > 1073741824 bytes (1.1 GB) copied, 22.4675 s, 47.8 MB/s
> > 
> > iostat -kx 1
> > 
> > Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
> > sda               0.00     0.00    0.00  178.00     0.00 89568.00  1006.38    74.35  417.71   4.80  85.40
> > sda               0.00     2.00    0.00  191.00     0.00 94428.00   988.77    53.34  219.03   4.34  82.90
> > sda               0.00    20.00    0.00  196.00     0.00 97712.00   997.06    71.11  337.45   4.77  93.50
> > sda               0.00     5.00    0.00  175.00     0.00 84648.00   967.41    54.03  316.44   5.06  88.60
> > sda               0.00     0.00    0.00  186.00     0.00 92432.00   993.89    56.22  267.54   5.38 100.00
> > sda               0.00     1.00    0.00  183.00     0.00 90156.00   985.31    37.99  325.55   4.33  79.20
> > sda               0.00     0.00    0.00  175.00     0.00 88692.00  1013.62    48.70  218.43   4.69  82.10
> > sda               0.00     0.00    0.00  196.00     0.00 97528.00   995.18    43.38  236.87   5.10 100.00
> > sda               0.00     0.00    0.00  179.00     0.00 88648.00   990.48    45.83  285.43   5.59 100.00
> > sda               0.00     0.00    0.00  178.00     0.00 88500.00   994.38    28.28  158.89   4.99  88.80
> > sda               0.00     0.00    0.00  194.00     0.00 95852.00   988.16    32.58  167.39   5.15 100.00
> > sda               0.00     2.00    0.00  215.00     0.00 105996.00   986.01    41.72  201.43   4.65 100.00
> > sda               0.00     4.00    0.00  173.00     0.00 84332.00   974.94    50.48  260.23   5.76  99.60
> > sda               0.00     0.00    0.00  182.00     0.00 90312.00   992.44    36.83  212.07   5.49 100.00
> > sda               0.00     8.00    0.00  195.00     0.00 95940.50   984.01    50.18  221.06   5.13 100.00
> > sda               0.00     1.00    0.00  220.00     0.00 108852.00   989.56    40.99  202.68   4.55 100.00
> > sda               0.00     2.00    0.00  161.00     0.00 80384.00   998.56    37.19  268.49   6.21 100.00
> > sda               0.00     4.00    0.00  182.00     0.00 90830.00   998.13    50.58  239.77   5.49 100.00
> > sda               0.00     0.00    0.00  197.00     0.00 94877.00   963.22    36.68  196.79   5.08 100.00
> > 
> > avg-cpu:  %user   %nice %system %iowait  %steal   %idle
> >            0.25    0.00   15.08   33.92    0.00   50.75
> >            0.25    0.00   14.54   35.09    0.00   50.13
> >            0.50    0.00   13.57   32.41    0.00   53.52
> >            0.50    0.00   11.28   36.84    0.00   51.38
> >            0.50    0.00   15.75   32.00    0.00   51.75
> >            0.50    0.00   10.50   34.00    0.00   55.00
> >            0.50    0.00   17.63   27.46    0.00   54.41
> >            0.50    0.00   15.08   30.90    0.00   53.52
> >            0.50    0.00   11.28   32.83    0.00   55.39
> >            0.75    0.00   16.79   26.82    0.00   55.64
> >            0.50    0.00   16.08   29.15    0.00   54.27
> >            0.50    0.00   13.50   30.50    0.00   55.50
> >            0.50    0.00   14.32   35.18    0.00   50.00
> >            0.50    0.00   12.06   33.92    0.00   53.52
> >            0.50    0.00   17.29   30.58    0.00   51.63
> >            0.50    0.00   15.08   29.65    0.00   54.77
> >            0.50    0.00   12.53   29.32    0.00   57.64
> >            0.50    0.00   15.29   31.83    0.00   52.38
> > 
> > The global dd numbers for comparison:
> > 
> > Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
> > sda               0.00     0.00    0.00  189.00     0.00 95752.00  1013.25   143.09  684.48   5.29 100.00
> > sda               0.00     0.00    0.00  208.00     0.00 105480.00  1014.23   143.06  733.29   4.81 100.00
> > sda               0.00     0.00    0.00  161.00     0.00 81924.00  1017.69   141.71  757.79   6.21 100.00
> > sda               0.00     0.00    0.00  217.00     0.00 109580.00  1009.95   143.09  749.55   4.61 100.10
> > sda               0.00     0.00    0.00  187.00     0.00 94728.00  1013.13   144.31  773.67   5.35 100.00
> > sda               0.00     0.00    0.00  189.00     0.00 95752.00  1013.25   144.14  742.00   5.29 100.00
> > sda               0.00     0.00    0.00  177.00     0.00 90032.00  1017.31   143.32  656.59   5.65 100.00
> > sda               0.00     0.00    0.00  215.00     0.00 108640.00  1010.60   142.90  817.54   4.65 100.00
> > sda               0.00     2.00    0.00  166.00     0.00 83858.00  1010.34   143.64  808.61   6.02 100.00
> > sda               0.00     0.00    0.00  186.00     0.00 92813.00   997.99   141.18  736.95   5.38 100.00
> > sda               0.00     0.00    0.00  206.00     0.00 104456.00  1014.14   146.27  729.33   4.85 100.00
> > sda               0.00     0.00    0.00  213.00     0.00 107024.00  1004.92   143.25  705.70   4.69 100.00
> > sda               0.00     0.00    0.00  188.00     0.00 95748.00  1018.60   141.82  764.78   5.32 100.00
> > 
> > avg-cpu:  %user   %nice %system %iowait  %steal   %idle
> >            0.51    0.00   11.22   52.30    0.00   35.97
> >            0.25    0.00   10.15   52.54    0.00   37.06
> >            0.25    0.00    5.01   56.64    0.00   38.10
> >            0.51    0.00   15.15   43.94    0.00   40.40
> >            0.25    0.00   12.12   48.23    0.00   39.39
> >            0.51    0.00   11.20   53.94    0.00   34.35
> >            0.26    0.00    9.72   51.41    0.00   38.62
> >            0.76    0.00    9.62   50.63    0.00   38.99
> >            0.51    0.00   10.46   53.32    0.00   35.71
> >            0.51    0.00    9.41   51.91    0.00   38.17
> >            0.25    0.00   10.69   49.62    0.00   39.44
> >            0.51    0.00   12.21   52.67    0.00   34.61
> >            0.51    0.00   11.45   53.18    0.00   34.86
> > 
> > XXX: commit NFS unstable pages via write_inode()
> > XXX: the added congestion_wait() may be undesirable in some situations
> > 
> 
> This second XXX may also not be redundant.

Sure. Removed.

> > CC: Jan Kara <jack@suse.cz>
> > CC: Mel Gorman <mgorman@suse.de>
> > Acked-by: Rik van Riel <riel@redhat.com>
> > CC: Greg Thelen <gthelen@google.com>
> > CC: Minchan Kim <minchan.kim@gmail.com>
> > Signed-off-by: Wu Fengguang <fengguang.wu@intel.com>
> > ---
> >  fs/fs-writeback.c                |  169 ++++++++++++++++++++++++++++-
> >  include/linux/backing-dev.h      |    2 
> >  include/linux/writeback.h        |    4 
> >  include/trace/events/writeback.h |   19 ++-
> >  mm/backing-dev.c                 |   35 ++++++
> >  mm/swap.c                        |    1 
> >  mm/vmscan.c                      |   32 +++--
> >  7 files changed, 245 insertions(+), 17 deletions(-)
> > 
> > - move congestion_wait() out of the page lock: it's blocking btrfs lock_delalloc_pages()
> > 
> > --- linux.orig/include/linux/backing-dev.h	2012-02-14 20:11:21.000000000 +0800
> > +++ linux/include/linux/backing-dev.h	2012-02-15 12:34:24.000000000 +0800
> > @@ -304,6 +304,8 @@ void clear_bdi_congested(struct backing_
> >  void set_bdi_congested(struct backing_dev_info *bdi, int sync);
> >  long congestion_wait(int sync, long timeout);
> >  long wait_iff_congested(struct zone *zone, int sync, long timeout);
> > +long reclaim_wait(long timeout);
> > +void reclaim_rotated(void);
> >  
> >  static inline bool bdi_cap_writeback_dirty(struct backing_dev_info *bdi)
> >  {
> > --- linux.orig/mm/backing-dev.c	2012-02-14 20:11:21.000000000 +0800
> > +++ linux/mm/backing-dev.c	2012-02-15 12:34:19.000000000 +0800
> > @@ -873,3 +873,38 @@ out:
> >  	return ret;
> >  }
> >  EXPORT_SYMBOL(wait_iff_congested);
> > +
> > +static DECLARE_WAIT_QUEUE_HEAD(reclaim_wqh);
> > +
> 
> Should this be declared on a per-NUMA node basis to avoid throttling on one
> node being woken up by activity on an unrelated node?  reclaim_rorated()
> is called from a context that has a page so looking up the waitqueue would
> be easy. Grep for place that initialise kswapd_wait and the initialisation
> code will be easier although watch that if a node is hot-removed that
> the queue is woken.

OK I've changed reclaim_wqh to per-NUMA node wqh. It may be further
expanded to per-memcg wqh in similar way.

But note that it will now only wake the node corresponding to one
random page in the pagevec. It should work well enough in practice,
though.

> > +/**
> > + * reclaim_wait - wait for some pages being rotated to the LRU tail
> > + * @timeout: timeout in jiffies
> > + *
> > + * Wait until @timeout, or when some (typically PG_reclaim under writeback)
> > + * pages rotated to the LRU so that page reclaim can make progress.
> > + */
> > +long reclaim_wait(long timeout)
> > +{
> > +	long ret;
> > +	unsigned long start = jiffies;
> > +	DEFINE_WAIT(wait);
> > +
> > +	prepare_to_wait(&reclaim_wqh, &wait, TASK_KILLABLE);
> > +	ret = io_schedule_timeout(timeout);
> > +	finish_wait(&reclaim_wqh, &wait);
> > +
> > +	trace_writeback_reclaim_wait(jiffies_to_usecs(timeout),
> > +				     jiffies_to_usecs(jiffies - start));
> > +
> > +	return ret;
> > +}
> > +EXPORT_SYMBOL(reclaim_wait);
> > +
> 
> Why do we export this? Only vmscan.c is calling it and I'm scratching my
> head trying to figure out why a kernel module would want to call it.

Err a typical copy&paste error... Removed this and further moved
reclaim_wait()/reclaim_rotated() to vmscan.c.

> > +void reclaim_rotated()
> > +{
> 
> style nit
> 
> void reclaim_rotated(void)

Updated, to...

        void reclaim_rotated(struct page *page)

which will wakeup tasks from the wqh

        &NODE_DATA(page_to_nid(page))->reclaim_wait

> > +	wait_queue_head_t *wqh = &reclaim_wqh;
> > +
> > +	if (waitqueue_active(wqh))
> > +		wake_up(wqh);
> > +}
> > +
> > --- linux.orig/mm/swap.c	2012-02-14 20:11:21.000000000 +0800
> > +++ linux/mm/swap.c	2012-02-15 12:27:35.000000000 +0800
> > @@ -253,6 +253,7 @@ static void pagevec_move_tail(struct pag
> >  
> >  	pagevec_lru_move_fn(pvec, pagevec_move_tail_fn, &pgmoved);
> >  	__count_vm_events(PGROTATED, pgmoved);
> > +	reclaim_rotated();
> >  }
> >  
> >  /*
> > --- linux.orig/mm/vmscan.c	2012-02-14 20:11:21.000000000 +0800
> > +++ linux/mm/vmscan.c	2012-02-16 17:23:17.000000000 +0800
> > @@ -767,7 +767,8 @@ static unsigned long shrink_page_list(st
> >  				      struct scan_control *sc,
> >  				      int priority,
> >  				      unsigned long *ret_nr_dirty,
> > -				      unsigned long *ret_nr_writeback)
> > +				      unsigned long *ret_nr_writeback,
> > +				      unsigned long *ret_nr_pgreclaim)
> >  {
> >  	LIST_HEAD(ret_pages);
> >  	LIST_HEAD(free_pages);
> > @@ -776,6 +777,7 @@ static unsigned long shrink_page_list(st
> >  	unsigned long nr_congested = 0;
> >  	unsigned long nr_reclaimed = 0;
> >  	unsigned long nr_writeback = 0;
> > +	unsigned long nr_pgreclaim = 0;
> >  
> >  	cond_resched();
> >  
> > @@ -813,6 +815,10 @@ static unsigned long shrink_page_list(st
> >  
> >  		if (PageWriteback(page)) {
> >  			nr_writeback++;
> > +			if (PageReclaim(page))
> > +				nr_pgreclaim++;
> > +			else
> > +				SetPageReclaim(page);
> >  			/*
> >  			 * Synchronous reclaim cannot queue pages for
> >  			 * writeback due to the possibility of stack overflow
> > @@ -874,12 +880,15 @@ static unsigned long shrink_page_list(st
> >  			nr_dirty++;
> >  
> >  			/*
> > -			 * Only kswapd can writeback filesystem pages to
> > -			 * avoid risk of stack overflow but do not writeback
> > -			 * unless under significant pressure.
> > +			 * run into the visited page again: we are scanning
> > +			 * faster than the flusher can writeout dirty pages
> >  			 */
> > -			if (page_is_file_cache(page) &&
> > -					(!current_is_kswapd() || priority >= DEF_PRIORITY - 2)) {
> > +			if (page_is_file_cache(page) && PageReclaim(page)) {
> > +				nr_pgreclaim++;
> > +				goto keep_locked;
> > +			}
> 
> This change means that kswapd is no longer doing any writeback from page
> reclaim.

Not really skipping all pageout()s. Here we skip PG_reclaim dirty
pages because there are already pageout works queued for them.

> Was that intended because it's not discussed in the changelog. I
> know writeback from kswapd is poor in terms of IO performance but it's a
> last resort for freeing a page when reclaim is in trouble. If we are to
> disable it and depend 100% on the flusher threads, it should be in its
> own patch for bisection reasons if nothing else.

I'm actually totally with your points.  The patch will still default
to pageout() if failed to queue pageout work for the page.

But yes, there is the possibility of pageout works delayed by some
sync works for a dozen of seconds. In this case the sync work may also
hit some PG_reclaim pages and do the clean-and-rotate job for us. But
it would certainly be much better to have some kind of guarantee...

> > +			if (page_is_file_cache(page) && mapping &&
> > +			    flush_inode_page(mapping, page, false) >= 0) {
> >  				/*
> >  				 * Immediately reclaim when written back.
> >  				 * Similar in principal to deactivate_page()
> > @@ -1028,6 +1037,7 @@ keep_lumpy:
> >  	count_vm_events(PGACTIVATE, pgactivate);
> >  	*ret_nr_dirty += nr_dirty;
> >  	*ret_nr_writeback += nr_writeback;
> > +	*ret_nr_pgreclaim += nr_pgreclaim;
> >  	return nr_reclaimed;
> >  }
> >  
> > @@ -1509,6 +1519,7 @@ shrink_inactive_list(unsigned long nr_to
> >  	unsigned long nr_file;
> >  	unsigned long nr_dirty = 0;
> >  	unsigned long nr_writeback = 0;
> > +	unsigned long nr_pgreclaim = 0;
> >  	isolate_mode_t reclaim_mode = ISOLATE_INACTIVE;
> >  	struct zone *zone = mz->zone;
> >  
> > @@ -1559,13 +1570,13 @@ shrink_inactive_list(unsigned long nr_to
> >  	spin_unlock_irq(&zone->lru_lock);
> >  
> >  	nr_reclaimed = shrink_page_list(&page_list, mz, sc, priority,
> > -						&nr_dirty, &nr_writeback);
> > +				&nr_dirty, &nr_writeback, &nr_pgreclaim);
> >  
> >  	/* Check if we should syncronously wait for writeback */
> >  	if (should_reclaim_stall(nr_taken, nr_reclaimed, priority, sc)) {
> >  		set_reclaim_mode(priority, sc, true);
> >  		nr_reclaimed += shrink_page_list(&page_list, mz, sc,
> > -					priority, &nr_dirty, &nr_writeback);
> > +			priority, &nr_dirty, &nr_writeback, &nr_pgreclaim);
> >  	}
> >  
> >  	spin_lock_irq(&zone->lru_lock);
> > @@ -1608,6 +1619,9 @@ shrink_inactive_list(unsigned long nr_to
> >  	 */
> >  	if (nr_writeback && nr_writeback >= (nr_taken >> (DEF_PRIORITY-priority)))
> >  		wait_iff_congested(zone, BLK_RW_ASYNC, HZ/10);
> > +	if (nr_pgreclaim && nr_pgreclaim >= (nr_taken >> (DEF_PRIORITY-priority)) &&
> > +	    (!global_reclaim(sc) || !zone_dirty_ok(zone)))
> > +		reclaim_wait(HZ/10);
> >  
> 
> I prefer this but it would be nice if there was a comment explaining it
> or at least expand the comment explaining how nr_writeback can lead to
> wait_iff_congested() being called.

Sure, done.

> >  	trace_mm_vmscan_lru_shrink_inactive(zone->zone_pgdat->node_id,
> >  		zone_idx(zone),
> > @@ -2382,8 +2396,6 @@ static unsigned long do_try_to_free_page
> >  		 */
> >  		writeback_threshold = sc->nr_to_reclaim + sc->nr_to_reclaim / 2;
> >  		if (total_scanned > writeback_threshold) {
> > -			wakeup_flusher_threads(laptop_mode ? 0 : total_scanned,
> > -						WB_REASON_TRY_TO_FREE_PAGES);
> >  			sc->may_writepage = 1;
> >  		}
> >  
> > --- linux.orig/fs/fs-writeback.c	2012-02-14 20:11:21.000000000 +0800
> > +++ linux/fs/fs-writeback.c	2012-02-15 12:27:35.000000000 +0800
> > @@ -41,6 +41,8 @@ struct wb_writeback_work {
> >  	long nr_pages;
> >  	struct super_block *sb;
> >  	unsigned long *older_than_this;
> > +	struct inode *inode;
> > +	pgoff_t offset;
> >  	enum writeback_sync_modes sync_mode;
> >  	unsigned int tagged_writepages:1;
> >  	unsigned int for_kupdate:1;
> > @@ -65,6 +67,27 @@ struct wb_writeback_work {
> >   */
> >  int nr_pdflush_threads;
> >  
> > +static mempool_t *wb_work_mempool;
> > +
> > +static void *wb_work_alloc(gfp_t gfp_mask, void *pool_data)
> > +{
> > +	/*
> > +	 * bdi_flush_inode_range() may be called on page reclaim
> > +	 */
> > +	if (current->flags & PF_MEMALLOC)
> > +		return NULL;
> > +
> 
> This check is why I worry about kswapd being unable to write pages at
> all. If the mempool is depleted for whatever reason, reclaim has no way
> of telling the flushers what work is needed or waking them. Potentially,
> we could be waiting a long time for pending flusher work to complete to
> free up a slot.  I recognise it may not be bad in practice because the
> pool is large and other work will be completing but it's why kswapd not
> writing back pages should be in its own patch.

See above, the patch still falls back to pageout() if we fail to queue
the work. The main problem is, it's possible for the *queued* work to
be delayed for long time by some other writeback works. It should be
trivial to make pageout works the No.1 priority for the flushers, but
then that may risk delaying sync() works for long time...

> > +	return kmalloc(sizeof(struct wb_writeback_work), gfp_mask);
> > +}
> > +
> > +static __init int wb_work_init(void)
> > +{
> > +	wb_work_mempool = mempool_create(1024,
> > +					 wb_work_alloc, mempool_kfree, NULL);
> > +	return wb_work_mempool ? 0 : -ENOMEM;
> > +}
> > +fs_initcall(wb_work_init);
> > +
> >  /**
> >   * writeback_in_progress - determine whether there is writeback in progress
> >   * @bdi: the device's backing_dev_info structure.
> > @@ -129,7 +152,7 @@ __bdi_start_writeback(struct backing_dev
> >  	 * This is WB_SYNC_NONE writeback, so if allocation fails just
> >  	 * wakeup the thread for old dirty data writeback
> >  	 */
> > -	work = kzalloc(sizeof(*work), GFP_ATOMIC);
> > +	work = mempool_alloc(wb_work_mempool, GFP_NOWAIT);
> >  	if (!work) {
> >  		if (bdi->wb.task) {
> >  			trace_writeback_nowork(bdi);
> > @@ -138,6 +161,7 @@ __bdi_start_writeback(struct backing_dev
> >  		return;
> >  	}
> >  
> > +	memset(work, 0, sizeof(*work));
> >  	work->sync_mode	= WB_SYNC_NONE;
> >  	work->nr_pages	= nr_pages;
> >  	work->range_cyclic = range_cyclic;
> > @@ -186,6 +210,125 @@ void bdi_start_background_writeback(stru
> >  	spin_unlock_bh(&bdi->wb_lock);
> >  }
> >  
> > +static bool extend_writeback_range(struct wb_writeback_work *work,
> > +				   pgoff_t offset,
> > +				   unsigned long write_around_pages)
> > +{
> 
> comment on what this function is for and what the return values mean.
> 
> "returns true if the wb_writeback_work now encompasses the request"
> 
> or something

Here it goes:

/*
 * Check if @work already covers @offset, or try to extend it to cover @offset.
 * Returns true if the wb_writeback_work now encompasses the requested offset.
 */
 
> > +	pgoff_t end = work->offset + work->nr_pages;
> > +
> > +	if (offset >= work->offset && offset < end)
> > +		return true;
> > +
> 
> This does not ensure that the full span of
> offset -> offset+write_around_pages is encompassed by work. All it
> checks is that the start of the requested range is going to be handled.

Oh this function does not work as you expected.

I'd better change the "write_around_pages" parameter name to "unit".

Every pageout work starts with [begin, begin+unit] and each time it's
extended, the added length will be N*unit.

extend_writeback_range() only aims to extend the work to encompass the
single page at @offset. "unit" merely serves as the extending unit.
 
> I guess it's ok because the page reclaims cares about is covered and
> avoids a situation where too much IO is being queued. It's unclear if
> this is what you intended though because you check for too much IO being
> queued in the next block.

Yeah both the number of pending pageout works and the chunk size of
each work will be limited.

> > +	/*
> > +	 * for sequential workloads with good locality, include up to 8 times
> > +	 * more data in one chunk
> > +	 */
> > +	if (work->nr_pages >= 8 * write_around_pages)
> > +		return false;
> > +
> > +	/* the unsigned comparison helps eliminate one compare */
> > +	if (work->offset - offset < write_around_pages) {
> > +		work->nr_pages += write_around_pages;
> > +		work->offset -= write_around_pages;
> > +		return true;
> > +	}
> > +
> > +	if (offset - end < write_around_pages) {
> > +		work->nr_pages += write_around_pages;
> > +		return true;
> > +	}
> > +
> > +	return false;
> > +}
> > +
> > +/*
> > + * schedule writeback on a range of inode pages.
> > + */
> > +static struct wb_writeback_work *
> > +bdi_flush_inode_range(struct backing_dev_info *bdi,
> > +		      struct inode *inode,
> > +		      pgoff_t offset,
> > +		      pgoff_t len,
> > +		      bool wait)
> > +{
> > +	struct wb_writeback_work *work;
> > +
> > +	if (!igrab(inode))
> > +		return ERR_PTR(-ENOENT);
> > +
> 
> Explain why the igrab is necessary. I think it's because we are calling
> this from page reclaim context and the only thing pinning the
> address_space is the page lock . If I'm right, it should be made clear
> in the comment for bdi_flush_inode_range that this should only be called
> from page reclaim context. Maybe even VM_BUG_ON if
> !(current->flags & PF_MEMALLOC)?

OK, I added this comment.

        /*
         * Grab the inode until work execution time. We are calling this from
         * page reclaim context and the only thing pinning the address_space
         * for the moment is the page lock.
         */

Maybe not VM_BUG_ON, since it won't hurt that much if called by
someone else ;)

> > +	work = mempool_alloc(wb_work_mempool, wait ? GFP_NOIO : GFP_NOWAIT);
> > +	if (!work) {
> > +		trace_printk("wb_work_mempool alloc fail\n");
> > +		return ERR_PTR(-ENOMEM);
> > +	}
> > +
> > +	memset(work, 0, sizeof(*work));
> > +	work->sync_mode		= WB_SYNC_NONE;
> > +	work->inode		= inode;
> > +	work->offset		= offset;
> > +	work->nr_pages		= len;
> > +	work->reason		= WB_REASON_PAGEOUT;
> > +
> > +	bdi_queue_work(bdi, work);
> > +
> > +	return work;
> > +}
> > +
> > +/*
> > + * Called by page reclaim code to flush the dirty page ASAP. Do write-around to
> > + * improve IO throughput. The nearby pages will have good chance to reside in
> > + * the same LRU list that vmscan is working on, and even close to each other
> > + * inside the LRU list in the common case of sequential read/write.
> > + *
> > + * ret > 0: success, found/reused a previous writeback work
> > + * ret = 0: success, allocated/queued a new writeback work
> > + * ret < 0: failed
> > + */
> > +long flush_inode_page(struct address_space *mapping,
> > +		      struct page *page,
> > +		      bool wait)
> > +{
> > +	struct backing_dev_info *bdi = mapping->backing_dev_info;
> > +	struct inode *inode = mapping->host;
> > +	struct wb_writeback_work *work;
> > +	unsigned long write_around_pages;
> > +	pgoff_t offset = page->index;
> > +	int i;
> > +	long ret = 0;
> > +
> > +	if (unlikely(!inode))
> > +		return -ENOENT;
> > +
> > +	/*
> > +	 * piggy back 8-15ms worth of data
> > +	 */
> > +	write_around_pages = bdi->avg_write_bandwidth + MIN_WRITEBACK_PAGES;
> > +	write_around_pages = rounddown_pow_of_two(write_around_pages) >> 6;
> > +
> > +	i = 1;
> > +	spin_lock_bh(&bdi->wb_lock);
> > +	list_for_each_entry_reverse(work, &bdi->work_list, list) {
> > +		if (work->inode != inode)
> > +			continue;
> > +		if (extend_writeback_range(work, offset, write_around_pages)) {
> > +			ret = i;
> > +			break;
> > +		}
> > +		if (i++ > 100)	/* limit search depth */
> > +			break;
> 
> No harm to move Jan's comment on depth limit search to here adding why
> 100 is as good as number as any to use.

Good idea. Here is the refined work pool throttling scheme:

/*
 * Tailored for vmscan which may submit lots of pageout works. The page reclaim
 * will try to slow down the pageout work submission rate when the queue size
 * grows to LOTS_OF_WRITEBACK_WORKS. flush_inode_page() will accordingly limit
 * its search depth to (2 * LOTS_OF_WRITEBACK_WORKS).
 *
 * Note that the limited search and work pool is not a big problem: 1024 IOs
 * under flight are typically more than enough to saturate the disk. And the
 * overheads of searching in the work list didn't even show up in perf report.
 */
#define WB_WORK_MEMPOOL_SIZE            1024
#define LOTS_OF_WRITEBACK_WORKS         (WB_WORK_MEMPOOL_SIZE / 8)

Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: reclaim the LRU lists full of dirty/writeback pages

[KAMEZAWA Hiroyuki]

On Tue, 14 Feb 2012 21:18:12 +0800
Wu Fengguang <fengguang.wu@intel.com> wrote:

> 
> --- linux.orig/include/linux/backing-dev.h	2012-02-14 19:43:06.000000000 +0800
> +++ linux/include/linux/backing-dev.h	2012-02-14 19:49:26.000000000 +0800
> @@ -304,6 +304,8 @@ void clear_bdi_congested(struct backing_
>  void set_bdi_congested(struct backing_dev_info *bdi, int sync);
>  long congestion_wait(int sync, long timeout);
>  long wait_iff_congested(struct zone *zone, int sync, long timeout);
> +long reclaim_wait(long timeout);
> +void reclaim_rotated(void);
>  
>  static inline bool bdi_cap_writeback_dirty(struct backing_dev_info *bdi)
>  {
> --- linux.orig/mm/backing-dev.c	2012-02-14 19:26:15.000000000 +0800
> +++ linux/mm/backing-dev.c	2012-02-14 20:09:45.000000000 +0800
> @@ -873,3 +873,38 @@ out:
>  	return ret;
>  }
>  EXPORT_SYMBOL(wait_iff_congested);
> +
> +static DECLARE_WAIT_QUEUE_HEAD(reclaim_wqh);
> +
> +/**
> + * reclaim_wait - wait for some pages being rotated to the LRU tail
> + * @timeout: timeout in jiffies
> + *
> + * Wait until @timeout, or when some (typically PG_reclaim under writeback)
> + * pages rotated to the LRU so that page reclaim can make progress.
> + */
> +long reclaim_wait(long timeout)
> +{
> +	long ret;
> +	unsigned long start = jiffies;
> +	DEFINE_WAIT(wait);
> +
> +	prepare_to_wait(&reclaim_wqh, &wait, TASK_KILLABLE);
> +	ret = io_schedule_timeout(timeout);
> +	finish_wait(&reclaim_wqh, &wait);
> +
> +	trace_writeback_reclaim_wait(jiffies_to_usecs(timeout),
> +				     jiffies_to_usecs(jiffies - start));
> +
> +	return ret;
> +}
> +EXPORT_SYMBOL(reclaim_wait);
> +
> +void reclaim_rotated()
> +{
> +	wait_queue_head_t *wqh = &reclaim_wqh;
> +
> +	if (waitqueue_active(wqh))
> +		wake_up(wqh);
> +}
> +

Thank you.

I like this approach. A nitpick is that this may wake up all waiters 
in the system when a memcg is rotated.

How about wait_event() + condition by bitmap (using per memcg unique IDs.) ?


Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: reclaim the LRU lists full of dirty/writeback pages

[Wu Fengguang]

On Thu, Feb 16, 2012 at 09:00:37AM +0900, KAMEZAWA Hiroyuki wrote:
> On Tue, 14 Feb 2012 21:18:12 +0800
> Wu Fengguang <fengguang.wu@intel.com> wrote:
> 
> > 
> > --- linux.orig/include/linux/backing-dev.h	2012-02-14 19:43:06.000000000 +0800
> > +++ linux/include/linux/backing-dev.h	2012-02-14 19:49:26.000000000 +0800
> > @@ -304,6 +304,8 @@ void clear_bdi_congested(struct backing_
> >  void set_bdi_congested(struct backing_dev_info *bdi, int sync);
> >  long congestion_wait(int sync, long timeout);
> >  long wait_iff_congested(struct zone *zone, int sync, long timeout);
> > +long reclaim_wait(long timeout);
> > +void reclaim_rotated(void);
> >  
> >  static inline bool bdi_cap_writeback_dirty(struct backing_dev_info *bdi)
> >  {
> > --- linux.orig/mm/backing-dev.c	2012-02-14 19:26:15.000000000 +0800
> > +++ linux/mm/backing-dev.c	2012-02-14 20:09:45.000000000 +0800
> > @@ -873,3 +873,38 @@ out:
> >  	return ret;
> >  }
> >  EXPORT_SYMBOL(wait_iff_congested);
> > +
> > +static DECLARE_WAIT_QUEUE_HEAD(reclaim_wqh);
> > +
> > +/**
> > + * reclaim_wait - wait for some pages being rotated to the LRU tail
> > + * @timeout: timeout in jiffies
> > + *
> > + * Wait until @timeout, or when some (typically PG_reclaim under writeback)
> > + * pages rotated to the LRU so that page reclaim can make progress.
> > + */
> > +long reclaim_wait(long timeout)
> > +{
> > +	long ret;
> > +	unsigned long start = jiffies;
> > +	DEFINE_WAIT(wait);
> > +
> > +	prepare_to_wait(&reclaim_wqh, &wait, TASK_KILLABLE);
> > +	ret = io_schedule_timeout(timeout);
> > +	finish_wait(&reclaim_wqh, &wait);
> > +
> > +	trace_writeback_reclaim_wait(jiffies_to_usecs(timeout),
> > +				     jiffies_to_usecs(jiffies - start));
> > +
> > +	return ret;
> > +}
> > +EXPORT_SYMBOL(reclaim_wait);
> > +
> > +void reclaim_rotated()
> > +{
> > +	wait_queue_head_t *wqh = &reclaim_wqh;
> > +
> > +	if (waitqueue_active(wqh))
> > +		wake_up(wqh);
> > +}
> > +
> 
> Thank you.
> 
> I like this approach. A nitpick is that this may wake up all waiters 
> in the system when a memcg is rotated.

Thank you. It sure helps to start it simple :-)

> How about wait_event() + condition by bitmap (using per memcg unique IDs.) ?

I'm not sure how to manage the bitmap. The idea in my mind is to

- maintain a memcg->pages_rotated counter

- in reclaim_wait(), grab the current ->pages_rotated value before
  going to wait, compare it to the new value on every wakeup, and
  return to the user when seeing a different ->pages_rotated value.
  (this cannot stop waking up multiple tasks in the same memcg...) 

Does that sound reasonable?

Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: reclaim the LRU lists full of dirty/writeback pages

[KAMEZAWA Hiroyuki]

On Thu, 16 Feb 2012 11:04:15 +0800
Wu Fengguang <fengguang.wu@intel.com> wrote:

> On Thu, Feb 16, 2012 at 09:00:37AM +0900, KAMEZAWA Hiroyuki wrote:
> > On Tue, 14 Feb 2012 21:18:12 +0800
> > Wu Fengguang <fengguang.wu@intel.com> wrote:
> > 
> > > 
> > > --- linux.orig/include/linux/backing-dev.h	2012-02-14 19:43:06.000000000 +0800
> > > +++ linux/include/linux/backing-dev.h	2012-02-14 19:49:26.000000000 +0800
> > > @@ -304,6 +304,8 @@ void clear_bdi_congested(struct backing_
> > >  void set_bdi_congested(struct backing_dev_info *bdi, int sync);
> > >  long congestion_wait(int sync, long timeout);
> > >  long wait_iff_congested(struct zone *zone, int sync, long timeout);
> > > +long reclaim_wait(long timeout);
> > > +void reclaim_rotated(void);
> > >  
> > >  static inline bool bdi_cap_writeback_dirty(struct backing_dev_info *bdi)
> > >  {
> > > --- linux.orig/mm/backing-dev.c	2012-02-14 19:26:15.000000000 +0800
> > > +++ linux/mm/backing-dev.c	2012-02-14 20:09:45.000000000 +0800
> > > @@ -873,3 +873,38 @@ out:
> > >  	return ret;
> > >  }
> > >  EXPORT_SYMBOL(wait_iff_congested);
> > > +
> > > +static DECLARE_WAIT_QUEUE_HEAD(reclaim_wqh);
> > > +
> > > +/**
> > > + * reclaim_wait - wait for some pages being rotated to the LRU tail
> > > + * @timeout: timeout in jiffies
> > > + *
> > > + * Wait until @timeout, or when some (typically PG_reclaim under writeback)
> > > + * pages rotated to the LRU so that page reclaim can make progress.
> > > + */
> > > +long reclaim_wait(long timeout)
> > > +{
> > > +	long ret;
> > > +	unsigned long start = jiffies;
> > > +	DEFINE_WAIT(wait);
> > > +
> > > +	prepare_to_wait(&reclaim_wqh, &wait, TASK_KILLABLE);
> > > +	ret = io_schedule_timeout(timeout);
> > > +	finish_wait(&reclaim_wqh, &wait);
> > > +
> > > +	trace_writeback_reclaim_wait(jiffies_to_usecs(timeout),
> > > +				     jiffies_to_usecs(jiffies - start));
> > > +
> > > +	return ret;
> > > +}
> > > +EXPORT_SYMBOL(reclaim_wait);
> > > +
> > > +void reclaim_rotated()
> > > +{
> > > +	wait_queue_head_t *wqh = &reclaim_wqh;
> > > +
> > > +	if (waitqueue_active(wqh))
> > > +		wake_up(wqh);
> > > +}
> > > +
> > 
> > Thank you.
> > 
> > I like this approach. A nitpick is that this may wake up all waiters 
> > in the system when a memcg is rotated.
> 
> Thank you. It sure helps to start it simple :-)
> 
> > How about wait_event() + condition by bitmap (using per memcg unique IDs.) ?
> 
> I'm not sure how to manage the bitmap. The idea in my mind is to
> 
> - maintain a memcg->pages_rotated counter
> 
> - in reclaim_wait(), grab the current ->pages_rotated value before
>   going to wait, compare it to the new value on every wakeup, and
>   return to the user when seeing a different ->pages_rotated value.
>   (this cannot stop waking up multiple tasks in the same memcg...) 
> 
> Does that sound reasonable?
> 

Maybe. But there may be problem in looking up memcg from page at every
rotation. I think it's ok to start with a way ignoring per-memcg status.
Sorry for noise.

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: reclaim the LRU lists full of dirty/writeback pages

[Wu Fengguang]

On Thu, Feb 16, 2012 at 12:52:21PM +0900, KAMEZAWA Hiroyuki wrote:
> On Thu, 16 Feb 2012 11:04:15 +0800
> Wu Fengguang <fengguang.wu@intel.com> wrote:
> 
> > On Thu, Feb 16, 2012 at 09:00:37AM +0900, KAMEZAWA Hiroyuki wrote:
> > > On Tue, 14 Feb 2012 21:18:12 +0800
> > > Wu Fengguang <fengguang.wu@intel.com> wrote:
> > > 
> > > > 
> > > > --- linux.orig/include/linux/backing-dev.h	2012-02-14 19:43:06.000000000 +0800
> > > > +++ linux/include/linux/backing-dev.h	2012-02-14 19:49:26.000000000 +0800
> > > > @@ -304,6 +304,8 @@ void clear_bdi_congested(struct backing_
> > > >  void set_bdi_congested(struct backing_dev_info *bdi, int sync);
> > > >  long congestion_wait(int sync, long timeout);
> > > >  long wait_iff_congested(struct zone *zone, int sync, long timeout);
> > > > +long reclaim_wait(long timeout);
> > > > +void reclaim_rotated(void);
> > > >  
> > > >  static inline bool bdi_cap_writeback_dirty(struct backing_dev_info *bdi)
> > > >  {
> > > > --- linux.orig/mm/backing-dev.c	2012-02-14 19:26:15.000000000 +0800
> > > > +++ linux/mm/backing-dev.c	2012-02-14 20:09:45.000000000 +0800
> > > > @@ -873,3 +873,38 @@ out:
> > > >  	return ret;
> > > >  }
> > > >  EXPORT_SYMBOL(wait_iff_congested);
> > > > +
> > > > +static DECLARE_WAIT_QUEUE_HEAD(reclaim_wqh);
> > > > +
> > > > +/**
> > > > + * reclaim_wait - wait for some pages being rotated to the LRU tail
> > > > + * @timeout: timeout in jiffies
> > > > + *
> > > > + * Wait until @timeout, or when some (typically PG_reclaim under writeback)
> > > > + * pages rotated to the LRU so that page reclaim can make progress.
> > > > + */
> > > > +long reclaim_wait(long timeout)
> > > > +{
> > > > +	long ret;
> > > > +	unsigned long start = jiffies;
> > > > +	DEFINE_WAIT(wait);
> > > > +
> > > > +	prepare_to_wait(&reclaim_wqh, &wait, TASK_KILLABLE);
> > > > +	ret = io_schedule_timeout(timeout);
> > > > +	finish_wait(&reclaim_wqh, &wait);
> > > > +
> > > > +	trace_writeback_reclaim_wait(jiffies_to_usecs(timeout),
> > > > +				     jiffies_to_usecs(jiffies - start));
> > > > +
> > > > +	return ret;
> > > > +}
> > > > +EXPORT_SYMBOL(reclaim_wait);
> > > > +
> > > > +void reclaim_rotated()
> > > > +{
> > > > +	wait_queue_head_t *wqh = &reclaim_wqh;
> > > > +
> > > > +	if (waitqueue_active(wqh))
> > > > +		wake_up(wqh);
> > > > +}
> > > > +
> > > 
> > > Thank you.
> > > 
> > > I like this approach. A nitpick is that this may wake up all waiters 
> > > in the system when a memcg is rotated.
> > 
> > Thank you. It sure helps to start it simple :-)
> > 
> > > How about wait_event() + condition by bitmap (using per memcg unique IDs.) ?
> > 
> > I'm not sure how to manage the bitmap. The idea in my mind is to
> > 
> > - maintain a memcg->pages_rotated counter
> > 
> > - in reclaim_wait(), grab the current ->pages_rotated value before
> >   going to wait, compare it to the new value on every wakeup, and
> >   return to the user when seeing a different ->pages_rotated value.
> >   (this cannot stop waking up multiple tasks in the same memcg...) 
> > 
> > Does that sound reasonable?
> > 
> 
> Maybe. But there may be problem in looking up memcg from page at every
> rotation. I think it's ok to start with a way ignoring per-memcg status.
> Sorry for noise.

When there comes such a need, we could take some sampled approach: for
each pagevec rotated, only dereference one of its pages and wakeup the
corresponding memcg task(s). That should do well enough in practice
even for random write patterns.

Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>