* memcg writeback (was Re: [Lsf-pc] [LSF/MM TOPIC] memcg topics.)
@ 2012-02-08 7:55 Greg Thelen
2012-02-08 9:31 ` Wu Fengguang
0 siblings, 1 reply; 33+ messages in thread
From: Greg Thelen @ 2012-02-08 7:55 UTC (permalink / raw)
To: Wu Fengguang
Cc: Jan Kara, bsingharora, Hugh Dickins, Michal Hocko, linux-mm,
Mel Gorman, Ying Han, hannes, lsf-pc, KAMEZAWA Hiroyuki
On Fri, Feb 3, 2012 at 1:40 AM, Wu Fengguang <fengguang.wu@intel.com> wrote:
> If moving dirty pages out of the memcg to the 20% global dirty pages
> pool on page reclaim, the above OOM can be avoided. It does change the
> meaning of memory.limit_in_bytes in that the memcg tasks can now
> actually consume more pages (up to the shared global 20% dirty limit).
This seems like an easy change, but unfortunately the global 20% pool
has some shortcomings for my needs:
1. the global 20% pool is not moderated. One cgroup can dominate it
and deny service to other cgroups.
2. the global 20% pool is free, unaccounted memory. Ideally cgroups only
use the amount of memory specified in their memory.limit_in_bytes. The
goal is to sell portions of a system. Global resource like the 20% are an
undesirable system-wide tax that's shared by jobs that may not even
perform buffered writes.
3. Setting aside 20% extra memory for system wide dirty buffers is a lot of
memory. This becomes a larger issue when the global dirty_ratio is
higher than 20%.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: memcg writeback (was Re: [Lsf-pc] [LSF/MM TOPIC] memcg topics.)
2012-02-08 7:55 memcg writeback (was Re: [Lsf-pc] [LSF/MM TOPIC] memcg topics.) Greg Thelen
@ 2012-02-08 9:31 ` Wu Fengguang
2012-02-08 20:54 ` Ying Han
2012-02-10 5:51 ` Greg Thelen
0 siblings, 2 replies; 33+ messages in thread
From: Wu Fengguang @ 2012-02-08 9:31 UTC (permalink / raw)
To: Greg Thelen
Cc: Jan Kara, bsingharora, Hugh Dickins, Michal Hocko, linux-mm,
Mel Gorman, Ying Han, hannes, lsf-pc, KAMEZAWA Hiroyuki
On Tue, Feb 07, 2012 at 11:55:05PM -0800, Greg Thelen wrote:
> On Fri, Feb 3, 2012 at 1:40 AM, Wu Fengguang <fengguang.wu@intel.com> wrote:
> > If moving dirty pages out of the memcg to the 20% global dirty pages
> > pool on page reclaim, the above OOM can be avoided. It does change the
> > meaning of memory.limit_in_bytes in that the memcg tasks can now
> > actually consume more pages (up to the shared global 20% dirty limit).
>
> This seems like an easy change, but unfortunately the global 20% pool
> has some shortcomings for my needs:
>
> 1. the global 20% pool is not moderated. One cgroup can dominate it
> and deny service to other cgroups.
It is moderated by balance_dirty_pages() -- in terms of dirty ratelimit.
And you have the freedom to control the bandwidth allocation with some
async write I/O controller.
Even though there is no direct control of dirty pages, we can roughly
get it as the side effect of rate control. Given
ratelimit_cgroup_A = 2 * ratelimit_cgroup_B
There will naturally be more dirty pages for cgroup A to be worked by
the flusher. And the dirty pages will be roughly balanced around
nr_dirty_cgroup_A = 2 * nr_dirty_cgroup_B
when writeout bandwidths for their dirty pages are equal.
> 2. the global 20% pool is free, unaccounted memory. Ideally cgroups only
> use the amount of memory specified in their memory.limit_in_bytes. The
> goal is to sell portions of a system. Global resource like the 20% are an
> undesirable system-wide tax that's shared by jobs that may not even
> perform buffered writes.
Right, it is the shortcoming.
> 3. Setting aside 20% extra memory for system wide dirty buffers is a lot of
> memory. This becomes a larger issue when the global dirty_ratio is
> higher than 20%.
Yeah the global pool scheme does mean that you'd better allocate at
most 80% memory to individual memory cgroups, otherwise it's possible
for a tiny memcg doing dd writes to push dirty pages to global LRU and
*squeeze* the size of other memcgs.
However I guess it should be mitigated by the fact that
- we typically already reserve some space for the root memcg
- 20% dirty ratio is mostly an overkill for large memory systems.
It's often enough to hold 10-30s worth of dirty data for them, which
is 1-3GB for one 100MB/s disk. This is the reason vm.dirty_bytes is
introduced: someone wants to do some <1% dirty ratio.
Thanks,
Fengguang
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: memcg writeback (was Re: [Lsf-pc] [LSF/MM TOPIC] memcg topics.)
2012-02-08 9:31 ` Wu Fengguang
@ 2012-02-08 20:54 ` Ying Han
2012-02-09 13:50 ` Wu Fengguang
2012-02-10 5:51 ` Greg Thelen
1 sibling, 1 reply; 33+ messages in thread
From: Ying Han @ 2012-02-08 20:54 UTC (permalink / raw)
To: Wu Fengguang
Cc: Greg Thelen, Jan Kara, bsingharora, Hugh Dickins, Michal Hocko,
linux-mm, Mel Gorman, hannes, lsf-pc, KAMEZAWA Hiroyuki
On Wed, Feb 8, 2012 at 1:31 AM, Wu Fengguang <fengguang.wu@intel.com> wrote:
> On Tue, Feb 07, 2012 at 11:55:05PM -0800, Greg Thelen wrote:
>> On Fri, Feb 3, 2012 at 1:40 AM, Wu Fengguang <fengguang.wu@intel.com> wrote:
>> > If moving dirty pages out of the memcg to the 20% global dirty pages
>> > pool on page reclaim, the above OOM can be avoided. It does change the
>> > meaning of memory.limit_in_bytes in that the memcg tasks can now
>> > actually consume more pages (up to the shared global 20% dirty limit).
>>
>> This seems like an easy change, but unfortunately the global 20% pool
>> has some shortcomings for my needs:
>>
>> 1. the global 20% pool is not moderated. One cgroup can dominate it
>> and deny service to other cgroups.
>
> It is moderated by balance_dirty_pages() -- in terms of dirty ratelimit.
> And you have the freedom to control the bandwidth allocation with some
> async write I/O controller.
>
> Even though there is no direct control of dirty pages, we can roughly
> get it as the side effect of rate control. Given
>
> ratelimit_cgroup_A = 2 * ratelimit_cgroup_B
>
> There will naturally be more dirty pages for cgroup A to be worked by
> the flusher. And the dirty pages will be roughly balanced around
>
> nr_dirty_cgroup_A = 2 * nr_dirty_cgroup_B
>
> when writeout bandwidths for their dirty pages are equal.
>
>> 2. the global 20% pool is free, unaccounted memory. Ideally cgroups only
>> use the amount of memory specified in their memory.limit_in_bytes. The
>> goal is to sell portions of a system. Global resource like the 20% are an
>> undesirable system-wide tax that's shared by jobs that may not even
>> perform buffered writes.
>
> Right, it is the shortcoming.
>
>> 3. Setting aside 20% extra memory for system wide dirty buffers is a lot of
>> memory. This becomes a larger issue when the global dirty_ratio is
>> higher than 20%.
>
> Yeah the global pool scheme does mean that you'd better allocate at
> most 80% memory to individual memory cgroups, otherwise it's possible
> for a tiny memcg doing dd writes to push dirty pages to global LRU and
> *squeeze* the size of other memcgs.
>
> However I guess it should be mitigated by the fact that
>
> - we typically already reserve some space for the root memcg
Can you give more details on that? AFAIK, we don't treat root cgroup
differently than other sub-cgroups, except root cgroup doesn't have
limit.
In general, I don't like the idea of shared pool in root for all the
dirty pages.
Imagining a system which has nothing running under root and every
application runs within sub-cgroup. It is easy to track and limit each
cgroup's memory usage, but not the pages being moved to root. We have
been experiencing difficulties of tracking pages being re-parented to
root, and this will make it even harder.
--Ying
>
> - 20% dirty ratio is mostly an overkill for large memory systems.
> It's often enough to hold 10-30s worth of dirty data for them, which
> is 1-3GB for one 100MB/s disk. This is the reason vm.dirty_bytes is
> introduced: someone wants to do some <1% dirty ratio.
>
> Thanks,
> Fengguang
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: memcg writeback (was Re: [Lsf-pc] [LSF/MM TOPIC] memcg topics.)
2012-02-08 20:54 ` Ying Han
@ 2012-02-09 13:50 ` Wu Fengguang
2012-02-13 18:40 ` Ying Han
0 siblings, 1 reply; 33+ messages in thread
From: Wu Fengguang @ 2012-02-09 13:50 UTC (permalink / raw)
To: Ying Han
Cc: Greg Thelen, Jan Kara, bsingharora, Hugh Dickins, Michal Hocko,
linux-mm, Mel Gorman, hannes, lsf-pc, KAMEZAWA Hiroyuki
On Wed, Feb 08, 2012 at 12:54:33PM -0800, Ying Han wrote:
> On Wed, Feb 8, 2012 at 1:31 AM, Wu Fengguang <fengguang.wu@intel.com> wrote:
> > On Tue, Feb 07, 2012 at 11:55:05PM -0800, Greg Thelen wrote:
> >> On Fri, Feb 3, 2012 at 1:40 AM, Wu Fengguang <fengguang.wu@intel.com> wrote:
> >> > If moving dirty pages out of the memcg to the 20% global dirty pages
> >> > pool on page reclaim, the above OOM can be avoided. It does change the
> >> > meaning of memory.limit_in_bytes in that the memcg tasks can now
> >> > actually consume more pages (up to the shared global 20% dirty limit).
> >>
> >> This seems like an easy change, but unfortunately the global 20% pool
> >> has some shortcomings for my needs:
> >>
> >> 1. the global 20% pool is not moderated. A One cgroup can dominate it
> >> A A and deny service to other cgroups.
> >
> > It is moderated by balance_dirty_pages() -- in terms of dirty ratelimit.
> > And you have the freedom to control the bandwidth allocation with some
> > async write I/O controller.
> >
> > Even though there is no direct control of dirty pages, we can roughly
> > get it as the side effect of rate control. Given
> >
> > A A A A ratelimit_cgroup_A = 2 * ratelimit_cgroup_B
> >
> > There will naturally be more dirty pages for cgroup A to be worked by
> > the flusher. And the dirty pages will be roughly balanced around
> >
> > A A A A nr_dirty_cgroup_A = 2 * nr_dirty_cgroup_B
> >
> > when writeout bandwidths for their dirty pages are equal.
> >
> >> 2. the global 20% pool is free, unaccounted memory. A Ideally cgroups only
> >> A A use the amount of memory specified in their memory.limit_in_bytes. A The
> >> A A goal is to sell portions of a system. A Global resource like the 20% are an
> >> A A undesirable system-wide tax that's shared by jobs that may not even
> >> A A perform buffered writes.
> >
> > Right, it is the shortcoming.
> >
> >> 3. Setting aside 20% extra memory for system wide dirty buffers is a lot of
> >> A A memory. A This becomes a larger issue when the global dirty_ratio is
> >> A A higher than 20%.
> >
> > Yeah the global pool scheme does mean that you'd better allocate at
> > most 80% memory to individual memory cgroups, otherwise it's possible
> > for a tiny memcg doing dd writes to push dirty pages to global LRU and
> > *squeeze* the size of other memcgs.
> >
> > However I guess it should be mitigated by the fact that
> >
> > - we typically already reserve some space for the root memcg
>
> Can you give more details on that? AFAIK, we don't treat root cgroup
> differently than other sub-cgroups, except root cgroup doesn't have
> limit.
OK. I'd imagine this to be the typical usage for desktop and quite a
few servers: a few cgroups are employed to limit the resource usage
for selected tasks (such as backups, background GUI tasks, cron tasks,
etc.). These systems are still running mainly in the global context.
> In general, I don't like the idea of shared pool in root for all the
> dirty pages.
>
> Imagining a system which has nothing running under root and every
> application runs within sub-cgroup. It is easy to track and limit each
> cgroup's memory usage, but not the pages being moved to root. We have
> been experiencing difficulties of tracking pages being re-parented to
> root, and this will make it even harder.
So you want to push memcg allocations to the hardware limits. This is
a worthwhile target for cloud servers that run a number of well
contained jobs.
I guess it can be achieved reasonably well with the global shared
dirty pool. Let's discuss the two major cases.
1) no change of behavior
For example, when the system memory is divided equally to 10 cgroups
each running 1 dd. In this case, the dirty pages will be contained
within the memcg LRUs. Page reclaim rarely encounters any dirty pages.
There is no moving to the global LRU, so no side effect at all.
2) small memcg squeezing other memcg(s)
When system memory is divided to 1 small memcg A and 1 large memcg B,
each running a dd task. In this case the dirty pages from A will be
moved to the global LRU, and global page reclaims will be triggered.
In the end it will be balanced around
- global LRU: 10% memory (which are A's dirty pages)
- memcg B: 90% memory
- memcg A: a tiny ignorable fraction of memory
Now job B uses 10% less memory than w/o the global dirty pool scheme.
I guess this is bad for some type of jobs.
However my question is, will the typical demand be more flexible?
Something like the "minimal" and "recommended" setup: "this job
requires at least XXX memory and better at YYY memory", rather than
some fixed size memory allocation.
The minimal requirement should be trivially satisfied by adding a
memcg watermark that protects the memcg LRU from being reclaimed
when dropped under it.
Then the cloud server could be configured to
sum(memcg.limit_in_bytes) / memtotal = 100%
sum(memcg.minimal_size) / memtotal < 100% - dirty_ratio
Which makes a simple and flexibly partitioned system.
Thanks,
Fengguang
> > - 20% dirty ratio is mostly an overkill for large memory systems.
> > A It's often enough to hold 10-30s worth of dirty data for them, which
> > A is 1-3GB for one 100MB/s disk. This is the reason vm.dirty_bytes is
> > A introduced: someone wants to do some <1% dirty ratio.
> >
> > Thanks,
> > Fengguang
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: memcg writeback (was Re: [Lsf-pc] [LSF/MM TOPIC] memcg topics.)
2012-02-08 9:31 ` Wu Fengguang
2012-02-08 20:54 ` Ying Han
@ 2012-02-10 5:51 ` Greg Thelen
2012-02-10 5:52 ` Greg Thelen
2012-02-10 11:47 ` Wu Fengguang
1 sibling, 2 replies; 33+ messages in thread
From: Greg Thelen @ 2012-02-10 5:51 UTC (permalink / raw)
To: Wu Fengguang
Cc: Jan Kara, bsingharora, Hugh Dickins, Michal Hocko, linux-mm,
Mel Gorman, Ying Han, hannes, KAMEZAWA Hiroyuki
(removed lsf-pc@lists.linux-foundation.org because this really isn't
program committee matter)
On Wed, Feb 8, 2012 at 1:31 AM, Wu Fengguang <fengguang.wu@intel.com> wrote:
> On Tue, Feb 07, 2012 at 11:55:05PM -0800, Greg Thelen wrote:
>> On Fri, Feb 3, 2012 at 1:40 AM, Wu Fengguang <fengguang.wu@intel.com> wrote:
>> > If moving dirty pages out of the memcg to the 20% global dirty pages
>> > pool on page reclaim, the above OOM can be avoided. It does change the
>> > meaning of memory.limit_in_bytes in that the memcg tasks can now
>> > actually consume more pages (up to the shared global 20% dirty limit).
>>
>> This seems like an easy change, but unfortunately the global 20% pool
>> has some shortcomings for my needs:
>>
>> 1. the global 20% pool is not moderated. One cgroup can dominate it
>> and deny service to other cgroups.
>
> It is moderated by balance_dirty_pages() -- in terms of dirty ratelimit.
> And you have the freedom to control the bandwidth allocation with some
> async write I/O controller.
>
> Even though there is no direct control of dirty pages, we can roughly
> get it as the side effect of rate control. Given
>
> ratelimit_cgroup_A = 2 * ratelimit_cgroup_B
>
> There will naturally be more dirty pages for cgroup A to be worked by
> the flusher. And the dirty pages will be roughly balanced around
>
> nr_dirty_cgroup_A = 2 * nr_dirty_cgroup_B
>
> when writeout bandwidths for their dirty pages are equal.
>
>> 2. the global 20% pool is free, unaccounted memory. Ideally cgroups only
>> use the amount of memory specified in their memory.limit_in_bytes. The
>> goal is to sell portions of a system. Global resource like the 20% are an
>> undesirable system-wide tax that's shared by jobs that may not even
>> perform buffered writes.
>
> Right, it is the shortcoming.
>
>> 3. Setting aside 20% extra memory for system wide dirty buffers is a lot of
>> memory. This becomes a larger issue when the global dirty_ratio is
>> higher than 20%.
>
> Yeah the global pool scheme does mean that you'd better allocate at
> most 80% memory to individual memory cgroups, otherwise it's possible
> for a tiny memcg doing dd writes to push dirty pages to global LRU and
> *squeeze* the size of other memcgs.
>
> However I guess it should be mitigated by the fact that
>
> - we typically already reserve some space for the root memcg
>
> - 20% dirty ratio is mostly an overkill for large memory systems.
> It's often enough to hold 10-30s worth of dirty data for them, which
> is 1-3GB for one 100MB/s disk. This is the reason vm.dirty_bytes is
> introduced: someone wants to do some <1% dirty ratio.
Have you encountered situations where it's desirable to have more than
20% dirty ratio? I imagine that if the dirty working set is larger
than 20% increasing dirty ratio would prevent rewrites.
Leaking dirty memory to a root global dirty pool is concerning. I
suspect that under some conditions such pages may remain remain in
root after writeback indefinitely as clean pages. I admit this may
not be the common case, but having such leaks into root can allow low
priority jobs access entire machine denying service to higher priority
jobs.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: memcg writeback (was Re: [Lsf-pc] [LSF/MM TOPIC] memcg topics.)
2012-02-10 5:51 ` Greg Thelen
@ 2012-02-10 5:52 ` Greg Thelen
2012-02-10 9:20 ` Wu Fengguang
2012-02-10 11:47 ` Wu Fengguang
1 sibling, 1 reply; 33+ messages in thread
From: Greg Thelen @ 2012-02-10 5:52 UTC (permalink / raw)
To: Wu Fengguang
Cc: KAMEZAWA Hiroyuki, linux-mm, hannes, Michal Hocko, bsingharora,
Hugh Dickins, Ying Han, Mel Gorman
(removed lsf-pc@lists.linux-foundation.org because this really isn't
program committee matter)
On Wed, Feb 1, 2012 at 11:52 PM, Wu Fengguang <fengguang.wu@intel.com> wrote:
> Unfortunately the memcg partitioning could fundamentally make the
> dirty throttling more bumpy.
>
> Imagine 10 memcgs each with
>
> - memcg_dirty_limit=50MB
> - 1 dd dirty task
>
> The flusher thread will be working on 10 inodes in turn, each time
> grabbing the next inode and taking ~0.5s to write ~50MB of its dirty
> pages to the disk. So each inode will be flushed on every ~5s.
Does the flusher thread need to write 50MB/inode in this case? Would
there be problems interleaving writes by declaring some max write
limit (e.g. 8 MiB/write). Such interleaving would be beneficial if
there are multiple memcg expecting service from the single bdi flusher
thread. I suspect certain filesystems might have increased
fragmentation with this, but I am not sure if appending writes can
easily expand an extent.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: memcg writeback (was Re: [Lsf-pc] [LSF/MM TOPIC] memcg topics.)
2012-02-10 5:52 ` Greg Thelen
@ 2012-02-10 9:20 ` Wu Fengguang
0 siblings, 0 replies; 33+ messages in thread
From: Wu Fengguang @ 2012-02-10 9:20 UTC (permalink / raw)
To: Greg Thelen
Cc: KAMEZAWA Hiroyuki, linux-mm, hannes, Michal Hocko, bsingharora,
Hugh Dickins, Ying Han, Mel Gorman
On Thu, Feb 09, 2012 at 09:52:03PM -0800, Greg Thelen wrote:
> (removed lsf-pc@lists.linux-foundation.org because this really isn't
> program committee matter)
>
> On Wed, Feb 1, 2012 at 11:52 PM, Wu Fengguang <fengguang.wu@intel.com> wrote:
> > Unfortunately the memcg partitioning could fundamentally make the
> > dirty throttling more bumpy.
> >
> > Imagine 10 memcgs each with
> >
> > - memcg_dirty_limit=50MB
> > - 1 dd dirty task
> >
> > The flusher thread will be working on 10 inodes in turn, each time
> > grabbing the next inode and taking ~0.5s to write ~50MB of its dirty
> > pages to the disk. So each inode will be flushed on every ~5s.
>
> Does the flusher thread need to write 50MB/inode in this case?
> Would there be problems interleaving writes by declaring some max
> write limit (e.g. 8 MiB/write).
ext4 actually forces write chunk size to be >=128MB for better write
throughput and less fragmentation, which also helps read performance.
Other filesystems use the VFS computed chunk size, which is defined
in writeback_chunk_size() as write_bandwith/2.
> Such interleaving would be beneficial if there are multiple memcg
> expecting service from the single bdi flusher thread.
Right, reducing the writeback chunk size will improve the memcg's
dirty pages smoothness right away.
> I suspect certain filesystems might have increased fragmentation
> with this, but I am not sure if appending writes can easily expand
> an extent.
To be exact, it's ext4 that will suffer from fragmentation with smaller
chunk sizes. Because it uses the size passed by ->writepages() as hint
to allocate extents. Perhaps this heuristic is somehow improvable.
XFS does not have the fragmentation issue (at least not affected by
the chunk size). However my old tests show that it costs much less
seeks and performs noticeably better with raised write chunk size.
Thanks,
Fengguang
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: memcg writeback (was Re: [Lsf-pc] [LSF/MM TOPIC] memcg topics.)
2012-02-10 5:51 ` Greg Thelen
2012-02-10 5:52 ` Greg Thelen
@ 2012-02-10 11:47 ` Wu Fengguang
2012-02-11 12:44 ` reclaim the LRU lists full of dirty/writeback pages Wu Fengguang
1 sibling, 1 reply; 33+ messages in thread
From: Wu Fengguang @ 2012-02-10 11:47 UTC (permalink / raw)
To: Greg Thelen
Cc: Jan Kara, bsingharora, Hugh Dickins, Michal Hocko, linux-mm,
Mel Gorman, Ying Han, hannes, KAMEZAWA Hiroyuki
On Thu, Feb 09, 2012 at 09:51:31PM -0800, Greg Thelen wrote:
> On Wed, Feb 8, 2012 at 1:31 AM, Wu Fengguang <fengguang.wu@intel.com> wrote:
> > On Tue, Feb 07, 2012 at 11:55:05PM -0800, Greg Thelen wrote:
> >> On Fri, Feb 3, 2012 at 1:40 AM, Wu Fengguang <fengguang.wu@intel.com> wrote:
> >> > If moving dirty pages out of the memcg to the 20% global dirty pages
> >> > pool on page reclaim, the above OOM can be avoided. It does change the
> >> > meaning of memory.limit_in_bytes in that the memcg tasks can now
> >> > actually consume more pages (up to the shared global 20% dirty limit).
> >>
> >> This seems like an easy change, but unfortunately the global 20% pool
> >> has some shortcomings for my needs:
> >>
> >> 1. the global 20% pool is not moderated. A One cgroup can dominate it
> >> A A and deny service to other cgroups.
> >
> > It is moderated by balance_dirty_pages() -- in terms of dirty ratelimit.
> > And you have the freedom to control the bandwidth allocation with some
> > async write I/O controller.
> >
> > Even though there is no direct control of dirty pages, we can roughly
> > get it as the side effect of rate control. Given
> >
> > A A A A ratelimit_cgroup_A = 2 * ratelimit_cgroup_B
> >
> > There will naturally be more dirty pages for cgroup A to be worked by
> > the flusher. And the dirty pages will be roughly balanced around
> >
> > A A A A nr_dirty_cgroup_A = 2 * nr_dirty_cgroup_B
> >
> > when writeout bandwidths for their dirty pages are equal.
> >
> >> 2. the global 20% pool is free, unaccounted memory. A Ideally cgroups only
> >> A A use the amount of memory specified in their memory.limit_in_bytes. A The
> >> A A goal is to sell portions of a system. A Global resource like the 20% are an
> >> A A undesirable system-wide tax that's shared by jobs that may not even
> >> A A perform buffered writes.
> >
> > Right, it is the shortcoming.
> >
> >> 3. Setting aside 20% extra memory for system wide dirty buffers is a lot of
> >> A A memory. A This becomes a larger issue when the global dirty_ratio is
> >> A A higher than 20%.
> >
> > Yeah the global pool scheme does mean that you'd better allocate at
> > most 80% memory to individual memory cgroups, otherwise it's possible
> > for a tiny memcg doing dd writes to push dirty pages to global LRU and
> > *squeeze* the size of other memcgs.
> >
> > However I guess it should be mitigated by the fact that
> >
> > - we typically already reserve some space for the root memcg
> >
> > - 20% dirty ratio is mostly an overkill for large memory systems.
> > A It's often enough to hold 10-30s worth of dirty data for them, which
> > A is 1-3GB for one 100MB/s disk. This is the reason vm.dirty_bytes is
> > A introduced: someone wants to do some <1% dirty ratio.
>
> Have you encountered situations where it's desirable to have more than
> 20% dirty ratio? I imagine that if the dirty working set is larger
> than 20% increasing dirty ratio would prevent rewrites.
Not encountered in person, but there will sure be such situations.
One may need to dirty some 40% sized in-memory data set and don't want
to be throttled and trigger lots of I/O. In this case increasing the
dirty ratio to 40% will do the job.
The less obvious condition for this to work is, the workload should
avoid much page allocations when working on the large fixed data set.
Otherwise page reclaim will keep running into dirty pages and act badly.
So it looks still pretty compatible with the "reparent to root memcg
on page reclaim" scheme if that job is put into some memcg. There will
be almost no dirty pages encountered during page reclaim, hence no
move and no any side effects.
But if there is another job doing heavy dirtying, that job will eat up
the global 40% dirty limit and heavily impact the above job. This is
one case the memcg dirty ratio can help a lot.
> Leaking dirty memory to a root global dirty pool is concerning. I
> suspect that under some conditions such pages may remain remain in
> root after writeback indefinitely as clean pages. I admit this may
> not be the common case, but having such leaks into root can allow low
> priority jobs access entire machine denying service to higher priority
> jobs.
You are right. DoS can be achieved by
loop {
dirty one more page
access all previously dirtied pages
}
Assuming only !PG_reference pages are moved to the global dirty pool,
it requires someone to access the page in order for it to stay in the
global LRU for one more cycle, and to access it frequently for keeping
it in global LRU indefinitely.
So yes, it's possible for some evil job to DoS the whole box. It will
be an issue when hosting jobs from untrusted sources (ie. Amazon style
cloud service), which I guess should be running inside KVM?
It should hardly happen in real workloads. If some job does manage to
do so, it probably means some kind of mis-configuration: the memcg is
configured way too small to hold the job's working set.
Thanks,
Fengguang
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 33+ messages in thread
* reclaim the LRU lists full of dirty/writeback pages
2012-02-10 11:47 ` Wu Fengguang
@ 2012-02-11 12:44 ` Wu Fengguang
2012-02-11 14:55 ` Rik van Riel
2012-02-14 10:19 ` Mel Gorman
0 siblings, 2 replies; 33+ messages in thread
From: Wu Fengguang @ 2012-02-11 12:44 UTC (permalink / raw)
To: Greg Thelen
Cc: Jan Kara, bsingharora, Hugh Dickins, Michal Hocko, linux-mm,
Mel Gorman, Ying Han, hannes, KAMEZAWA Hiroyuki, Rik van Riel,
Minchan Kim
On Fri, Feb 10, 2012 at 07:47:06PM +0800, Wu Fengguang wrote:
> On Thu, Feb 09, 2012 at 09:51:31PM -0800, Greg Thelen wrote:
> > Have you encountered situations where it's desirable to have more than
> > 20% dirty ratio? I imagine that if the dirty working set is larger
> > than 20% increasing dirty ratio would prevent rewrites.
> One may need to dirty some 40% sized in-memory data set and don't want
> to be throttled and trigger lots of I/O. In this case increasing the
> dirty ratio to 40% will do the job.
> But if there is another job doing heavy dirtying, that job will eat up
> the global 40% dirty limit and heavily impact the above job. This is
> one case the memcg dirty ratio can help a lot.
>
> > Leaking dirty memory to a root global dirty pool is concerning. I
> > suspect that under some conditions such pages may remain remain in
> > root after writeback indefinitely as clean pages. I admit this may
> > not be the common case, but having such leaks into root can allow low
> > priority jobs access entire machine denying service to higher priority
> > jobs.
>
> You are right. DoS can be achieved by
>
> loop {
> dirty one more page
> access all previously dirtied pages
> }
So there are situations that prefer the dirty pages to be strictly
contained within the memcg.
For these use cases it looks worthwhile to improve the page reclaim
algorithms to handle the 100% dirty zone well. I'd regard this as a
much more promising direction than memcg dirty ratio, because efforts
on this is going to benefit the general kernel as a whole.
The below patch aims to be the first step towards the goal. It turns
out to work pretty well for avoiding OOM, with reasonably good I/O
throughput and low CPU overheads.
Hopefully the page reclaim can be further improved to make the 100%
dirty zone a seriously supported and well performed case.
Thanks,
Fengguang
---
Subject: writeback: introduce the pageout work
Date: Thu Jul 29 14:41:19 CST 2010
This relays file pageout IOs to the flusher threads.
The ultimate target is to gracefully handle the LRU lists full of
dirty/writeback pages.
1) I/O efficiency
The flusher will piggy back the around 1MB dirty pages for I/O (XXX:
make the chunk size adaptive to the bdi write bandwidth).
This takes advantage of the time/spacial locality in most workloads: the
nearby pages of one file are typically populated into the LRU at the same
time, hence will likely be close to each other in the LRU list. Writing
them in one shot helps clean more pages effectively for page reclaim.
2) OOM avoidance and scan rate control
Typically we do LRU scan w/o rate control and quickly get enough clean
pages for the LRU lists not full of dirty pages.
Or we can still get a number of freshly cleaned pages (moved to LRU tail
by end_page_writeback()) when the queued pageout I/O is completed within
tens of milli-seconds.
However if the LRU list is small and full of dirty pages, it can be
quickly fully scanned and go OOM before the flusher manages to clean
enough pages.
Here a simple yet reliable scheme is employed to avoid OOM and keep scan
rate in sync with the I/O rate:
if (PageReclaim(page))
congestion_wait();
PG_reclaim plays the key role. When dirty pages are encountered, we
queue I/O for it, set PG_reclaim and put it back to the LRU head.
So if PG_reclaim pages are encountered again, it means the dirty page
has not yet been cleaned by the flusher after a full zone scan. It
indicates we are scanning more fast than I/O and shall take a snap.
The runtime behavior on a fully dirtied small LRU list would be:
It will start with a quick scan of the list, queuing all pages for I/O.
Then the scan will be slowed down by the PG_reclaim pages *adaptively*
to match the I/O bandwidth.
3) writeback work coordinations
To avoid memory allocations at page reclaim, a mempool for struct
wb_writeback_work is created.
wakeup_flusher_threads() is removed because it can easily delay the
more oriented pageout works and even exhaust the mempool reservations.
It's often not I/O efficient by submitting writeback works with small
->nr_pages.
Background/periodic works will quit automatically (as done in another
patch), so as to clean the pages under reclaim ASAP. However for now the
sync work can still block us for long time.
Jan Kara: limit the search scope. Note that the limited search and work
pool is not a big problem: 1000 IOs under flight are typically more than
enough to saturate the disk. And the overheads of searching in the work
list didn't even show up in the perf report.
4) test case
Run 2 dd tasks in a 100MB memcg (a very handy test case from Greg Thelen):
mkdir /cgroup/x
echo 100M > /cgroup/x/memory.limit_in_bytes
echo $$ > /cgroup/x/tasks
for i in `seq 2`
do
dd if=/dev/zero of=/fs/f$i bs=1k count=1M &
done
Before patch, the dd tasks are quickly OOM killed.
After patch, they run well with reasonably good performance and overheads:
1073741824 bytes (1.1 GB) copied, 22.2196 s, 48.3 MB/s
1073741824 bytes (1.1 GB) copied, 22.4675 s, 47.8 MB/s
iostat -kx 1
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 178.00 0.00 89568.00 1006.38 74.35 417.71 4.80 85.40
sda 0.00 2.00 0.00 191.00 0.00 94428.00 988.77 53.34 219.03 4.34 82.90
sda 0.00 20.00 0.00 196.00 0.00 97712.00 997.06 71.11 337.45 4.77 93.50
sda 0.00 5.00 0.00 175.00 0.00 84648.00 967.41 54.03 316.44 5.06 88.60
sda 0.00 0.00 0.00 186.00 0.00 92432.00 993.89 56.22 267.54 5.38 100.00
sda 0.00 1.00 0.00 183.00 0.00 90156.00 985.31 37.99 325.55 4.33 79.20
sda 0.00 0.00 0.00 175.00 0.00 88692.00 1013.62 48.70 218.43 4.69 82.10
sda 0.00 0.00 0.00 196.00 0.00 97528.00 995.18 43.38 236.87 5.10 100.00
sda 0.00 0.00 0.00 179.00 0.00 88648.00 990.48 45.83 285.43 5.59 100.00
sda 0.00 0.00 0.00 178.00 0.00 88500.00 994.38 28.28 158.89 4.99 88.80
sda 0.00 0.00 0.00 194.00 0.00 95852.00 988.16 32.58 167.39 5.15 100.00
sda 0.00 2.00 0.00 215.00 0.00 105996.00 986.01 41.72 201.43 4.65 100.00
sda 0.00 4.00 0.00 173.00 0.00 84332.00 974.94 50.48 260.23 5.76 99.60
sda 0.00 0.00 0.00 182.00 0.00 90312.00 992.44 36.83 212.07 5.49 100.00
sda 0.00 8.00 0.00 195.00 0.00 95940.50 984.01 50.18 221.06 5.13 100.00
sda 0.00 1.00 0.00 220.00 0.00 108852.00 989.56 40.99 202.68 4.55 100.00
sda 0.00 2.00 0.00 161.00 0.00 80384.00 998.56 37.19 268.49 6.21 100.00
sda 0.00 4.00 0.00 182.00 0.00 90830.00 998.13 50.58 239.77 5.49 100.00
sda 0.00 0.00 0.00 197.00 0.00 94877.00 963.22 36.68 196.79 5.08 100.00
avg-cpu: %user %nice %system %iowait %steal %idle
0.25 0.00 15.08 33.92 0.00 50.75
0.25 0.00 14.54 35.09 0.00 50.13
0.50 0.00 13.57 32.41 0.00 53.52
0.50 0.00 11.28 36.84 0.00 51.38
0.50 0.00 15.75 32.00 0.00 51.75
0.50 0.00 10.50 34.00 0.00 55.00
0.50 0.00 17.63 27.46 0.00 54.41
0.50 0.00 15.08 30.90 0.00 53.52
0.50 0.00 11.28 32.83 0.00 55.39
0.75 0.00 16.79 26.82 0.00 55.64
0.50 0.00 16.08 29.15 0.00 54.27
0.50 0.00 13.50 30.50 0.00 55.50
0.50 0.00 14.32 35.18 0.00 50.00
0.50 0.00 12.06 33.92 0.00 53.52
0.50 0.00 17.29 30.58 0.00 51.63
0.50 0.00 15.08 29.65 0.00 54.77
0.50 0.00 12.53 29.32 0.00 57.64
0.50 0.00 15.29 31.83 0.00 52.38
The global dd iostat for comparison:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 189.00 0.00 95752.00 1013.25 143.09 684.48 5.29 100.00
sda 0.00 0.00 0.00 208.00 0.00 105480.00 1014.23 143.06 733.29 4.81 100.00
sda 0.00 0.00 0.00 161.00 0.00 81924.00 1017.69 141.71 757.79 6.21 100.00
sda 0.00 0.00 0.00 217.00 0.00 109580.00 1009.95 143.09 749.55 4.61 100.10
sda 0.00 0.00 0.00 187.00 0.00 94728.00 1013.13 144.31 773.67 5.35 100.00
sda 0.00 0.00 0.00 189.00 0.00 95752.00 1013.25 144.14 742.00 5.29 100.00
sda 0.00 0.00 0.00 177.00 0.00 90032.00 1017.31 143.32 656.59 5.65 100.00
sda 0.00 0.00 0.00 215.00 0.00 108640.00 1010.60 142.90 817.54 4.65 100.00
sda 0.00 2.00 0.00 166.00 0.00 83858.00 1010.34 143.64 808.61 6.02 100.00
sda 0.00 0.00 0.00 186.00 0.00 92813.00 997.99 141.18 736.95 5.38 100.00
sda 0.00 0.00 0.00 206.00 0.00 104456.00 1014.14 146.27 729.33 4.85 100.00
sda 0.00 0.00 0.00 213.00 0.00 107024.00 1004.92 143.25 705.70 4.69 100.00
sda 0.00 0.00 0.00 188.00 0.00 95748.00 1018.60 141.82 764.78 5.32 100.00
avg-cpu: %user %nice %system %iowait %steal %idle
0.51 0.00 11.22 52.30 0.00 35.97
0.25 0.00 10.15 52.54 0.00 37.06
0.25 0.00 5.01 56.64 0.00 38.10
0.51 0.00 15.15 43.94 0.00 40.40
0.25 0.00 12.12 48.23 0.00 39.39
0.51 0.00 11.20 53.94 0.00 34.35
0.26 0.00 9.72 51.41 0.00 38.62
0.76 0.00 9.62 50.63 0.00 38.99
0.51 0.00 10.46 53.32 0.00 35.71
0.51 0.00 9.41 51.91 0.00 38.17
0.25 0.00 10.69 49.62 0.00 39.44
0.51 0.00 12.21 52.67 0.00 34.61
0.51 0.00 11.45 53.18 0.00 34.86
Note that it's data for XFS. ext4 seems to have some problem with the
workload: the majority pages are found to be writeback pages, and the
flusher ends up blocking on the unconditional wait_on_page_writeback()
in write_cache_pages_da() from time to time...
XXX: commit NFS unstable pages via write_inode()
XXX: the added congestion_wait() may be undesirable in some situations
CC: Jan Kara <jack@suse.cz>
CC: Mel Gorman <mgorman@suse.de>
CC: Rik van Riel <riel@redhat.com>
CC: Greg Thelen <gthelen@google.com>
CC: Minchan Kim <minchan.kim@gmail.com>
Signed-off-by: Wu Fengguang <fengguang.wu@intel.com>
---
fs/fs-writeback.c | 165 ++++++++++++++++++++++++++++-
include/linux/writeback.h | 4
include/trace/events/writeback.h | 12 +-
mm/vmscan.c | 17 +-
4 files changed, 184 insertions(+), 14 deletions(-)
--- linux.orig/mm/vmscan.c 2012-02-03 21:42:21.000000000 +0800
+++ linux/mm/vmscan.c 2012-02-11 17:28:54.000000000 +0800
@@ -813,6 +813,8 @@ static unsigned long shrink_page_list(st
if (PageWriteback(page)) {
nr_writeback++;
+ if (PageReclaim(page))
+ congestion_wait(BLK_RW_ASYNC, HZ/10);
/*
* Synchronous reclaim cannot queue pages for
* writeback due to the possibility of stack overflow
@@ -874,12 +876,15 @@ static unsigned long shrink_page_list(st
nr_dirty++;
/*
- * Only kswapd can writeback filesystem pages to
- * avoid risk of stack overflow but do not writeback
- * unless under significant pressure.
+ * run into the visited page again: we are scanning
+ * faster than the flusher can writeout dirty pages
*/
- if (page_is_file_cache(page) &&
- (!current_is_kswapd() || priority >= DEF_PRIORITY - 2)) {
+ if (page_is_file_cache(page) && PageReclaim(page)) {
+ congestion_wait(BLK_RW_ASYNC, HZ/10);
+ goto keep_locked;
+ }
+ if (page_is_file_cache(page) && mapping &&
+ flush_inode_page(mapping, page, true) >= 0) {
/*
* Immediately reclaim when written back.
* Similar in principal to deactivate_page()
@@ -2382,8 +2387,6 @@ static unsigned long do_try_to_free_page
*/
writeback_threshold = sc->nr_to_reclaim + sc->nr_to_reclaim / 2;
if (total_scanned > writeback_threshold) {
- wakeup_flusher_threads(laptop_mode ? 0 : total_scanned,
- WB_REASON_TRY_TO_FREE_PAGES);
sc->may_writepage = 1;
}
--- linux.orig/fs/fs-writeback.c 2012-02-03 21:42:16.000000000 +0800
+++ linux/fs/fs-writeback.c 2012-02-11 18:24:24.000000000 +0800
@@ -35,12 +35,21 @@
#define MIN_WRITEBACK_PAGES (4096UL >> (PAGE_CACHE_SHIFT - 10))
/*
+ * When flushing an inode page (for page reclaim), try to piggy back up to
+ * 1MB nearby pages for IO efficiency. These pages will have good opportunity
+ * to be in the same LRU list.
+ */
+#define WRITE_AROUND_PAGES (1024UL >> (PAGE_CACHE_SHIFT - 10))
+
+/*
* Passed into wb_writeback(), essentially a subset of writeback_control
*/
struct wb_writeback_work {
long nr_pages;
struct super_block *sb;
unsigned long *older_than_this;
+ struct inode *inode;
+ pgoff_t offset;
enum writeback_sync_modes sync_mode;
unsigned int tagged_writepages:1;
unsigned int for_kupdate:1;
@@ -65,6 +74,27 @@ struct wb_writeback_work {
*/
int nr_pdflush_threads;
+static mempool_t *wb_work_mempool;
+
+static void *wb_work_alloc(gfp_t gfp_mask, void *pool_data)
+{
+ /*
+ * bdi_flush_inode_range() may be called on page reclaim
+ */
+ if (current->flags & PF_MEMALLOC)
+ return NULL;
+
+ return kmalloc(sizeof(struct wb_writeback_work), gfp_mask);
+}
+
+static __init int wb_work_init(void)
+{
+ wb_work_mempool = mempool_create(1024,
+ wb_work_alloc, mempool_kfree, NULL);
+ return wb_work_mempool ? 0 : -ENOMEM;
+}
+fs_initcall(wb_work_init);
+
/**
* writeback_in_progress - determine whether there is writeback in progress
* @bdi: the device's backing_dev_info structure.
@@ -129,7 +159,7 @@ __bdi_start_writeback(struct backing_dev
* This is WB_SYNC_NONE writeback, so if allocation fails just
* wakeup the thread for old dirty data writeback
*/
- work = kzalloc(sizeof(*work), GFP_ATOMIC);
+ work = mempool_alloc(wb_work_mempool, GFP_NOWAIT);
if (!work) {
if (bdi->wb.task) {
trace_writeback_nowork(bdi);
@@ -138,6 +168,7 @@ __bdi_start_writeback(struct backing_dev
return;
}
+ memset(work, 0, sizeof(*work));
work->sync_mode = WB_SYNC_NONE;
work->nr_pages = nr_pages;
work->range_cyclic = range_cyclic;
@@ -186,6 +217,114 @@ void bdi_start_background_writeback(stru
spin_unlock_bh(&bdi->wb_lock);
}
+static bool extend_writeback_range(struct wb_writeback_work *work,
+ pgoff_t offset)
+{
+ pgoff_t end = work->offset + work->nr_pages;
+
+ if (offset >= work->offset && offset < end)
+ return true;
+
+ if (work->nr_pages >= 8 * WRITE_AROUND_PAGES)
+ return false;
+
+ /* the unsigned comparison helps eliminate one compare */
+ if (work->offset - offset < WRITE_AROUND_PAGES) {
+ work->nr_pages += WRITE_AROUND_PAGES;
+ work->offset -= WRITE_AROUND_PAGES;
+ return true;
+ }
+
+ if (offset - end < WRITE_AROUND_PAGES) {
+ work->nr_pages += WRITE_AROUND_PAGES;
+ return true;
+ }
+
+ return false;
+}
+
+/*
+ * schedule writeback on a range of inode pages.
+ */
+static struct wb_writeback_work *
+bdi_flush_inode_range(struct backing_dev_info *bdi,
+ struct inode *inode,
+ pgoff_t offset,
+ pgoff_t len,
+ bool wait)
+{
+ struct wb_writeback_work *work;
+
+ if (!igrab(inode))
+ return ERR_PTR(-ENOENT);
+
+ work = mempool_alloc(wb_work_mempool, wait ? GFP_NOIO : GFP_NOWAIT);
+ if (!work)
+ return ERR_PTR(-ENOMEM);
+
+ memset(work, 0, sizeof(*work));
+ work->sync_mode = WB_SYNC_NONE;
+ work->inode = inode;
+ work->offset = offset;
+ work->nr_pages = len;
+ work->reason = WB_REASON_PAGEOUT;
+
+ bdi_queue_work(bdi, work);
+
+ return work;
+}
+
+/*
+ * Called by page reclaim code to flush the dirty page ASAP. Do write-around to
+ * improve IO throughput. The nearby pages will have good chance to reside in
+ * the same LRU list that vmscan is working on, and even close to each other
+ * inside the LRU list in the common case of sequential read/write.
+ *
+ * ret > 0: success, found/reused a previous writeback work
+ * ret = 0: success, allocated/queued a new writeback work
+ * ret < 0: failed
+ */
+long flush_inode_page(struct address_space *mapping,
+ struct page *page,
+ bool wait)
+{
+ struct backing_dev_info *bdi = mapping->backing_dev_info;
+ struct inode *inode = mapping->host;
+ pgoff_t offset = page->index;
+ pgoff_t len = 0;
+ struct wb_writeback_work *work;
+ long ret = -ENOENT;
+
+ if (unlikely(!inode))
+ goto out;
+
+ len = 1;
+ spin_lock_bh(&bdi->wb_lock);
+ list_for_each_entry_reverse(work, &bdi->work_list, list) {
+ if (work->inode != inode)
+ continue;
+ if (extend_writeback_range(work, offset)) {
+ ret = len;
+ offset = work->offset;
+ len = work->nr_pages;
+ break;
+ }
+ if (len++ > 100) /* limit search depth */
+ break;
+ }
+ spin_unlock_bh(&bdi->wb_lock);
+
+ if (ret > 0)
+ goto out;
+
+ offset = round_down(offset, WRITE_AROUND_PAGES);
+ len = WRITE_AROUND_PAGES;
+ work = bdi_flush_inode_range(bdi, inode, offset, len, wait);
+ ret = IS_ERR(work) ? PTR_ERR(work) : 0;
+out:
+ return ret;
+}
+
/*
* Remove the inode from the writeback list it is on.
*/
@@ -833,6 +972,23 @@ static unsigned long get_nr_dirty_pages(
get_nr_dirty_inodes();
}
+static long wb_flush_inode(struct bdi_writeback *wb,
+ struct wb_writeback_work *work)
+{
+ struct writeback_control wbc = {
+ .sync_mode = WB_SYNC_NONE,
+ .nr_to_write = LONG_MAX,
+ .range_start = work->offset << PAGE_CACHE_SHIFT,
+ .range_end = (work->offset + work->nr_pages - 1)
+ << PAGE_CACHE_SHIFT,
+ };
+
+ do_writepages(work->inode->i_mapping, &wbc);
+ iput(work->inode);
+
+ return LONG_MAX - wbc.nr_to_write;
+}
+
static long wb_check_background_flush(struct bdi_writeback *wb)
{
if (over_bground_thresh(wb->bdi)) {
@@ -905,7 +1061,10 @@ long wb_do_writeback(struct bdi_writebac
trace_writeback_exec(bdi, work);
- wrote += wb_writeback(wb, work);
+ if (work->inode)
+ wrote += wb_flush_inode(wb, work);
+ else
+ wrote += wb_writeback(wb, work);
/*
* Notify the caller of completion if this is a synchronous
@@ -914,7 +1073,7 @@ long wb_do_writeback(struct bdi_writebac
if (work->done)
complete(work->done);
else
- kfree(work);
+ mempool_free(work, wb_work_mempool);
}
/*
--- linux.orig/include/trace/events/writeback.h 2012-02-10 21:54:14.000000000 +0800
+++ linux/include/trace/events/writeback.h 2012-02-11 16:49:18.000000000 +0800
@@ -23,7 +23,7 @@
#define WB_WORK_REASON \
{WB_REASON_BACKGROUND, "background"}, \
- {WB_REASON_TRY_TO_FREE_PAGES, "try_to_free_pages"}, \
+ {WB_REASON_PAGEOUT, "pageout"}, \
{WB_REASON_SYNC, "sync"}, \
{WB_REASON_PERIODIC, "periodic"}, \
{WB_REASON_LAPTOP_TIMER, "laptop_timer"}, \
@@ -45,6 +45,8 @@ DECLARE_EVENT_CLASS(writeback_work_class
__field(int, range_cyclic)
__field(int, for_background)
__field(int, reason)
+ __field(unsigned long, ino)
+ __field(unsigned long, offset)
),
TP_fast_assign(
strncpy(__entry->name, dev_name(bdi->dev), 32);
@@ -55,9 +57,11 @@ DECLARE_EVENT_CLASS(writeback_work_class
__entry->range_cyclic = work->range_cyclic;
__entry->for_background = work->for_background;
__entry->reason = work->reason;
+ __entry->ino = work->inode ? work->inode->i_ino : 0;
+ __entry->offset = work->offset;
),
TP_printk("bdi %s: sb_dev %d:%d nr_pages=%ld sync_mode=%d "
- "kupdate=%d range_cyclic=%d background=%d reason=%s",
+ "kupdate=%d range_cyclic=%d background=%d reason=%s ino=%lu offset=%lu",
__entry->name,
MAJOR(__entry->sb_dev), MINOR(__entry->sb_dev),
__entry->nr_pages,
@@ -65,7 +69,9 @@ DECLARE_EVENT_CLASS(writeback_work_class
__entry->for_kupdate,
__entry->range_cyclic,
__entry->for_background,
- __print_symbolic(__entry->reason, WB_WORK_REASON)
+ __print_symbolic(__entry->reason, WB_WORK_REASON),
+ __entry->ino,
+ __entry->offset
)
);
#define DEFINE_WRITEBACK_WORK_EVENT(name) \
--- linux.orig/include/linux/writeback.h 2012-02-11 09:53:53.000000000 +0800
+++ linux/include/linux/writeback.h 2012-02-11 16:49:36.000000000 +0800
@@ -40,7 +40,7 @@ enum writeback_sync_modes {
*/
enum wb_reason {
WB_REASON_BACKGROUND,
- WB_REASON_TRY_TO_FREE_PAGES,
+ WB_REASON_PAGEOUT,
WB_REASON_SYNC,
WB_REASON_PERIODIC,
WB_REASON_LAPTOP_TIMER,
@@ -94,6 +94,8 @@ long writeback_inodes_wb(struct bdi_writ
enum wb_reason reason);
long wb_do_writeback(struct bdi_writeback *wb, int force_wait);
void wakeup_flusher_threads(long nr_pages, enum wb_reason reason);
+long flush_inode_page(struct address_space *mapping, struct page *page,
+ bool wait);
/* writeback.h requires fs.h; it, too, is not included from here. */
static inline void wait_on_inode(struct inode *inode)
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: reclaim the LRU lists full of dirty/writeback pages
2012-02-11 12:44 ` reclaim the LRU lists full of dirty/writeback pages Wu Fengguang
@ 2012-02-11 14:55 ` Rik van Riel
2012-02-12 3:10 ` Wu Fengguang
2012-02-14 10:19 ` Mel Gorman
1 sibling, 1 reply; 33+ messages in thread
From: Rik van Riel @ 2012-02-11 14:55 UTC (permalink / raw)
To: Wu Fengguang
Cc: Greg Thelen, Jan Kara, bsingharora, Hugh Dickins, Michal Hocko,
linux-mm, Mel Gorman, Ying Han, hannes, KAMEZAWA Hiroyuki,
Minchan Kim
On 02/11/2012 07:44 AM, Wu Fengguang wrote:
> Note that it's data for XFS. ext4 seems to have some problem with the
> workload: the majority pages are found to be writeback pages, and the
> flusher ends up blocking on the unconditional wait_on_page_writeback()
> in write_cache_pages_da() from time to time...
>
> XXX: commit NFS unstable pages via write_inode()
> XXX: the added congestion_wait() may be undesirable in some situations
Even with these caveats, this seems to be the right way forward.
> CC: Jan Kara<jack@suse.cz>
> CC: Mel Gorman<mgorman@suse.de>
> CC: Rik van Riel<riel@redhat.com>
> CC: Greg Thelen<gthelen@google.com>
> CC: Minchan Kim<minchan.kim@gmail.com>
> Signed-off-by: Wu Fengguang<fengguang.wu@intel.com>
Acked-by: Rik van Riel <riel@redhat.com>
--
All rights reversed
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: reclaim the LRU lists full of dirty/writeback pages
2012-02-11 14:55 ` Rik van Riel
@ 2012-02-12 3:10 ` Wu Fengguang
2012-02-12 6:45 ` Wu Fengguang
2012-02-13 15:43 ` Jan Kara
0 siblings, 2 replies; 33+ messages in thread
From: Wu Fengguang @ 2012-02-12 3:10 UTC (permalink / raw)
To: Rik van Riel
Cc: Greg Thelen, Jan Kara, bsingharora, Hugh Dickins, Michal Hocko,
linux-mm, Mel Gorman, Ying Han, hannes, KAMEZAWA Hiroyuki,
Minchan Kim
On Sat, Feb 11, 2012 at 09:55:38AM -0500, Rik van Riel wrote:
> On 02/11/2012 07:44 AM, Wu Fengguang wrote:
>
> >Note that it's data for XFS. ext4 seems to have some problem with the
> >workload: the majority pages are found to be writeback pages, and the
> >flusher ends up blocking on the unconditional wait_on_page_writeback()
> >in write_cache_pages_da() from time to time...
Sorry I overlooked the WB_SYNC_NONE test before the wait_on_page_writeback()
call! And the issue can no longer be reproduce anyway. ext4 performs pretty
good now, here is the result for one single memcg dd:
dd if=/dev/zero of=/fs/f$i bs=4k count=1M
4294967296 bytes (4.3 GB) copied, 44.5759 s, 96.4 MB/s
iostat -kx 3
avg-cpu: %user %nice %system %iowait %steal %idle
0.25 0.00 11.03 28.54 0.00 60.19
0.25 0.00 13.71 16.65 0.00 69.39
0.17 0.00 8.41 24.81 0.00 66.61
0.25 0.00 15.00 19.63 0.00 65.12
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 17.00 0.00 178.33 0.00 90694.67 1017.14 111.34 520.88 5.45 97.23
sda 0.00 0.00 0.00 193.67 0.00 98816.00 1020.48 86.22 496.81 4.81 93.07
sda 0.00 3.33 0.00 182.33 0.00 92345.33 1012.93 101.14 623.98 5.49 100.03
sda 0.00 3.00 0.00 187.00 0.00 95586.67 1022.32 89.36 441.70 4.96 92.70
> >XXX: commit NFS unstable pages via write_inode()
> >XXX: the added congestion_wait() may be undesirable in some situations
>
> Even with these caveats, this seems to be the right way forward.
> Acked-by: Rik van Riel <riel@redhat.com>
Thank you!
Here is the updated patch.
- ~10ms write around chunk size, adaptive to the bdi bandwith
- cleanup flush_inode_page()
Thanks,
Fengguang
---
Subject: writeback: introduce the pageout work
Date: Thu Jul 29 14:41:19 CST 2010
This relays file pageout IOs to the flusher threads.
The ultimate target is to gracefully handle the LRU lists full of
dirty/writeback pages.
1) I/O efficiency
The flusher will piggy back the nearby ~10ms worth of dirty pages for I/O.
This takes advantage of the time/spacial locality in most workloads: the
nearby pages of one file are typically populated into the LRU at the same
time, hence will likely be close to each other in the LRU list. Writing
them in one shot helps clean more pages effectively for page reclaim.
2) OOM avoidance and scan rate control
Typically we do LRU scan w/o rate control and quickly get enough clean
pages for the LRU lists not full of dirty pages.
Or we can still get a number of freshly cleaned pages (moved to LRU tail
by end_page_writeback()) when the queued pageout I/O is completed within
tens of milli-seconds.
However if the LRU list is small and full of dirty pages, it can be
quickly fully scanned and go OOM before the flusher manages to clean
enough pages.
A simple yet reliable scheme is employed to avoid OOM and keep scan rate
in sync with the I/O rate:
if (PageReclaim(page))
congestion_wait(HZ/10);
PG_reclaim plays the key role. When dirty pages are encountered, we
queue I/O for it, set PG_reclaim and put it back to the LRU head.
So if PG_reclaim pages are encountered again, it means the dirty page
has not yet been cleaned by the flusher after a full zone scan. It
indicates we are scanning more fast than I/O and shall take a snap.
The runtime behavior on a fully dirtied small LRU list would be:
It will start with a quick scan of the list, queuing all pages for I/O.
Then the scan will be slowed down by the PG_reclaim pages *adaptively*
to match the I/O bandwidth.
3) writeback work coordinations
To avoid memory allocations at page reclaim, a mempool for struct
wb_writeback_work is created.
wakeup_flusher_threads() is removed because it can easily delay the
more oriented pageout works and even exhaust the mempool reservations.
It's also found to not I/O efficient by frequently submitting writeback
works with small ->nr_pages.
Background/periodic works will quit automatically, so as to clean the
pages under reclaim ASAP. However for now the sync work can still block
us for long time.
Jan Kara: limit the search scope. Note that the limited search and work
pool is not a big problem: 1000 IOs under flight are typically more than
enough to saturate the disk. And the overheads of searching in the work
list didn't even show up in the perf report.
4) test case
Run 2 dd tasks in a 100MB memcg (a very handy test case from Greg Thelen):
mkdir /cgroup/x
echo 100M > /cgroup/x/memory.limit_in_bytes
echo $$ > /cgroup/x/tasks
for i in `seq 2`
do
dd if=/dev/zero of=/fs/f$i bs=1k count=1M &
done
Before patch, the dd tasks are quickly OOM killed.
After patch, they run well with reasonably good performance and overheads:
1073741824 bytes (1.1 GB) copied, 22.2196 s, 48.3 MB/s
1073741824 bytes (1.1 GB) copied, 22.4675 s, 47.8 MB/s
iostat -kx 1
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 178.00 0.00 89568.00 1006.38 74.35 417.71 4.80 85.40
sda 0.00 2.00 0.00 191.00 0.00 94428.00 988.77 53.34 219.03 4.34 82.90
sda 0.00 20.00 0.00 196.00 0.00 97712.00 997.06 71.11 337.45 4.77 93.50
sda 0.00 5.00 0.00 175.00 0.00 84648.00 967.41 54.03 316.44 5.06 88.60
sda 0.00 0.00 0.00 186.00 0.00 92432.00 993.89 56.22 267.54 5.38 100.00
sda 0.00 1.00 0.00 183.00 0.00 90156.00 985.31 37.99 325.55 4.33 79.20
sda 0.00 0.00 0.00 175.00 0.00 88692.00 1013.62 48.70 218.43 4.69 82.10
sda 0.00 0.00 0.00 196.00 0.00 97528.00 995.18 43.38 236.87 5.10 100.00
sda 0.00 0.00 0.00 179.00 0.00 88648.00 990.48 45.83 285.43 5.59 100.00
sda 0.00 0.00 0.00 178.00 0.00 88500.00 994.38 28.28 158.89 4.99 88.80
sda 0.00 0.00 0.00 194.00 0.00 95852.00 988.16 32.58 167.39 5.15 100.00
sda 0.00 2.00 0.00 215.00 0.00 105996.00 986.01 41.72 201.43 4.65 100.00
sda 0.00 4.00 0.00 173.00 0.00 84332.00 974.94 50.48 260.23 5.76 99.60
sda 0.00 0.00 0.00 182.00 0.00 90312.00 992.44 36.83 212.07 5.49 100.00
sda 0.00 8.00 0.00 195.00 0.00 95940.50 984.01 50.18 221.06 5.13 100.00
sda 0.00 1.00 0.00 220.00 0.00 108852.00 989.56 40.99 202.68 4.55 100.00
sda 0.00 2.00 0.00 161.00 0.00 80384.00 998.56 37.19 268.49 6.21 100.00
sda 0.00 4.00 0.00 182.00 0.00 90830.00 998.13 50.58 239.77 5.49 100.00
sda 0.00 0.00 0.00 197.00 0.00 94877.00 963.22 36.68 196.79 5.08 100.00
avg-cpu: %user %nice %system %iowait %steal %idle
0.25 0.00 15.08 33.92 0.00 50.75
0.25 0.00 14.54 35.09 0.00 50.13
0.50 0.00 13.57 32.41 0.00 53.52
0.50 0.00 11.28 36.84 0.00 51.38
0.50 0.00 15.75 32.00 0.00 51.75
0.50 0.00 10.50 34.00 0.00 55.00
0.50 0.00 17.63 27.46 0.00 54.41
0.50 0.00 15.08 30.90 0.00 53.52
0.50 0.00 11.28 32.83 0.00 55.39
0.75 0.00 16.79 26.82 0.00 55.64
0.50 0.00 16.08 29.15 0.00 54.27
0.50 0.00 13.50 30.50 0.00 55.50
0.50 0.00 14.32 35.18 0.00 50.00
0.50 0.00 12.06 33.92 0.00 53.52
0.50 0.00 17.29 30.58 0.00 51.63
0.50 0.00 15.08 29.65 0.00 54.77
0.50 0.00 12.53 29.32 0.00 57.64
0.50 0.00 15.29 31.83 0.00 52.38
The global dd numbers for comparison:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 189.00 0.00 95752.00 1013.25 143.09 684.48 5.29 100.00
sda 0.00 0.00 0.00 208.00 0.00 105480.00 1014.23 143.06 733.29 4.81 100.00
sda 0.00 0.00 0.00 161.00 0.00 81924.00 1017.69 141.71 757.79 6.21 100.00
sda 0.00 0.00 0.00 217.00 0.00 109580.00 1009.95 143.09 749.55 4.61 100.10
sda 0.00 0.00 0.00 187.00 0.00 94728.00 1013.13 144.31 773.67 5.35 100.00
sda 0.00 0.00 0.00 189.00 0.00 95752.00 1013.25 144.14 742.00 5.29 100.00
sda 0.00 0.00 0.00 177.00 0.00 90032.00 1017.31 143.32 656.59 5.65 100.00
sda 0.00 0.00 0.00 215.00 0.00 108640.00 1010.60 142.90 817.54 4.65 100.00
sda 0.00 2.00 0.00 166.00 0.00 83858.00 1010.34 143.64 808.61 6.02 100.00
sda 0.00 0.00 0.00 186.00 0.00 92813.00 997.99 141.18 736.95 5.38 100.00
sda 0.00 0.00 0.00 206.00 0.00 104456.00 1014.14 146.27 729.33 4.85 100.00
sda 0.00 0.00 0.00 213.00 0.00 107024.00 1004.92 143.25 705.70 4.69 100.00
sda 0.00 0.00 0.00 188.00 0.00 95748.00 1018.60 141.82 764.78 5.32 100.00
avg-cpu: %user %nice %system %iowait %steal %idle
0.51 0.00 11.22 52.30 0.00 35.97
0.25 0.00 10.15 52.54 0.00 37.06
0.25 0.00 5.01 56.64 0.00 38.10
0.51 0.00 15.15 43.94 0.00 40.40
0.25 0.00 12.12 48.23 0.00 39.39
0.51 0.00 11.20 53.94 0.00 34.35
0.26 0.00 9.72 51.41 0.00 38.62
0.76 0.00 9.62 50.63 0.00 38.99
0.51 0.00 10.46 53.32 0.00 35.71
0.51 0.00 9.41 51.91 0.00 38.17
0.25 0.00 10.69 49.62 0.00 39.44
0.51 0.00 12.21 52.67 0.00 34.61
0.51 0.00 11.45 53.18 0.00 34.86
XXX: commit NFS unstable pages via write_inode()
XXX: the added congestion_wait() may be undesirable in some situations
CC: Jan Kara <jack@suse.cz>
CC: Mel Gorman <mgorman@suse.de>
Acked-by: Rik van Riel <riel@redhat.com>
CC: Greg Thelen <gthelen@google.com>
CC: Minchan Kim <minchan.kim@gmail.com>
Signed-off-by: Wu Fengguang <fengguang.wu@intel.com>
---
fs/fs-writeback.c | 167 ++++++++++++++++++++++++++++-
include/linux/writeback.h | 4
include/trace/events/writeback.h | 12 +-
mm/vmscan.c | 17 +-
4 files changed, 186 insertions(+), 14 deletions(-)
--- linux.orig/mm/vmscan.c 2012-02-11 19:59:40.000000000 +0800
+++ linux/mm/vmscan.c 2012-02-11 20:07:48.000000000 +0800
@@ -813,6 +813,8 @@ static unsigned long shrink_page_list(st
if (PageWriteback(page)) {
nr_writeback++;
+ if (PageReclaim(page))
+ congestion_wait(BLK_RW_ASYNC, HZ/10);
/*
* Synchronous reclaim cannot queue pages for
* writeback due to the possibility of stack overflow
@@ -874,12 +876,15 @@ static unsigned long shrink_page_list(st
nr_dirty++;
/*
- * Only kswapd can writeback filesystem pages to
- * avoid risk of stack overflow but do not writeback
- * unless under significant pressure.
+ * run into the visited page again: we are scanning
+ * faster than the flusher can writeout dirty pages
*/
- if (page_is_file_cache(page) &&
- (!current_is_kswapd() || priority >= DEF_PRIORITY - 2)) {
+ if (page_is_file_cache(page) && PageReclaim(page)) {
+ congestion_wait(BLK_RW_ASYNC, HZ/10);
+ goto keep_locked;
+ }
+ if (page_is_file_cache(page) && mapping &&
+ flush_inode_page(mapping, page, true) >= 0) {
/*
* Immediately reclaim when written back.
* Similar in principal to deactivate_page()
@@ -2382,8 +2387,6 @@ static unsigned long do_try_to_free_page
*/
writeback_threshold = sc->nr_to_reclaim + sc->nr_to_reclaim / 2;
if (total_scanned > writeback_threshold) {
- wakeup_flusher_threads(laptop_mode ? 0 : total_scanned,
- WB_REASON_TRY_TO_FREE_PAGES);
sc->may_writepage = 1;
}
--- linux.orig/fs/fs-writeback.c 2012-02-11 19:59:40.000000000 +0800
+++ linux/fs/fs-writeback.c 2012-02-12 10:54:05.000000000 +0800
@@ -41,6 +41,8 @@ struct wb_writeback_work {
long nr_pages;
struct super_block *sb;
unsigned long *older_than_this;
+ struct inode *inode;
+ pgoff_t offset;
enum writeback_sync_modes sync_mode;
unsigned int tagged_writepages:1;
unsigned int for_kupdate:1;
@@ -65,6 +67,27 @@ struct wb_writeback_work {
*/
int nr_pdflush_threads;
+static mempool_t *wb_work_mempool;
+
+static void *wb_work_alloc(gfp_t gfp_mask, void *pool_data)
+{
+ /*
+ * bdi_flush_inode_range() may be called on page reclaim
+ */
+ if (current->flags & PF_MEMALLOC)
+ return NULL;
+
+ return kmalloc(sizeof(struct wb_writeback_work), gfp_mask);
+}
+
+static __init int wb_work_init(void)
+{
+ wb_work_mempool = mempool_create(1024,
+ wb_work_alloc, mempool_kfree, NULL);
+ return wb_work_mempool ? 0 : -ENOMEM;
+}
+fs_initcall(wb_work_init);
+
/**
* writeback_in_progress - determine whether there is writeback in progress
* @bdi: the device's backing_dev_info structure.
@@ -129,7 +152,7 @@ __bdi_start_writeback(struct backing_dev
* This is WB_SYNC_NONE writeback, so if allocation fails just
* wakeup the thread for old dirty data writeback
*/
- work = kzalloc(sizeof(*work), GFP_ATOMIC);
+ work = mempool_alloc(wb_work_mempool, GFP_NOWAIT);
if (!work) {
if (bdi->wb.task) {
trace_writeback_nowork(bdi);
@@ -138,6 +161,7 @@ __bdi_start_writeback(struct backing_dev
return;
}
+ memset(work, 0, sizeof(*work));
work->sync_mode = WB_SYNC_NONE;
work->nr_pages = nr_pages;
work->range_cyclic = range_cyclic;
@@ -186,6 +210,123 @@ void bdi_start_background_writeback(stru
spin_unlock_bh(&bdi->wb_lock);
}
+static bool extend_writeback_range(struct wb_writeback_work *work,
+ pgoff_t offset,
+ unsigned long write_around_pages)
+{
+ pgoff_t end = work->offset + work->nr_pages;
+
+ if (offset >= work->offset && offset < end)
+ return true;
+
+ /*
+ * for sequential workloads with good locality, include up to 8 times
+ * more data in one chunk
+ */
+ if (work->nr_pages >= 8 * write_around_pages)
+ return false;
+
+ /* the unsigned comparison helps eliminate one compare */
+ if (work->offset - offset < write_around_pages) {
+ work->nr_pages += write_around_pages;
+ work->offset -= write_around_pages;
+ return true;
+ }
+
+ if (offset - end < write_around_pages) {
+ work->nr_pages += write_around_pages;
+ return true;
+ }
+
+ return false;
+}
+
+/*
+ * schedule writeback on a range of inode pages.
+ */
+static struct wb_writeback_work *
+bdi_flush_inode_range(struct backing_dev_info *bdi,
+ struct inode *inode,
+ pgoff_t offset,
+ pgoff_t len,
+ bool wait)
+{
+ struct wb_writeback_work *work;
+
+ if (!igrab(inode))
+ return ERR_PTR(-ENOENT);
+
+ work = mempool_alloc(wb_work_mempool, wait ? GFP_NOIO : GFP_NOWAIT);
+ if (!work)
+ return ERR_PTR(-ENOMEM);
+
+ memset(work, 0, sizeof(*work));
+ work->sync_mode = WB_SYNC_NONE;
+ work->inode = inode;
+ work->offset = offset;
+ work->nr_pages = len;
+ work->reason = WB_REASON_PAGEOUT;
+
+ bdi_queue_work(bdi, work);
+
+ return work;
+}
+
+/*
+ * Called by page reclaim code to flush the dirty page ASAP. Do write-around to
+ * improve IO throughput. The nearby pages will have good chance to reside in
+ * the same LRU list that vmscan is working on, and even close to each other
+ * inside the LRU list in the common case of sequential read/write.
+ *
+ * ret > 0: success, found/reused a previous writeback work
+ * ret = 0: success, allocated/queued a new writeback work
+ * ret < 0: failed
+ */
+long flush_inode_page(struct address_space *mapping,
+ struct page *page,
+ bool wait)
+{
+ struct backing_dev_info *bdi = mapping->backing_dev_info;
+ struct inode *inode = mapping->host;
+ struct wb_writeback_work *work;
+ unsigned long write_around_pages;
+ pgoff_t offset = page->index;
+ int i;
+ long ret = 0;
+
+ if (unlikely(!inode))
+ return -ENOENT;
+
+ /*
+ * piggy back 8-15ms worth of data
+ */
+ write_around_pages = bdi->avg_write_bandwidth + MIN_WRITEBACK_PAGES;
+ write_around_pages = rounddown_pow_of_two(write_around_pages) >> 6;
+
+ i = 1;
+ spin_lock_bh(&bdi->wb_lock);
+ list_for_each_entry_reverse(work, &bdi->work_list, list) {
+ if (work->inode != inode)
+ continue;
+ if (extend_writeback_range(work, offset, write_around_pages)) {
+ ret = i;
+ break;
+ }
+ if (i++ > 100) /* limit search depth */
+ break;
+ }
+ spin_unlock_bh(&bdi->wb_lock);
+
+ if (!ret) {
+ offset = round_down(offset, write_around_pages);
+ work = bdi_flush_inode_range(bdi, inode,
+ offset, write_around_pages, wait);
+ if (IS_ERR(work))
+ ret = PTR_ERR(work);
+ }
+ return ret;
+}
+
/*
* Remove the inode from the writeback list it is on.
*/
@@ -833,6 +974,23 @@ static unsigned long get_nr_dirty_pages(
get_nr_dirty_inodes();
}
+static long wb_flush_inode(struct bdi_writeback *wb,
+ struct wb_writeback_work *work)
+{
+ struct writeback_control wbc = {
+ .sync_mode = WB_SYNC_NONE,
+ .nr_to_write = LONG_MAX,
+ .range_start = work->offset << PAGE_CACHE_SHIFT,
+ .range_end = (work->offset + work->nr_pages - 1)
+ << PAGE_CACHE_SHIFT,
+ };
+
+ do_writepages(work->inode->i_mapping, &wbc);
+ iput(work->inode);
+
+ return LONG_MAX - wbc.nr_to_write;
+}
+
static long wb_check_background_flush(struct bdi_writeback *wb)
{
if (over_bground_thresh(wb->bdi)) {
@@ -905,7 +1063,10 @@ long wb_do_writeback(struct bdi_writebac
trace_writeback_exec(bdi, work);
- wrote += wb_writeback(wb, work);
+ if (work->inode)
+ wrote += wb_flush_inode(wb, work);
+ else
+ wrote += wb_writeback(wb, work);
/*
* Notify the caller of completion if this is a synchronous
@@ -914,7 +1075,7 @@ long wb_do_writeback(struct bdi_writebac
if (work->done)
complete(work->done);
else
- kfree(work);
+ mempool_free(work, wb_work_mempool);
}
/*
--- linux.orig/include/trace/events/writeback.h 2012-02-11 19:59:40.000000000 +0800
+++ linux/include/trace/events/writeback.h 2012-02-11 20:07:48.000000000 +0800
@@ -23,7 +23,7 @@
#define WB_WORK_REASON \
{WB_REASON_BACKGROUND, "background"}, \
- {WB_REASON_TRY_TO_FREE_PAGES, "try_to_free_pages"}, \
+ {WB_REASON_PAGEOUT, "pageout"}, \
{WB_REASON_SYNC, "sync"}, \
{WB_REASON_PERIODIC, "periodic"}, \
{WB_REASON_LAPTOP_TIMER, "laptop_timer"}, \
@@ -45,6 +45,8 @@ DECLARE_EVENT_CLASS(writeback_work_class
__field(int, range_cyclic)
__field(int, for_background)
__field(int, reason)
+ __field(unsigned long, ino)
+ __field(unsigned long, offset)
),
TP_fast_assign(
strncpy(__entry->name, dev_name(bdi->dev), 32);
@@ -55,9 +57,11 @@ DECLARE_EVENT_CLASS(writeback_work_class
__entry->range_cyclic = work->range_cyclic;
__entry->for_background = work->for_background;
__entry->reason = work->reason;
+ __entry->ino = work->inode ? work->inode->i_ino : 0;
+ __entry->offset = work->offset;
),
TP_printk("bdi %s: sb_dev %d:%d nr_pages=%ld sync_mode=%d "
- "kupdate=%d range_cyclic=%d background=%d reason=%s",
+ "kupdate=%d range_cyclic=%d background=%d reason=%s ino=%lu offset=%lu",
__entry->name,
MAJOR(__entry->sb_dev), MINOR(__entry->sb_dev),
__entry->nr_pages,
@@ -65,7 +69,9 @@ DECLARE_EVENT_CLASS(writeback_work_class
__entry->for_kupdate,
__entry->range_cyclic,
__entry->for_background,
- __print_symbolic(__entry->reason, WB_WORK_REASON)
+ __print_symbolic(__entry->reason, WB_WORK_REASON),
+ __entry->ino,
+ __entry->offset
)
);
#define DEFINE_WRITEBACK_WORK_EVENT(name) \
--- linux.orig/include/linux/writeback.h 2012-02-11 19:59:40.000000000 +0800
+++ linux/include/linux/writeback.h 2012-02-11 20:07:48.000000000 +0800
@@ -40,7 +40,7 @@ enum writeback_sync_modes {
*/
enum wb_reason {
WB_REASON_BACKGROUND,
- WB_REASON_TRY_TO_FREE_PAGES,
+ WB_REASON_PAGEOUT,
WB_REASON_SYNC,
WB_REASON_PERIODIC,
WB_REASON_LAPTOP_TIMER,
@@ -94,6 +94,8 @@ long writeback_inodes_wb(struct bdi_writ
enum wb_reason reason);
long wb_do_writeback(struct bdi_writeback *wb, int force_wait);
void wakeup_flusher_threads(long nr_pages, enum wb_reason reason);
+long flush_inode_page(struct address_space *mapping, struct page *page,
+ bool wait);
/* writeback.h requires fs.h; it, too, is not included from here. */
static inline void wait_on_inode(struct inode *inode)
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: reclaim the LRU lists full of dirty/writeback pages
2012-02-12 3:10 ` Wu Fengguang
@ 2012-02-12 6:45 ` Wu Fengguang
2012-02-13 15:43 ` Jan Kara
1 sibling, 0 replies; 33+ messages in thread
From: Wu Fengguang @ 2012-02-12 6:45 UTC (permalink / raw)
To: Rik van Riel
Cc: Greg Thelen, Jan Kara, bsingharora, Hugh Dickins, Michal Hocko,
linux-mm, Mel Gorman, Ying Han, hannes, KAMEZAWA Hiroyuki,
Minchan Kim
[-- Attachment #1: Type: text/plain, Size: 409223 bytes --]
> ext4 performs pretty good now, here is the result for one single
> memcg dd:
>
> dd if=/dev/zero of=/fs/f$i bs=4k count=1M
Here is one full writeback trace collected with
echo 1 > /debug/tracing/events/writeback/enable
echo 0 > /debug/tracing/events/writeback/global_dirty_state/enable
echo 0 > /debug/tracing/events/writeback/wbc_writepage/enable
echo 0 > /debug/tracing/events/writeback/writeback_wait_iff_congested/enable
Looking at the reason=pageout lines, the chunk size at queue time is
mostly nr_pages=256, and since it's a sequential dd with very good
locality, lots of the pageout works have been extended to
nr_pages=2048 at the time of execution.
The progress is a bit bumpy in that the writeback_queue/writeback_exec
lines and the writeback_congestion_wait lines tend to come together in
batches. According to the attached graphs (generated on a private
task_io trace event), the dd write progress is reasonably smooth.
There are several sudden jump of both x and y values, maybe it's
caused by lost trace samples.
# tracer: nop
#
# entries-in-buffer/entries-written: 127889/168807 #P:4
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
dd-4267 [002] .... 8250.326788: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8250.335124: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8250.343750: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8250.356916: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [002] .... 8250.361203: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [002] .... 8251.323386: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [002] .... 8251.363306: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [002] .... 8251.405320: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [002] .... 8251.427430: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [002] .... 8251.458557: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [002] .... 8251.476925: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8251.592810: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [002] .... 8251.647229: writeback_congestion_wait: usec_timeout=100000 usec_delayed=54000
dd-4267 [002] .... 8251.682142: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [002] .... 8251.728420: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [002] .... 8251.754083: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
flush-8:0-4272 [002] .... 8251.819186: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=24320
flush-8:0-4272 [002] .... 8251.823270: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=24320
flush-8:0-4272 [002] .... 8251.826068: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=26368
flush-8:0-4272 [002] .... 8251.832939: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=28416
flush-8:0-4272 [002] .... 8251.839785: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=30464
flush-8:0-4272 [002] .... 8251.846554: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=32512
flush-8:0-4272 [002] .... 8251.852810: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=34560
flush-8:0-4272 [002] .... 8251.858865: writeback_start: bdi 8:0: sb_dev 0:0 nr_pages=9223372036854775807 sync_mode=0 kupdate=0 range_cyclic=1 background=1 reason=background ino=0 offset=0
flush-8:0-4272 [002] .... 8251.858866: writeback_queue_io: bdi 8:0: older=4302923541 age=0 enqueue=0 reason=background
flush-8:0-4272 [002] .... 8251.860726: writeback_single_inode: bdi 8:0: ino=13 state=I_DIRTY_PAGES dirtied_when=4302921148 age=55 index=37632 to_write=1024 wrote=1024
flush-8:0-4272 [002] .... 8251.860729: writeback_written: bdi 8:0: sb_dev 0:0 nr_pages=9223372036854774783 sync_mode=0 kupdate=0 range_cyclic=1 background=1 reason=background ino=0 offset=0
flush-8:0-4272 [002] .... 8251.860730: writeback_start: bdi 8:0: sb_dev 0:0 nr_pages=9223372036854774783 sync_mode=0 kupdate=0 range_cyclic=1 background=1 reason=background ino=0 offset=0
flush-8:0-4272 [002] .... 8251.860731: writeback_queue_io: bdi 8:0: older=4302923543 age=0 enqueue=0 reason=background
flush-8:0-4272 [002] .... 8252.607338: writeback_single_inode: bdi 8:0: ino=13 state=I_DIRTY_PAGES dirtied_when=4302921148 age=55 index=51712 to_write=1024 wrote=1024
flush-8:0-4272 [002] .... 8252.607341: writeback_written: bdi 8:0: sb_dev 0:0 nr_pages=9223372036854774783 sync_mode=0 kupdate=0 range_cyclic=1 background=1 reason=background ino=0 offset=0
flush-8:0-4272 [002] .... 8252.607342: writeback_start: bdi 8:0: sb_dev 0:0 nr_pages=9223372036854774783 sync_mode=0 kupdate=0 range_cyclic=1 background=1 reason=background ino=0 offset=0
flush-8:0-4272 [002] .... 8252.607343: writeback_queue_io: bdi 8:0: older=4302924290 age=0 enqueue=0 reason=background
flush-8:0-4272 [002] .... 8253.026787: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=63232
flush-8:0-4272 [002] .... 8253.026915: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=512 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=63232
flush-8:0-4272 [002] .... 8253.027360: writeback_pages_written: 316
flush-8:0-4272 [002] .... 8253.027437: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=63744
flush-8:0-4272 [002] .... 8253.029744: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=63744
flush-8:0-4272 [002] .... 8253.030492: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1024 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=64512
flush-8:0-4272 [002] .... 8253.033488: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=65280
flush-8:0-4272 [002] .... 8253.036827: writeback_pages_written: 3584
flush-8:0-4272 [002] .... 8253.040560: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=67328
flush-8:0-4272 [002] .... 8253.042129: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=67328
flush-8:0-4272 [002] .... 8253.045098: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=69376
flush-8:0-4272 [002] .... 8253.050144: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=71424
##### CPU 3 buffer started ####
flush-8:0-4272 [002] .... 8253.485155: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=75008
flush-8:0-4272 [002] .... 8253.488248: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=75008
flush-8:0-4272 [002] .... 8253.488254: writeback_pages_written: 256
dd-4267 [003] .... 8253.587708: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [002] .... 8253.616408: writeback_congestion_wait: usec_timeout=100000 usec_delayed=28000
dd-4267 [002] .... 8253.627826: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
flush-8:0-4272 [002] .... 8253.678663: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=75264
flush-8:0-4272 [002] .... 8253.682232: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=75264
flush-8:0-4272 [002] .... 8253.682240: writeback_start: bdi 8:0: sb_dev 0:0 nr_pages=9223372036854775807 sync_mode=0 kupdate=0 range_cyclic=1 background=1 reason=background ino=0 offset=0
flush-8:0-4272 [002] .... 8253.682240: writeback_queue_io: bdi 8:0: older=4302925365 age=0 enqueue=0 reason=background
flush-8:0-4272 [002] .... 8253.684064: writeback_single_inode: bdi 8:0: ino=13 state=I_DIRTY_PAGES dirtied_when=4302921148 age=55 index=76544 to_write=1024 wrote=1024
flush-8:0-4272 [002] .... 8253.684067: writeback_written: bdi 8:0: sb_dev 0:0 nr_pages=9223372036854774783 sync_mode=0 kupdate=0 range_cyclic=1 background=1 reason=background ino=0 offset=0
flush-8:0-4272 [002] .... 8253.684069: writeback_pages_written: 1280
dd-4267 [002] .... 8253.730357: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [002] .... 8253.814751: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [002] .... 8253.850639: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8253.870770: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [002] .... 8253.897749: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [002] .... 8254.014484: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [002] .... 8254.113342: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
flush-8:0-4272 [002] .... 8254.123898: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=76544
flush-8:0-4272 [002] .... 8254.128327: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=76544
flush-8:0-4272 [002] .... 8254.131253: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=78592
flush-8:0-4272 [002] .... 8254.138062: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=80640
flush-8:0-4272 [002] .... 8254.144703: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=82688
flush-8:0-4272 [002] .... 8254.230122: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=84736
flush-8:0-4272 [002] .... 8254.235649: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=86784
flush-8:0-4272 [002] .... 8254.241491: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=88832
flush-8:0-4272 [002] .... 8254.246782: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1024 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=90880
flush-8:0-4272 [002] .... 8254.249905: writeback_pages_written: 15360
flush-8:0-4272 [002] .... 8254.282817: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=91904
flush-8:0-4272 [002] .... 8254.285648: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=91904
flush-8:0-4272 [002] .... 8254.289279: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1280 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=93952
flush-8:0-4272 [002] .... 8254.292899: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=94976
flush-8:0-4272 [002] .... 8254.295917: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=97024
flush-8:0-4272 [002] .... 8254.299985: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=99072
flush-8:0-4272 [002] .... 8254.300865: writeback_pages_written: 7424
dd-4267 [003] .... 8254.393293: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [002] .... 8254.536096: writeback_congestion_wait: usec_timeout=100000 usec_delayed=82000
dd-4267 [002] .... 8254.625314: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=99328
dd-4267 [002] .... 8254.625340: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=99328
dd-4267 [002] .... 8254.626413: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [002] .... 8254.630694: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
flush-0:16-4223 [003] .... 8255.307759: writeback_start: bdi 0:16: sb_dev 0:0 nr_pages=45399 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-0:16-4223 [003] .... 8255.307762: writeback_queue_io: bdi 0:16: older=4302896992 age=30000 enqueue=0 reason=periodic
flush-0:16-4223 [003] .... 8255.307762: writeback_written: bdi 0:16: sb_dev 0:0 nr_pages=45399 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-0:16-4223 [003] .... 8255.307764: writeback_pages_written: 0
dd-4267 [002] .... 8255.318648: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=99584
dd-4267 [002] .... 8255.318664: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=99584
dd-4267 [002] .... 8255.320138: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=101632
dd-4267 [002] .... 8255.321638: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=103680
dd-4267 [002] .... 8255.323080: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=105728
dd-4267 [002] .... 8255.324538: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=107776
dd-4267 [002] .... 8255.326007: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=109824
dd-4267 [002] .... 8255.327498: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=111872
dd-4267 [002] .... 8255.328994: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=113920
dd-4267 [002] .... 8255.330478: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=115968
dd-4267 [002] .... 8255.332007: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=118016
dd-4267 [002] .... 8255.333557: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=120064
dd-4267 [002] .... 8255.335096: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=122112
dd-4267 [002] .... 8255.341094: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8255.345525: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [002] .... 8255.350156: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
flush-8:0-4272 [003] .... 8255.767124: writeback_single_inode: bdi 8:0: ino=13 state=I_DIRTY_SYNC|I_DIRTY_DATASYNC|I_DIRTY_PAGES dirtied_when=4302921148 age=55 index=127007 to_write=1024 wrote=2079
flush-8:0-4272 [003] .... 8255.767127: writeback_written: bdi 8:0: sb_dev 0:0 nr_pages=21763 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-8:0-4272 [003] .... 8255.767128: writeback_start: bdi 8:0: sb_dev 0:0 nr_pages=21763 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-8:0-4272 [003] .... 8255.767128: writeback_queue_io: bdi 8:0: older=4302897451 age=30000 enqueue=0 reason=periodic
dd-4267 [002] .... 8255.913486: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
flush-8:0-4272 [003] .... 8256.148983: writeback_single_inode: bdi 8:0: ino=13 state=I_DIRTY_PAGES dirtied_when=4302921148 age=55 index=141755 to_write=1024 wrote=5901
flush-8:0-4272 [003] .... 8256.148987: writeback_written: bdi 8:0: sb_dev 0:0 nr_pages=7015 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-8:0-4272 [003] .... 8256.148988: writeback_start: bdi 8:0: sb_dev 0:0 nr_pages=7015 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-8:0-4272 [003] .... 8256.148988: writeback_queue_io: bdi 8:0: older=4302897833 age=30000 enqueue=0 reason=periodic
dd-4267 [002] .... 8256.369200: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
flush-8:0-4272 [003] .... 8256.628529: writeback_single_inode: bdi 8:0: ino=13 state=I_DIRTY_PAGES dirtied_when=4302921148 age=55 index=150440 to_write=1024 wrote=6452
flush-8:0-4272 [003] .... 8256.628533: writeback_written: bdi 8:0: sb_dev 0:0 nr_pages=-1670 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-8:0-4272 [003] .... 8256.628536: writeback_pages_written: 50856
flush-8:0-4272 [003] .... 8256.628538: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=147456
flush-8:0-4272 [003] .... 8256.628545: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=149504
flush-8:0-4272 [003] .... 8256.630190: writeback_pages_written: 1112
dd-4267 [002] .... 8256.693086: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [003] .... 8256.909199: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [003] .... 8257.016875: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [003] .... 8257.041563: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [003] .... 8257.106900: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [003] .... 8257.246224: writeback_congestion_wait: usec_timeout=100000 usec_delayed=8000
dd-4267 [003] .... 8257.271030: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [003] .... 8257.348551: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [003] .... 8257.381224: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
flush-8:0-4272 [003] .... 8257.393446: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=151552
flush-8:0-4272 [003] .... 8257.398224: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=151552
flush-8:0-4272 [003] .... 8257.400851: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=153600
dd-4267 [003] .... 8257.434020: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [003] .... 8257.541621: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [003] .... 8257.840373: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [002] .... 8258.031388: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [002] .... 8258.054534: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
flush-8:0-4272 [002] .... 8258.104190: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=175872
flush-8:0-4272 [002] .... 8258.107511: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=512 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=175872
flush-8:0-4272 [002] .... 8258.107923: writeback_pages_written: 512
dd-4267 [002] .... 8258.126205: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [002] .... 8258.130305: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=176384
dd-4267 [002] .... 8258.130881: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [002] .... 8258.133116: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [002] .... 8258.137594: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [002] .... 8258.257151: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [002] .... 8258.357770: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [002] .... 8258.406325: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [002] .... 8258.432124: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [002] .... 8258.452275: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [002] .... 8258.472362: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [002] .... 8258.492576: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
flush-8:0-4272 [002] .... 8258.518908: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=176640
flush-8:0-4272 [002] .... 8258.523446: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=176640
flush-8:0-4272 [002] .... 8258.523453: writeback_pages_written: 256
dd-4267 [002] .... 8258.545795: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
flush-8:0-4272 [002] .... 8258.559572: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=176896
flush-8:0-4272 [002] .... 8258.564143: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=176896
flush-8:0-4272 [002] .... 8258.567279: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=178944
flush-8:0-4272 [002] .... 8258.575307: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=180992
dd-4267 [002] .... 8258.779190: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=192256
dd-4267 [002] .... 8258.779212: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=192256
dd-4267 [002] .... 8258.784476: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=193280
dd-4267 [002] .... 8258.784497: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=193280
dd-4267 [002] .... 8258.786546: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=195328
dd-4267 [002] .... 8258.788222: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=197376
dd-4267 [002] .... 8258.789834: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=199424
dd-4267 [003] .... 8259.091794: writeback_congestion_wait: usec_timeout=100000 usec_delayed=9000
dd-4267 [003] .... 8259.103932: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [003] .... 8259.142635: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
flush-8:0-4272 [003] .... 8259.166727: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=200448
flush-8:0-4272 [003] .... 8259.169783: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=200448
flush-8:0-4272 [003] .... 8259.169790: writeback_pages_written: 256
dd-4267 [003] .... 8259.183714: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
flush-8:0-4272 [003] .... 8259.211788: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=200704
flush-8:0-4272 [003] .... 8259.214924: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=200704
flush-8:0-4272 [003] .... 8259.217800: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=202752
flush-8:0-4272 [003] .... 8259.222920: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1024 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=204800
flush-8:0-4272 [003] .... 8259.226184: writeback_pages_written: 5120
dd-4267 [003] .... 8259.257092: writeback_congestion_wait: usec_timeout=100000 usec_delayed=35000
dd-4267 [003] .... 8259.275740: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=205824
dd-4267 [003] .... 8259.275756: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=205824
dd-4267 [003] .... 8259.277624: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [003] .... 8259.298203: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [003] .... 8259.318745: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [003] .... 8259.451826: writeback_congestion_wait: usec_timeout=100000 usec_delayed=54000
dd-4267 [003] .... 8259.465856: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [003] .... 8259.499584: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [003] .... 8259.518780: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [003] .... 8259.538457: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [003] .... 8259.558070: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
flush-8:0-4272 [003] .... 8259.570146: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=206080
flush-8:0-4272 [003] .... 8259.573737: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=512 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=206080
flush-8:0-4272 [003] .... 8259.574164: writeback_pages_written: 512
dd-4267 [003] .... 8259.577670: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
flush-8:0-4272 [003] .... 8259.594106: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=206592
flush-8:0-4272 [003] .... 8259.597848: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=206592
flush-8:0-4272 [003] .... 8259.600521: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=208640
flush-8:0-4272 [003] .... 8259.606747: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=210688
flush-8:0-4272 [003] .... 8259.612775: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=212736
flush-8:0-4272 [003] .... 8259.618277: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1792 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=214784
dd-4267 [003] .... 8259.656171: writeback_congestion_wait: usec_timeout=100000 usec_delayed=37000
dd-4267 [003] .... 8259.890390: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=216576
dd-4267 [003] .... 8259.890404: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=216576
dd-4267 [003] .... 8259.891900: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=218624
dd-4267 [003] .... 8259.895754: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=220672
dd-4267 [003] .... 8259.897308: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=222720
dd-4267 [003] .... 8259.902780: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [003] .... 8259.908417: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [003] .... 8260.079480: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [003] .... 8260.084487: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=224768
dd-4267 [003] .... 8260.084506: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=224768
dd-4267 [003] .... 8260.107416: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [003] .... 8260.173675: writeback_congestion_wait: usec_timeout=100000 usec_delayed=38000
dd-4267 [003] .... 8261.028667: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [003] .... 8261.148521: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
flush-8:0-4272 [002] .... 8261.283270: writeback_start: bdi 8:0: sb_dev 0:0 nr_pages=26057 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-8:0-4272 [002] .... 8261.283272: writeback_queue_io: bdi 8:0: older=4302902970 age=30000 enqueue=0 reason=periodic
flush-8:0-4272 [002] .... 8261.285234: writeback_single_inode: bdi 8:0: ino=13 state=I_DIRTY_PAGES dirtied_when=4302921148 age=55 index=250624 to_write=1024 wrote=1024
flush-8:0-4272 [002] .... 8261.285237: writeback_written: bdi 8:0: sb_dev 0:0 nr_pages=25033 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-8:0-4272 [002] .... 8261.285237: writeback_start: bdi 8:0: sb_dev 0:0 nr_pages=25033 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-8:0-4272 [002] .... 8261.285238: writeback_queue_io: bdi 8:0: older=4302902972 age=30000 enqueue=0 reason=periodic
flush-8:0-4272 [002] .... 8261.556566: writeback_single_inode: bdi 8:0: ino=13 state=I_DIRTY_PAGES dirtied_when=4302921148 age=55 index=258435 to_write=1024 wrote=4472
flush-8:0-4272 [002] .... 8261.556570: writeback_written: bdi 8:0: sb_dev 0:0 nr_pages=17222 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-8:0-4272 [002] .... 8261.556571: writeback_start: bdi 8:0: sb_dev 0:0 nr_pages=17222 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-8:0-4272 [002] .... 8261.556571: writeback_queue_io: bdi 8:0: older=4302903244 age=30000 enqueue=0 reason=periodic
dd-4267 [003] .... 8261.828188: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [003] .... 8261.930822: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=272384
dd-4267 [003] .... 8261.934217: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=274432
dd-4267 [003] .... 8262.188991: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
flush-8:0-4272 [002] .... 8262.287291: writeback_single_inode: bdi 8:0: ino=13 state=I_DIRTY_PAGES dirtied_when=4302921148 age=55 index=276437 to_write=1024 wrote=6356
flush-8:0-4272 [002] .... 8262.287296: writeback_written: bdi 8:0: sb_dev 0:0 nr_pages=-780 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-8:0-4272 [002] .... 8262.287298: writeback_pages_written: 51413
flush-8:0-4272 [002] .... 8262.287300: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=272384
flush-8:0-4272 [002] .... 8262.287308: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1280 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=274432
flush-8:0-4272 [002] .... 8262.287312: writeback_pages_written: 0
##### CPU 1 buffer started ####
dd-4267 [001] .... 8262.412091: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=276224
flush-8:0-4272 [002] .... 8262.412109: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=276224
dd-4267 [001] .... 8262.412113: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=276224
flush-8:0-4272 [002] .... 8262.412273: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=512 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=276224
dd-4267 [001] .... 8262.412843: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
flush-8:0-4272 [002] .... 8262.413228: writeback_pages_written: 299
dd-4267 [002] .... 8262.432804: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [002] .... 8262.452739: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [002] .... 8262.472704: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [002] .... 8262.546051: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [002] .... 8262.585998: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [002] .... 8262.635683: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8262.655672: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8262.675597: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8262.695554: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8262.717581: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [002] .... 8262.738142: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [002] .... 8262.758666: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [002] .... 8262.780860: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [002] .... 8262.801393: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8262.811726: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [002] .... 8262.821976: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [002] .... 8262.842505: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
flush-8:0-4272 [002] .... 8262.855614: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=276736
flush-8:0-4272 [002] .... 8262.862079: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=276736
dd-4267 [001] .... 8262.863088: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=280832
dd-4267 [001] .... 8262.864811: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=282880
flush-8:0-4272 [002] .... 8262.865154: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=278784
dd-4267 [001] .... 8262.866948: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=284928
dd-4267 [001] .... 8262.868516: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=286976
dd-4267 [002] .... 8262.873680: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
flush-8:0-4272 [001] .... 8263.074676: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=282880
flush-8:0-4272 [001] .... 8263.082377: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=284928
flush-8:0-4272 [001] .... 8263.088659: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1792 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=286976
flush-8:0-4272 [001] .... 8263.093872: writeback_pages_written: 12032
dd-4267 [002] .... 8263.162787: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=288768
flush-8:0-4272 [001] .... 8263.162806: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=288768
dd-4267 [002] .... 8263.162807: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=288768
dd-4267 [002] .... 8263.166798: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=290816
flush-8:0-4272 [001] .... 8263.167706: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=288768
dd-4267 [002] .... 8263.168416: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=292864
dd-4267 [002] .... 8263.170061: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=294912
flush-8:0-4272 [001] .... 8263.172041: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=290816
dd-4267 [002] .... 8263.174392: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=296960
dd-4267 [002] .... 8263.178723: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=299008
flush-8:0-4272 [001] .... 8263.178800: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=292864
dd-4267 [002] .... 8263.180744: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [002] .... 8263.184819: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
flush-8:0-4272 [001] .... 8263.184948: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=294912
flush-8:0-4272 [001] .... 8263.189635: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=296960
flush-8:0-4272 [001] .... 8263.193882: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1280 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=299008
flush-8:0-4272 [001] .... 8263.196897: writeback_pages_written: 11520
dd-4267 [001] .... 8263.371538: writeback_congestion_wait: usec_timeout=100000 usec_delayed=22000
dd-4267 [001] .... 8263.391105: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [001] .... 8263.397755: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [001] .... 8263.417164: writeback_congestion_wait: usec_timeout=100000 usec_delayed=15000
dd-4267 [003] .... 8263.421709: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [001] .... 8263.424214: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [001] .... 8263.466360: writeback_congestion_wait: usec_timeout=100000 usec_delayed=43000
dd-4267 [001] .... 8263.482457: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=300288
flush-8:0-4272 [003] .... 8263.482473: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=300288
dd-4267 [001] .... 8263.482480: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=300288
dd-4267 [001] .... 8263.483957: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=302336
dd-4267 [001] .... 8263.485413: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=304384
flush-8:0-4272 [003] .... 8263.485563: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=300288
flush-8:0-4272 [003] .... 8263.488444: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=302336
flush-8:0-4272 [003] .... 8263.493375: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1792 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=304384
flush-8:0-4272 [003] .... 8263.497569: writeback_pages_written: 5888
dd-4267 [003] .... 8263.586187: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [003] .... 8263.621915: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=306176
flush-8:0-4272 [001] .... 8263.621932: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=306176
dd-4267 [003] .... 8263.621932: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=306176
flush-8:0-4272 [001] .... 8263.624277: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=306176
flush-8:0-4272 [001] .... 8263.624284: writeback_pages_written: 256
dd-4267 [001] .... 8263.713731: writeback_congestion_wait: usec_timeout=100000 usec_delayed=92000
dd-4267 [003] .... 8263.762825: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [001] .... 8263.772571: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8263.782397: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8263.786346: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [003] .... 8263.792063: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8263.794373: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [001] .... 8263.802026: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [001] .... 8263.823655: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=306432
flush-8:0-4272 [003] .... 8263.823672: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=306432
dd-4267 [001] .... 8263.823673: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=306432
dd-4267 [001] .... 8263.826272: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [001] .... 8263.828813: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [001] .... 8263.831328: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [003] .... 8263.835578: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [001] .... 8263.843013: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8263.852831: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8263.862617: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [003] .... 8263.872450: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8263.894518: writeback_congestion_wait: usec_timeout=100000 usec_delayed=18000
dd-4267 [001] .... 8263.898954: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [001] .... 8263.942418: writeback_congestion_wait: usec_timeout=100000 usec_delayed=44000
dd-4267 [001] .... 8263.962011: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=306688
dd-4267 [001] .... 8263.962030: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=306688
dd-4267 [001] .... 8263.963721: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=308736
dd-4267 [001] .... 8263.965316: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=310784
dd-4267 [001] .... 8263.966932: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=312832
dd-4267 [003] .... 8263.969872: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [001] .... 8264.003468: writeback_congestion_wait: usec_timeout=100000 usec_delayed=31000
dd-4267 [001] .... 8264.005330: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [001] .... 8264.007494: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [001] .... 8264.172407: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [003] .... 8264.179889: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [003] .... 8264.185841: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [003] .... 8264.190127: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [001] .... 8264.196118: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [003] .... 8264.200459: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [003] .... 8264.206405: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [003] .... 8264.210702: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [001] .... 8264.216654: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [003] .... 8264.221002: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [003] .... 8264.226948: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [003] .... 8264.231270: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [001] .... 8264.237169: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [003] .... 8264.241569: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [003] .... 8264.247491: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [003] .... 8264.249307: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=313856
dd-4267 [003] .... 8264.249328: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=313856
dd-4267 [003] .... 8264.250997: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=315904
dd-4267 [003] .... 8264.251768: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [003] .... 8264.255542: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=317952
dd-4267 [003] .... 8264.257148: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=320000
dd-4267 [001] .... 8264.257726: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [001] .... 8264.262047: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [001] .... 8264.268005: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8264.270266: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=321792
dd-4267 [001] .... 8264.271910: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=323840
dd-4267 [003] .... 8264.373787: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [001] .... 8264.530715: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
flush-8:0-4272 [002] .... 8264.602404: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=325120
flush-8:0-4272 [002] .... 8264.605571: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=325120
flush-8:0-4272 [002] .... 8264.608617: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=327168
flush-8:0-4272 [002] .... 8264.613823: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1280 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=329216
flush-8:0-4272 [002] .... 8264.617055: writeback_pages_written: 5376
dd-4267 [003] .... 8264.706635: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [003] .... 8264.777165: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=330496
flush-8:0-4272 [002] .... 8264.777184: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=330496
dd-4267 [003] .... 8264.777184: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=330496
flush-8:0-4272 [002] .... 8264.780903: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=330496
flush-8:0-4272 [002] .... 8264.780910: writeback_pages_written: 256
dd-4267 [002] .... 8264.781921: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [002] .... 8264.821810: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [002] .... 8264.845760: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [002] .... 8264.875710: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
flush-8:0-4272 [002] .... 8264.887380: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=330752
flush-8:0-4272 [002] .... 8264.890912: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=330752
flush-8:0-4272 [002] .... 8264.890919: writeback_pages_written: 256
dd-4267 [002] .... 8264.901596: writeback_congestion_wait: usec_timeout=100000 usec_delayed=8000
dd-4267 [002] .... 8264.941549: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [002] .... 8264.965702: writeback_congestion_wait: usec_timeout=100000 usec_delayed=8000
flush-8:0-4272 [002] .... 8264.983529: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=331008
flush-8:0-4272 [002] .... 8264.987449: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=331008
flush-8:0-4272 [002] .... 8264.987456: writeback_pages_written: 256
dd-4267 [002] .... 8264.991277: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
flush-8:0-4272 [002] .... 8265.017144: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=331264
flush-8:0-4272 [002] .... 8265.021013: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=331264
flush-8:0-4272 [002] .... 8265.024087: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=333312
flush-8:0-4272 [002] .... 8265.030297: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=335360
flush-8:0-4272 [002] .... 8265.036048: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1024 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=337408
flush-8:0-4272 [002] .... 8265.039951: writeback_pages_written: 7168
dd-4267 [002] .... 8265.057223: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8265.077361: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [002] .... 8265.097530: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8265.117633: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8265.143609: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [002] .... 8265.163795: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8265.183921: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8265.204076: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
flush-8:0-4272 [002] .... 8265.211688: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=338432
flush-8:0-4272 [002] .... 8265.215220: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=338432
flush-8:0-4272 [002] .... 8265.215226: writeback_pages_written: 256
dd-4267 [002] .... 8265.224208: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
flush-8:0-4272 [002] .... 8265.231866: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=338688
flush-8:0-4272 [002] .... 8265.235287: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=338688
flush-8:0-4272 [002] .... 8265.238119: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=340736
flush-8:0-4272 [002] .... 8265.243774: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=342784
flush-8:0-4272 [002] .... 8265.249266: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1536 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=344832
flush-0:16-4223 [003] .... 8265.302236: writeback_start: bdi 0:16: sb_dev 0:0 nr_pages=30929 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-0:16-4223 [003] .... 8265.302237: writeback_queue_io: bdi 0:16: older=4302906992 age=30000 enqueue=0 reason=periodic
flush-0:16-4223 [003] .... 8265.302238: writeback_written: bdi 0:16: sb_dev 0:0 nr_pages=30929 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-0:16-4223 [003] .... 8265.302243: writeback_pages_written: 0
dd-4267 [002] .... 8265.346236: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [002] .... 8265.415236: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=346368
dd-4267 [002] .... 8265.415253: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=346368
dd-4267 [002] .... 8265.416806: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=348416
dd-4267 [002] .... 8265.593749: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [002] .... 8265.597894: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=349696
dd-4267 [002] .... 8265.597911: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=349696
dd-4267 [002] .... 8265.599469: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=351744
dd-4267 [002] .... 8265.601033: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=353792
dd-4267 [002] .... 8265.603547: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
flush-8:0-4272 [001] .... 8265.604475: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=351744
flush-8:0-4272 [001] .... 8265.612575: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1024 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=353792
flush-8:0-4272 [001] .... 8265.615654: writeback_pages_written: 5120
flush-8:0-4272 [001] .... 8265.779243: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=354816
dd-4267 [002] .... 8265.780668: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
flush-8:0-4272 [003] .... 8266.597438: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=354816
flush-8:0-4272 [003] .... 8266.597448: writeback_start: bdi 8:0: sb_dev 0:0 nr_pages=34303 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-8:0-4272 [003] .... 8266.597449: writeback_queue_io: bdi 8:0: older=4302908287 age=30000 enqueue=0 reason=periodic
flush-8:0-4272 [003] .... 8266.599938: writeback_single_inode: bdi 8:0: ino=13 state=I_DIRTY_PAGES dirtied_when=4302921148 age=55 index=356096 to_write=1024 wrote=1024
flush-8:0-4272 [003] .... 8266.599941: writeback_written: bdi 8:0: sb_dev 0:0 nr_pages=33279 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-8:0-4272 [003] .... 8266.599942: writeback_start: bdi 8:0: sb_dev 0:0 nr_pages=33279 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-8:0-4272 [003] .... 8266.599942: writeback_queue_io: bdi 8:0: older=4302908290 age=30000 enqueue=0 reason=periodic
flush-8:0-4272 [003] .... 8266.622191: writeback_single_inode: bdi 8:0: ino=13 state=I_DIRTY_PAGES dirtied_when=4302921148 age=55 index=367701 to_write=1024 wrote=11605
flush-8:0-4272 [003] .... 8266.622196: writeback_written: bdi 8:0: sb_dev 0:0 nr_pages=21674 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-8:0-4272 [003] .... 8266.622196: writeback_start: bdi 8:0: sb_dev 0:0 nr_pages=21674 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-8:0-4272 [003] .... 8266.622197: writeback_queue_io: bdi 8:0: older=4302908312 age=30000 enqueue=0 reason=periodic
flush-8:0-4272 [003] .... 8266.624595: writeback_single_inode: bdi 8:0: ino=13 state=I_DIRTY_PAGES dirtied_when=4302921148 age=55 index=368861 to_write=1024 wrote=1160
flush-8:0-4272 [003] .... 8266.624597: writeback_written: bdi 8:0: sb_dev 0:0 nr_pages=20514 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-8:0-4272 [003] .... 8266.624597: writeback_start: bdi 8:0: sb_dev 0:0 nr_pages=20514 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-8:0-4272 [003] .... 8266.624598: writeback_queue_io: bdi 8:0: older=4302908315 age=30000 enqueue=0 reason=periodic
flush-8:0-4272 [003] .... 8266.625314: writeback_single_inode: bdi 8:0: ino=13 state=I_DIRTY_PAGES dirtied_when=4302938315 age=38 index=368861 to_write=1024 wrote=145
flush-8:0-4272 [003] .... 8266.625316: writeback_written: bdi 8:0: sb_dev 0:0 nr_pages=20369 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-8:0-4272 [003] .... 8266.625317: writeback_start: bdi 8:0: sb_dev 0:0 nr_pages=20369 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-8:0-4272 [003] .... 8266.625317: writeback_queue_io: bdi 8:0: older=4302908315 age=30000 enqueue=0 reason=periodic
flush-8:0-4272 [003] .... 8266.625317: writeback_written: bdi 8:0: sb_dev 0:0 nr_pages=20369 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-8:0-4272 [003] .... 8266.625319: writeback_pages_written: 14190
flush-8:0-4272 [003] .... 8266.825892: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=368896
flush-8:0-4272 [003] .... 8266.826219: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=368896
flush-8:0-4272 [003] .... 8266.827173: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1536 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=369408
flush-8:0-4272 [003] .... 8266.829558: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=370688
flush-8:0-4272 [003] .... 8266.832712: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=372736
flush-8:0-4272 [003] .... 8266.838252: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=374784
flush-8:0-4272 [003] .... 8266.842368: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=376832
dd-4267 [002] .... 8266.935447: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
flush-8:0-4272 [001] .... 8266.999637: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=378880
flush-8:0-4272 [001] .... 8267.145618: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1280 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=380928
flush-8:0-4272 [001] .... 8267.148792: writeback_pages_written: 13202
dd-4267 [001] .... 8267.351195: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [001] .... 8267.361164: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [003] .... 8267.370981: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8267.380830: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8267.390569: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8267.400401: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [003] .... 8267.410227: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [001] .... 8267.420027: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [001] .... 8267.429869: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8267.439597: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [003] .... 8267.449465: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8267.459294: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [001] .... 8267.469116: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8267.477442: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [003] .... 8267.481950: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [001] .... 8267.486445: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [001] .... 8267.490959: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [001] .... 8267.609943: writeback_congestion_wait: usec_timeout=100000 usec_delayed=97000
dd-4267 [003] .... 8267.614031: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [001] .... 8267.618624: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [001] .... 8267.628386: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [003] .... 8267.635454: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [001] .... 8267.639944: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [001] .... 8267.644410: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [001] .... 8267.648982: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [003] .... 8267.654300: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [001] .... 8267.658700: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [002] .... 8267.667206: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [001] .... 8267.669674: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [001] .... 8267.674305: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [001] .... 8267.706797: writeback_congestion_wait: usec_timeout=100000 usec_delayed=18000
dd-4267 [001] .... 8267.711275: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [003] .... 8267.719838: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [001] .... 8267.729680: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [001] .... 8267.737881: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [001] .... 8267.742316: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [001] .... 8267.752567: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=382208
flush-8:0-4272 [003] .... 8267.752583: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=382208
dd-4267 [001] .... 8267.752584: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=382208
dd-4267 [001] .... 8267.754070: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=384256
dd-4267 [001] .... 8267.755594: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=386304
dd-4267 [001] .... 8267.757105: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=388352
flush-8:0-4272 [003] .... 8267.757895: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=382208
dd-4267 [001] .... 8267.758599: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=390400
dd-4267 [001] .... 8267.760122: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=392448
flush-8:0-4272 [003] .... 8267.760868: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=384256
flush-8:0-4272 [003] .... 8267.768745: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=386304
flush-8:0-4272 [003] .... 8267.776742: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=388352
flush-8:0-4272 [003] .... 8267.783814: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=390400
flush-8:0-4272 [003] .... 8267.790773: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=392448
flush-8:0-4272 [003] .... 8267.797382: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=394496
flush-8:0-4272 [003] .... 8267.803995: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=396544
flush-8:0-4272 [003] .... 8267.809316: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=398592
flush-8:0-4272 [003] .... 8267.813952: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=400640
flush-8:0-4272 [003] .... 8267.818094: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=402688
dd-4267 [002] .... 8267.905908: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
flush-8:0-4272 [001] .... 8267.938284: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1792 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=404736
flush-8:0-4272 [001] .... 8267.943129: writeback_pages_written: 24320
dd-4267 [001] .... 8268.163764: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8268.173986: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [003] .... 8268.184334: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [001] .... 8268.194576: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8268.198511: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=406528
flush-8:0-4272 [003] .... 8268.198524: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=406528
dd-4267 [001] .... 8268.198525: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=406528
flush-8:0-4272 [003] .... 8268.202131: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1280 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=406528
flush-8:0-4272 [003] .... 8268.204478: writeback_pages_written: 1280
dd-4267 [001] .... 8268.204959: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8268.209828: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=407808
flush-8:0-4272 [003] .... 8268.209837: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=407808
dd-4267 [001] .... 8268.209837: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=407808
flush-8:0-4272 [003] .... 8268.212352: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=407808
flush-8:0-4272 [003] .... 8268.212358: writeback_pages_written: 256
dd-4267 [001] .... 8268.215084: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [003] .... 8268.227002: writeback_congestion_wait: usec_timeout=100000 usec_delayed=8000
dd-4267 [001] .... 8268.237303: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8268.247561: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8268.257823: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [003] .... 8268.268081: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8268.278409: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8268.288799: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [001] .... 8268.298937: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [003] .... 8268.309285: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8268.319485: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8268.323556: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=408064
flush-8:0-4272 [003] .... 8268.323569: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=408064
dd-4267 [001] .... 8268.323570: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=408064
flush-8:0-4272 [003] .... 8268.326837: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=408064
flush-8:0-4272 [003] .... 8268.326843: writeback_pages_written: 256
dd-4267 [001] .... 8268.329773: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [003] .... 8268.333919: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [001] .... 8268.339844: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8268.345842: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8268.350165: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [003] .... 8268.356091: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8268.358321: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [001] .... 8268.362952: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [001] .... 8268.383814: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [001] .... 8268.437266: writeback_congestion_wait: usec_timeout=100000 usec_delayed=52000
dd-4267 [001] .... 8268.442101: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [001] .... 8268.444583: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [003] .... 8268.448725: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [003] .... 8268.457661: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [001] .... 8268.467635: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8268.477643: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [001] .... 8268.487599: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8268.497414: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8268.501582: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [001] .... 8268.507385: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [003] .... 8268.511569: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [001] .... 8268.517350: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8268.521564: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [001] .... 8268.527355: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [003] .... 8268.531537: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [001] .... 8268.533925: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [001] .... 8268.539707: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8268.561714: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=408320
flush-8:0-4272 [003] .... 8268.561729: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=408320
dd-4267 [001] .... 8268.561736: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=408320
dd-4267 [001] .... 8268.562225: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [001] .... 8268.564550: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
flush-8:0-4272 [003] .... 8268.566727: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=408320
flush-8:0-4272 [003] .... 8268.566734: writeback_pages_written: 256
dd-4267 [001] .... 8268.567268: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [001] .... 8268.569268: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [003] .... 8268.575561: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [001] .... 8268.579702: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [001] .... 8268.585533: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8268.589691: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [003] .... 8268.595494: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8268.599670: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [001] .... 8268.605495: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8268.609659: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [003] .... 8268.615463: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [003] .... 8268.617783: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=408576
flush-8:0-4272 [001] .... 8268.617795: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=408576
dd-4267 [003] .... 8268.617796: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=408576
dd-4267 [003] .... 8268.619270: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=410624
dd-4267 [003] .... 8268.622449: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=412672
flush-8:0-4272 [001] .... 8268.622860: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=408576
dd-4267 [003] .... 8268.623989: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=414720
flush-8:0-4272 [001] .... 8268.625976: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=410624
dd-4267 [003] .... 8268.635422: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
flush-8:0-4272 [001] .... 8268.840617: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=412672
flush-8:0-4272 [001] .... 8268.848404: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=414720
dd-4267 [003] .... 8268.930278: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=419584
dd-4267 [003] .... 8268.930296: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=419584
dd-4267 [003] .... 8268.931994: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=421632
dd-4267 [003] .... 8268.934099: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=423680
dd-4267 [003] .... 8268.935726: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=425728
dd-4267 [003] .... 8268.937352: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=427776
dd-4267 [003] .... 8268.939046: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=429824
dd-4267 [001] .... 8269.039200: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [003] .... 8269.139135: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [001] .... 8269.239185: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [001] .... 8269.408981: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [001] .... 8269.584400: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=431104
dd-4267 [001] .... 8269.584469: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=431104
dd-4267 [001] .... 8269.593152: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=431360
dd-4267 [001] .... 8269.597285: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=431616
dd-4267 [001] .... 8269.597492: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=431616
dd-4267 [001] .... 8269.598967: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=433664
dd-4267 [001] .... 8269.614402: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=434944
dd-4267 [001] .... 8269.614421: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=434944
dd-4267 [001] .... 8269.616055: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=436992
dd-4267 [002] .... 8269.637753: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [002] .... 8269.680296: writeback_congestion_wait: usec_timeout=100000 usec_delayed=29000
dd-4267 [002] .... 8269.817372: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
flush-8:0-4272 [002] .... 8269.864374: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=437248
flush-8:0-4272 [002] .... 8269.868771: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=437248
dd-4267 [001] .... 8269.871123: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=443392
flush-8:0-4272 [002] .... 8269.871919: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=439296
dd-4267 [001] .... 8269.875734: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=445440
dd-4267 [001] .... 8269.877352: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=447488
flush-8:0-4272 [002] .... 8269.878824: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=441344
dd-4267 [001] .... 8269.881258: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=449536
dd-4267 [002] .... 8269.885307: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8269.889882: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
flush-8:0-4272 [002] .... 8269.936593: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=443392
flush-8:0-4272 [002] .... 8269.942755: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=445440
flush-8:0-4272 [002] .... 8269.948672: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=447488
flush-8:0-4272 [002] .... 8269.953747: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=449536
flush-8:0-4272 [002] .... 8269.956079: writeback_pages_written: 13056
dd-4267 [001] .... 8270.015147: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=450304
flush-8:0-4272 [002] .... 8270.015168: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=450304
dd-4267 [001] .... 8270.015174: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=450304
flush-8:0-4272 [002] .... 8270.018425: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=450304
flush-8:0-4272 [002] .... 8270.019354: writeback_pages_written: 768
dd-4267 [002] .... 8270.064531: writeback_congestion_wait: usec_timeout=100000 usec_delayed=48000
dd-4267 [002] .... 8270.106521: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=451072
dd-4267 [002] .... 8270.106542: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=451072
dd-4267 [002] .... 8270.109830: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=452352
dd-4267 [002] .... 8270.111279: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=454400
flush-8:0-4272 [002] .... 8270.274267: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=455424
flush-8:0-4272 [002] .... 8270.277302: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=455424
flush-8:0-4272 [002] .... 8270.277308: writeback_pages_written: 256
dd-4267 [002] .... 8270.281460: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
flush-0:16-4223 [001] .... 8270.300792: writeback_start: bdi 0:16: sb_dev 0:0 nr_pages=34352 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-0:16-4223 [001] .... 8270.300795: writeback_queue_io: bdi 0:16: older=4302911993 age=30000 enqueue=1 reason=periodic
flush-0:16-4223 [001] .... 8270.300805: writeback_single_inode: bdi 0:16: ino=4874359 state= dirtied_when=4302911988 age=65 index=0 to_write=1024 wrote=0
flush-0:16-4223 [001] .... 8270.300808: writeback_written: bdi 0:16: sb_dev 0:0 nr_pages=34352 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-0:16-4223 [001] .... 8270.300809: writeback_start: bdi 0:16: sb_dev 0:0 nr_pages=34352 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-0:16-4223 [001] .... 8270.300809: writeback_queue_io: bdi 0:16: older=4302911993 age=30000 enqueue=0 reason=periodic
flush-0:16-4223 [001] .... 8270.300809: writeback_written: bdi 0:16: sb_dev 0:0 nr_pages=34352 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-0:16-4223 [001] .... 8270.300814: writeback_pages_written: 0
dd-4267 [002] .... 8270.302002: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [002] .... 8270.316621: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8270.343276: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [002] .... 8270.373944: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [002] .... 8270.394471: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [002] .... 8270.415038: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8270.435568: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [002] .... 8270.472530: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [002] .... 8270.513637: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [002] .... 8270.517612: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=455680
dd-4267 [002] .... 8270.517625: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=455680
dd-4267 [002] .... 8270.554764: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [002] .... 8270.558840: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=455936
dd-4267 [002] .... 8270.558852: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=455936
dd-4267 [002] .... 8270.560339: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=457984
dd-4267 [002] .... 8270.561810: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=460032
dd-4267 [002] .... 8270.564781: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8270.575044: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
flush-8:0-4272 [002] .... 8270.590611: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=461056
flush-8:0-4272 [002] .... 8270.594701: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=461056
flush-8:0-4272 [002] .... 8270.598284: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=463104
flush-8:0-4272 [002] .... 8270.600496: writeback_pages_written: 2304
dd-4267 [002] .... 8270.601943: writeback_congestion_wait: usec_timeout=100000 usec_delayed=9000
dd-4267 [002] .... 8270.633486: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
flush-8:0-4272 [002] .... 8270.657444: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=463360
flush-8:0-4272 [002] .... 8270.660617: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=463360
flush-8:0-4272 [002] .... 8270.660624: writeback_pages_written: 256
dd-4267 [002] .... 8270.673407: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [002] .... 8270.703164: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [002] .... 8270.723126: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [002] .... 8270.743057: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [002] .... 8270.883761: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=463616
dd-4267 [002] .... 8270.883782: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=463616
dd-4267 [002] .... 8270.885303: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=465664
dd-4267 [002] .... 8270.886785: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=467712
dd-4267 [002] .... 8270.888382: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=469760
dd-4267 [002] .... 8270.889945: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=471808
flush-8:0-4272 [001] .... 8271.327896: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=467712
flush-8:0-4272 [001] .... 8271.334733: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=469760
flush-8:0-4272 [001] .... 8271.340404: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=471808
flush-8:0-4272 [001] .... 8271.345681: writeback_pages_written: 10240
dd-4267 [002] .... 8271.421367: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=473856
flush-8:0-4272 [001] .... 8271.421387: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=473856
dd-4267 [002] .... 8271.421390: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=473856
dd-4267 [002] .... 8271.423065: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=475904
dd-4267 [002] .... 8271.425186: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=477952
flush-8:0-4272 [001] .... 8271.425640: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=473856
dd-4267 [002] .... 8271.426874: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=480000
flush-8:0-4272 [001] .... 8271.492067: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=475904
flush-8:0-4272 [001] .... 8271.497902: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=477952
flush-8:0-4272 [001] .... 8271.502398: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1280 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=480000
flush-8:0-4272 [001] .... 8271.505244: writeback_pages_written: 7424
dd-4267 [002] .... 8271.678718: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [001] .... 8271.777600: writeback_congestion_wait: usec_timeout=100000 usec_delayed=98000
dd-4267 [001] .... 8271.817785: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [001] .... 8271.827867: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8271.837939: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [003] .... 8271.847990: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [001] .... 8271.858059: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8271.868133: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8271.878146: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [003] .... 8271.888271: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8271.898364: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8271.903945: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [001] .... 8271.908242: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [003] .... 8271.910635: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [001] .... 8271.918477: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [001] .... 8271.941108: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [001] .... 8271.945668: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [001] .... 8271.948169: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [001] .... 8271.954621: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [001] .... 8271.958741: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=481280
flush-8:0-4272 [003] .... 8271.958757: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=481280
dd-4267 [001] .... 8271.958758: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=481280
dd-4267 [001] .... 8271.960266: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=483328
dd-4267 [001] .... 8271.961761: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=485376
flush-8:0-4272 [003] .... 8271.962732: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=481280
dd-4267 [001] .... 8271.963264: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=487424
dd-4267 [001] .... 8271.964441: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [001] .... 8271.974773: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [003] .... 8272.009232: writeback_congestion_wait: usec_timeout=100000 usec_delayed=30000
dd-4267 [001] .... 8272.029378: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [001] .... 8272.035038: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [003] .... 8272.039234: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [001] .... 8272.045124: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8272.049333: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [001] .... 8272.051819: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [003] .... 8272.057558: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [003] .... 8272.078026: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [001] .... 8272.082691: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [001] .... 8272.090003: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [003] .... 8272.100194: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8272.110246: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8272.120279: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8272.130351: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [003] .... 8272.136822: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [001] .... 8272.142901: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [001] .... 8272.183805: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=488192
dd-4267 [001] .... 8272.183823: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=488192
dd-4267 [001] .... 8272.195352: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=489216
dd-4267 [001] .... 8272.195373: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=489216
dd-4267 [001] .... 8272.197069: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=491264
dd-4267 [001] .... 8272.198688: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=493312
dd-4267 [001] .... 8272.200844: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=495360
dd-4267 [001] .... 8272.202448: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=497408
dd-4267 [003] .... 8272.302380: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [001] .... 8272.402340: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
##### CPU 0 buffer started ####
dd-4267 [000] .... 8272.587442: writeback_congestion_wait: usec_timeout=100000 usec_delayed=95000
dd-4267 [000] .... 8272.589707: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [000] .... 8272.594405: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8272.603546: writeback_congestion_wait: usec_timeout=100000 usec_delayed=8000
dd-4267 [000] .... 8272.613529: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8272.617066: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=498176
flush-8:0-4272 [002] .... 8272.617081: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=498176
dd-4267 [000] .... 8272.617086: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=498176
dd-4267 [000] .... 8272.619094: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=500224
flush-8:0-4272 [002] .... 8272.620566: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=498176
dd-4267 [000] .... 8272.620634: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=502272
dd-4267 [000] .... 8272.622174: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=504320
flush-8:0-4272 [002] .... 8272.623787: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=500224
flush-8:0-4272 [002] .... 8272.629276: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=502272
flush-8:0-4272 [002] .... 8272.633956: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1792 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=504320
flush-8:0-4272 [002] .... 8272.637736: writeback_pages_written: 7936
dd-4267 [003] .... 8272.728244: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [003] .... 8272.816879: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=506112
flush-8:0-4272 [002] .... 8272.816896: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=506112
dd-4267 [003] .... 8272.816897: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=506112
flush-8:0-4272 [002] .... 8272.820965: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=506112
flush-8:0-4272 [002] .... 8272.822144: writeback_pages_written: 768
dd-4267 [000] .... 8272.883681: writeback_congestion_wait: usec_timeout=100000 usec_delayed=66000
dd-4267 [000] .... 8272.901202: writeback_congestion_wait: usec_timeout=100000 usec_delayed=9000
dd-4267 [000] .... 8272.911472: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8272.921776: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8272.932042: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [000] .... 8272.942300: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [002] .... 8272.952536: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8272.962856: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8272.972948: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [002] .... 8272.977268: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8272.983253: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8272.989219: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8272.993531: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8272.999473: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8273.003785: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8273.009743: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8273.014080: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8273.020012: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8273.024305: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8273.030315: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8273.033647: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=506880
flush-8:0-4272 [002] .... 8273.033663: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=506880
dd-4267 [000] .... 8273.033664: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=506880
dd-4267 [000] .... 8273.035857: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=508928
flush-8:0-4272 [002] .... 8273.037953: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=506880
flush-8:0-4272 [000] .... 8273.118847: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=508928
flush-8:0-4272 [000] .... 8273.125008: writeback_pages_written: 2816
dd-4267 [001] .... 8273.201257: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=509696
flush-8:0-4272 [000] .... 8273.201275: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=509696
dd-4267 [001] .... 8273.201283: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=509696
dd-4267 [001] .... 8273.202988: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=511744
dd-4267 [001] .... 8273.204095: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [001] .... 8273.204351: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
flush-8:0-4272 [000] .... 8273.207289: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=509696
flush-8:0-4272 [000] .... 8273.210162: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=511744
dd-4267 [001] .... 8273.214288: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
flush-8:0-4272 [000] .... 8273.214741: writeback_pages_written: 2816
dd-4267 [002] .... 8273.220237: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8273.224624: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8273.230590: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8273.234929: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [002] .... 8273.240847: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8273.245152: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8273.252550: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [000] .... 8273.256650: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8273.262386: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8273.266457: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8273.272133: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8273.274530: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [002] .... 8273.276786: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8273.281514: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [000] .... 8273.303716: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=512512
flush-8:0-4272 [002] .... 8273.303734: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=512512
dd-4267 [000] .... 8273.303742: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=512512
dd-4267 [000] .... 8273.305253: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=514560
dd-4267 [000] .... 8273.306756: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=516608
flush-8:0-4272 [002] .... 8273.308082: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=512512
dd-4267 [000] .... 8273.308310: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=518656
dd-4267 [001] .... 8273.309955: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=520704
flush-8:0-4272 [002] .... 8273.310937: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=514560
flush-8:0-4272 [002] .... 8273.317219: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=516608
flush-8:0-4272 [002] .... 8273.322499: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=518656
flush-8:0-4272 [002] .... 8273.327288: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=520704
flush-8:0-4272 [002] .... 8273.331707: writeback_pages_written: 10240
dd-4267 [003] .... 8273.411759: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [003] .... 8273.467526: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=522752
flush-8:0-4272 [002] .... 8273.467545: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=522752
dd-4267 [003] .... 8273.467547: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=522752
dd-4267 [003] .... 8273.469243: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=524800
dd-4267 [003] .... 8273.470853: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=526848
flush-8:0-4272 [002] .... 8273.471531: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=522752
dd-4267 [003] .... 8273.472458: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
flush-8:0-4272 [002] .... 8273.476836: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=524800
flush-8:0-4272 [002] .... 8273.482005: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1280 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=526848
flush-8:0-4272 [002] .... 8273.484987: writeback_pages_written: 5376
dd-4267 [001] .... 8273.576718: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [001] .... 8273.630560: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=528128
flush-8:0-4272 [002] .... 8273.630579: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=528128
dd-4267 [001] .... 8273.630580: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=528128
dd-4267 [001] .... 8273.632915: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=530176
flush-8:0-4272 [002] .... 8273.634217: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=528128
flush-8:0-4272 [002] .... 8273.637881: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1280 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=530176
flush-8:0-4272 [002] .... 8273.641495: writeback_pages_written: 3328
dd-4267 [003] .... 8273.733643: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [000] .... 8273.803010: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8273.812834: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8273.822636: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [002] .... 8273.832432: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [000] .... 8273.842319: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8273.852182: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8273.861874: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [002] .... 8273.872685: writeback_congestion_wait: usec_timeout=100000 usec_delayed=8000
dd-4267 [000] .... 8273.877222: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [000] .... 8273.881324: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8273.887043: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [002] .... 8273.891135: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8273.896818: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8273.900890: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8273.906661: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [002] .... 8273.910747: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8273.916474: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8273.920550: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8273.926259: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [002] .... 8273.930361: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8273.936114: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8273.940195: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8273.942458: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=531456
flush-8:0-4272 [002] .... 8273.942469: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=531456
dd-4267 [000] .... 8273.942470: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=531456
dd-4267 [000] .... 8273.943968: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=533504
dd-4267 [000] .... 8273.945828: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
flush-8:0-4272 [002] .... 8273.946273: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=531456
flush-8:0-4272 [002] .... 8273.949248: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1536 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=533504
dd-4267 [000] .... 8273.949900: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
flush-8:0-4272 [002] .N.. 8273.954420: writeback_pages_written: 3584
dd-4267 [002] .... 8273.955696: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8273.959774: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8273.965545: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [002] .... 8273.969610: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8273.975340: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8273.979456: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8273.987689: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8273.992204: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8274.026101: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [002] .... 8274.031783: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8274.035939: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8274.040032: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8274.045692: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [002] .... 8274.049762: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8274.055579: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8274.059648: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8274.065335: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [002] .... 8274.069439: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8274.075170: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8274.079279: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8274.086684: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [002] .... 8274.091051: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8274.097019: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8274.102975: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8274.107277: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8274.111257: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=535040
flush-8:0-4272 [002] .... 8274.111272: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=535040
dd-4267 [000] .... 8274.111280: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=535040
dd-4267 [000] .... 8274.114916: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=537088
flush-8:0-4272 [002] .... 8274.115545: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=535040
flush-8:0-4272 [002] .... 8274.118567: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1280 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=537088
dd-4267 [001] .... 8274.120214: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=538112
dd-4267 [001] .... 8274.123035: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
flush-8:0-4272 [002] .... 8274.123531: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=538112
flush-8:0-4272 [002] .... 8274.124428: writeback_pages_written: 3840
dd-4267 [001] .... 8274.125289: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8274.127722: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8274.130996: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [002] .... 8274.133237: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8274.135492: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [002] .... 8274.175907: writeback_congestion_wait: usec_timeout=100000 usec_delayed=40000
dd-4267 [000] .... 8274.290285: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [000] .... 8274.336268: writeback_congestion_wait: usec_timeout=100000 usec_delayed=46000
dd-4267 [002] .... 8274.340912: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [000] .... 8274.347609: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8274.357660: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [002] .... 8274.367884: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8274.373602: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8274.377990: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8274.383934: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8274.386279: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=538880
flush-8:0-4272 [002] .... 8274.386291: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=538880
dd-4267 [000] .... 8274.386292: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=538880
dd-4267 [000] .... 8274.387774: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=540928
flush-8:0-4272 [002] .... 8274.390547: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=538880
dd-4267 [000] .... 8274.391322: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=542976
dd-4267 [000] .... 8274.392891: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=545024
flush-8:0-4272 [000] .... 8274.536676: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=540928
flush-8:0-4272 [000] .... 8274.543142: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=542976
flush-8:0-4272 [000] .... 8274.549982: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1536 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=545024
flush-8:0-4272 [000] .... 8274.554681: writeback_pages_written: 7680
dd-4267 [001] .... 8274.657074: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=546560
flush-8:0-4272 [000] .... 8274.657090: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=546560
dd-4267 [001] .... 8274.657093: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=546560
dd-4267 [001] .... 8274.661141: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=548608
flush-8:0-4272 [000] .... 8274.662544: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=546560
dd-4267 [001] .... 8274.662787: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=550656
flush-8:0-4272 [000] .... 8274.666715: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=548608
dd-4267 [001] .... 8274.666972: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=552704
dd-4267 [001] .... 8274.670755: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=554752
dd-4267 [001] .... 8274.673577: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
flush-8:0-4272 [000] .... 8274.674196: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=550656
dd-4267 [001] .... 8274.677756: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
flush-8:0-4272 [000] .... 8274.680418: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=552704
flush-8:0-4272 [000] .... 8274.685640: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1024 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=554752
flush-8:0-4272 [000] .... 8274.688771: writeback_pages_written: 9216
dd-4267 [003] .... 8274.779091: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [002] .... 8274.843553: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [000] .... 8274.853430: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8274.863405: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [000] .... 8274.870132: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [002] .... 8274.874634: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [000] .... 8274.878968: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8274.883432: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8274.887848: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8274.930432: writeback_congestion_wait: usec_timeout=100000 usec_delayed=37000
dd-4267 [000] .... 8274.932637: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [000] .... 8274.943276: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [000] .... 8274.943281: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=555776
flush-8:0-4272 [002] .... 8274.943292: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=555776
flush-8:0-4272 [002] .... 8274.947234: writeback_pages_written: 256
dd-4267 [000] .... 8274.947540: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=556032
flush-8:0-4272 [002] .... 8274.947548: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=556032
dd-4267 [000] .... 8274.947549: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=556032
dd-4267 [000] .... 8274.949050: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=558080
flush-8:0-4272 [002] .... 8274.950731: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=556032
flush-8:0-4272 [002] .... 8274.954025: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1024 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=558080
flush-8:0-4272 [002] .... 8274.958120: writeback_pages_written: 3072
dd-4267 [001] .... 8274.962461: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=559104
flush-8:0-4272 [002] .... 8274.962480: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=559104
dd-4267 [001] .... 8274.962481: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=559104
flush-8:0-4272 [002] .... 8274.965051: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1024 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=559104
flush-8:0-4272 [002] .... 8274.966440: writeback_pages_written: 1024
dd-4267 [001] .... 8274.971338: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=560128
flush-8:0-4272 [002] .... 8274.971354: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=560128
dd-4267 [001] .... 8274.971355: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=560128
flush-8:0-4272 [002] .... 8274.973875: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=560128
flush-8:0-4272 [002] .... 8274.974803: writeback_pages_written: 768
dd-4267 [001] .... 8274.976380: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=560896
flush-8:0-4272 [002] .... 8274.976392: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=560896
dd-4267 [001] .... 8274.976393: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=560896
flush-8:0-4272 [002] .... 8274.980232: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1024 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=560896
dd-4267 [001] .... 8274.981786: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=561920
flush-8:0-4272 [002] .... 8274.982195: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=512 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=561920
dd-4267 [001] .... 8274.982196: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=562176
flush-8:0-4272 [002] .... 8274.985722: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1536 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=562176
flush-8:0-4272 [002] .... 8274.988800: writeback_pages_written: 2816
dd-4267 [003] .... 8275.082867: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [000] .... 8275.128437: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8275.138326: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=563712
flush-8:0-4272 [002] .... 8275.138340: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=563712
dd-4267 [000] .... 8275.138341: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=563712
flush-8:0-4272 [002] .... 8275.141461: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=563712
flush-8:0-4272 [002] .... 8275.141469: writeback_pages_written: 256
dd-4267 [002] .... 8275.194016: writeback_congestion_wait: usec_timeout=100000 usec_delayed=55000
dd-4267 [000] .... 8275.198421: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8275.292553: writeback_congestion_wait: usec_timeout=100000 usec_delayed=69000
dd-4267 [000] .... 8275.296650: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=563968
flush-8:0-4272 [002] .... 8275.296664: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=563968
dd-4267 [000] .... 8275.296665: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=563968
dd-4267 [000] .... 8275.297205: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
flush-0:16-4223 [003] .... 8275.297726: writeback_start: bdi 0:16: sb_dev 0:0 nr_pages=35970 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-0:16-4223 [003] .... 8275.297730: writeback_queue_io: bdi 0:16: older=4302916993 age=30000 enqueue=2 reason=periodic
flush-0:16-4223 [003] .... 8275.297741: writeback_single_inode: bdi 0:16: ino=1573485 state= dirtied_when=4302912037 age=65 index=0 to_write=1024 wrote=0
flush-0:16-4223 [003] .... 8275.297765: writeback_single_inode: bdi 0:16: ino=1573501 state= dirtied_when=4302915356 age=61 index=24 to_write=1024 wrote=1
flush-0:16-4223 [003] .... 8275.297769: writeback_written: bdi 0:16: sb_dev 0:0 nr_pages=35969 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-0:16-4223 [003] .... 8275.297770: writeback_start: bdi 0:16: sb_dev 0:0 nr_pages=35969 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-0:16-4223 [003] .... 8275.297770: writeback_queue_io: bdi 0:16: older=4302916993 age=30000 enqueue=0 reason=periodic
flush-0:16-4223 [003] .... 8275.297771: writeback_written: bdi 0:16: sb_dev 0:0 nr_pages=35969 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-0:16-4223 [003] .... 8275.297777: writeback_pages_written: 1
flush-8:0-4272 [002] .... 8275.300356: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=563968
flush-8:0-4272 [002] .... 8275.300363: writeback_pages_written: 256
dd-4267 [000] .... 8275.301842: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8275.307614: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [002] .... 8275.312153: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8275.319068: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8275.328375: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [002] .... 8275.338401: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [002] .... 8275.342574: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=564224
flush-8:0-4272 [000] .... 8275.342588: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=564224
dd-4267 [002] .... 8275.342589: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=564224
dd-4267 [002] .... 8275.346229: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=566272
flush-8:0-4272 [000] .... 8275.346586: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=564224
dd-4267 [002] .... 8275.347786: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=568320
flush-8:0-4272 [000] .... 8275.349746: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=566272
dd-4267 [002] .... 8275.351519: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=570368
dd-4267 [002] .... 8275.353984: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
flush-8:0-4272 [000] .... 8275.355780: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=568320
dd-4267 [002] .... 8275.356369: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
flush-8:0-4272 [000] .... 8275.361337: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=570368
flush-8:0-4272 [000] .... 8275.363817: writeback_pages_written: 6912
dd-4267 [000] .... 8275.455701: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [000] .... 8275.493067: writeback_congestion_wait: usec_timeout=100000 usec_delayed=14000
dd-4267 [002] .... 8275.513157: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8275.523329: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8275.533356: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8275.536894: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=571136
flush-8:0-4272 [002] .... 8275.536907: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=571136
dd-4267 [000] .... 8275.536915: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=571136
flush-8:0-4272 [002] .... 8275.540007: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=571136
flush-8:0-4272 [002] .... 8275.540783: writeback_pages_written: 768
dd-4267 [000] .... 8275.549196: writeback_congestion_wait: usec_timeout=100000 usec_delayed=12000
dd-4267 [002] .... 8275.559066: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8275.569257: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8275.574835: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8275.576992: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=571904
flush-8:0-4272 [002] .... 8275.577003: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=571904
dd-4267 [000] .... 8275.577010: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=571904
dd-4267 [000] .... 8275.579035: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
flush-8:0-4272 [002] .... 8275.580062: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=512 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=571904
flush-8:0-4272 [002] .... 8275.580471: writeback_pages_written: 512
dd-4267 [002] .... 8275.584884: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8275.590767: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8275.595004: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8275.600852: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8275.607811: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8275.611310: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=572416
flush-8:0-4272 [002] .... 8275.611324: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=572416
dd-4267 [000] .... 8275.611327: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=572416
dd-4267 [000] .... 8275.612088: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
flush-8:0-4272 [002] .... 8275.614606: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=572416
flush-8:0-4272 [002] .... 8275.614613: writeback_pages_written: 256
dd-4267 [000] .... 8275.616597: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [002] .... 8275.620899: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [002] .... 8275.622607: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=572672
flush-8:0-4272 [000] .... 8275.622616: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=572672
dd-4267 [002] .... 8275.622617: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=572672
flush-8:0-4272 [000] .... 8275.626265: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=572672
flush-8:0-4272 [000] .... 8275.626271: writeback_pages_written: 256
dd-4267 [000] .... 8275.632606: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8275.642731: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8275.656678: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=572928
flush-8:0-4272 [002] .... 8275.656693: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=572928
dd-4267 [000] .... 8275.656694: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=572928
dd-4267 [000] .... 8275.658185: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=574976
flush-8:0-4272 [002] .... 8275.660256: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=572928
dd-4267 [002] .... 8275.663561: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
flush-8:0-4272 [000] .... 8275.762227: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1280 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=574976
flush-8:0-4272 [000] .... 8275.768192: writeback_pages_written: 3328
dd-4267 [001] .... 8275.813954: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=576256
flush-8:0-4272 [000] .... 8275.813972: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=576256
dd-4267 [001] .... 8275.813973: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=576256
flush-8:0-4272 [000] .... 8275.818853: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1280 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=576256
dd-4267 [001] .... 8275.819749: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=577536
flush-8:0-4272 [000] .... 8275.821425: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=577536
dd-4267 [001] .... 8275.821426: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=579328
dd-4267 [001] .... 8275.825312: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
flush-8:0-4272 [000] .... 8275.829046: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1024 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=579328
flush-8:0-4272 [000] .... 8275.830311: writeback_pages_written: 4096
dd-4267 [000] .... 8275.835347: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [002] .... 8275.841226: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8275.845447: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8276.548367: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=580352
flush-8:0-4272 [002] .... 8276.548384: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=580352
dd-4267 [000] .... 8276.548400: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=580352
dd-4267 [000] .... 8276.550364: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=582400
dd-4267 [000] .... 8276.551926: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=584448
dd-4267 [000] .... 8276.553518: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=586496
flush-8:0-4272 [002] .... 8276.554547: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=580352
dd-4267 [000] .... 8276.555127: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=588544
dd-4267 [000] .... 8276.556649: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=590592
flush-8:0-4272 [002] .... 8276.557449: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=582400
dd-4267 [001] .... 8276.560430: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=592640
dd-4267 [001] .... 8276.562084: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=594688
dd-4267 [001] .... 8276.563704: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=596736
dd-4267 [001] .... 8276.565324: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=598784
flush-8:0-4272 [002] .... 8276.565402: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=584448
dd-4267 [001] .... 8276.566863: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=600832
dd-4267 [001] .... 8276.568420: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=602880
flush-8:0-4272 [002] .... 8276.573699: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=586496
flush-8:0-4272 [002] .... 8276.580114: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=588544
flush-8:0-4272 [002] .... 8276.585744: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=590592
flush-8:0-4272 [002] .... 8276.590904: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=592640
flush-8:0-4272 [002] .... 8276.595589: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=594688
flush-8:0-4272 [002] .... 8276.599917: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=596736
flush-8:0-4272 [002] .... 8276.603987: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=598784
dd-4267 [003] .... 8276.668972: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [001] .... 8276.769045: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
flush-8:0-4272 [002] .... 8276.890790: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=600832
flush-8:0-4272 [002] .... 8276.895314: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1792 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=602880
flush-8:0-4272 [002] .... 8276.898568: writeback_pages_written: 24320
dd-4267 [003] .... 8276.908887: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [001] .... 8277.050810: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [000] .... 8277.200263: writeback_congestion_wait: usec_timeout=100000 usec_delayed=88000
dd-4267 [000] .... 8277.258031: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=604672
flush-8:0-4272 [002] .... 8277.258049: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=604672
dd-4267 [000] .... 8277.258050: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=604672
dd-4267 [000] .... 8277.258823: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
flush-8:0-4272 [002] .... 8277.261572: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=604672
flush-8:0-4272 [002] .... 8277.261579: writeback_start: bdi 8:0: sb_dev 0:0 nr_pages=35048 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-8:0-4272 [002] .... 8277.261580: writeback_queue_io: bdi 8:0: older=4302918957 age=30000 enqueue=0 reason=periodic
flush-8:0-4272 [002] .... 8277.261580: writeback_written: bdi 8:0: sb_dev 0:0 nr_pages=35048 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-8:0-4272 [002] .... 8277.261582: writeback_pages_written: 256
dd-4267 [000] .... 8277.262937: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8277.268671: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [002] .... 8277.272757: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8277.278481: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8277.282585: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8277.288315: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [002] .... 8277.292395: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8277.298115: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8277.302215: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8277.307931: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [002] .... 8277.312008: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8277.317735: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8277.324837: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8277.355709: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8277.364584: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8277.374398: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8277.382588: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [002] .... 8277.392402: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [000] .... 8277.397901: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8277.402057: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8277.407739: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [002] .... 8277.411818: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8277.417560: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8277.421658: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8277.427424: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [002] .... 8277.431473: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [000] .... 8277.437191: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8277.441271: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8277.447015: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [002] .... 8277.451087: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8277.456772: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8277.460844: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8277.466614: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [002] .... 8277.470713: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8277.476407: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8277.480519: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8277.486224: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [002] .... 8277.490324: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8277.496067: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8277.500124: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8277.505806: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [002] .... 8277.509968: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8277.515667: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8277.519729: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8277.521881: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [002] .... 8277.524031: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8277.526217: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8277.528385: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8277.530862: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [002] .... 8277.533032: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [000] .... 8277.539390: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [000] .... 8277.546494: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [002] .... 8277.550594: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8277.556326: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8277.560414: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8277.566132: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [002] .... 8277.570213: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8277.575939: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8277.580028: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8277.585719: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [002] .... 8277.589773: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8277.595538: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8277.599643: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8277.605356: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [002] .... 8277.609463: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8277.615151: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8277.619253: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8277.624990: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [002] .... 8277.629080: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8277.630998: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=604928
flush-8:0-4272 [000] .... 8277.631010: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=604928
dd-4267 [002] .... 8277.631011: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=604928
dd-4267 [002] .... 8277.632513: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=606976
dd-4267 [002] .... 8277.634731: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
flush-8:0-4272 [000] .... 8277.636659: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=604928
dd-4267 [002] .... 8277.638506: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=609024
dd-4267 [002] .... 8277.640040: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=611072
dd-4267 [002] .... 8277.641550: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=613120
dd-4267 [002] .... 8277.643094: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=615168
flush-8:0-4272 [000] .... 8277.715839: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=606976
flush-8:0-4272 [000] .... 8277.724516: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=609024
flush-8:0-4272 [000] .... 8277.733064: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=611072
flush-8:0-4272 [000] .... 8277.740840: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=613120
flush-8:0-4272 [000] .... 8277.748062: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=615168
flush-8:0-4272 [000] .... 8277.755291: writeback_pages_written: 12288
dd-4267 [001] .... 8277.802545: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=617216
flush-8:0-4272 [000] .... 8277.802565: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=617216
dd-4267 [001] .... 8277.802566: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=617216
dd-4267 [001] .... 8277.804416: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=619264
dd-4267 [001] .... 8277.806516: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=621312
flush-8:0-4272 [000] .... 8277.808332: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=617216
flush-8:0-4272 [000] .... 8277.813105: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=619264
dd-4267 [001] .... 8277.813218: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=623360
dd-4267 [001] .... 8277.815522: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=625408
flush-8:0-4272 [000] .... 8277.820794: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=621312
dd-4267 [001] .... 8277.821744: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=627456
flush-8:0-4272 [000] .... 8277.826650: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=623360
dd-4267 [001] .... 8277.826949: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
flush-8:0-4272 [000] .... 8277.832483: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=625408
flush-8:0-4272 [000] .... 8277.837046: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1536 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=627456
flush-8:0-4272 [000] .... 8277.840357: writeback_pages_written: 11776
dd-4267 [003] .... 8277.931348: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [000] .... 8278.051459: writeback_congestion_wait: usec_timeout=100000 usec_delayed=55000
dd-4267 [000] .... 8278.116343: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=628992
flush-8:0-4272 [002] .... 8278.116363: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=628992
dd-4267 [000] .... 8278.116364: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=628992
dd-4267 [000] .... 8278.118347: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=631040
dd-4267 [000] .... 8278.119860: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=633088
flush-8:0-4272 [001] .... 8278.120392: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=628992
dd-4267 [000] .... 8278.123157: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
flush-8:0-4272 [001] .... 8278.123728: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=631040
flush-8:0-4272 [001] .... 8278.130201: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1280 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=633088
flush-8:0-4272 [001] .... 8278.133845: writeback_pages_written: 5376
dd-4267 [002] .... 8278.225775: writeback_congestion_wait: usec_timeout=100000 usec_delayed=99000
dd-4267 [002] .... 8278.294088: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=634368
flush-8:0-4272 [001] .... 8278.294107: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=634368
dd-4267 [002] .... 8278.294109: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=634368
flush-8:0-4272 [001] .... 8278.297397: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=634368
flush-8:0-4272 [001] .... 8278.297405: writeback_pages_written: 256
dd-4267 [002] .... 8278.303627: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8278.308159: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [000] .... 8278.359684: writeback_congestion_wait: usec_timeout=100000 usec_delayed=51000
dd-4267 [000] .... 8278.362293: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [002] .... 8278.414805: writeback_congestion_wait: usec_timeout=100000 usec_delayed=52000
dd-4267 [000] .... 8278.417186: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8278.419534: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8278.421935: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8278.424848: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [002] .... 8278.523938: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [000] .... 8278.609350: writeback_congestion_wait: usec_timeout=100000 usec_delayed=85000
dd-4267 [002] .... 8278.611904: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8278.614392: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8278.617127: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8278.621765: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [002] .... 8278.629512: writeback_congestion_wait: usec_timeout=100000 usec_delayed=8000
dd-4267 [000] .... 8278.639410: writeback_congestion_wait: usec_timeout=100000 usec_delayed=10000
dd-4267 [003] .... 8278.649387: writeback_congestion_wait: usec_timeout=100000 usec_delayed=10000
dd-4267 [001] .... 8278.655016: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8278.659199: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8278.664970: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8278.744840: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8278.748992: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8278.754791: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [003] .... 8278.758983: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8278.764795: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8278.768952: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8278.774753: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [003] .... 8278.778942: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [001] .... 8278.784729: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [001] .... 8278.788914: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [001] .... 8278.794714: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [003] .... 8278.798872: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [001] .... 8278.804677: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [001] .... 8278.808866: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [001] .... 8278.814669: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [003] .... 8278.818829: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [001] .... 8278.824640: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [001] .... 8278.828858: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [001] .... 8278.834601: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [003] .... 8278.838792: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [000] .... 8278.901654: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8278.908163: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=634624
flush-8:0-4272 [001] .... 8278.908182: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=634624
dd-4267 [000] .... 8278.908650: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=634624
dd-4267 [000] .... 8278.911296: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
flush-8:0-4272 [001] .... 8278.913786: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=634624
flush-8:0-4272 [001] .... 8278.913794: writeback_pages_written: 256
dd-4267 [002] .... 8278.921513: writeback_congestion_wait: usec_timeout=100000 usec_delayed=8000
dd-4267 [000] .... 8278.925513: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [000] .... 8278.931324: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8278.935488: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8278.941321: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8278.945468: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8278.951282: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [002] .... 8278.955458: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8278.961249: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8278.964003: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=634880
flush-8:0-4272 [001] .... 8278.964020: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=634880
dd-4267 [000] .... 8278.964035: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=634880
dd-4267 [000] .... 8278.966502: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=636928
dd-4267 [000] .... 8278.968844: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=638976
flush-8:0-4272 [001] .... 8278.969634: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=634880
dd-4267 [000] .... 8278.971204: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=641024
flush-8:0-4272 [001] .... 8278.973125: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=636928
dd-4267 [000] .... 8278.975599: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=643072
dd-4267 [000] .... 8278.977264: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=645120
flush-8:0-4272 [001] .... 8278.982123: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=638976
dd-4267 [000] .... 8278.985235: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=647168
dd-4267 [000] .... 8278.986899: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=649216
dd-4267 [000] .... 8278.988478: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=651264
flush-8:0-4272 [001] .... 8279.107478: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=641024
flush-8:0-4272 [001] .... 8279.115590: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=643072
flush-8:0-4272 [001] .... 8279.122629: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=645120
flush-8:0-4272 [001] .... 8279.128637: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=647168
flush-8:0-4272 [001] .... 8279.134624: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=649216
flush-8:0-4272 [001] .... 8279.139908: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1024 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=651264
flush-8:0-4272 [001] .... 8279.143584: writeback_pages_written: 17408
dd-4267 [000] .... 8279.191039: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=652288
flush-8:0-4272 [001] .... 8279.191060: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=652288
dd-4267 [000] .... 8279.191070: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=652288
dd-4267 [000] .... 8279.192824: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=654336
flush-8:0-4272 [001] .... 8279.194897: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=652288
flush-8:0-4272 [001] .... 8279.198344: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1280 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=654336
dd-4267 [000] .... 8279.198346: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=655360
dd-4267 [000] .... 8279.199880: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=657408
flush-8:0-4272 [001] .... 8279.202409: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=655360
flush-8:0-4272 [001] .... 8279.205272: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1792 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=657408
flush-8:0-4272 [001] .... 8279.209400: writeback_pages_written: 6912
dd-4267 [002] .... 8279.300548: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [003] .... 8279.420643: writeback_congestion_wait: usec_timeout=100000 usec_delayed=59000
dd-4267 [003] .... 8279.473709: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=659200
flush-8:0-4272 [001] .... 8279.473725: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=659200
dd-4267 [003] .... 8279.473735: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=659200
flush-8:0-4272 [001] .... 8279.477129: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=512 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=659200
flush-8:0-4272 [001] .... 8279.477537: writeback_pages_written: 512
dd-4267 [001] .... 8279.567996: writeback_congestion_wait: usec_timeout=100000 usec_delayed=94000
dd-4267 [001] .... 8279.570412: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [001] .... 8279.575103: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [001] .... 8279.583971: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [001] .... 8279.594089: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [003] .... 8279.605732: writeback_congestion_wait: usec_timeout=100000 usec_delayed=8000
dd-4267 [001] .... 8279.615825: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8279.625850: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8279.635943: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [003] .... 8279.646070: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8279.656125: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8279.666165: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8279.676215: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [003] .... 8279.686308: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [001] .... 8279.696389: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [001] .... 8279.706454: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8279.716539: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8279.720112: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=659712
flush-8:0-4272 [003] .... 8279.720126: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=659712
dd-4267 [001] .... 8279.720127: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=659712
dd-4267 [001] .... 8279.721617: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=661760
dd-4267 [001] .... 8279.723124: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=663808
flush-8:0-4272 [003] .... 8279.724157: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=659712
dd-4267 [003] .... 8279.728278: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
flush-8:0-4272 [000] .... 8279.728765: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=661760
flush-8:0-4272 [000] .... 8279.736675: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1024 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=663808
dd-4267 [001] .... 8279.738442: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
flush-8:0-4272 [000] .... 8279.740568: writeback_pages_written: 5120
dd-4267 [001] .... 8279.748446: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [001] .... 8279.758431: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [003] .... 8279.768498: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [001] .... 8279.778569: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8279.788654: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [001] .... 8279.798693: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [003] .... 8279.808736: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [001] .... 8279.818831: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8279.828949: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8279.839007: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8279.842997: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=664832
flush-8:0-4272 [000] .... 8279.843014: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=664832
dd-4267 [001] .... 8279.843016: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=664832
flush-8:0-4272 [000] .... 8279.847182: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=664832
flush-8:0-4272 [000] .... 8279.847191: writeback_pages_written: 256
dd-4267 [003] .... 8279.849076: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8279.853108: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [001] .... 8279.863346: writeback_congestion_wait: usec_timeout=100000 usec_delayed=9000
dd-4267 [001] .... 8279.874975: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [003] .... 8279.880557: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [001] .... 8279.884813: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [001] .... 8279.890544: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8279.894781: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [003] .... 8279.900525: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8279.904786: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [001] .... 8279.910495: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8279.914742: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [003] .... 8279.920501: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8279.924716: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [001] .... 8279.930462: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8279.934704: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [003] .... 8279.940466: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8279.944675: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [001] .... 8279.950443: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8279.954671: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [003] .... 8279.960415: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8279.964640: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [001] .... 8279.970407: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8279.974635: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [003] .... 8279.980386: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8279.984609: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [001] .... 8279.990382: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8279.992689: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=665088
flush-8:0-4272 [000] .... 8279.992705: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=665088
dd-4267 [001] .... 8279.992706: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=665088
dd-4267 [001] .... 8279.996636: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=667136
flush-8:0-4272 [000] .... 8279.998271: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=665088
dd-4267 [001] .... 8279.998348: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=669184
dd-4267 [001] .... 8279.999931: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=671232
flush-8:0-4272 [000] .... 8280.002961: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=667136
dd-4267 [001] .... 8280.004506: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
flush-8:0-4272 [000] .... 8280.010048: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=669184
dd-4267 [001] .... 8280.010313: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8280.012905: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=673280
flush-8:0-4272 [000] .... 8280.016172: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=671232
dd-4267 [001] .... 8280.017137: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=675328
dd-4267 [001] .... 8280.018724: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [002] .... 8280.020975: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8280.023205: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [003] .... 8280.025762: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
flush-8:0-4272 [002] .... 8280.166717: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=673280
flush-8:0-4272 [002] .... 8280.172594: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1280 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=675328
flush-8:0-4272 [002] .... 8280.176266: writeback_pages_written: 11520
dd-4267 [003] .... 8280.266612: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=676608
flush-8:0-4272 [002] .... 8280.266632: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=676608
dd-4267 [003] .... 8280.266635: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=676608
dd-4267 [003] .... 8280.268438: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=678656
flush-8:0-4272 [002] .... 8280.270275: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=676608
dd-4267 [003] .... 8280.272860: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=680704
flush-8:0-4272 [002] .... 8280.273667: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=678656
dd-4267 [003] .... 8280.277004: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=682752
flush-8:0-4272 [002] .... 8280.279511: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=680704
flush-8:0-4272 [002] .... 8280.284580: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=682752
flush-8:0-4272 [002] .... 8280.286725: writeback_pages_written: 6912
flush-0:16-4223 [001] .... 8280.294981: writeback_start: bdi 0:16: sb_dev 0:0 nr_pages=29073 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-0:16-4223 [001] .... 8280.294985: writeback_queue_io: bdi 0:16: older=4302921993 age=30000 enqueue=3 reason=periodic
flush-0:16-4223 [001] .... 8280.294994: writeback_single_inode: bdi 0:16: ino=3080195 state= dirtied_when=4302920855 age=56 index=0 to_write=1024 wrote=0
flush-0:16-4223 [001] .... 8280.295020: writeback_single_inode: bdi 0:16: ino=1573416 state= dirtied_when=4302921098 age=56 index=0 to_write=1024 wrote=1
flush-0:16-4223 [001] .... 8280.295032: writeback_single_inode: bdi 0:16: ino=1573486 state= dirtied_when=4302921098 age=56 index=52481 to_write=1024 wrote=1
flush-0:16-4223 [001] .... 8280.295035: writeback_written: bdi 0:16: sb_dev 0:0 nr_pages=29071 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-0:16-4223 [001] .... 8280.295036: writeback_start: bdi 0:16: sb_dev 0:0 nr_pages=29071 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-0:16-4223 [001] .... 8280.295037: writeback_queue_io: bdi 0:16: older=4302921993 age=30000 enqueue=0 reason=periodic
flush-0:16-4223 [001] .... 8280.295037: writeback_written: bdi 0:16: sb_dev 0:0 nr_pages=29071 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-0:16-4223 [001] .... 8280.295042: writeback_pages_written: 2
dd-4267 [001] .... 8280.297036: writeback_congestion_wait: usec_timeout=100000 usec_delayed=20000
dd-4267 [003] .... 8280.408987: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [002] .... 8280.467718: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8280.477905: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [000] .... 8280.488203: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8280.492193: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=683520
flush-8:0-4272 [002] .... 8280.492204: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=683520
dd-4267 [000] .... 8280.492205: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=683520
dd-4267 [000] .... 8280.494904: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
flush-8:0-4272 [002] .... 8280.495455: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=683520
flush-8:0-4272 [002] .... 8280.495461: writeback_pages_written: 256
dd-4267 [002] .... 8280.499328: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8280.503823: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8280.508200: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8280.516395: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=683776
flush-8:0-4272 [002] .... 8280.516410: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=683776
dd-4267 [000] .... 8280.516410: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=683776
dd-4267 [000] .... 8280.516691: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [000] .... 8280.516877: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
flush-8:0-4272 [002] .... 8280.520252: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=683776
flush-8:0-4272 [002] .... 8280.520259: writeback_pages_written: 256
dd-4267 [000] .... 8280.521328: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8280.525593: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8280.613120: writeback_congestion_wait: usec_timeout=100000 usec_delayed=84000
dd-4267 [000] .... 8280.621118: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8280.628140: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [002] .... 8280.636293: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8280.640774: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8280.645305: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [000] .... 8280.649723: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [002] .... 8280.657246: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8280.661750: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8280.670022: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=684032
flush-8:0-4272 [002] .... 8280.670036: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=684032
dd-4267 [000] .... 8280.670040: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=684032
dd-4267 [000] .... 8280.671552: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=686080
dd-4267 [000] .... 8280.673073: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=688128
flush-8:0-4272 [002] .... 8280.674106: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=684032
flush-8:0-4272 [001] .... 8280.677134: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=686080
dd-4267 [000] .... 8280.680892: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=690176
flush-8:0-4272 [001] .... 8280.685604: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=688128
dd-4267 [000] .... 8280.686261: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
flush-8:0-4272 [001] .... 8280.691619: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=690176
flush-8:0-4272 [001] .... 8280.693880: writeback_pages_written: 6400
dd-4267 [000] .... 8280.699811: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8280.725768: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [002] .... 8280.735583: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8280.745458: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8280.749433: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=690432
flush-8:0-4272 [001] .... 8280.749453: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=690432
dd-4267 [000] .... 8280.749455: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=690432
flush-8:0-4272 [001] .... 8280.753038: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=690432
flush-8:0-4272 [001] .... 8280.753044: writeback_pages_written: 256
dd-4267 [000] .... 8280.755184: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8280.763233: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [002] .... 8280.767808: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8280.772434: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8280.776929: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8280.781392: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [002] .... 8280.785741: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8280.790431: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8280.805974: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=690688
flush-8:0-4272 [001] .... 8280.805990: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=690688
dd-4267 [000] .... 8280.805995: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=690688
flush-8:0-4272 [001] .... 8280.810334: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=690688
flush-8:0-4272 [001] .... 8280.810343: writeback_pages_written: 256
dd-4267 [002] .... 8280.835906: writeback_congestion_wait: usec_timeout=100000 usec_delayed=30000
dd-4267 [002] .... 8280.942587: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [000] .... 8280.999893: writeback_congestion_wait: usec_timeout=100000 usec_delayed=57000
dd-4267 [000] .... 8281.002359: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [000] .... 8281.004620: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [002] .... 8281.007144: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8281.011505: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8281.015104: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=690944
flush-8:0-4272 [001] .... 8281.015120: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=690944
dd-4267 [000] .... 8281.015121: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=690944
dd-4267 [000] .... 8281.018879: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
flush-8:0-4272 [001] .... 8281.019801: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=690944
flush-8:0-4272 [001] .... 8281.021129: writeback_pages_written: 768
dd-4267 [000] .... 8281.023390: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=691712
flush-8:0-4272 [001] .... 8281.023405: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=691712
dd-4267 [000] .... 8281.023406: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=691712
dd-4267 [000] .... 8281.025078: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=693760
dd-4267 [000] .... 8281.026726: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=695808
flush-8:0-4272 [001] .... 8281.027870: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=691712
dd-4267 [000] .... 8281.028680: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
flush-8:0-4272 [001] .... 8281.032562: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=693760
dd-4267 [000] .... 8281.034082: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=697856
dd-4267 [002] .... 8281.038551: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
flush-8:0-4272 [001] .... 8281.039182: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=695808
dd-4267 [002] .... 8281.043151: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=699904
flush-8:0-4272 [001] .... 8281.045416: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=697856
flush-8:0-4272 [001] .... 8281.050590: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1280 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=699904
flush-8:0-4272 [001] .... 8281.054085: writeback_pages_written: 9472
dd-4267 [000] .... 8281.146586: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [000] .... 8281.267657: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=701184
flush-8:0-4272 [001] .... 8281.267676: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=701184
dd-4267 [000] .... 8281.267682: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=701184
dd-4267 [000] .... 8281.269420: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=703232
flush-8:0-4272 [001] .... 8281.273221: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=701184
dd-4267 [000] .... 8281.275742: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=705280
flush-8:0-4272 [001] .... 8281.276549: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=703232
dd-4267 [000] .... 8281.277912: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=707328
dd-4267 [003] .... 8281.280371: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
flush-8:0-4272 [001] .... 8281.318655: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=705280
flush-8:0-4272 [001] .... 8281.324616: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=707328
flush-8:0-4272 [001] .... 8281.328030: writeback_pages_written: 6912
dd-4267 [002] .... 8281.450334: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [003] .... 8281.461260: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [003] .... 8281.483632: writeback_congestion_wait: usec_timeout=100000 usec_delayed=20000
dd-4267 [003] .... 8281.486146: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [001] .... 8281.488493: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [003] .... 8281.491058: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [003] .... 8281.495253: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [003] .... 8281.503364: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [003] .... 8281.513429: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [003] .... 8281.522995: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [003] .... 8281.532819: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [003] .... 8281.542599: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [003] .... 8281.546558: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=708096
flush-8:0-4272 [001] .... 8281.546572: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=708096
dd-4267 [003] .... 8281.546572: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=708096
dd-4267 [001] .... 8281.556507: writeback_congestion_wait: usec_timeout=100000 usec_delayed=10000
flush-8:0-4272 [003] .... 8282.192050: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=708096
flush-8:0-4272 [003] .... 8282.192057: writeback_pages_written: 256
dd-4267 [003] .... 8282.363940: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=708352
flush-8:0-4272 [001] .... 8282.363957: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=708352
dd-4267 [003] .... 8282.363965: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=708352
dd-4267 [003] .... 8282.365488: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=710400
dd-4267 [003] .... 8282.366985: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=712448
dd-4267 [003] .... 8282.368510: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=714496
flush-8:0-4272 [001] .... 8282.369497: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=708352
dd-4267 [003] .... 8282.370048: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=716544
dd-4267 [003] .... 8282.371633: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=718592
flush-8:0-4272 [000] .... 8282.374078: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=710400
dd-4267 [003] .... 8282.378185: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=720640
flush-8:0-4272 [000] .... 8282.385965: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=712448
dd-4267 [003] .... 8282.389259: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=722688
dd-4267 [003] .... 8282.390925: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=724736
flush-8:0-4272 [000] .... 8282.394370: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=714496
flush-8:0-4272 [000] .... 8282.403366: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=716544
flush-8:0-4272 [000] .... 8282.411168: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=718592
flush-8:0-4272 [000] .... 8282.418447: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=720640
dd-4267 [003] .... 8282.419012: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=726784
dd-4267 [003] .... 8282.420711: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=728832
flush-8:0-4272 [000] .... 8282.425497: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=722688
flush-8:0-4272 [000] .... 8282.432174: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=724736
dd-4267 [003] .... 8282.435607: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=730880
flush-8:0-4272 [000] .... 8282.437550: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=726784
flush-8:0-4272 [000] .... 8282.443269: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=728832
flush-8:0-4272 [000] .... 8282.447516: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1792 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=730880
dd-4267 [001] .... 8282.536821: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
flush-8:0-4272 [002] .... 8282.594711: writeback_start: bdi 8:0: sb_dev 0:0 nr_pages=28847 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-8:0-4272 [002] .... 8282.594714: writeback_queue_io: bdi 8:0: older=4302924294 age=30000 enqueue=0 reason=periodic
flush-8:0-4272 [002] .... 8282.594714: writeback_written: bdi 8:0: sb_dev 0:0 nr_pages=28847 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-8:0-4272 [002] .... 8282.594717: writeback_pages_written: 24320
dd-4267 [003] .... 8282.702707: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [003] .... 8282.766257: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=732672
flush-8:0-4272 [002] .... 8282.766275: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=732672
dd-4267 [003] .... 8282.766276: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=732672
dd-4267 [000] .... 8282.769626: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
flush-8:0-4272 [002] .... 8282.770439: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=732672
flush-8:0-4272 [002] .... 8282.770448: writeback_pages_written: 256
dd-4267 [000] .... 8282.792970: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=732928
flush-8:0-4272 [002] .... 8282.792985: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=732928
dd-4267 [000] .... 8282.792991: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=732928
flush-8:0-4272 [002] .... 8282.796612: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=512 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=732928
flush-8:0-4272 [002] .... 8282.797095: writeback_pages_written: 512
dd-4267 [001] .... 8282.803815: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=733440
flush-8:0-4272 [002] .... 8282.803828: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=733440
dd-4267 [000] .... 8282.805987: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
flush-8:0-4272 [002] .... 8282.807590: writeback_pages_written: 256
dd-4267 [002] .... 8282.816053: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8282.826012: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [000] .... 8282.836053: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8282.840123: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [002] .... 8282.862607: writeback_congestion_wait: usec_timeout=100000 usec_delayed=21000
dd-4267 [000] .... 8282.866842: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8282.872692: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8282.876853: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [002] .... 8282.882737: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8282.887007: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8282.892840: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8282.897051: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [002] .... 8282.902856: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8282.907142: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8282.912979: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8282.917185: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [002] .... 8282.923050: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8282.927261: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8282.933120: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8282.937351: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [002] .... 8282.943169: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8282.947400: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8282.953258: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8282.957497: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8282.963315: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8282.965832: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8282.967943: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8282.976215: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=733696
flush-8:0-4272 [002] .... 8282.976230: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=733696
dd-4267 [000] .... 8282.976230: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=733696
dd-4267 [000] .... 8282.977340: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
flush-8:0-4272 [002] .... 8282.980514: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=733696
dd-4267 [000] .... 8282.981192: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=734464
flush-8:0-4272 [002] .... 8282.981312: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=734464
dd-4267 [000] .... 8282.981745: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8282.981778: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=734464
flush-8:0-4272 [002] .... 8282.985084: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=734464
flush-8:0-4272 [002] .... 8282.985091: writeback_pages_written: 1024
dd-4267 [000] .... 8282.985665: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=734720
flush-8:0-4272 [002] .... 8282.985674: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=734720
dd-4267 [000] .... 8282.985675: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=734720
flush-8:0-4272 [002] .... 8282.989807: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=512 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=734720
flush-8:0-4272 [002] .... 8282.990230: writeback_pages_written: 512
dd-4267 [003] .... 8283.089416: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [000] .... 8283.122929: writeback_congestion_wait: usec_timeout=100000 usec_delayed=33000
dd-4267 [000] .... 8283.135611: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8283.145666: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8283.155723: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [002] .... 8283.165791: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8283.176075: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [000] .... 8283.181528: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8283.185688: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8283.187835: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=735232
flush-8:0-4272 [002] .... 8283.187848: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=735232
dd-4267 [000] .... 8283.187849: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=735232
dd-4267 [000] .... 8283.191527: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
flush-8:0-4272 [002] .... 8283.192426: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=512 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=735232
flush-8:0-4272 [002] .... 8283.192847: writeback_pages_written: 512
dd-4267 [000] .... 8283.195835: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8283.201634: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8283.205915: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8283.211685: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8283.215994: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8283.221809: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8283.226052: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [002] .... 8283.231889: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8283.236089: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8283.241969: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8283.246186: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [002] .... 8283.252004: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8283.256226: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8283.262106: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8283.266341: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8283.273554: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [000] .... 8283.277824: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8283.283655: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8283.287868: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8283.293646: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8283.297947: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8283.303809: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8283.312730: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8283.328557: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=735744
flush-8:0-4272 [002] .... 8283.328574: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=735744
dd-4267 [000] .... 8283.328582: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=735744
dd-4267 [000] .... 8283.330129: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=737792
dd-4267 [000] .... 8283.331848: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=739840
dd-4267 [000] .... 8283.333393: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=741888
flush-8:0-4272 [002] .... 8283.333676: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=735744
dd-4267 [000] .... 8283.334935: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=743936
dd-4267 [000] .... 8283.337887: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8283.338115: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
flush-8:0-4272 [002] .... 8283.431725: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=737792
flush-8:0-4272 [002] .... 8283.439317: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=739840
flush-8:0-4272 [002] .... 8283.447089: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=741888
flush-8:0-4272 [002] .... 8283.454001: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1792 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=743936
flush-8:0-4272 [002] .... 8283.459772: writeback_pages_written: 9984
dd-4267 [001] .... 8283.537146: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=745728
flush-8:0-4272 [002] .... 8283.537162: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=745728
dd-4267 [001] .... 8283.537171: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=745728
dd-4267 [001] .... 8283.539419: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=747776
dd-4267 [001] .... 8283.541208: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=749824
flush-8:0-4272 [002] .... 8283.541833: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=745728
flush-8:0-4272 [002] .... 8283.545118: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=747776
dd-4267 [001] .... 8283.545748: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=751872
dd-4267 [001] .... 8283.547488: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=753920
flush-8:0-4272 [002] .... 8283.552959: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=749824
dd-4267 [001] .... 8283.558579: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=755968
flush-8:0-4272 [002] .... 8283.559396: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=751872
dd-4267 [001] .... 8283.561724: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
flush-8:0-4272 [002] .... 8283.565992: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=753920
flush-8:0-4272 [002] .... 8283.570817: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1792 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=755968
flush-8:0-4272 [002] .... 8283.574693: writeback_pages_written: 12032
dd-4267 [003] .... 8283.668117: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [000] .... 8283.743847: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8283.747107: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=757760
flush-8:0-4272 [002] .... 8283.747123: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=757760
dd-4267 [000] .... 8283.747124: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=757760
flush-8:0-4272 [002] .... 8283.751308: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1792 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=757760
dd-4267 [000] .... 8283.751309: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=759296
dd-4267 [000] .... 8283.752829: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=761344
flush-8:0-4272 [002] .... 8283.753579: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=759296
dd-4267 [001] .... 8283.756564: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=763392
flush-8:0-4272 [002] .... 8283.756906: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=761344
flush-8:0-4272 [002] .... 8283.762110: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=763392
flush-8:0-4272 [002] .... 8283.764271: writeback_pages_written: 6400
dd-4267 [003] .... 8283.857011: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [002] .... 8283.973375: writeback_congestion_wait: usec_timeout=100000 usec_delayed=55000
dd-4267 [002] .... 8284.011625: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=764160
flush-8:0-4272 [000] .... 8284.011640: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=764160
dd-4267 [002] .... 8284.011645: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=764160
dd-4267 [002] .... 8284.013208: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
flush-8:0-4272 [000] .... 8284.015143: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=764160
flush-8:0-4272 [000] .... 8284.015149: writeback_pages_written: 256
dd-4267 [000] .... 8284.019069: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8284.023215: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8284.030639: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [002] .... 8284.034941: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8284.040920: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [000] .... 8284.051372: writeback_congestion_wait: usec_timeout=100000 usec_delayed=8000
dd-4267 [000] .... 8284.061660: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [002] .... 8284.071910: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [000] .... 8284.082199: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [000] .... 8284.092449: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8284.102756: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8284.106747: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=764416
flush-8:0-4272 [002] .... 8284.106761: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=764416
dd-4267 [000] .... 8284.106772: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=764416
flush-8:0-4272 [002] .... 8284.110412: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=764416
flush-8:0-4272 [002] .... 8284.110420: writeback_pages_written: 256
dd-4267 [002] .... 8284.113017: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [002] .... 8284.117024: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=764672
flush-8:0-4272 [000] .... 8284.117035: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=764672
dd-4267 [002] .... 8284.117041: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=764672
flush-8:0-4272 [000] .... 8284.120264: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=764672
flush-8:0-4272 [000] .... 8284.120271: writeback_pages_written: 256
dd-4267 [000] .... 8284.123300: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8284.133583: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [000] .... 8284.145288: writeback_congestion_wait: usec_timeout=100000 usec_delayed=8000
dd-4267 [002] .... 8284.149617: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [000] .... 8284.153959: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8284.159928: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [000] .... 8284.164244: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [002] .... 8284.170143: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8284.174494: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8284.180462: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8284.184792: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8284.190718: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8284.195060: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8284.201037: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8284.205349: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8284.211282: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8284.215607: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8284.217861: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=764928
flush-8:0-4272 [002] .... 8284.217873: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=764928
dd-4267 [000] .... 8284.217883: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=764928
dd-4267 [000] .... 8284.219412: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=766976
flush-8:0-4272 [002] .... 8284.222662: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=764928
dd-4267 [000] .... 8284.224852: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=769024
flush-8:0-4272 [001] .... 8284.226978: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=766976
dd-4267 [000] .... 8284.231789: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
flush-8:0-4272 [001] .... 8284.234737: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1280 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=769024
dd-4267 [000] .... 8284.236127: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8284.242083: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
flush-8:0-4272 [001] .... 8284.537975: writeback_pages_written: 5376
dd-4267 [002] .... 8284.712091: writeback_congestion_wait: usec_timeout=100000 usec_delayed=97000
dd-4267 [002] .... 8284.714592: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [002] .... 8284.718902: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [002] .... 8284.721113: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [002] .... 8284.723558: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8284.729274: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8284.730891: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=770304
flush-8:0-4272 [001] .... 8284.730909: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=770304
dd-4267 [000] .... 8284.731346: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=770304
dd-4267 [000] .... 8284.733058: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=772352
flush-8:0-4272 [001] .... 8284.735492: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=770304
dd-4267 [000] .... 8284.736899: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=774400
dd-4267 [000] .... 8284.738556: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=776448
flush-8:0-4272 [001] .... 8284.738766: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=772352
dd-4267 [000] .... 8284.743348: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
flush-8:0-4272 [001] .... 8284.745968: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=774400
dd-4267 [000] .... 8284.749665: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=778496
flush-8:0-4272 [001] .... 8284.751572: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=776448
dd-4267 [000] .... 8284.753153: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
flush-8:0-4272 [001] .... 8284.757398: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=778496
dd-4267 [000] .... 8284.757400: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=779008
flush-8:0-4272 [001] .... 8284.760124: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1024 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=779008
flush-8:0-4272 [001] .... 8284.761353: writeback_pages_written: 9728
dd-4267 [002] .... 8284.857465: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [002] .... 8284.912787: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=780032
flush-8:0-4272 [001] .... 8284.912808: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=780032
dd-4267 [002] .... 8284.912815: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=780032
flush-8:0-4272 [001] .... 8284.916025: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=780032
flush-8:0-4272 [001] .... 8284.918844: writeback_pages_written: 2048
dd-4267 [003] .... 8284.940686: writeback_congestion_wait: usec_timeout=100000 usec_delayed=26000
dd-4267 [003] .... 8284.978581: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=782080
flush-8:0-4272 [001] .... 8284.978599: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=782080
dd-4267 [003] .... 8284.978602: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=782080
flush-8:0-4272 [001] .... 8284.982521: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=512 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=782080
flush-8:0-4272 [001] .... 8284.982946: writeback_pages_written: 512
dd-4267 [003] .... 8284.983056: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=782592
flush-8:0-4272 [001] .... 8284.983063: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=782592
dd-4267 [003] .... 8284.983064: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=782592
dd-4267 [003] .... 8284.984577: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=784640
flush-8:0-4272 [001] .... 8284.985859: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=782592
flush-8:0-4272 [000] .... 8284.989292: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1280 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=784640
dd-4267 [003] .... 8284.990536: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=785664
dd-4267 [003] .... 8284.992120: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=787712
flush-8:0-4272 [000] .... 8284.994372: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=785664
flush-8:0-4272 [000] .... 8284.997795: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=787712
flush-8:0-4272 [000] .... 8285.000496: writeback_pages_written: 5888
dd-4267 [001] .... 8285.092449: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [001] .... 8285.177444: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=788480
flush-8:0-4272 [000] .... 8285.177462: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=788480
dd-4267 [001] .... 8285.177462: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=788480
dd-4267 [001] .... 8285.179141: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
flush-8:0-4272 [000] .... 8285.181333: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=788480
flush-8:0-4272 [000] .... 8285.181342: writeback_pages_written: 256
dd-4267 [001] .... 8285.183347: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=788736
flush-8:0-4272 [000] .... 8285.183364: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=788736
dd-4267 [001] .... 8285.183373: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=788736
flush-8:0-4272 [000] .... 8285.185778: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=788736
flush-8:0-4272 [000] .... 8285.185784: writeback_pages_written: 256
dd-4267 [001] .... 8285.188015: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8285.197816: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [003] .... 8285.207603: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8285.217253: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8285.219617: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [003] .... 8285.224224: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [003] .... 8285.260785: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=788992
flush-8:0-4272 [000] .... 8285.260803: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=788992
dd-4267 [003] .... 8285.260804: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=788992
dd-4267 [003] .... 8285.262513: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=791040
flush-8:0-4272 [000] .... 8285.265222: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=788992
flush-8:0-4272 [000] .... 8285.269537: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1792 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=791040
dd-4267 [001] .... 8285.270972: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
flush-8:0-4272 [000] .... 8285.275330: writeback_pages_written: 3840
dd-4267 [003] .... 8285.324595: writeback_congestion_wait: usec_timeout=100000 usec_delayed=46000
dd-4267 [001] .... 8285.428150: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [001] .... 8285.428951: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [001] .... 8285.433539: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [003] .... 8285.440157: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [001] .... 8285.449269: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [001] .... 8285.455291: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=792832
flush-8:0-4272 [000] .... 8285.455308: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=792832
dd-4267 [001] .... 8285.455310: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=792832
dd-4267 [001] .... 8285.457006: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=794880
dd-4267 [001] .... 8285.458912: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
flush-8:0-4272 [000] .... 8285.459455: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=792832
flush-8:0-4272 [000] .... 8285.462753: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=794880
dd-4267 [003] .... 8285.462940: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
flush-8:0-4272 [000] .... 8285.465164: writeback_pages_written: 2304
dd-4267 [003] .... 8285.465311: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=795136
flush-8:0-4272 [000] .... 8285.465328: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=795136
dd-4267 [003] .... 8285.465330: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=795136
flush-8:0-4272 [000] .... 8285.467314: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=795136
flush-8:0-4272 [000] .... 8285.467319: writeback_pages_written: 256
dd-4267 [001] .... 8285.478696: writeback_congestion_wait: usec_timeout=100000 usec_delayed=13000
dd-4267 [002] .... 8285.511501: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8285.519624: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [002] .... 8285.529305: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [002] .... 8285.529483: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [000] .... 8285.541032: writeback_congestion_wait: usec_timeout=100000 usec_delayed=9000
dd-4267 [000] .... 8285.577136: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=795392
flush-8:0-4272 [002] .... 8285.577156: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=795392
dd-4267 [000] .... 8285.577638: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=795392
flush-8:0-4272 [002] .... 8285.581251: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=512 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=795392
flush-8:0-4272 [002] .... 8285.581647: writeback_pages_written: 512
dd-4267 [002] .... 8285.676988: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [000] .... 8285.698727: writeback_congestion_wait: usec_timeout=100000 usec_delayed=21000
dd-4267 [000] .... 8285.701218: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8285.710431: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8285.718230: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8285.722253: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=795904
flush-8:0-4272 [002] .... 8285.722266: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=795904
dd-4267 [000] .... 8285.722267: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=795904
dd-4267 [000] .... 8285.723781: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=797952
dd-4267 [000] .... 8285.725289: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=800000
flush-8:0-4272 [002] .... 8285.726260: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=795904
dd-4267 [000] .... 8285.728248: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
flush-8:0-4272 [002] .... 8285.729667: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=797952
flush-8:0-4272 [002] .... 8285.735947: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1536 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=800000
flush-8:0-4272 [002] .... 8285.740515: writeback_pages_written: 5632
dd-4267 [000] .... 8285.743317: writeback_congestion_wait: usec_timeout=100000 usec_delayed=11000
dd-4267 [000] .... 8285.748737: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=801536
flush-8:0-4272 [002] .... 8285.748749: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=801536
dd-4267 [000] .... 8285.748750: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=801536
dd-4267 [000] .... 8285.750301: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=803584
flush-8:0-4272 [002] .... 8285.751451: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=801536
dd-4267 [000] .... 8285.789981: writeback_congestion_wait: usec_timeout=100000 usec_delayed=39000
flush-8:0-4272 [000] .... 8285.820051: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1024 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=803584
flush-8:0-4272 [000] .... 8285.824204: writeback_pages_written: 3072
dd-4267 [002] .... 8285.877041: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=804608
flush-8:0-4272 [000] .... 8285.877059: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=804608
dd-4267 [002] .... 8285.877059: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=804608
flush-8:0-4272 [000] .... 8285.880219: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=804608
dd-4267 [002] .... 8285.881976: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
flush-8:0-4272 [000] .... 8285.883304: writeback_pages_written: 2048
dd-4267 [000] .... 8285.913064: writeback_congestion_wait: usec_timeout=100000 usec_delayed=27000
dd-4267 [000] .... 8285.953294: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8285.993092: writeback_congestion_wait: usec_timeout=100000 usec_delayed=40000
dd-4267 [000] .... 8286.017921: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=806656
flush-8:0-4272 [002] .... 8286.017941: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=806656
dd-4267 [000] .... 8286.017942: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=806656
flush-8:0-4272 [002] .... 8286.022344: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=806656
flush-8:0-4272 [002] .... 8286.022351: writeback_pages_written: 256
dd-4267 [000] .... 8286.022457: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=806912
flush-8:0-4272 [002] .... 8286.022463: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=806912
dd-4267 [000] .... 8286.022464: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=806912
flush-8:0-4272 [002] .... 8286.025804: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1024 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=806912
dd-4267 [001] .... 8286.025808: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=807680
flush-8:0-4272 [002] .... 8286.027152: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1792 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=807680
dd-4267 [001] .... 8286.027154: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=809216
flush-8:0-4272 [002] .... 8286.030162: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1536 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=809216
dd-4267 [001] .... 8286.030934: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=810752
flush-8:0-4272 [002] .... 8286.032432: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1792 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=810752
dd-4267 [001] .... 8286.032436: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=812288
flush-8:0-4272 [002] .... 8286.037770: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=812288
flush-8:0-4272 [002] .... 8286.038525: writeback_pages_written: 6144
dd-4267 [003] .... 8286.131784: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [003] .... 8286.193153: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=813056
flush-8:0-4272 [002] .... 8286.193171: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=813056
dd-4267 [003] .... 8286.193182: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=813056
dd-4267 [000] .... 8286.194538: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
flush-8:0-4272 [002] .... 8286.197515: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=512 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=813056
flush-8:0-4272 [002] .... 8286.198149: writeback_pages_written: 512
dd-4267 [000] .... 8286.204432: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [002] .... 8286.208413: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [000] .... 8286.214233: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8286.218379: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8286.224189: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [002] .... 8286.228384: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8286.234169: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8286.238355: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8286.244159: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [002] .... 8286.248349: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8286.254131: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8286.258310: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8286.268537: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [002] .... 8286.278455: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8286.288438: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8286.298422: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [002] .... 8286.308378: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [002] .... 8286.311951: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=813568
flush-8:0-4272 [000] .... 8286.311964: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=813568
dd-4267 [002] .... 8286.311974: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=813568
dd-4267 [002] .... 8286.313470: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=815616
flush-8:0-4272 [000] .... 8286.316042: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=813568
dd-4267 [002] .... 8286.318069: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
flush-8:0-4272 [000] .... 8286.319085: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1536 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=815616
dd-4267 [001] .... 8286.323896: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
flush-8:0-4272 [000] .... 8286.324156: writeback_pages_written: 3584
dd-4267 [002] .... 8286.328179: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8286.333974: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8286.338144: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8286.343939: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [002] .... 8286.348136: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8286.353914: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8286.358096: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8286.363902: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [002] .... 8286.368092: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8286.375470: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8286.379691: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8286.390024: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [000] .... 8286.394213: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=817152
flush-8:0-4272 [002] .... 8286.394227: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=817152
dd-4267 [000] .... 8286.394228: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=817152
dd-4267 [000] .... 8286.395740: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=819200
flush-8:0-4272 [002] .... 8286.397700: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=817152
dd-4267 [000] .... 8286.399729: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8286.399908: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
flush-8:0-4272 [002] .... 8286.400793: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=819200
dd-4267 [000] .... 8286.403696: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=821248
flush-8:0-4272 [002] .... 8286.406096: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=512 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=821248
flush-8:0-4272 [002] .... 8286.408455: writeback_pages_written: 4608
dd-4267 [000] .... 8286.490429: writeback_congestion_wait: usec_timeout=100000 usec_delayed=86000
dd-4267 [000] .... 8287.253038: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=821760
flush-8:0-4272 [002] .... 8287.253056: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=821760
dd-4267 [000] .... 8287.253111: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=821760
dd-4267 [000] .... 8287.256585: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=823808
dd-4267 [000] .... 8287.258183: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=825856
flush-8:0-4272 [002] .... 8287.259493: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=821760
dd-4267 [001] .... 8287.259950: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=827904
dd-4267 [001] .... 8287.261742: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=829952
flush-8:0-4272 [002] .... 8287.262677: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=823808
dd-4267 [001] .... 8287.264192: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=832000
dd-4267 [001] .... 8287.265825: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=834048
dd-4267 [001] .... 8287.267436: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=836096
dd-4267 [001] .... 8287.269029: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=838144
dd-4267 [001] .... 8287.270652: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=840192
flush-8:0-4272 [002] .... 8287.271798: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=825856
dd-4267 [001] .... 8287.272253: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=842240
dd-4267 [001] .... 8287.273814: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=844288
flush-8:0-4272 [002] .... 8287.280257: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=827904
flush-8:0-4272 [002] .... 8287.286640: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=829952
flush-8:0-4272 [002] .... 8287.292224: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=832000
flush-8:0-4272 [002] .... 8287.297298: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=834048
dd-4267 [003] .... 8287.375066: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
flush-8:0-4272 [000] .... 8287.433845: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=836096
flush-8:0-4272 [000] .... 8287.439490: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=838144
flush-8:0-4272 [000] .... 8287.444495: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=840192
flush-8:0-4272 [000] .... 8287.449427: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=842240
dd-4267 [001] .... 8287.568053: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
flush-8:0-4272 [002] .... 8287.589239: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=844288
flush-8:0-4272 [002] .... 8287.594059: writeback_start: bdi 8:0: sb_dev 0:0 nr_pages=25531 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-8:0-4272 [002] .... 8287.594061: writeback_queue_io: bdi 8:0: older=4302929296 age=30000 enqueue=1 reason=periodic
flush-8:0-4272 [002] .... 8287.594217: writeback_single_inode: bdi 8:0: ino=0 state= dirtied_when=4302926220 age=51 index=62423040 to_write=1024 wrote=60
flush-8:0-4272 [002] .... 8287.594220: writeback_written: bdi 8:0: sb_dev 0:0 nr_pages=25471 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-8:0-4272 [002] .... 8287.594220: writeback_start: bdi 8:0: sb_dev 0:0 nr_pages=25471 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-8:0-4272 [002] .... 8287.594220: writeback_queue_io: bdi 8:0: older=4302929296 age=30000 enqueue=0 reason=periodic
flush-8:0-4272 [002] .... 8287.594221: writeback_written: bdi 8:0: sb_dev 0:0 nr_pages=25471 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-8:0-4272 [002] .... 8287.594223: writeback_pages_written: 24636
dd-4267 [003] .... 8287.725915: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [000] .... 8287.867240: writeback_congestion_wait: usec_timeout=100000 usec_delayed=68000
dd-4267 [000] .... 8287.929273: writeback_congestion_wait: usec_timeout=100000 usec_delayed=29000
dd-4267 [000] .... 8287.933932: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [002] .... 8287.943436: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8287.953437: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8287.963311: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8287.968957: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8287.971328: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [000] .... 8287.975932: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [000] .... 8287.996004: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8288.000541: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [002] .... 8288.007408: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8288.017420: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8288.024017: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8288.029668: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [002] .... 8288.034148: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8288.038464: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [000] .... 8288.042966: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [002] .... 8288.145630: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [000] .... 8288.198092: writeback_congestion_wait: usec_timeout=100000 usec_delayed=52000
dd-4267 [000] .... 8288.214718: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8288.224982: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [002] .... 8288.235294: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8288.245537: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [000] .... 8288.255814: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [000] .... 8288.262402: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8288.270398: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [002] .... 8288.281573: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8288.285848: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8288.293826: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [002] .... 8288.298364: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [000] .... 8288.306692: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8288.316956: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [002] .... 8288.327285: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8288.337529: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [000] .... 8288.343307: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [000] .... 8288.350208: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8288.360343: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8288.364844: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8288.368427: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=846336
flush-8:0-4272 [002] .... 8288.368440: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=846336
dd-4267 [000] .... 8288.368442: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=846336
dd-4267 [000] .... 8288.374199: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=848384
flush-8:0-4272 [002] .... 8288.374913: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=846336
dd-4267 [001] .... 8288.376663: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=850432
flush-8:0-4272 [002] .... 8288.377993: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=848384
dd-4267 [001] .... 8288.378387: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=852480
dd-4267 [001] .... 8288.380037: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=854528
dd-4267 [001] .... 8288.381692: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=856576
flush-8:0-4272 [002] .... 8288.387743: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=850432
dd-4267 [001] .... 8288.390190: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=858624
flush-8:0-4272 [002] .... 8288.395961: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=852480
flush-8:0-4272 [002] .... 8288.403948: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=854528
dd-4267 [001] .... 8288.404992: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=860672
dd-4267 [001] .... 8288.406709: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=862720
dd-4267 [001] .... 8288.408423: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=864768
dd-4267 [001] .... 8288.410722: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=866816
flush-8:0-4272 [002] .... 8288.410728: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=856576
dd-4267 [001] .... 8288.412372: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=868864
flush-8:0-4272 [002] .... 8288.417731: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=858624
flush-8:0-4272 [002] .... 8288.422959: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=860672
flush-8:0-4272 [002] .... 8288.427513: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=862720
flush-8:0-4272 [002] .... 8288.431706: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=864768
flush-8:0-4272 [002] .... 8288.435613: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=866816
dd-4267 [003] .... 8288.513425: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [001] .... 8288.613432: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
flush-8:0-4272 [002] .... 8288.807055: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1792 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=868864
flush-8:0-4272 [002] .... 8288.813588: writeback_pages_written: 24320
dd-4267 [003] .... 8288.964246: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [003] .... 8289.018315: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=870656
flush-8:0-4272 [002] .... 8289.018333: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=870656
dd-4267 [003] .... 8289.018334: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=870656
flush-8:0-4272 [002] .... 8289.022448: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=512 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=870656
flush-8:0-4272 [002] .... 8289.023068: writeback_pages_written: 512
dd-4267 [000] .N.. 8289.043118: writeback_congestion_wait: usec_timeout=100000 usec_delayed=24000
dd-4267 [000] .... 8289.062610: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8289.072452: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8289.082237: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [002] .... 8289.092057: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8289.101888: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8289.113107: writeback_congestion_wait: usec_timeout=100000 usec_delayed=8000
dd-4267 [000] .... 8289.117139: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=871168
flush-8:0-4272 [002] .... 8289.117151: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=871168
dd-4267 [000] .... 8289.117152: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=871168
flush-8:0-4272 [002] .... 8289.120609: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=871168
flush-8:0-4272 [002] .... 8289.121378: writeback_pages_written: 768
dd-4267 [000] .... 8289.122889: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [002] .... 8289.132666: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8289.140943: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8289.145446: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8289.149985: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [002] .... 8289.154423: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8289.169836: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8289.174289: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [002] .... 8289.239886: writeback_congestion_wait: usec_timeout=100000 usec_delayed=62000
dd-4267 [000] .... 8289.244347: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8289.248919: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8289.258780: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8289.268549: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8289.278371: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8289.288216: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [002] .... 8289.297985: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [000] .... 8289.307821: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8289.317661: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8289.327410: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [002] .... 8289.337217: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [000] .... 8289.347038: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8289.353471: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8289.359251: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [002] .... 8289.363735: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [000] .... 8289.368137: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8289.372635: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [000] .... 8289.377206: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8289.480585: writeback_congestion_wait: usec_timeout=100000 usec_delayed=81000
dd-4267 [000] .... 8289.482791: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [000] .... 8289.485066: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [002] .... 8289.487555: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8289.495298: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8289.505137: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [000] .... 8289.514910: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [002] .... 8289.520479: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8289.527108: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [000] .... 8289.544021: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=871936
flush-8:0-4272 [002] .... 8289.544039: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=871936
dd-4267 [000] .... 8289.544046: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=871936
dd-4267 [000] .... 8289.545565: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=873984
dd-4267 [000] .... 8289.547139: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=876032
dd-4267 [000] .... 8289.548677: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=878080
flush-8:0-4272 [002] .... 8289.549822: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=871936
dd-4267 [000] .... 8289.553126: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=880128
flush-8:0-4272 [001] .... 8289.554779: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=873984
flush-8:0-4272 [001] .... 8289.567484: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=876032
dd-4267 [000] .... 8289.568912: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=882176
dd-4267 [000] .... 8289.570643: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=884224
dd-4267 [000] .... 8289.572336: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=886272
dd-4267 [000] .... 8289.574000: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=888320
flush-8:0-4272 [001] .... 8289.575450: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=878080
dd-4267 [000] .... 8289.575687: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=890368
dd-4267 [000] .... 8289.577346: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=892416
dd-4267 [000] .... 8289.579606: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=894464
flush-8:0-4272 [001] .... 8289.583703: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=880128
flush-8:0-4272 [001] .... 8289.590178: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=882176
flush-8:0-4272 [001] .... 8289.595612: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=884224
flush-8:0-4272 [001] .... 8289.600583: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=886272
flush-8:0-4272 [001] .... 8289.605480: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=888320
flush-8:0-4272 [001] .... 8289.610298: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=890368
flush-8:0-4272 [001] .... 8289.614567: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=892416
dd-4267 [002] .... 8289.679818: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
flush-8:0-4272 [001] .... 8289.790935: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1280 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=894464
flush-8:0-4272 [001] .... 8289.794239: writeback_pages_written: 23808
dd-4267 [000] .... 8289.830712: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [000] .... 8289.979274: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=895744
flush-8:0-4272 [001] .... 8289.979295: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=895744
dd-4267 [000] .... 8289.979299: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=895744
flush-8:0-4272 [001] .... 8289.984149: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1024 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=895744
flush-8:0-4272 [001] .... 8289.985910: writeback_pages_written: 1024
dd-4267 [003] .... 8289.986879: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [001] .... 8289.989278: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [001] .... 8289.993746: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [001] .... 8290.022891: writeback_congestion_wait: usec_timeout=100000 usec_delayed=29000
dd-4267 [003] .... 8290.138516: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [001] .... 8290.238484: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [001] .... 8290.258044: writeback_congestion_wait: usec_timeout=100000 usec_delayed=19000
dd-4267 [001] .... 8290.272386: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [003] .... 8290.282389: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8290.292359: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8290.302381: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [003] .... 8290.312326: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8290.330620: writeback_congestion_wait: usec_timeout=100000 usec_delayed=15000
dd-4267 [001] .... 8290.340609: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [001] .... 8290.350579: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [003] .... 8290.360570: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [001] .... 8290.370533: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8290.380548: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [001] .... 8290.390506: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [003] .... 8290.400468: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8290.410469: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [001] .... 8290.420238: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [003] .... 8290.424433: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [001] .... 8290.430244: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [001] .... 8290.434405: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [001] .... 8290.440201: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [003] .... 8290.444394: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [001] .... 8290.450177: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [001] .... 8290.454365: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [001] .... 8290.460164: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [003] .... 8290.464360: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [001] .... 8290.466690: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [001] .... 8290.472687: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8290.494518: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [001] .... 8290.499431: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [001] .... 8290.506101: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [001] .... 8290.514874: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [001] .... 8290.525026: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [003] .... 8290.535090: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8290.545126: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8290.550820: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [001] .... 8290.555010: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [003] .... 8290.560856: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8290.573413: writeback_congestion_wait: usec_timeout=100000 usec_delayed=11000
dd-4267 [001] .... 8290.579244: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8290.585075: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [003] .... 8290.589299: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [003] .... 8290.591103: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=896768
flush-8:0-4272 [001] .... 8290.591114: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=896768
dd-4267 [003] .... 8290.591115: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=896768
dd-4267 [003] .... 8290.592629: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=898816
dd-4267 [003] .... 8290.594151: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=900864
dd-4267 [003] .... 8290.595687: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=902912
flush-8:0-4272 [001] .... 8290.596742: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=896768
flush-8:0-4272 [001] .... 8290.599560: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=898816
dd-4267 [003] .... 8290.599589: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=904960
dd-4267 [000] .... 8290.601759: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=907008
dd-4267 [000] .... 8290.604180: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=909056
dd-4267 [000] .... 8290.606776: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8290.606967: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
flush-8:0-4272 [001] .... 8290.608163: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=900864
dd-4267 [003] .... 8290.616932: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
flush-8:0-4272 [001] .... 8290.807466: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=902912
flush-8:0-4272 [001] .... 8290.815087: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=904960
flush-8:0-4272 [000] .... 8290.825087: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=907008
flush-8:0-4272 [000] .... 8290.833291: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=909056
flush-8:0-4272 [000] .... 8290.837230: writeback_pages_written: 13056
dd-4267 [003] .... 8290.898320: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=909824
flush-8:0-4272 [000] .... 8290.898341: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=909824
dd-4267 [003] .... 8290.898342: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=909824
dd-4267 [003] .... 8290.900086: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=911872
dd-4267 [003] .... 8290.902263: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=913920
flush-8:0-4272 [000] .... 8290.903241: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=909824
dd-4267 [003] .... 8290.903966: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=915968
dd-4267 [003] .... 8290.905605: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=918016
flush-8:0-4272 [000] .... 8290.907371: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=911872
flush-8:0-4272 [000] .... 8290.914048: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=913920
flush-8:0-4272 [000] .... 8290.919004: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=915968
flush-8:0-4272 [000] .... 8290.923475: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=918016
flush-8:0-4272 [000] .... 8290.927621: writeback_pages_written: 10240
dd-4267 [001] .... 8291.007124: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [001] .... 8291.055558: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=920064
flush-8:0-4272 [000] .... 8291.055576: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=920064
dd-4267 [001] .... 8291.055577: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=920064
flush-8:0-4272 [000] .... 8291.058663: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=920064
flush-8:0-4272 [000] .... 8291.058672: writeback_pages_written: 256
dd-4267 [000] .... 8291.120020: writeback_congestion_wait: usec_timeout=100000 usec_delayed=65000
dd-4267 [000] .... 8291.228552: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=920320
flush-8:0-4272 [002] .... 8291.228569: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=920320
dd-4267 [000] .... 8291.228570: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=920320
dd-4267 [000] .... 8291.230580: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=922368
dd-4267 [000] .... 8291.232100: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=924416
flush-8:0-4272 [002] .... 8291.232583: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=920320
flush-8:0-4272 [002] .... 8291.235968: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=922368
flush-8:0-4272 [002] .... 8291.241987: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=924416
dd-4267 [000] .... 8291.242861: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [000] .... 8291.246767: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=926464
flush-8:0-4272 [002] .... 8291.247352: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=926464
flush-8:0-4272 [002] .... 8291.249590: writeback_pages_written: 6400
dd-4267 [000] .... 8291.303192: writeback_congestion_wait: usec_timeout=100000 usec_delayed=57000
dd-4267 [002] .... 8291.418747: writeback_congestion_wait: usec_timeout=100000 usec_delayed=70000
dd-4267 [000] .... 8291.421290: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [002] .... 8291.430170: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8291.438818: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [000] .... 8291.448613: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8291.452546: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [002] .... 8291.458291: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8291.463059: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8291.501991: writeback_congestion_wait: usec_timeout=100000 usec_delayed=8000
dd-4267 [002] .... 8291.511822: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [000] .... 8291.521615: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [000] .... 8291.531477: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8291.541259: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8291.545313: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=926720
flush-8:0-4272 [002] .... 8291.545327: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=926720
dd-4267 [000] .... 8291.545329: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=926720
dd-4267 [000] .... 8291.546859: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=928768
flush-8:0-4272 [002] .... 8291.549677: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=926720
dd-4267 [000] .... 8291.550805: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [000] .... 8291.550981: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
flush-8:0-4272 [002] .... 8291.553082: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1024 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=928768
dd-4267 [000] .... 8291.554588: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=929536
dd-4267 [001] .... 8291.556446: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=931584
flush-8:0-4272 [002] .... 8291.558013: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=929536
dd-4267 [001] .... 8291.558925: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=933632
flush-8:0-4272 [002] .... 8291.560978: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=931584
dd-4267 [001] .... 8291.561309: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=935680
dd-4267 [000] .... 8291.566775: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
flush-8:0-4272 [002] .... 8291.567625: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=933632
flush-8:0-4272 [000] .... 8291.691938: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1024 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=935680
flush-8:0-4272 [000] .... 8291.696544: writeback_pages_written: 9984
dd-4267 [000] .... 8291.755429: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8291.765681: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [000] .... 8291.769854: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=936704
flush-8:0-4272 [002] .... 8291.769869: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=936704
dd-4267 [000] .... 8291.769870: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=936704
dd-4267 [000] .... 8291.771371: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
flush-8:0-4272 [002] .... 8291.773234: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=936704
flush-8:0-4272 [002] .... 8291.773241: writeback_pages_written: 256
dd-4267 [002] .... 8291.775760: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8291.781707: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [000] .... 8291.784087: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8291.788791: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [002] .... 8291.809228: writeback_congestion_wait: usec_timeout=100000 usec_delayed=12000
dd-4267 [000] .... 8291.811942: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8291.819305: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=936960
flush-8:0-4272 [002] .... 8291.819318: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=936960
dd-4267 [000] .... 8291.819319: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=936960
dd-4267 [000] .... 8291.820882: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=939008
dd-4267 [000] .... 8291.822428: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=941056
flush-8:0-4272 [002] .... 8291.822757: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=936960
dd-4267 [000] .... 8291.824434: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=943104
flush-8:0-4272 [002] .... 8291.825686: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=939008
dd-4267 [000] .... 8291.827048: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8291.827226: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
flush-8:0-4272 [002] .... 8291.831471: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=941056
flush-8:0-4272 [002] .... 8291.836478: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1536 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=943104
flush-8:0-4272 [002] .... 8291.839973: writeback_pages_written: 7680
dd-4267 [002] .... 8291.930645: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [002] .... 8291.992029: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=944640
flush-8:0-4272 [000] .... 8291.992046: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=944640
dd-4267 [002] .... 8291.992056: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=944640
flush-8:0-4272 [000] .... 8291.994832: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=944640
flush-8:0-4272 [000] .... 8291.994838: writeback_pages_written: 256
dd-4267 [002] .... 8292.030123: writeback_congestion_wait: usec_timeout=100000 usec_delayed=38000
dd-4267 [002] .... 8292.050503: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [002] .... 8292.054831: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [000] .... 8292.060738: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [002] .... 8292.065103: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8292.071061: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [002] .... 8292.075356: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8292.081306: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [002] .... 8292.087263: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [002] .... 8292.091595: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8292.097536: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [002] .... 8292.122806: writeback_congestion_wait: usec_timeout=100000 usec_delayed=24000
dd-4267 [002] .... 8292.127164: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8292.168759: writeback_congestion_wait: usec_timeout=100000 usec_delayed=38000
dd-4267 [000] .... 8292.173114: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8292.235518: writeback_congestion_wait: usec_timeout=100000 usec_delayed=59000
dd-4267 [002] .... 8292.240176: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8292.247496: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8292.257469: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [002] .... 8292.267454: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [002] .... 8292.273257: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [000] .... 8292.277430: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [002] .... 8292.283231: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [002] .... 8292.287413: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [002] .... 8292.293190: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [002] .... 8292.297383: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8292.303189: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [002] .... 8292.305193: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=944896
flush-8:0-4272 [000] .... 8292.305204: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=944896
dd-4267 [002] .... 8292.305205: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=944896
dd-4267 [002] .... 8292.306738: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=946944
dd-4267 [002] .... 8292.309833: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=948992
dd-4267 [000] .... 8292.313157: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
flush-8:0-4272 [000] .... 8292.735920: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=944896
flush-8:0-4272 [000] .... 8292.740409: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=946944
flush-8:0-4272 [001] .... 8292.748894: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=948992
flush-8:0-4272 [001] .... 8292.752966: writeback_start: bdi 8:0: sb_dev 0:0 nr_pages=33828 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-8:0-4272 [001] .... 8292.752968: writeback_queue_io: bdi 8:0: older=4302934457 age=30000 enqueue=0 reason=periodic
flush-8:0-4272 [001] .... 8292.752968: writeback_written: bdi 8:0: sb_dev 0:0 nr_pages=33828 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-8:0-4272 [001] .... 8292.752971: writeback_pages_written: 4864
dd-4267 [002] .... 8292.865367: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=949760
flush-8:0-4272 [001] .... 8292.865385: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=949760
dd-4267 [002] .... 8292.865386: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=949760
dd-4267 [002] .... 8292.867161: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=951808
dd-4267 [002] .... 8292.868811: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=953856
dd-4267 [002] .... 8292.870466: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=955904
flush-8:0-4272 [001] .... 8292.871705: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=949760
dd-4267 [002] .... 8292.873150: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=957952
dd-4267 [002] .... 8292.875731: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=960000
flush-8:0-4272 [001] .... 8292.876282: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=951808
flush-8:0-4272 [001] .... 8292.883855: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=953856
flush-8:0-4272 [001] .... 8292.889576: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=955904
flush-8:0-4272 [001] .... 8292.894750: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=957952
flush-8:0-4272 [001] .... 8292.899538: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=960000
flush-8:0-4272 [001] .... 8292.901892: writeback_pages_written: 11008
dd-4267 [000] .... 8292.976950: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [001] .... 8293.020761: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=960768
flush-8:0-4272 [003] .... 8293.020778: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=960768
dd-4267 [001] .... 8293.021248: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=960768
flush-8:0-4272 [003] .... 8293.023781: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=960768
flush-8:0-4272 [003] .... 8293.023789: writeback_pages_written: 256
dd-4267 [001] .... 8293.083843: writeback_congestion_wait: usec_timeout=100000 usec_delayed=62000
dd-4267 [003] .... 8293.097927: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8293.108016: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [001] .... 8293.126388: writeback_congestion_wait: usec_timeout=100000 usec_delayed=14000
dd-4267 [001] .... 8293.136458: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [003] .... 8293.146577: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8293.156636: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8293.166659: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8293.170791: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=961024
flush-8:0-4272 [003] .... 8293.170804: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=961024
dd-4267 [001] .... 8293.170805: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=961024
dd-4267 [001] .... 8293.172326: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=963072
dd-4267 [001] .... 8293.173830: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=965120
flush-8:0-4272 [003] .... 8293.174168: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=961024
dd-4267 [001] .... 8293.175352: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=967168
flush-8:0-4272 [003] .... 8293.177053: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=963072
flush-8:0-4272 [003] .... 8293.182445: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=965120
dd-4267 [002] .... 8293.282859: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
flush-8:0-4272 [003] .... 8293.321004: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=967168
flush-8:0-4272 [003] .... 8293.325879: writeback_pages_written: 8192
dd-4267 [002] .... 8293.424762: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [002] .... 8293.428997: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8293.434825: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [002] .... 8293.439025: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [002] .... 8293.444901: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [002] .... 8293.449071: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [002] .... 8293.454973: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [002] .... 8293.459202: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8293.465002: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [002] .... 8293.469260: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8293.475132: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [002] .... 8293.479335: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8293.485174: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [002] .... 8293.489424: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8293.495268: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [002] .... 8293.499474: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8293.505310: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [002] .... 8293.509565: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8293.515405: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [002] .... 8293.519611: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [003] .... 8293.529871: writeback_congestion_wait: usec_timeout=100000 usec_delayed=8000
dd-4267 [001] .... 8293.539968: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [001] .... 8293.550034: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8293.560097: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [003] .... 8293.570116: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [003] .... 8293.574101: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=969216
flush-8:0-4272 [001] .... 8293.574114: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=969216
dd-4267 [003] .... 8293.574119: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=969216
flush-8:0-4272 [001] .... 8293.578134: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=969216
flush-8:0-4272 [001] .... 8293.578142: writeback_pages_written: 256
dd-4267 [001] .... 8293.580245: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8293.583993: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=969472
flush-8:0-4272 [003] .... 8293.584005: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=969472
dd-4267 [001] .... 8293.584006: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=969472
dd-4267 [001] .... 8293.585670: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=971520
flush-8:0-4272 [003] .... 8293.587582: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=969472
dd-4267 [001] .... 8293.590259: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
flush-8:0-4272 [003] .... 8293.590489: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=512 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=971520
flush-8:0-4272 [003] .... 8293.593743: writeback_pages_written: 2560
dd-4267 [001] .... 8293.594087: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=972032
flush-8:0-4272 [003] .... 8293.594095: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=972032
dd-4267 [001] .... 8293.594104: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=972032
dd-4267 [001] .... 8293.595677: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=974080
flush-8:0-4272 [003] .... 8293.596789: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=972032
flush-8:0-4272 [003] .... 8293.599663: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=974080
flush-8:0-4272 [003] .... 8293.601695: writeback_pages_written: 2304
dd-4267 [001] .... 8293.620507: writeback_congestion_wait: usec_timeout=100000 usec_delayed=24000
dd-4267 [001] .... 8293.640663: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [003] .... 8293.648906: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [001] .... 8293.657732: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [001] .... 8293.662163: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [003] .... 8293.666667: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [001] .... 8293.719584: writeback_congestion_wait: usec_timeout=100000 usec_delayed=48000
dd-4267 [001] .... 8293.721790: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [001] .... 8293.724056: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [001] .... 8293.731220: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [001] .... 8293.741252: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8293.748399: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [001] .... 8293.752893: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [001] .... 8293.757387: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [003] .... 8293.761705: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [001] .... 8293.768565: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8293.773111: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [003] .... 8293.784513: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [001] .... 8293.794486: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [001] .... 8293.804558: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8293.816096: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [000] .... 8293.820338: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [002] .... 8293.822648: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [002] .... 8293.824862: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [003] .... 8293.827337: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [001] .... 8293.831982: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [001] .... 8293.850276: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=974336
flush-8:0-4272 [003] .... 8293.850291: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=974336
dd-4267 [001] .... 8293.850292: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=974336
flush-8:0-4272 [003] .... 8293.854749: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=974336
flush-8:0-4272 [003] .... 8293.854756: writeback_pages_written: 256
dd-4267 [003] .... 8293.949419: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [001] .... 8293.950099: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [001] .... 8293.954661: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [003] .... 8293.957135: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [003] .... 8293.959495: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=974592
flush-8:0-4272 [001] .... 8293.959505: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=974592
dd-4267 [003] .... 8293.959506: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=974592
flush-8:0-4272 [001] .... 8293.964429: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=974592
dd-4267 [003] .... 8293.964831: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=976640
dd-4267 [003] .... 8293.966420: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=978688
flush-8:0-4272 [001] .... 8293.967125: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=976640
dd-4267 [003] .... 8293.967957: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=980736
flush-8:0-4272 [001] .... 8293.974557: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=978688
dd-4267 [000] .... 8293.976671: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=982784
dd-4267 [000] .... 8293.979998: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=984832
dd-4267 [000] .... 8293.980544: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
flush-8:0-4272 [001] .... 8293.981418: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=980736
flush-8:0-4272 [001] .... 8293.987940: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=982784
dd-4267 [000] .... 8293.988730: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
flush-8:0-4272 [001] .... 8293.994540: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=984832
flush-8:0-4272 [001] .... 8293.996489: writeback_pages_written: 10496
dd-4267 [001] .... 8294.029115: writeback_congestion_wait: usec_timeout=100000 usec_delayed=27000
dd-4267 [001] .... 8294.064816: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=985088
flush-8:0-4272 [003] .... 8294.064833: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=985088
dd-4267 [001] .... 8294.064834: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=985088
flush-8:0-4272 [003] .... 8294.067992: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=985088
flush-8:0-4272 [003] .... 8294.068000: writeback_pages_written: 256
dd-4267 [001] .... 8294.068875: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8294.078863: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8294.088870: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [003] .... 8294.098835: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8294.108811: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8294.118806: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8294.122874: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=985344
flush-8:0-4272 [003] .... 8294.122890: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=985344
dd-4267 [001] .... 8294.122895: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=985344
dd-4267 [001] .... 8294.124491: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=987392
dd-4267 [001] .... 8294.126057: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=989440
flush-8:0-4272 [003] .N.. 8294.126285: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=985344
dd-4267 [001] .... 8294.127632: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=991488
flush-8:0-4272 [003] .... 8294.129080: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=987392
flush-8:0-4272 [003] .... 8294.134718: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=989440
flush-8:0-4272 [003] .... 8294.139698: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1792 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=991488
flush-8:0-4272 [003] .... 8294.143930: writeback_pages_written: 7936
dd-4267 [000] .... 8294.154721: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=993280
flush-8:0-4272 [003] .... 8294.154740: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=993280
dd-4267 [000] .... 8294.154741: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=993280
flush-8:0-4272 [003] .... 8294.156714: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=993280
flush-8:0-4272 [003] .... 8294.156722: writeback_pages_written: 256
dd-4267 [002] .... 8294.254308: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [002] .... 8294.351041: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=993536
flush-8:0-4272 [003] .... 8294.351061: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=993536
dd-4267 [002] .... 8294.351067: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=993536
flush-8:0-4272 [003] .... 8294.355385: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=993536
flush-8:0-4272 [003] .... 8294.355392: writeback_pages_written: 256
dd-4267 [003] .... 8294.429885: writeback_congestion_wait: usec_timeout=100000 usec_delayed=74000
dd-4267 [001] .... 8294.432726: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [001] .... 8294.435228: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [001] .... 8294.449908: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [001] .... 8294.455841: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [003] .... 8294.461793: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8294.466140: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [001] .N.. 8294.472105: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8294.476408: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [003] .... 8294.482332: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8294.486666: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8294.492631: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8294.496951: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [003] .... 8294.502897: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8294.507247: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [001] .... 8294.513196: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8294.517488: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [003] .... 8294.523451: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8294.527785: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8294.533730: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8294.538058: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [003] .... 8294.544007: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [001] .... 8294.548348: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [001] .... 8294.613136: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [001] .... 8294.617232: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8294.622950: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [003] .... 8294.627046: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [001] .... 8294.632738: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [003] .... 8294.646693: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [001] .... 8294.656647: writeback_congestion_wait: usec_timeout=100000 usec_delayed=10000
dd-4267 [001] .... 8294.662184: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8294.676087: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [001] .... 8294.680177: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8294.685900: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [003] .... 8294.690000: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [001] .... 8294.695720: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [001] .... 8294.699792: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8294.705508: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [003] .... 8294.709607: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8294.715336: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8294.719415: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8294.725125: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [003] .... 8294.729222: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8294.734952: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8294.739050: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8294.744735: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [003] .... 8294.748838: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8294.754564: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8294.758690: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8294.768628: writeback_congestion_wait: usec_timeout=100000 usec_delayed=10000
dd-4267 [001] .... 8294.827256: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [000] .... 8294.829463: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [000] .... 8294.831574: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8294.833749: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [002] .... 8294.835917: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [003] .... 8294.849642: writeback_congestion_wait: usec_timeout=100000 usec_delayed=13000
dd-4267 [001] .... 8294.851883: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [001] .... 8294.926921: writeback_congestion_wait: usec_timeout=100000 usec_delayed=71000
dd-4267 [003] .... 8294.929117: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [003] .... 8294.931419: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
dd-4267 [003] .... 8294.936759: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=993792
flush-8:0-4272 [001] .... 8294.936771: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=993792
dd-4267 [003] .... 8294.936776: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=993792
dd-4267 [003] .... 8294.938346: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=995840
dd-4267 [003] .... 8294.939895: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=997888
dd-4267 [003] .... 8294.941461: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=999936
flush-8:0-4272 [001] .... 8294.942432: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=993792
dd-4267 [003] .... 8294.943044: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1001984
dd-4267 [003] .... 8294.944594: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1004032
flush-8:0-4272 [001] .... 8294.945567: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=995840
dd-4267 [003] .... 8294.948509: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1006080
dd-4267 [000] .... 8294.950142: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1008128
flush-8:0-4272 [000] .... 8294.954509: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=997888
flush-8:0-4272 [000] .... 8295.060073: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=999936
flush-8:0-4272 [001] .... 8295.070258: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1001984
flush-8:0-4272 [001] .... 8295.079359: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1004032
flush-8:0-4272 [001] .... 8295.086159: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1006080
flush-8:0-4272 [001] .... 8295.092211: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1024 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1008128
flush-8:0-4272 [001] .... 8295.095943: writeback_pages_written: 15360
dd-4267 [002] .... 8295.149629: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1009152
flush-8:0-4272 [001] .... 8295.149649: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1009152
dd-4267 [002] .... 8295.149661: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1009152
flush-8:0-4272 [001] .... 8295.154294: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=768 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1009152
dd-4267 [002] .... 8295.154608: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1009920
flush-8:0-4272 [001] .... 8295.155299: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1024 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1009920
dd-4267 [002] .... 8295.155309: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1010688
dd-4267 [002] .... 8295.156976: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1012736
dd-4267 [002] .... 8295.158592: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1014784
flush-8:0-4272 [001] .... 8295.159917: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1010688
flush-8:0-4272 [001] .... 8295.163902: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1012736
dd-4267 [002] .... 8295.164768: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1016832
flush-8:0-4272 [001] .... 8295.170302: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1014784
flush-8:0-4272 [001] .... 8295.175152: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=1536 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1016832
flush-8:0-4272 [001] .... 8295.178552: writeback_pages_written: 9216
dd-4267 [000] .... 8295.265787: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [003] .... 8295.368201: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8295.377759: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [001] .... 8295.381826: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [003] .... 8295.387549: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [001] .... 8295.391664: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [001] .... 8295.397376: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8295.401466: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [003] .... 8295.407182: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [001] .... 8295.411274: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [001] .... 8295.417013: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8295.421116: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [003] .... 8295.426809: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [001] .... 8295.430867: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [001] .... 8295.436645: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8295.446602: writeback_congestion_wait: usec_timeout=100000 usec_delayed=7000
dd-4267 [001] .... 8295.450132: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1018368
flush-8:0-4272 [003] .... 8295.450147: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1018368
dd-4267 [001] .... 8295.450152: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1018368
dd-4267 [001] .... 8295.454012: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1020416
flush-8:0-4272 [003] .... 8295.454533: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1018368
dd-4267 [001] .... 8295.455564: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1022464
flush-8:0-4272 [003] .... 8295.457413: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1020416
dd-4267 [000] .... 8295.459310: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1024512
dd-4267 [001] .... 8295.460273: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
flush-8:0-4272 [003] .... 8295.463336: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1022464
flush-8:0-4272 [003] .... 8295.468614: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1024512
flush-8:0-4272 [003] .... 8295.470473: writeback_pages_written: 6400
dd-4267 [003] .... 8295.561536: writeback_congestion_wait: usec_timeout=100000 usec_delayed=100000
dd-4267 [000] .... 8295.619605: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1024768
flush-8:0-4272 [003] .... 8295.619628: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1024768
dd-4267 [000] .... 8295.619629: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1024768
flush-8:0-4272 [003] .... 8295.624325: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1024768
flush-8:0-4272 [003] .... 8295.624334: writeback_pages_written: 256
dd-4267 [001] .... 8295.626436: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [001] .... 8295.630738: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [001] .... 8295.636693: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [003] .... 8295.641010: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [001] .... 8295.646977: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8295.651282: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [001] .... 8295.657228: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [003] .... 8295.661569: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [001] .... 8295.667523: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [001] .... 8295.671798: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [001] .... 8295.677742: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [003] .... 8295.682120: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [001] .... 8295.688077: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [001] .... 8295.698626: writeback_congestion_wait: usec_timeout=100000 usec_delayed=9000
dd-4267 [001] .... 8295.708811: writeback_congestion_wait: usec_timeout=100000 usec_delayed=5000
dd-4267 [003] .... 8295.719075: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [001] .... 8295.724796: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [001] .... 8295.729170: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [001] .... 8295.735113: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [003] .... 8295.739452: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [003] .... 8295.741771: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1025024
flush-8:0-4272 [001] .... 8295.741782: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1025024
dd-4267 [003] .... 8295.741783: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1025024
dd-4267 [003] .... 8295.745338: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
flush-8:0-4272 [001] .... 8295.745929: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1025024
flush-8:0-4272 [001] .... 8295.745935: writeback_pages_written: 256
dd-4267 [001] .... 8295.749693: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [001] .... 8295.755677: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [003] .... 8295.760008: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [001] .... 8295.765967: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [001] .... 8295.770295: writeback_congestion_wait: usec_timeout=100000 usec_delayed=1000
dd-4267 [001] .... 8295.776226: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [003] .... 8295.780561: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [001] .... 8295.786491: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8295.790761: writeback_congestion_wait: usec_timeout=100000 usec_delayed=3000
dd-4267 [001] .... 8295.796727: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [003] .... 8295.801103: writeback_congestion_wait: usec_timeout=100000 usec_delayed=2000
dd-4267 [001] .... 8295.807061: writeback_congestion_wait: usec_timeout=100000 usec_delayed=4000
dd-4267 [001] .... 8295.808918: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1025280
flush-8:0-4272 [003] .... 8295.808931: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1025280
dd-4267 [001] .... 8295.808942: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1025280
dd-4267 [001] .... 8295.810446: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1027328
dd-4267 [001] .... 8295.811934: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1029376
flush-8:0-4272 [003] .... 8295.813114: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1025280
dd-4267 [001] .... 8295.815817: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1031424
flush-8:0-4272 [003] .... 8295.816182: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1027328
dd-4267 [000] .... 8295.821315: writeback_queue: bdi 8:0: sb_dev 0:0 nr_pages=256 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1033472
flush-8:0-4272 [003] .... 8295.822702: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1029376
dd-4267 [000] .... 8295.827544: writeback_congestion_wait: usec_timeout=100000 usec_delayed=6000
dd-4267 [000] .... 8295.827734: writeback_congestion_wait: usec_timeout=100000 usec_delayed=0
flush-8:0-4272 [003] .... 8295.828932: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=2048 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1031424
dd-4267 [001] .... 8295.864365: writeback_congestion_wait: usec_timeout=100000 usec_delayed=33000
flush-8:0-4272 [001] .... 8296.051145: writeback_exec: bdi 8:0: sb_dev 0:0 nr_pages=512 sync_mode=0 kupdate=0 range_cyclic=0 background=0 reason=pageout ino=13 offset=1033472
flush-8:0-4272 [001] .... 8296.054689: writeback_pages_written: 8704
flush-8:0-4272 [003] .... 8301.051487: writeback_start: bdi 8:0: sb_dev 0:0 nr_pages=21838 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-8:0-4272 [003] .... 8301.051491: writeback_queue_io: bdi 8:0: older=4302942761 age=30000 enqueue=1 reason=periodic
flush-8:0-4272 [003] .... 8301.051498: writeback_single_inode: bdi 8:0: ino=13 state= dirtied_when=4302938315 age=39 index=0 to_write=1024 wrote=0
flush-8:0-4272 [003] .... 8301.051501: writeback_written: bdi 8:0: sb_dev 0:0 nr_pages=21838 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-8:0-4272 [003] .... 8301.051502: writeback_start: bdi 8:0: sb_dev 0:0 nr_pages=21838 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-8:0-4272 [003] .... 8301.051502: writeback_queue_io: bdi 8:0: older=4302942761 age=30000 enqueue=0 reason=periodic
flush-8:0-4272 [003] .... 8301.051502: writeback_written: bdi 8:0: sb_dev 0:0 nr_pages=21838 sync_mode=0 kupdate=1 range_cyclic=1 background=0 reason=periodic ino=0 offset=0
flush-8:0-4272 [003] .... 8301.051505: writeback_pages_written: 0
[-- Attachment #2: balance_dirty_pages-task-bw-300.png --]
[-- Type: image/png, Size: 41820 bytes --]
[-- Attachment #3: balance_dirty_pages-task-bw.png --]
[-- Type: image/png, Size: 34173 bytes --]
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: reclaim the LRU lists full of dirty/writeback pages
2012-02-12 3:10 ` Wu Fengguang
2012-02-12 6:45 ` Wu Fengguang
@ 2012-02-13 15:43 ` Jan Kara
2012-02-14 10:03 ` Wu Fengguang
1 sibling, 1 reply; 33+ messages in thread
From: Jan Kara @ 2012-02-13 15:43 UTC (permalink / raw)
To: Wu Fengguang
Cc: Rik van Riel, Greg Thelen, Jan Kara, bsingharora, Hugh Dickins,
Michal Hocko, linux-mm, Mel Gorman, Ying Han, hannes,
KAMEZAWA Hiroyuki, Minchan Kim
On Sun 12-02-12 11:10:29, Wu Fengguang wrote:
> On Sat, Feb 11, 2012 at 09:55:38AM -0500, Rik van Riel wrote:
> > On 02/11/2012 07:44 AM, Wu Fengguang wrote:
> >
> > >Note that it's data for XFS. ext4 seems to have some problem with the
> > >workload: the majority pages are found to be writeback pages, and the
> > >flusher ends up blocking on the unconditional wait_on_page_writeback()
> > >in write_cache_pages_da() from time to time...
>
> Sorry I overlooked the WB_SYNC_NONE test before the wait_on_page_writeback()
> call! And the issue can no longer be reproduce anyway. ext4 performs pretty
> good now, here is the result for one single memcg dd:
>
> dd if=/dev/zero of=/fs/f$i bs=4k count=1M
>
> 4294967296 bytes (4.3 GB) copied, 44.5759 s, 96.4 MB/s
>
> iostat -kx 3
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 0.25 0.00 11.03 28.54 0.00 60.19
> 0.25 0.00 13.71 16.65 0.00 69.39
> 0.17 0.00 8.41 24.81 0.00 66.61
> 0.25 0.00 15.00 19.63 0.00 65.12
>
> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
> sda 0.00 17.00 0.00 178.33 0.00 90694.67 1017.14 111.34 520.88 5.45 97.23
> sda 0.00 0.00 0.00 193.67 0.00 98816.00 1020.48 86.22 496.81 4.81 93.07
> sda 0.00 3.33 0.00 182.33 0.00 92345.33 1012.93 101.14 623.98 5.49 100.03
> sda 0.00 3.00 0.00 187.00 0.00 95586.67 1022.32 89.36 441.70 4.96 92.70
>
> > >XXX: commit NFS unstable pages via write_inode()
> > >XXX: the added congestion_wait() may be undesirable in some situations
> >
> > Even with these caveats, this seems to be the right way forward.
>
> > Acked-by: Rik van Riel <riel@redhat.com>
>
> Thank you!
>
> Here is the updated patch.
> - ~10ms write around chunk size, adaptive to the bdi bandwith
> - cleanup flush_inode_page()
>
> Thanks,
> Fengguang
> ---
> Subject: writeback: introduce the pageout work
> Date: Thu Jul 29 14:41:19 CST 2010
>
> This relays file pageout IOs to the flusher threads.
>
> The ultimate target is to gracefully handle the LRU lists full of
> dirty/writeback pages.
>
> 1) I/O efficiency
>
> The flusher will piggy back the nearby ~10ms worth of dirty pages for I/O.
>
> This takes advantage of the time/spacial locality in most workloads: the
> nearby pages of one file are typically populated into the LRU at the same
> time, hence will likely be close to each other in the LRU list. Writing
> them in one shot helps clean more pages effectively for page reclaim.
>
> 2) OOM avoidance and scan rate control
>
> Typically we do LRU scan w/o rate control and quickly get enough clean
> pages for the LRU lists not full of dirty pages.
>
> Or we can still get a number of freshly cleaned pages (moved to LRU tail
> by end_page_writeback()) when the queued pageout I/O is completed within
> tens of milli-seconds.
>
> However if the LRU list is small and full of dirty pages, it can be
> quickly fully scanned and go OOM before the flusher manages to clean
> enough pages.
>
> A simple yet reliable scheme is employed to avoid OOM and keep scan rate
> in sync with the I/O rate:
>
> if (PageReclaim(page))
> congestion_wait(HZ/10);
>
> PG_reclaim plays the key role. When dirty pages are encountered, we
> queue I/O for it, set PG_reclaim and put it back to the LRU head.
> So if PG_reclaim pages are encountered again, it means the dirty page
> has not yet been cleaned by the flusher after a full zone scan. It
> indicates we are scanning more fast than I/O and shall take a snap.
>
> The runtime behavior on a fully dirtied small LRU list would be:
> It will start with a quick scan of the list, queuing all pages for I/O.
> Then the scan will be slowed down by the PG_reclaim pages *adaptively*
> to match the I/O bandwidth.
>
> 3) writeback work coordinations
>
> To avoid memory allocations at page reclaim, a mempool for struct
> wb_writeback_work is created.
>
> wakeup_flusher_threads() is removed because it can easily delay the
> more oriented pageout works and even exhaust the mempool reservations.
> It's also found to not I/O efficient by frequently submitting writeback
> works with small ->nr_pages.
>
> Background/periodic works will quit automatically, so as to clean the
> pages under reclaim ASAP. However for now the sync work can still block
> us for long time.
>
> Jan Kara: limit the search scope. Note that the limited search and work
> pool is not a big problem: 1000 IOs under flight are typically more than
> enough to saturate the disk. And the overheads of searching in the work
> list didn't even show up in the perf report.
>
> 4) test case
>
> Run 2 dd tasks in a 100MB memcg (a very handy test case from Greg Thelen):
>
> mkdir /cgroup/x
> echo 100M > /cgroup/x/memory.limit_in_bytes
> echo $$ > /cgroup/x/tasks
>
> for i in `seq 2`
> do
> dd if=/dev/zero of=/fs/f$i bs=1k count=1M &
> done
>
> Before patch, the dd tasks are quickly OOM killed.
> After patch, they run well with reasonably good performance and overheads:
>
> 1073741824 bytes (1.1 GB) copied, 22.2196 s, 48.3 MB/s
> 1073741824 bytes (1.1 GB) copied, 22.4675 s, 47.8 MB/s
I wonder what happens if you run:
mkdir /cgroup/x
echo 100M > /cgroup/x/memory.limit_in_bytes
echo $$ > /cgroup/x/tasks
for (( i = 0; i < 2; i++ )); do
mkdir /fs/d$i
for (( j = 0; j < 5000; j++ )); do
dd if=/dev/zero of=/fs/d$i/f$j bs=1k count=50
done &
done
Because for small files the writearound logic won't help much... Also the
number of work items queued might become interesting.
Another common case to test - run 'slapadd' command in each cgroup to
create big LDAP database. That does pretty much random IO on a big mmaped
DB file.
> +/*
> + * schedule writeback on a range of inode pages.
> + */
> +static struct wb_writeback_work *
> +bdi_flush_inode_range(struct backing_dev_info *bdi,
> + struct inode *inode,
> + pgoff_t offset,
> + pgoff_t len,
> + bool wait)
> +{
> + struct wb_writeback_work *work;
> +
> + if (!igrab(inode))
> + return ERR_PTR(-ENOENT);
One technical note here: If the inode is deleted while it is queued, this
reference will keep it living until flusher thread gets to it. Then when
flusher thread puts its reference, the inode will get deleted in flusher
thread context. I don't see an immediate problem in that but it might be
surprising sometimes. Another problem I see is that if you try to
unmount the filesystem while the work item is queued, you'll get EBUSY for
no apparent reason (for userspace).
Honza
--
Jan Kara <jack@suse.cz>
SUSE Labs, CR
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: memcg writeback (was Re: [Lsf-pc] [LSF/MM TOPIC] memcg topics.)
2012-02-09 13:50 ` Wu Fengguang
@ 2012-02-13 18:40 ` Ying Han
0 siblings, 0 replies; 33+ messages in thread
From: Ying Han @ 2012-02-13 18:40 UTC (permalink / raw)
To: Wu Fengguang
Cc: Greg Thelen, Jan Kara, bsingharora, Hugh Dickins, Michal Hocko,
linux-mm, Mel Gorman, hannes, lsf-pc, KAMEZAWA Hiroyuki
On Thu, Feb 9, 2012 at 5:50 AM, Wu Fengguang <fengguang.wu@intel.com> wrote:
> On Wed, Feb 08, 2012 at 12:54:33PM -0800, Ying Han wrote:
>> On Wed, Feb 8, 2012 at 1:31 AM, Wu Fengguang <fengguang.wu@intel.com> wrote:
>> > On Tue, Feb 07, 2012 at 11:55:05PM -0800, Greg Thelen wrote:
>> >> On Fri, Feb 3, 2012 at 1:40 AM, Wu Fengguang <fengguang.wu@intel.com> wrote:
>> >> > If moving dirty pages out of the memcg to the 20% global dirty pages
>> >> > pool on page reclaim, the above OOM can be avoided. It does change the
>> >> > meaning of memory.limit_in_bytes in that the memcg tasks can now
>> >> > actually consume more pages (up to the shared global 20% dirty limit).
>> >>
>> >> This seems like an easy change, but unfortunately the global 20% pool
>> >> has some shortcomings for my needs:
>> >>
>> >> 1. the global 20% pool is not moderated. One cgroup can dominate it
>> >> and deny service to other cgroups.
>> >
>> > It is moderated by balance_dirty_pages() -- in terms of dirty ratelimit.
>> > And you have the freedom to control the bandwidth allocation with some
>> > async write I/O controller.
>> >
>> > Even though there is no direct control of dirty pages, we can roughly
>> > get it as the side effect of rate control. Given
>> >
>> > ratelimit_cgroup_A = 2 * ratelimit_cgroup_B
>> >
>> > There will naturally be more dirty pages for cgroup A to be worked by
>> > the flusher. And the dirty pages will be roughly balanced around
>> >
>> > nr_dirty_cgroup_A = 2 * nr_dirty_cgroup_B
>> >
>> > when writeout bandwidths for their dirty pages are equal.
>> >
>> >> 2. the global 20% pool is free, unaccounted memory. Ideally cgroups only
>> >> use the amount of memory specified in their memory.limit_in_bytes. The
>> >> goal is to sell portions of a system. Global resource like the 20% are an
>> >> undesirable system-wide tax that's shared by jobs that may not even
>> >> perform buffered writes.
>> >
>> > Right, it is the shortcoming.
>> >
>> >> 3. Setting aside 20% extra memory for system wide dirty buffers is a lot of
>> >> memory. This becomes a larger issue when the global dirty_ratio is
>> >> higher than 20%.
>> >
>> > Yeah the global pool scheme does mean that you'd better allocate at
>> > most 80% memory to individual memory cgroups, otherwise it's possible
>> > for a tiny memcg doing dd writes to push dirty pages to global LRU and
>> > *squeeze* the size of other memcgs.
>> >
>> > However I guess it should be mitigated by the fact that
>> >
>> > - we typically already reserve some space for the root memcg
>>
>> Can you give more details on that? AFAIK, we don't treat root cgroup
>> differently than other sub-cgroups, except root cgroup doesn't have
>> limit.
>
> OK. I'd imagine this to be the typical usage for desktop and quite a
> few servers: a few cgroups are employed to limit the resource usage
> for selected tasks (such as backups, background GUI tasks, cron tasks,
> etc.). These systems are still running mainly in the global context.
The use case makes senses, but still not sure about the "reservation
for root" part.
For other tasks not running under cgroups, they runs under global
context as you said. However, there is no memory limit for root cgroup
and it will only trigger global reclaim when running short of memory.
It doesn't sounds like a straight-forward configuration for
environments requires memory isolation badly. The worst part is the
unpredictability, which we don't have control of how much
dirty-and-later-clean pages being leaked to root and stays.
--Ying
>
>> In general, I don't like the idea of shared pool in root for all the
>> dirty pages.
>>
>> Imagining a system which has nothing running under root and every
>> application runs within sub-cgroup. It is easy to track and limit each
>> cgroup's memory usage, but not the pages being moved to root. We have
>> been experiencing difficulties of tracking pages being re-parented to
>> root, and this will make it even harder.
>
> So you want to push memcg allocations to the hardware limits. This is
> a worthwhile target for cloud servers that run a number of well
> contained jobs.
>
> I guess it can be achieved reasonably well with the global shared
> dirty pool. Let's discuss the two major cases.
>
> 1) no change of behavior
>
> For example, when the system memory is divided equally to 10 cgroups
> each running 1 dd. In this case, the dirty pages will be contained
> within the memcg LRUs. Page reclaim rarely encounters any dirty pages.
> There is no moving to the global LRU, so no side effect at all.
>
> 2) small memcg squeezing other memcg(s)
>
> When system memory is divided to 1 small memcg A and 1 large memcg B,
> each running a dd task. In this case the dirty pages from A will be
> moved to the global LRU, and global page reclaims will be triggered.
>
> In the end it will be balanced around
>
> - global LRU: 10% memory (which are A's dirty pages)
> - memcg B: 90% memory
> - memcg A: a tiny ignorable fraction of memory
>
> Now job B uses 10% less memory than w/o the global dirty pool scheme.
> I guess this is bad for some type of jobs.
>
> However my question is, will the typical demand be more flexible?
> Something like the "minimal" and "recommended" setup: "this job
> requires at least XXX memory and better at YYY memory", rather than
> some fixed size memory allocation.
>
> The minimal requirement should be trivially satisfied by adding a
> memcg watermark that protects the memcg LRU from being reclaimed
> when dropped under it.
>
> Then the cloud server could be configured to
>
> sum(memcg.limit_in_bytes) / memtotal = 100%
> sum(memcg.minimal_size) / memtotal < 100% - dirty_ratio
>
> Which makes a simple and flexibly partitioned system.
>
> Thanks,
> Fengguang
>
>> > - 20% dirty ratio is mostly an overkill for large memory systems.
>> > It's often enough to hold 10-30s worth of dirty data for them, which
>> > is 1-3GB for one 100MB/s disk. This is the reason vm.dirty_bytes is
>> > introduced: someone wants to do some <1% dirty ratio.
>> >
>> > Thanks,
>> > Fengguang
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: reclaim the LRU lists full of dirty/writeback pages
2012-02-13 15:43 ` Jan Kara
@ 2012-02-14 10:03 ` Wu Fengguang
2012-02-14 13:29 ` Jan Kara
0 siblings, 1 reply; 33+ messages in thread
From: Wu Fengguang @ 2012-02-14 10:03 UTC (permalink / raw)
To: Jan Kara
Cc: Rik van Riel, Greg Thelen, bsingharora, Hugh Dickins,
Michal Hocko, linux-mm, Mel Gorman, Ying Han, hannes,
KAMEZAWA Hiroyuki, Minchan Kim
[-- Attachment #1: Type: text/plain, Size: 32079 bytes --]
On Mon, Feb 13, 2012 at 04:43:13PM +0100, Jan Kara wrote:
> On Sun 12-02-12 11:10:29, Wu Fengguang wrote:
> > 4) test case
> >
> > Run 2 dd tasks in a 100MB memcg (a very handy test case from Greg Thelen):
> >
> > mkdir /cgroup/x
> > echo 100M > /cgroup/x/memory.limit_in_bytes
> > echo $$ > /cgroup/x/tasks
> >
> > for i in `seq 2`
> > do
> > dd if=/dev/zero of=/fs/f$i bs=1k count=1M &
> > done
> >
> > Before patch, the dd tasks are quickly OOM killed.
> > After patch, they run well with reasonably good performance and overheads:
> >
> > 1073741824 bytes (1.1 GB) copied, 22.2196 s, 48.3 MB/s
> > 1073741824 bytes (1.1 GB) copied, 22.4675 s, 47.8 MB/s
> I wonder what happens if you run:
> mkdir /cgroup/x
> echo 100M > /cgroup/x/memory.limit_in_bytes
> echo $$ > /cgroup/x/tasks
>
> for (( i = 0; i < 2; i++ )); do
> mkdir /fs/d$i
> for (( j = 0; j < 5000; j++ )); do
> dd if=/dev/zero of=/fs/d$i/f$j bs=1k count=50
> done &
> done
That's a very good case, thanks!
> Because for small files the writearound logic won't help much...
Right, it also means the native background work cannot be more I/O
efficient than the pageout works, except for the overheads of more
work items..
> Also the number of work items queued might become interesting.
It turns out that the 1024 mempool reservations are not exhausted at
all (the below patch as a trace_printk on alloc failure and it didn't
trigger at all).
Here is the representative iostat lines on XFS (full "iostat -kx 1 20" log attached):
avg-cpu: %user %nice %system %iowait %steal %idle
0.80 0.00 6.03 0.03 0.00 93.14
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 205.00 0.00 163.00 0.00 16900.00 207.36 4.09 21.63 1.88 30.70
The attached dirtied/written progress graph looks interesting.
Although the iostat disk utilization is low, the "dirtied" progress
line is pretty straight and there is no single congestion_wait event
in the trace log. Which makes me wonder if there are some unknown
blocking issues in the way.
> Another common case to test - run 'slapadd' command in each cgroup to
> create big LDAP database. That does pretty much random IO on a big mmaped
> DB file.
I've not used this. Will it need some configuration and data feed?
fio looks more handy to me for emulating mmap random IO.
> > +/*
> > + * schedule writeback on a range of inode pages.
> > + */
> > +static struct wb_writeback_work *
> > +bdi_flush_inode_range(struct backing_dev_info *bdi,
> > + struct inode *inode,
> > + pgoff_t offset,
> > + pgoff_t len,
> > + bool wait)
> > +{
> > + struct wb_writeback_work *work;
> > +
> > + if (!igrab(inode))
> > + return ERR_PTR(-ENOENT);
> One technical note here: If the inode is deleted while it is queued, this
> reference will keep it living until flusher thread gets to it. Then when
> flusher thread puts its reference, the inode will get deleted in flusher
> thread context. I don't see an immediate problem in that but it might be
> surprising sometimes. Another problem I see is that if you try to
> unmount the filesystem while the work item is queued, you'll get EBUSY for
> no apparent reason (for userspace).
Yeah, we need to make umount work.
And I find the pageout works seem to have some problems with ext4.
For example, this can be easily triggered with 10 dd tasks running
inside the 100MB limited memcg:
[18006.858109] INFO: task jbd2/sda1-8:51294 blocked for more than 120 seconds.
[18006.866425] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[18006.876096] jbd2/sda1-8 D 0000000000000000 5464 51294 2 0x00000000
[18006.884729] ffff88040b097c70 0000000000000046 ffff880823032310 ffff88040b096000
[18006.894356] 00000000001d2f00 00000000001d2f00 ffff8808230322a0 00000000001d2f00
[18006.904000] ffff88040b097fd8 00000000001d2f00 ffff88040b097fd8 00000000001d2f00
[18006.913652] Call Trace:
[18006.916901] [<ffffffff8103d4af>] ? native_sched_clock+0x29/0x70
[18006.924134] [<ffffffff81232aab>] ? jbd2_journal_commit_transaction+0x1d0/0x1281
[18006.933324] [<ffffffff8109660d>] ? local_clock+0x41/0x5a
[18006.939879] [<ffffffff810b0ddd>] ? lock_release_holdtime+0xa3/0xac
[18006.947410] [<ffffffff81232aab>] ? jbd2_journal_commit_transaction+0x1d0/0x1281
[18006.956607] [<ffffffff81a57904>] schedule+0x5a/0x5c
[18006.962677] [<ffffffff81232ab0>] jbd2_journal_commit_transaction+0x1d5/0x1281
[18006.971683] [<ffffffff8103d4af>] ? native_sched_clock+0x29/0x70
[18006.978933] [<ffffffff810738ce>] ? try_to_del_timer_sync+0xba/0xc8
[18006.986452] [<ffffffff8109660d>] ? local_clock+0x41/0x5a
[18006.992999] [<ffffffff8108683a>] ? wake_up_bit+0x2a/0x2a
[18006.999542] [<ffffffff810738ce>] ? try_to_del_timer_sync+0xba/0xc8
[18007.007062] [<ffffffff81073a6f>] ? del_timer_sync+0xbb/0xce
[18007.013898] [<ffffffff810739b4>] ? process_timeout+0x10/0x10
[18007.020835] [<ffffffff81237bc1>] kjournald2+0xcf/0x242
[18007.027187] [<ffffffff8108683a>] ? wake_up_bit+0x2a/0x2a
[18007.033733] [<ffffffff81237af2>] ? commit_timeout+0x10/0x10
[18007.040574] [<ffffffff81086384>] kthread+0x95/0x9d
[18007.046542] [<ffffffff81a61134>] kernel_thread_helper+0x4/0x10
[18007.053675] [<ffffffff81a591b4>] ? retint_restore_args+0x13/0x13
[18007.061003] [<ffffffff810862ef>] ? __init_kthread_worker+0x5b/0x5b
[18007.068521] [<ffffffff81a61130>] ? gs_change+0x13/0x13
[18007.074878] no locks held by jbd2/sda1-8/51294.
Sometimes I also catch dd/ext4lazyinit/flush all stalling in start_this_handle:
[17985.439567] dd D 0000000000000007 3616 61440 1 0x00000004
[17985.448088] ffff88080d71b9b8 0000000000000046 ffff88081ec80070 ffff88080d71a000
[17985.457545] 00000000001d2f00 00000000001d2f00 ffff88081ec80000 00000000001d2f00
[17985.467168] ffff88080d71bfd8 00000000001d2f00 ffff88080d71bfd8 00000000001d2f00
[17985.476647] Call Trace:
[17985.479843] [<ffffffff8103d4af>] ? native_sched_clock+0x29/0x70
[17985.487025] [<ffffffff81230b9d>] ? start_this_handle+0x357/0x4ed
[17985.494313] [<ffffffff8109660d>] ? local_clock+0x41/0x5a
[17985.500815] [<ffffffff810b0ddd>] ? lock_release_holdtime+0xa3/0xac
[17985.508287] [<ffffffff81230b9d>] ? start_this_handle+0x357/0x4ed
[17985.515575] [<ffffffff81a57904>] schedule+0x5a/0x5c
[17985.521588] [<ffffffff81230c39>] start_this_handle+0x3f3/0x4ed
[17985.528669] [<ffffffff81147820>] ? kmem_cache_free+0xfa/0x13a
[17985.545142] [<ffffffff8108683a>] ? wake_up_bit+0x2a/0x2a
[17985.551650] [<ffffffff81230f0e>] jbd2__journal_start+0xb0/0xf6
[17985.558732] [<ffffffff811f7ad7>] ? ext4_dirty_inode+0x1d/0x4c
[17985.565716] [<ffffffff81230f67>] jbd2_journal_start+0x13/0x15
[17985.572703] [<ffffffff8120e3e9>] ext4_journal_start_sb+0x13f/0x157
[17985.580172] [<ffffffff8109660d>] ? local_clock+0x41/0x5a
[17985.586680] [<ffffffff811f7ad7>] ext4_dirty_inode+0x1d/0x4c
[17985.593472] [<ffffffff81176827>] __mark_inode_dirty+0x2e/0x1cc
[17985.600552] [<ffffffff81168e84>] file_update_time+0xe4/0x106
[17985.607441] [<ffffffff811079f6>] __generic_file_aio_write+0x254/0x364
[17985.615202] [<ffffffff81a565da>] ? mutex_lock_nested+0x2e4/0x2f3
[17985.622488] [<ffffffff81107b50>] ? generic_file_aio_write+0x4a/0xc1
[17985.630057] [<ffffffff81107b6c>] generic_file_aio_write+0x66/0xc1
[17985.637442] [<ffffffff811ef72b>] ext4_file_write+0x1f9/0x251
[17985.644330] [<ffffffff8109660d>] ? local_clock+0x41/0x5a
[17985.650835] [<ffffffff8118809e>] ? fsnotify+0x222/0x27b
[17985.657238] [<ffffffff81153612>] do_sync_write+0xce/0x10b
[17985.663844] [<ffffffff8118809e>] ? fsnotify+0x222/0x27b
[17985.670243] [<ffffffff81187ef8>] ? fsnotify+0x7c/0x27b
[17985.676561] [<ffffffff81153dbe>] vfs_write+0xb8/0x157
[17985.682767] [<ffffffff81154075>] sys_write+0x4d/0x77
[17985.688878] [<ffffffff81a5fce9>] system_call_fastpath+0x16/0x1b
and jbd2 in
[17983.623657] jbd2/sda1-8 D 0000000000000000 5464 51294 2 0x00000000
[17983.632173] ffff88040b097c70 0000000000000046 ffff880823032310 ffff88040b096000
[17983.641640] 00000000001d2f00 00000000001d2f00 ffff8808230322a0 00000000001d2f00
[17983.651119] ffff88040b097fd8 00000000001d2f00 ffff88040b097fd8 00000000001d2f00
[17983.660603] Call Trace:
[17983.663808] [<ffffffff8103d4af>] ? native_sched_clock+0x29/0x70
[17983.670997] [<ffffffff81232aab>] ? jbd2_journal_commit_transaction+0x1d0/0x1281
[17983.680124] [<ffffffff8109660d>] ? local_clock+0x41/0x5a
[17983.686638] [<ffffffff810b0ddd>] ? lock_release_holdtime+0xa3/0xac
[17983.694108] [<ffffffff81232aab>] ? jbd2_journal_commit_transaction+0x1d0/0x1281
[17983.703243] [<ffffffff81a57904>] schedule+0x5a/0x5c
[17983.709262] [<ffffffff81232ab0>] jbd2_journal_commit_transaction+0x1d5/0x1281
[17983.718195] [<ffffffff8103d4af>] ? native_sched_clock+0x29/0x70
[17983.725392] [<ffffffff810738ce>] ? try_to_del_timer_sync+0xba/0xc8
[17983.732867] [<ffffffff8109660d>] ? local_clock+0x41/0x5a
[17983.739374] [<ffffffff8108683a>] ? wake_up_bit+0x2a/0x2a
[17983.745864] [<ffffffff810738ce>] ? try_to_del_timer_sync+0xba/0xc8
[17983.753343] [<ffffffff81073a6f>] ? del_timer_sync+0xbb/0xce
[17983.760137] [<ffffffff810739b4>] ? process_timeout+0x10/0x10
[17983.767041] [<ffffffff81237bc1>] kjournald2+0xcf/0x242
[17983.773361] [<ffffffff8108683a>] ? wake_up_bit+0x2a/0x2a
[17983.779863] [<ffffffff81237af2>] ? commit_timeout+0x10/0x10
[17983.786665] [<ffffffff81086384>] kthread+0x95/0x9d
[17983.792585] [<ffffffff81a61134>] kernel_thread_helper+0x4/0x10
[17983.799670] [<ffffffff81a591b4>] ? retint_restore_args+0x13/0x13
[17983.806948] [<ffffffff810862ef>] ? __init_kthread_worker+0x5b/0x5b
Here is the updated patch used in the new tests. It moves
congestion_wait() out of the page lock and make flush_inode_page() no
longer wait for memory allocation (looks unnecessary).
Thanks,
Fengguang
---
Subject: writeback: introduce the pageout work
Date: Thu Jul 29 14:41:19 CST 2010
This relays file pageout IOs to the flusher threads.
The ultimate target is to gracefully handle the LRU lists full of
dirty/writeback pages.
1) I/O efficiency
The flusher will piggy back the nearby ~10ms worth of dirty pages for I/O.
This takes advantage of the time/spacial locality in most workloads: the
nearby pages of one file are typically populated into the LRU at the same
time, hence will likely be close to each other in the LRU list. Writing
them in one shot helps clean more pages effectively for page reclaim.
2) OOM avoidance and scan rate control
Typically we do LRU scan w/o rate control and quickly get enough clean
pages for the LRU lists not full of dirty pages.
Or we can still get a number of freshly cleaned pages (moved to LRU tail
by end_page_writeback()) when the queued pageout I/O is completed within
tens of milli-seconds.
However if the LRU list is small and full of dirty pages, it can be
quickly fully scanned and go OOM before the flusher manages to clean
enough pages.
A simple yet reliable scheme is employed to avoid OOM and keep scan rate
in sync with the I/O rate:
if (PageReclaim(page))
congestion_wait(HZ/10);
PG_reclaim plays the key role. When dirty pages are encountered, we
queue I/O for it, set PG_reclaim and put it back to the LRU head.
So if PG_reclaim pages are encountered again, it means the dirty page
has not yet been cleaned by the flusher after a full zone scan. It
indicates we are scanning more fast than I/O and shall take a snap.
The runtime behavior on a fully dirtied small LRU list would be:
It will start with a quick scan of the list, queuing all pages for I/O.
Then the scan will be slowed down by the PG_reclaim pages *adaptively*
to match the I/O bandwidth.
3) writeback work coordinations
To avoid memory allocations at page reclaim, a mempool for struct
wb_writeback_work is created.
wakeup_flusher_threads() is removed because it can easily delay the
more oriented pageout works and even exhaust the mempool reservations.
It's also found to not I/O efficient by frequently submitting writeback
works with small ->nr_pages.
Background/periodic works will quit automatically, so as to clean the
pages under reclaim ASAP. However for now the sync work can still block
us for long time.
Jan Kara: limit the search scope. Note that the limited search and work
pool is not a big problem: 1000 IOs under flight are typically more than
enough to saturate the disk. And the overheads of searching in the work
list didn't even show up in the perf report.
4) test case
Run 2 dd tasks in a 100MB memcg (a very handy test case from Greg Thelen):
mkdir /cgroup/x
echo 100M > /cgroup/x/memory.limit_in_bytes
echo $$ > /cgroup/x/tasks
for i in `seq 2`
do
dd if=/dev/zero of=/fs/f$i bs=1k count=1M &
done
Before patch, the dd tasks are quickly OOM killed.
After patch, they run well with reasonably good performance and overheads:
1073741824 bytes (1.1 GB) copied, 22.2196 s, 48.3 MB/s
1073741824 bytes (1.1 GB) copied, 22.4675 s, 47.8 MB/s
iostat -kx 1
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 178.00 0.00 89568.00 1006.38 74.35 417.71 4.80 85.40
sda 0.00 2.00 0.00 191.00 0.00 94428.00 988.77 53.34 219.03 4.34 82.90
sda 0.00 20.00 0.00 196.00 0.00 97712.00 997.06 71.11 337.45 4.77 93.50
sda 0.00 5.00 0.00 175.00 0.00 84648.00 967.41 54.03 316.44 5.06 88.60
sda 0.00 0.00 0.00 186.00 0.00 92432.00 993.89 56.22 267.54 5.38 100.00
sda 0.00 1.00 0.00 183.00 0.00 90156.00 985.31 37.99 325.55 4.33 79.20
sda 0.00 0.00 0.00 175.00 0.00 88692.00 1013.62 48.70 218.43 4.69 82.10
sda 0.00 0.00 0.00 196.00 0.00 97528.00 995.18 43.38 236.87 5.10 100.00
sda 0.00 0.00 0.00 179.00 0.00 88648.00 990.48 45.83 285.43 5.59 100.00
sda 0.00 0.00 0.00 178.00 0.00 88500.00 994.38 28.28 158.89 4.99 88.80
sda 0.00 0.00 0.00 194.00 0.00 95852.00 988.16 32.58 167.39 5.15 100.00
sda 0.00 2.00 0.00 215.00 0.00 105996.00 986.01 41.72 201.43 4.65 100.00
sda 0.00 4.00 0.00 173.00 0.00 84332.00 974.94 50.48 260.23 5.76 99.60
sda 0.00 0.00 0.00 182.00 0.00 90312.00 992.44 36.83 212.07 5.49 100.00
sda 0.00 8.00 0.00 195.00 0.00 95940.50 984.01 50.18 221.06 5.13 100.00
sda 0.00 1.00 0.00 220.00 0.00 108852.00 989.56 40.99 202.68 4.55 100.00
sda 0.00 2.00 0.00 161.00 0.00 80384.00 998.56 37.19 268.49 6.21 100.00
sda 0.00 4.00 0.00 182.00 0.00 90830.00 998.13 50.58 239.77 5.49 100.00
sda 0.00 0.00 0.00 197.00 0.00 94877.00 963.22 36.68 196.79 5.08 100.00
avg-cpu: %user %nice %system %iowait %steal %idle
0.25 0.00 15.08 33.92 0.00 50.75
0.25 0.00 14.54 35.09 0.00 50.13
0.50 0.00 13.57 32.41 0.00 53.52
0.50 0.00 11.28 36.84 0.00 51.38
0.50 0.00 15.75 32.00 0.00 51.75
0.50 0.00 10.50 34.00 0.00 55.00
0.50 0.00 17.63 27.46 0.00 54.41
0.50 0.00 15.08 30.90 0.00 53.52
0.50 0.00 11.28 32.83 0.00 55.39
0.75 0.00 16.79 26.82 0.00 55.64
0.50 0.00 16.08 29.15 0.00 54.27
0.50 0.00 13.50 30.50 0.00 55.50
0.50 0.00 14.32 35.18 0.00 50.00
0.50 0.00 12.06 33.92 0.00 53.52
0.50 0.00 17.29 30.58 0.00 51.63
0.50 0.00 15.08 29.65 0.00 54.77
0.50 0.00 12.53 29.32 0.00 57.64
0.50 0.00 15.29 31.83 0.00 52.38
The global dd numbers for comparison:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 189.00 0.00 95752.00 1013.25 143.09 684.48 5.29 100.00
sda 0.00 0.00 0.00 208.00 0.00 105480.00 1014.23 143.06 733.29 4.81 100.00
sda 0.00 0.00 0.00 161.00 0.00 81924.00 1017.69 141.71 757.79 6.21 100.00
sda 0.00 0.00 0.00 217.00 0.00 109580.00 1009.95 143.09 749.55 4.61 100.10
sda 0.00 0.00 0.00 187.00 0.00 94728.00 1013.13 144.31 773.67 5.35 100.00
sda 0.00 0.00 0.00 189.00 0.00 95752.00 1013.25 144.14 742.00 5.29 100.00
sda 0.00 0.00 0.00 177.00 0.00 90032.00 1017.31 143.32 656.59 5.65 100.00
sda 0.00 0.00 0.00 215.00 0.00 108640.00 1010.60 142.90 817.54 4.65 100.00
sda 0.00 2.00 0.00 166.00 0.00 83858.00 1010.34 143.64 808.61 6.02 100.00
sda 0.00 0.00 0.00 186.00 0.00 92813.00 997.99 141.18 736.95 5.38 100.00
sda 0.00 0.00 0.00 206.00 0.00 104456.00 1014.14 146.27 729.33 4.85 100.00
sda 0.00 0.00 0.00 213.00 0.00 107024.00 1004.92 143.25 705.70 4.69 100.00
sda 0.00 0.00 0.00 188.00 0.00 95748.00 1018.60 141.82 764.78 5.32 100.00
avg-cpu: %user %nice %system %iowait %steal %idle
0.51 0.00 11.22 52.30 0.00 35.97
0.25 0.00 10.15 52.54 0.00 37.06
0.25 0.00 5.01 56.64 0.00 38.10
0.51 0.00 15.15 43.94 0.00 40.40
0.25 0.00 12.12 48.23 0.00 39.39
0.51 0.00 11.20 53.94 0.00 34.35
0.26 0.00 9.72 51.41 0.00 38.62
0.76 0.00 9.62 50.63 0.00 38.99
0.51 0.00 10.46 53.32 0.00 35.71
0.51 0.00 9.41 51.91 0.00 38.17
0.25 0.00 10.69 49.62 0.00 39.44
0.51 0.00 12.21 52.67 0.00 34.61
0.51 0.00 11.45 53.18 0.00 34.86
XXX: commit NFS unstable pages via write_inode()
XXX: the added congestion_wait() may be undesirable in some situations
CC: Jan Kara <jack@suse.cz>
CC: Mel Gorman <mgorman@suse.de>
Acked-by: Rik van Riel <riel@redhat.com>
CC: Greg Thelen <gthelen@google.com>
CC: Minchan Kim <minchan.kim@gmail.com>
Signed-off-by: Wu Fengguang <fengguang.wu@intel.com>
---
fs/fs-writeback.c | 169 ++++++++++++++++++++++++++++-
include/linux/writeback.h | 4
include/trace/events/writeback.h | 12 +-
mm/vmscan.c | 35 ++++--
4 files changed, 202 insertions(+), 18 deletions(-)
- move congestion_wait() out of the page lock: it's blocking btrfs lock_delalloc_pages()
--- linux.orig/mm/vmscan.c 2012-02-12 21:27:28.000000000 +0800
+++ linux/mm/vmscan.c 2012-02-13 12:14:20.000000000 +0800
@@ -767,7 +767,8 @@ static unsigned long shrink_page_list(st
struct scan_control *sc,
int priority,
unsigned long *ret_nr_dirty,
- unsigned long *ret_nr_writeback)
+ unsigned long *ret_nr_writeback,
+ unsigned long *ret_nr_pgreclaim)
{
LIST_HEAD(ret_pages);
LIST_HEAD(free_pages);
@@ -776,6 +777,7 @@ static unsigned long shrink_page_list(st
unsigned long nr_congested = 0;
unsigned long nr_reclaimed = 0;
unsigned long nr_writeback = 0;
+ unsigned long nr_pgreclaim = 0;
cond_resched();
@@ -813,6 +815,10 @@ static unsigned long shrink_page_list(st
if (PageWriteback(page)) {
nr_writeback++;
+ if (PageReclaim(page))
+ nr_pgreclaim++;
+ else
+ SetPageReclaim(page);
/*
* Synchronous reclaim cannot queue pages for
* writeback due to the possibility of stack overflow
@@ -874,12 +880,15 @@ static unsigned long shrink_page_list(st
nr_dirty++;
/*
- * Only kswapd can writeback filesystem pages to
- * avoid risk of stack overflow but do not writeback
- * unless under significant pressure.
+ * run into the visited page again: we are scanning
+ * faster than the flusher can writeout dirty pages
*/
- if (page_is_file_cache(page) &&
- (!current_is_kswapd() || priority >= DEF_PRIORITY - 2)) {
+ if (page_is_file_cache(page) && PageReclaim(page)) {
+ nr_pgreclaim++;
+ goto keep_locked;
+ }
+ if (page_is_file_cache(page) && mapping &&
+ flush_inode_page(mapping, page, false) >= 0) {
/*
* Immediately reclaim when written back.
* Similar in principal to deactivate_page()
@@ -1028,6 +1037,7 @@ keep_lumpy:
count_vm_events(PGACTIVATE, pgactivate);
*ret_nr_dirty += nr_dirty;
*ret_nr_writeback += nr_writeback;
+ *ret_nr_pgreclaim += nr_pgreclaim;
return nr_reclaimed;
}
@@ -1087,8 +1097,10 @@ int __isolate_lru_page(struct page *page
*/
if (mode & (ISOLATE_CLEAN|ISOLATE_ASYNC_MIGRATE)) {
/* All the caller can do on PageWriteback is block */
- if (PageWriteback(page))
+ if (PageWriteback(page)) {
+ SetPageReclaim(page);
return ret;
+ }
if (PageDirty(page)) {
struct address_space *mapping;
@@ -1509,6 +1521,7 @@ shrink_inactive_list(unsigned long nr_to
unsigned long nr_file;
unsigned long nr_dirty = 0;
unsigned long nr_writeback = 0;
+ unsigned long nr_pgreclaim = 0;
isolate_mode_t reclaim_mode = ISOLATE_INACTIVE;
struct zone *zone = mz->zone;
@@ -1559,13 +1572,13 @@ shrink_inactive_list(unsigned long nr_to
spin_unlock_irq(&zone->lru_lock);
nr_reclaimed = shrink_page_list(&page_list, mz, sc, priority,
- &nr_dirty, &nr_writeback);
+ &nr_dirty, &nr_writeback, &nr_pgreclaim);
/* Check if we should syncronously wait for writeback */
if (should_reclaim_stall(nr_taken, nr_reclaimed, priority, sc)) {
set_reclaim_mode(priority, sc, true);
nr_reclaimed += shrink_page_list(&page_list, mz, sc,
- priority, &nr_dirty, &nr_writeback);
+ priority, &nr_dirty, &nr_writeback, &nr_pgreclaim);
}
spin_lock_irq(&zone->lru_lock);
@@ -1608,6 +1621,8 @@ shrink_inactive_list(unsigned long nr_to
*/
if (nr_writeback && nr_writeback >= (nr_taken >> (DEF_PRIORITY-priority)))
wait_iff_congested(zone, BLK_RW_ASYNC, HZ/10);
+ if (nr_pgreclaim)
+ congestion_wait(BLK_RW_ASYNC, HZ/10);
trace_mm_vmscan_lru_shrink_inactive(zone->zone_pgdat->node_id,
zone_idx(zone),
@@ -2382,8 +2397,6 @@ static unsigned long do_try_to_free_page
*/
writeback_threshold = sc->nr_to_reclaim + sc->nr_to_reclaim / 2;
if (total_scanned > writeback_threshold) {
- wakeup_flusher_threads(laptop_mode ? 0 : total_scanned,
- WB_REASON_TRY_TO_FREE_PAGES);
sc->may_writepage = 1;
}
--- linux.orig/fs/fs-writeback.c 2012-02-12 21:27:28.000000000 +0800
+++ linux/fs/fs-writeback.c 2012-02-13 12:15:50.000000000 +0800
@@ -41,6 +41,8 @@ struct wb_writeback_work {
long nr_pages;
struct super_block *sb;
unsigned long *older_than_this;
+ struct inode *inode;
+ pgoff_t offset;
enum writeback_sync_modes sync_mode;
unsigned int tagged_writepages:1;
unsigned int for_kupdate:1;
@@ -65,6 +67,27 @@ struct wb_writeback_work {
*/
int nr_pdflush_threads;
+static mempool_t *wb_work_mempool;
+
+static void *wb_work_alloc(gfp_t gfp_mask, void *pool_data)
+{
+ /*
+ * bdi_flush_inode_range() may be called on page reclaim
+ */
+ if (current->flags & PF_MEMALLOC)
+ return NULL;
+
+ return kmalloc(sizeof(struct wb_writeback_work), gfp_mask);
+}
+
+static __init int wb_work_init(void)
+{
+ wb_work_mempool = mempool_create(1024,
+ wb_work_alloc, mempool_kfree, NULL);
+ return wb_work_mempool ? 0 : -ENOMEM;
+}
+fs_initcall(wb_work_init);
+
/**
* writeback_in_progress - determine whether there is writeback in progress
* @bdi: the device's backing_dev_info structure.
@@ -129,7 +152,7 @@ __bdi_start_writeback(struct backing_dev
* This is WB_SYNC_NONE writeback, so if allocation fails just
* wakeup the thread for old dirty data writeback
*/
- work = kzalloc(sizeof(*work), GFP_ATOMIC);
+ work = mempool_alloc(wb_work_mempool, GFP_NOWAIT);
if (!work) {
if (bdi->wb.task) {
trace_writeback_nowork(bdi);
@@ -138,6 +161,7 @@ __bdi_start_writeback(struct backing_dev
return;
}
+ memset(work, 0, sizeof(*work));
work->sync_mode = WB_SYNC_NONE;
work->nr_pages = nr_pages;
work->range_cyclic = range_cyclic;
@@ -186,6 +210,125 @@ void bdi_start_background_writeback(stru
spin_unlock_bh(&bdi->wb_lock);
}
+static bool extend_writeback_range(struct wb_writeback_work *work,
+ pgoff_t offset,
+ unsigned long write_around_pages)
+{
+ pgoff_t end = work->offset + work->nr_pages;
+
+ if (offset >= work->offset && offset < end)
+ return true;
+
+ /*
+ * for sequential workloads with good locality, include up to 8 times
+ * more data in one chunk
+ */
+ if (work->nr_pages >= 8 * write_around_pages)
+ return false;
+
+ /* the unsigned comparison helps eliminate one compare */
+ if (work->offset - offset < write_around_pages) {
+ work->nr_pages += write_around_pages;
+ work->offset -= write_around_pages;
+ return true;
+ }
+
+ if (offset - end < write_around_pages) {
+ work->nr_pages += write_around_pages;
+ return true;
+ }
+
+ return false;
+}
+
+/*
+ * schedule writeback on a range of inode pages.
+ */
+static struct wb_writeback_work *
+bdi_flush_inode_range(struct backing_dev_info *bdi,
+ struct inode *inode,
+ pgoff_t offset,
+ pgoff_t len,
+ bool wait)
+{
+ struct wb_writeback_work *work;
+
+ if (!igrab(inode))
+ return ERR_PTR(-ENOENT);
+
+ work = mempool_alloc(wb_work_mempool, wait ? GFP_NOIO : GFP_NOWAIT);
+ if (!work) {
+ trace_printk("wb_work_mempool alloc fail\n");
+ return ERR_PTR(-ENOMEM);
+ }
+
+ memset(work, 0, sizeof(*work));
+ work->sync_mode = WB_SYNC_NONE;
+ work->inode = inode;
+ work->offset = offset;
+ work->nr_pages = len;
+ work->reason = WB_REASON_PAGEOUT;
+
+ bdi_queue_work(bdi, work);
+
+ return work;
+}
+
+/*
+ * Called by page reclaim code to flush the dirty page ASAP. Do write-around to
+ * improve IO throughput. The nearby pages will have good chance to reside in
+ * the same LRU list that vmscan is working on, and even close to each other
+ * inside the LRU list in the common case of sequential read/write.
+ *
+ * ret > 0: success, found/reused a previous writeback work
+ * ret = 0: success, allocated/queued a new writeback work
+ * ret < 0: failed
+ */
+long flush_inode_page(struct address_space *mapping,
+ struct page *page,
+ bool wait)
+{
+ struct backing_dev_info *bdi = mapping->backing_dev_info;
+ struct inode *inode = mapping->host;
+ struct wb_writeback_work *work;
+ unsigned long write_around_pages;
+ pgoff_t offset = page->index;
+ int i;
+ long ret = 0;
+
+ if (unlikely(!inode))
+ return -ENOENT;
+
+ /*
+ * piggy back 8-15ms worth of data
+ */
+ write_around_pages = bdi->avg_write_bandwidth + MIN_WRITEBACK_PAGES;
+ write_around_pages = rounddown_pow_of_two(write_around_pages) >> 6;
+
+ i = 1;
+ spin_lock_bh(&bdi->wb_lock);
+ list_for_each_entry_reverse(work, &bdi->work_list, list) {
+ if (work->inode != inode)
+ continue;
+ if (extend_writeback_range(work, offset, write_around_pages)) {
+ ret = i;
+ break;
+ }
+ if (i++ > 100) /* limit search depth */
+ break;
+ }
+ spin_unlock_bh(&bdi->wb_lock);
+
+ if (!ret) {
+ offset = round_down(offset, write_around_pages);
+ work = bdi_flush_inode_range(bdi, inode,
+ offset, write_around_pages, wait);
+ if (IS_ERR(work))
+ ret = PTR_ERR(work);
+ }
+ return ret;
+}
+
/*
* Remove the inode from the writeback list it is on.
*/
@@ -833,6 +976,23 @@ static unsigned long get_nr_dirty_pages(
get_nr_dirty_inodes();
}
+static long wb_flush_inode(struct bdi_writeback *wb,
+ struct wb_writeback_work *work)
+{
+ struct writeback_control wbc = {
+ .sync_mode = WB_SYNC_NONE,
+ .nr_to_write = LONG_MAX,
+ .range_start = work->offset << PAGE_CACHE_SHIFT,
+ .range_end = (work->offset + work->nr_pages - 1)
+ << PAGE_CACHE_SHIFT,
+ };
+
+ do_writepages(work->inode->i_mapping, &wbc);
+ iput(work->inode);
+
+ return LONG_MAX - wbc.nr_to_write;
+}
+
static long wb_check_background_flush(struct bdi_writeback *wb)
{
if (over_bground_thresh(wb->bdi)) {
@@ -905,7 +1065,10 @@ long wb_do_writeback(struct bdi_writebac
trace_writeback_exec(bdi, work);
- wrote += wb_writeback(wb, work);
+ if (work->inode)
+ wrote += wb_flush_inode(wb, work);
+ else
+ wrote += wb_writeback(wb, work);
/*
* Notify the caller of completion if this is a synchronous
@@ -914,7 +1077,7 @@ long wb_do_writeback(struct bdi_writebac
if (work->done)
complete(work->done);
else
- kfree(work);
+ mempool_free(work, wb_work_mempool);
}
/*
--- linux.orig/include/trace/events/writeback.h 2012-02-12 21:27:33.000000000 +0800
+++ linux/include/trace/events/writeback.h 2012-02-12 21:27:34.000000000 +0800
@@ -23,7 +23,7 @@
#define WB_WORK_REASON \
{WB_REASON_BACKGROUND, "background"}, \
- {WB_REASON_TRY_TO_FREE_PAGES, "try_to_free_pages"}, \
+ {WB_REASON_PAGEOUT, "pageout"}, \
{WB_REASON_SYNC, "sync"}, \
{WB_REASON_PERIODIC, "periodic"}, \
{WB_REASON_LAPTOP_TIMER, "laptop_timer"}, \
@@ -45,6 +45,8 @@ DECLARE_EVENT_CLASS(writeback_work_class
__field(int, range_cyclic)
__field(int, for_background)
__field(int, reason)
+ __field(unsigned long, ino)
+ __field(unsigned long, offset)
),
TP_fast_assign(
strncpy(__entry->name, dev_name(bdi->dev), 32);
@@ -55,9 +57,11 @@ DECLARE_EVENT_CLASS(writeback_work_class
__entry->range_cyclic = work->range_cyclic;
__entry->for_background = work->for_background;
__entry->reason = work->reason;
+ __entry->ino = work->inode ? work->inode->i_ino : 0;
+ __entry->offset = work->offset;
),
TP_printk("bdi %s: sb_dev %d:%d nr_pages=%ld sync_mode=%d "
- "kupdate=%d range_cyclic=%d background=%d reason=%s",
+ "kupdate=%d range_cyclic=%d background=%d reason=%s ino=%lu offset=%lu",
__entry->name,
MAJOR(__entry->sb_dev), MINOR(__entry->sb_dev),
__entry->nr_pages,
@@ -65,7 +69,9 @@ DECLARE_EVENT_CLASS(writeback_work_class
__entry->for_kupdate,
__entry->range_cyclic,
__entry->for_background,
- __print_symbolic(__entry->reason, WB_WORK_REASON)
+ __print_symbolic(__entry->reason, WB_WORK_REASON),
+ __entry->ino,
+ __entry->offset
)
);
#define DEFINE_WRITEBACK_WORK_EVENT(name) \
--- linux.orig/include/linux/writeback.h 2012-02-12 21:27:28.000000000 +0800
+++ linux/include/linux/writeback.h 2012-02-12 21:27:34.000000000 +0800
@@ -40,7 +40,7 @@ enum writeback_sync_modes {
*/
enum wb_reason {
WB_REASON_BACKGROUND,
- WB_REASON_TRY_TO_FREE_PAGES,
+ WB_REASON_PAGEOUT,
WB_REASON_SYNC,
WB_REASON_PERIODIC,
WB_REASON_LAPTOP_TIMER,
@@ -94,6 +94,8 @@ long writeback_inodes_wb(struct bdi_writ
enum wb_reason reason);
long wb_do_writeback(struct bdi_writeback *wb, int force_wait);
void wakeup_flusher_threads(long nr_pages, enum wb_reason reason);
+long flush_inode_page(struct address_space *mapping, struct page *page,
+ bool wait);
/* writeback.h requires fs.h; it, too, is not included from here. */
static inline void wait_on_inode(struct inode *inode)
[-- Attachment #2: global_dirtied_written.png --]
[-- Type: image/png, Size: 37386 bytes --]
[-- Attachment #3: iostat --]
[-- Type: text/plain, Size: 6586 bytes --]
Linux 3.3.0-rc3-flush-page+ (snb) 02/14/2012 _x86_64_ (32 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
0.03 0.00 0.05 0.38 0.00 99.54
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.20 33.00 0.05 2.91 0.16 1025.66 694.73 1.50 508.37 6.10 1.80
avg-cpu: %user %nice %system %iowait %steal %idle
0.90 0.00 5.72 0.00 0.00 93.38
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.00 1.00 0.00 8.00 0.00 16.00 0.00 1.00 1.00 0.10
avg-cpu: %user %nice %system %iowait %steal %idle
0.90 0.00 5.82 0.00 0.00 93.28
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
avg-cpu: %user %nice %system %iowait %steal %idle
0.90 0.00 5.81 0.00 0.00 93.29
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
avg-cpu: %user %nice %system %iowait %steal %idle
0.83 0.00 5.84 0.00 0.00 93.33
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
avg-cpu: %user %nice %system %iowait %steal %idle
0.83 0.00 5.94 0.00 0.00 93.23
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 370.00 0.00 71.00 0.00 12480.00 351.55 11.17 71.76 3.56 25.30
avg-cpu: %user %nice %system %iowait %steal %idle
0.80 0.00 5.84 0.00 0.00 93.35
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 65.00 0.00 51.00 0.00 13104.00 513.88 2.85 170.22 4.00 20.40
avg-cpu: %user %nice %system %iowait %steal %idle
0.87 0.00 5.93 0.00 0.00 93.20
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 379.00 0.00 134.00 0.00 30004.00 447.82 15.43 116.99 4.11 55.10
avg-cpu: %user %nice %system %iowait %steal %idle
0.90 0.00 5.91 0.00 0.00 93.19
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 133.00 0.00 114.00 0.00 12792.00 224.42 3.20 28.12 2.30 26.20
avg-cpu: %user %nice %system %iowait %steal %idle
0.83 0.00 5.96 0.00 0.00 93.21
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 367.00 0.00 114.00 0.00 24960.00 437.89 12.55 110.12 4.35 49.60
avg-cpu: %user %nice %system %iowait %steal %idle
0.80 0.00 6.03 0.03 0.00 93.14
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 205.00 0.00 163.00 0.00 16900.00 207.36 4.09 21.63 1.88 30.70
avg-cpu: %user %nice %system %iowait %steal %idle
0.87 0.00 5.98 0.00 0.00 93.16
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 219.00 0.00 182.00 0.00 19292.00 212.00 4.42 21.41 1.97 35.80
avg-cpu: %user %nice %system %iowait %steal %idle
0.93 0.00 5.87 0.00 0.00 93.20
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 201.00 0.00 135.00 0.00 21216.00 314.31 6.39 55.44 2.97 40.10
avg-cpu: %user %nice %system %iowait %steal %idle
0.93 0.00 5.99 0.00 0.00 93.08
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 329.00 0.00 166.00 0.00 24336.00 293.20 10.42 58.39 3.19 53.00
avg-cpu: %user %nice %system %iowait %steal %idle
0.80 0.00 5.96 0.00 0.00 93.24
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 258.00 0.00 181.00 0.00 16848.00 186.17 4.01 17.64 1.36 24.70
avg-cpu: %user %nice %system %iowait %steal %idle
0.90 0.00 5.92 0.00 0.00 93.18
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 166.00 0.00 88.00 0.00 20488.00 465.64 6.09 86.72 4.68 41.20
avg-cpu: %user %nice %system %iowait %steal %idle
0.86 0.00 6.05 0.00 0.00 93.08
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 566.00 0.00 194.00 0.00 34528.00 355.96 20.50 78.32 3.16 61.30
avg-cpu: %user %nice %system %iowait %steal %idle
0.86 0.00 5.79 0.00 0.00 93.34
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 12.00 0.00 5616.00 936.00 0.77 506.67 9.58 11.50
avg-cpu: %user %nice %system %iowait %steal %idle
0.90 0.00 5.93 0.00 0.00 93.18
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 259.00 0.00 173.00 0.00 22464.00 259.70 4.79 27.67 2.09 36.10
avg-cpu: %user %nice %system %iowait %steal %idle
0.74 0.00 6.16 0.00 0.00 93.10
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 190.00 0.00 163.00 0.00 18304.00 224.59 3.67 22.50 1.78 29.00
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: reclaim the LRU lists full of dirty/writeback pages
2012-02-11 12:44 ` reclaim the LRU lists full of dirty/writeback pages Wu Fengguang
2012-02-11 14:55 ` Rik van Riel
@ 2012-02-14 10:19 ` Mel Gorman
2012-02-14 13:18 ` Wu Fengguang
1 sibling, 1 reply; 33+ messages in thread
From: Mel Gorman @ 2012-02-14 10:19 UTC (permalink / raw)
To: Wu Fengguang
Cc: Greg Thelen, Jan Kara, bsingharora, Hugh Dickins, Michal Hocko,
linux-mm, Ying Han, hannes, KAMEZAWA Hiroyuki, Rik van Riel,
Minchan Kim
On Sat, Feb 11, 2012 at 08:44:45PM +0800, Wu Fengguang wrote:
> <SNIP>
> --- linux.orig/mm/vmscan.c 2012-02-03 21:42:21.000000000 +0800
> +++ linux/mm/vmscan.c 2012-02-11 17:28:54.000000000 +0800
> @@ -813,6 +813,8 @@ static unsigned long shrink_page_list(st
>
> if (PageWriteback(page)) {
> nr_writeback++;
> + if (PageReclaim(page))
> + congestion_wait(BLK_RW_ASYNC, HZ/10);
> /*
> * Synchronous reclaim cannot queue pages for
> * writeback due to the possibility of stack overflow
I didn't look closely at the rest of the patch, I'm just focusing on the
congestion_wait part. You called this out yourself but this is in fact
really really bad. If this is in place and a user copies a large amount of
data to slow storage like a USB stick, the system will stall severely. A
parallel streaming reader will certainly have major issues as it will enter
page reclaim, find a bunch of dirty USB-backed pages at the end of the LRU
(20% of memory potentially) and stall for HZ/10 on each one of them. How
badly each process is affected will vary.
For the OOM problem, a more reasonable stopgap might be to identify when
a process is scanning a memcg at high priority and encountered all
PageReclaim with no forward progress and to congestion_wait() if that
situation occurs. A preferable way would be to wait until the flusher
wakes up a waiter on PageReclaim pages to be written out because we want
to keep moving way from congestion_wait() if at all possible.
Another possibility would be to relook at LRU_IMMEDIATE but right now it
requires a page flag and I haven't devised a way around that. Besides,
it would only address the problem of PageREclaim pages being encountered,
it would not handle the case where a memcg was filled with PageReclaim pages.
--
Mel Gorman
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: reclaim the LRU lists full of dirty/writeback pages
2012-02-14 10:19 ` Mel Gorman
@ 2012-02-14 13:18 ` Wu Fengguang
2012-02-14 13:35 ` Wu Fengguang
` (2 more replies)
0 siblings, 3 replies; 33+ messages in thread
From: Wu Fengguang @ 2012-02-14 13:18 UTC (permalink / raw)
To: Mel Gorman
Cc: Greg Thelen, Jan Kara, bsingharora, Hugh Dickins, Michal Hocko,
linux-mm, Ying Han, hannes, KAMEZAWA Hiroyuki, Rik van Riel,
Minchan Kim
On Tue, Feb 14, 2012 at 10:19:31AM +0000, Mel Gorman wrote:
> On Sat, Feb 11, 2012 at 08:44:45PM +0800, Wu Fengguang wrote:
> > <SNIP>
> > --- linux.orig/mm/vmscan.c 2012-02-03 21:42:21.000000000 +0800
> > +++ linux/mm/vmscan.c 2012-02-11 17:28:54.000000000 +0800
> > @@ -813,6 +813,8 @@ static unsigned long shrink_page_list(st
> >
> > if (PageWriteback(page)) {
> > nr_writeback++;
> > + if (PageReclaim(page))
> > + congestion_wait(BLK_RW_ASYNC, HZ/10);
> > /*
> > * Synchronous reclaim cannot queue pages for
> > * writeback due to the possibility of stack overflow
>
> I didn't look closely at the rest of the patch, I'm just focusing on the
> congestion_wait part. You called this out yourself but this is in fact
> really really bad. If this is in place and a user copies a large amount of
> data to slow storage like a USB stick, the system will stall severely. A
> parallel streaming reader will certainly have major issues as it will enter
> page reclaim, find a bunch of dirty USB-backed pages at the end of the LRU
> (20% of memory potentially) and stall for HZ/10 on each one of them. How
> badly each process is affected will vary.
Cannot agree any more the principle...me just want to demonstrate the
idea first :-)
> For the OOM problem, a more reasonable stopgap might be to identify when
> a process is scanning a memcg at high priority and encountered all
> PageReclaim with no forward progress and to congestion_wait() if that
> situation occurs. A preferable way would be to wait until the flusher
> wakes up a waiter on PageReclaim pages to be written out because we want
> to keep moving way from congestion_wait() if at all possible.
Good points! Below is the more serious page reclaim changes.
The dirty/writeback pages may often come close to each other in the
LRU list, so the local test during a 32-page scan may still trigger
reclaim waits unnecessarily. Some global information on the percent
of dirty/writeback pages in the LRU list may help. Anyway the added
tests should still be much better than no protection.
A global wait queue and reclaim_wait() is introduced. The waiters will
be wakeup when pages are rotated by end_page_writeback() or lru drain.
I have to say its effectiveness depends on the filesystem... ext4
and btrfs do fluent IO completions, so reclaim_wait() works pretty
well:
dd-14560 [017] .... 1360.894605: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=10000
dd-14560 [017] .... 1360.904456: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=8000
dd-14560 [017] .... 1360.908293: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
dd-14560 [017] .... 1360.923960: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=15000
dd-14560 [017] .... 1360.927810: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
dd-14560 [017] .... 1360.931656: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
dd-14560 [017] .... 1360.943503: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=10000
dd-14560 [017] .... 1360.953289: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=7000
dd-14560 [017] .... 1360.957177: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
dd-14560 [017] .... 1360.972949: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=15000
However XFS does IO completions in very large batches (there may be
only several big IO completions in one second). So reclaim_wait()
mostly end up waiting to the full HZ/10 timeout:
dd-4177 [008] .... 866.367661: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
dd-4177 [010] .... 866.567583: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
dd-4177 [012] .... 866.767458: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
dd-4177 [013] .... 866.867419: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
dd-4177 [008] .... 867.167266: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
dd-4177 [010] .... 867.367168: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
dd-4177 [012] .... 867.818950: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
dd-4177 [013] .... 867.918905: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
dd-4177 [013] .... 867.971657: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=52000
dd-4177 [013] .... 867.971812: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=0
dd-4177 [008] .... 868.355700: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
dd-4177 [010] .... 868.700515: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> Another possibility would be to relook at LRU_IMMEDIATE but right now it
> requires a page flag and I haven't devised a way around that. Besides,
> it would only address the problem of PageREclaim pages being encountered,
> it would not handle the case where a memcg was filled with PageReclaim pages.
I also considered things like LRU_IMMEDIATE, however got no clear idea yet.
Since the simple "wait on PG_reclaim" approach appears to work for this
memcg dd case, it effectively disables me to think any further ;-)
For the single dd inside memcg, ext4 is now working pretty well, with
least CPU overheads:
(running from another test box, so not directly comparable with old tests)
avg-cpu: %user %nice %system %iowait %steal %idle
0.03 0.00 0.85 5.35 0.00 93.77
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 112.00 0.00 57348.00 1024.07 81.66 1045.21 8.93 100.00
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 0.69 4.07 0.00 95.24
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 142.00 0.00 112.00 0.00 56832.00 1014.86 127.94 790.04 8.93 100.00
And xfs a bit less fluent:
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 3.79 2.54 0.00 93.68
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 108.00 0.00 54644.00 1011.93 48.13 1044.83 8.44 91.20
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 3.38 3.88 0.00 92.74
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 105.00 0.00 53156.00 1012.50 128.50 451.90 9.25 97.10
btrfs also looks good:
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 8.05 3.85 0.00 88.10
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 108.00 0.00 53248.00 986.07 88.11 643.99 9.26 100.00
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 4.04 2.51 0.00 93.45
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 112.00 0.00 57344.00 1024.00 91.58 998.41 8.93 100.00
Thanks,
Fengguang
---
--- linux.orig/include/linux/backing-dev.h 2012-02-14 19:43:06.000000000 +0800
+++ linux/include/linux/backing-dev.h 2012-02-14 19:49:26.000000000 +0800
@@ -304,6 +304,8 @@ void clear_bdi_congested(struct backing_
void set_bdi_congested(struct backing_dev_info *bdi, int sync);
long congestion_wait(int sync, long timeout);
long wait_iff_congested(struct zone *zone, int sync, long timeout);
+long reclaim_wait(long timeout);
+void reclaim_rotated(void);
static inline bool bdi_cap_writeback_dirty(struct backing_dev_info *bdi)
{
--- linux.orig/mm/backing-dev.c 2012-02-14 19:26:15.000000000 +0800
+++ linux/mm/backing-dev.c 2012-02-14 20:09:45.000000000 +0800
@@ -873,3 +873,38 @@ out:
return ret;
}
EXPORT_SYMBOL(wait_iff_congested);
+
+static DECLARE_WAIT_QUEUE_HEAD(reclaim_wqh);
+
+/**
+ * reclaim_wait - wait for some pages being rotated to the LRU tail
+ * @timeout: timeout in jiffies
+ *
+ * Wait until @timeout, or when some (typically PG_reclaim under writeback)
+ * pages rotated to the LRU so that page reclaim can make progress.
+ */
+long reclaim_wait(long timeout)
+{
+ long ret;
+ unsigned long start = jiffies;
+ DEFINE_WAIT(wait);
+
+ prepare_to_wait(&reclaim_wqh, &wait, TASK_KILLABLE);
+ ret = io_schedule_timeout(timeout);
+ finish_wait(&reclaim_wqh, &wait);
+
+ trace_writeback_reclaim_wait(jiffies_to_usecs(timeout),
+ jiffies_to_usecs(jiffies - start));
+
+ return ret;
+}
+EXPORT_SYMBOL(reclaim_wait);
+
+void reclaim_rotated()
+{
+ wait_queue_head_t *wqh = &reclaim_wqh;
+
+ if (waitqueue_active(wqh))
+ wake_up(wqh);
+}
+
--- linux.orig/mm/swap.c 2012-02-14 19:40:10.000000000 +0800
+++ linux/mm/swap.c 2012-02-14 19:45:13.000000000 +0800
@@ -253,6 +253,7 @@ static void pagevec_move_tail(struct pag
pagevec_lru_move_fn(pvec, pagevec_move_tail_fn, &pgmoved);
__count_vm_events(PGROTATED, pgmoved);
+ reclaim_rotated();
}
/*
--- linux.orig/mm/vmscan.c 2012-02-14 17:53:27.000000000 +0800
+++ linux/mm/vmscan.c 2012-02-14 19:44:11.000000000 +0800
@@ -767,7 +767,8 @@ static unsigned long shrink_page_list(st
struct scan_control *sc,
int priority,
unsigned long *ret_nr_dirty,
- unsigned long *ret_nr_writeback)
+ unsigned long *ret_nr_writeback,
+ unsigned long *ret_nr_pgreclaim)
{
LIST_HEAD(ret_pages);
LIST_HEAD(free_pages);
@@ -776,6 +777,7 @@ static unsigned long shrink_page_list(st
unsigned long nr_congested = 0;
unsigned long nr_reclaimed = 0;
unsigned long nr_writeback = 0;
+ unsigned long nr_pgreclaim = 0;
cond_resched();
@@ -813,6 +815,10 @@ static unsigned long shrink_page_list(st
if (PageWriteback(page)) {
nr_writeback++;
+ if (PageReclaim(page))
+ nr_pgreclaim++;
+ else
+ SetPageReclaim(page);
/*
* Synchronous reclaim cannot queue pages for
* writeback due to the possibility of stack overflow
@@ -874,12 +880,15 @@ static unsigned long shrink_page_list(st
nr_dirty++;
/*
- * Only kswapd can writeback filesystem pages to
- * avoid risk of stack overflow but do not writeback
- * unless under significant pressure.
+ * run into the visited page again: we are scanning
+ * faster than the flusher can writeout dirty pages
*/
- if (page_is_file_cache(page) &&
- (!current_is_kswapd() || priority >= DEF_PRIORITY - 2)) {
+ if (page_is_file_cache(page) && PageReclaim(page)) {
+ nr_pgreclaim++;
+ goto keep_locked;
+ }
+ if (page_is_file_cache(page) && mapping &&
+ flush_inode_page(mapping, page, false) >= 0) {
/*
* Immediately reclaim when written back.
* Similar in principal to deactivate_page()
@@ -1028,6 +1037,7 @@ keep_lumpy:
count_vm_events(PGACTIVATE, pgactivate);
*ret_nr_dirty += nr_dirty;
*ret_nr_writeback += nr_writeback;
+ *ret_nr_pgreclaim += nr_pgreclaim;
return nr_reclaimed;
}
@@ -1087,8 +1097,10 @@ int __isolate_lru_page(struct page *page
*/
if (mode & (ISOLATE_CLEAN|ISOLATE_ASYNC_MIGRATE)) {
/* All the caller can do on PageWriteback is block */
- if (PageWriteback(page))
+ if (PageWriteback(page)) {
+ SetPageReclaim(page);
return ret;
+ }
if (PageDirty(page)) {
struct address_space *mapping;
@@ -1509,6 +1521,7 @@ shrink_inactive_list(unsigned long nr_to
unsigned long nr_file;
unsigned long nr_dirty = 0;
unsigned long nr_writeback = 0;
+ unsigned long nr_pgreclaim = 0;
isolate_mode_t reclaim_mode = ISOLATE_INACTIVE;
struct zone *zone = mz->zone;
@@ -1559,13 +1572,13 @@ shrink_inactive_list(unsigned long nr_to
spin_unlock_irq(&zone->lru_lock);
nr_reclaimed = shrink_page_list(&page_list, mz, sc, priority,
- &nr_dirty, &nr_writeback);
+ &nr_dirty, &nr_writeback, &nr_pgreclaim);
/* Check if we should syncronously wait for writeback */
if (should_reclaim_stall(nr_taken, nr_reclaimed, priority, sc)) {
set_reclaim_mode(priority, sc, true);
nr_reclaimed += shrink_page_list(&page_list, mz, sc,
- priority, &nr_dirty, &nr_writeback);
+ priority, &nr_dirty, &nr_writeback, &nr_pgreclaim);
}
spin_lock_irq(&zone->lru_lock);
@@ -1608,6 +1621,8 @@ shrink_inactive_list(unsigned long nr_to
*/
if (nr_writeback && nr_writeback >= (nr_taken >> (DEF_PRIORITY-priority)))
wait_iff_congested(zone, BLK_RW_ASYNC, HZ/10);
+ if (nr_pgreclaim && nr_pgreclaim >= (nr_taken >> (DEF_PRIORITY-priority)))
+ reclaim_wait(HZ/10);
trace_mm_vmscan_lru_shrink_inactive(zone->zone_pgdat->node_id,
zone_idx(zone),
@@ -2382,8 +2397,6 @@ static unsigned long do_try_to_free_page
*/
writeback_threshold = sc->nr_to_reclaim + sc->nr_to_reclaim / 2;
if (total_scanned > writeback_threshold) {
- wakeup_flusher_threads(laptop_mode ? 0 : total_scanned,
- WB_REASON_TRY_TO_FREE_PAGES);
sc->may_writepage = 1;
}
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: reclaim the LRU lists full of dirty/writeback pages
2012-02-14 10:03 ` Wu Fengguang
@ 2012-02-14 13:29 ` Jan Kara
2012-02-16 4:00 ` Wu Fengguang
0 siblings, 1 reply; 33+ messages in thread
From: Jan Kara @ 2012-02-14 13:29 UTC (permalink / raw)
To: Wu Fengguang
Cc: Jan Kara, Rik van Riel, Greg Thelen, bsingharora, Hugh Dickins,
Michal Hocko, linux-mm, Mel Gorman, Ying Han, hannes,
KAMEZAWA Hiroyuki, Minchan Kim
On Tue 14-02-12 18:03:48, Wu Fengguang wrote:
> On Mon, Feb 13, 2012 at 04:43:13PM +0100, Jan Kara wrote:
> > On Sun 12-02-12 11:10:29, Wu Fengguang wrote:
>
> > > 4) test case
> > >
> > > Run 2 dd tasks in a 100MB memcg (a very handy test case from Greg Thelen):
> > >
> > > mkdir /cgroup/x
> > > echo 100M > /cgroup/x/memory.limit_in_bytes
> > > echo $$ > /cgroup/x/tasks
> > >
> > > for i in `seq 2`
> > > do
> > > dd if=/dev/zero of=/fs/f$i bs=1k count=1M &
> > > done
> > >
> > > Before patch, the dd tasks are quickly OOM killed.
> > > After patch, they run well with reasonably good performance and overheads:
> > >
> > > 1073741824 bytes (1.1 GB) copied, 22.2196 s, 48.3 MB/s
> > > 1073741824 bytes (1.1 GB) copied, 22.4675 s, 47.8 MB/s
> > I wonder what happens if you run:
> > mkdir /cgroup/x
> > echo 100M > /cgroup/x/memory.limit_in_bytes
> > echo $$ > /cgroup/x/tasks
> >
> > for (( i = 0; i < 2; i++ )); do
> > mkdir /fs/d$i
> > for (( j = 0; j < 5000; j++ )); do
> > dd if=/dev/zero of=/fs/d$i/f$j bs=1k count=50
> > done &
> > done
>
> That's a very good case, thanks!
>
> > Because for small files the writearound logic won't help much...
>
> Right, it also means the native background work cannot be more I/O
> efficient than the pageout works, except for the overheads of more
> work items..
Yes, that's true.
> > Also the number of work items queued might become interesting.
>
> It turns out that the 1024 mempool reservations are not exhausted at
> all (the below patch as a trace_printk on alloc failure and it didn't
> trigger at all).
>
> Here is the representative iostat lines on XFS (full "iostat -kx 1 20" log attached):
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 0.80 0.00 6.03 0.03 0.00 93.14
>
> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
> sda 0.00 205.00 0.00 163.00 0.00 16900.00 207.36 4.09 21.63 1.88 30.70
>
> The attached dirtied/written progress graph looks interesting.
> Although the iostat disk utilization is low, the "dirtied" progress
> line is pretty straight and there is no single congestion_wait event
> in the trace log. Which makes me wonder if there are some unknown
> blocking issues in the way.
Interesting. I'd also expect we should block in reclaim path. How fast
can dd threads progress when there is no cgroup involved?
> > Another common case to test - run 'slapadd' command in each cgroup to
> > create big LDAP database. That does pretty much random IO on a big mmaped
> > DB file.
>
> I've not used this. Will it need some configuration and data feed?
> fio looks more handy to me for emulating mmap random IO.
Yes, fio can generate random mmap IO. It's just that this is a real life
workload. So it is not completely random, it happens on several files and
is also interleaved with other memory allocations from DB. I can send you
the config files and data feed if you are interested.
> > > +/*
> > > + * schedule writeback on a range of inode pages.
> > > + */
> > > +static struct wb_writeback_work *
> > > +bdi_flush_inode_range(struct backing_dev_info *bdi,
> > > + struct inode *inode,
> > > + pgoff_t offset,
> > > + pgoff_t len,
> > > + bool wait)
> > > +{
> > > + struct wb_writeback_work *work;
> > > +
> > > + if (!igrab(inode))
> > > + return ERR_PTR(-ENOENT);
> > One technical note here: If the inode is deleted while it is queued, this
> > reference will keep it living until flusher thread gets to it. Then when
> > flusher thread puts its reference, the inode will get deleted in flusher
> > thread context. I don't see an immediate problem in that but it might be
> > surprising sometimes. Another problem I see is that if you try to
> > unmount the filesystem while the work item is queued, you'll get EBUSY for
> > no apparent reason (for userspace).
>
> Yeah, we need to make umount work.
The positive thing is that if the inode is reaped while the work item is
queue, we know all that needed to be done is done. So we don't really need
to pin the inode.
> And I find the pageout works seem to have some problems with ext4.
> For example, this can be easily triggered with 10 dd tasks running
> inside the 100MB limited memcg:
So journal thread is getting stuck while committing transaction. Most
likely waiting for some dd thread to stop a transaction so that commit can
proceed. The processes waiting in start_this_handle() are just secondary
effect resulting from the first problem. It might be interesting to get
stack traces of all bloked processes when the journal thread is stuck.
> [18006.858109] INFO: task jbd2/sda1-8:51294 blocked for more than 120 seconds.
> [18006.866425] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [18006.876096] jbd2/sda1-8 D 0000000000000000 5464 51294 2 0x00000000
> [18006.884729] ffff88040b097c70 0000000000000046 ffff880823032310 ffff88040b096000
> [18006.894356] 00000000001d2f00 00000000001d2f00 ffff8808230322a0 00000000001d2f00
> [18006.904000] ffff88040b097fd8 00000000001d2f00 ffff88040b097fd8 00000000001d2f00
> [18006.913652] Call Trace:
> [18006.916901] [<ffffffff8103d4af>] ? native_sched_clock+0x29/0x70
> [18006.924134] [<ffffffff81232aab>] ? jbd2_journal_commit_transaction+0x1d0/0x1281
> [18006.933324] [<ffffffff8109660d>] ? local_clock+0x41/0x5a
> [18006.939879] [<ffffffff810b0ddd>] ? lock_release_holdtime+0xa3/0xac
> [18006.947410] [<ffffffff81232aab>] ? jbd2_journal_commit_transaction+0x1d0/0x1281
> [18006.956607] [<ffffffff81a57904>] schedule+0x5a/0x5c
> [18006.962677] [<ffffffff81232ab0>] jbd2_journal_commit_transaction+0x1d5/0x1281
> [18006.971683] [<ffffffff8103d4af>] ? native_sched_clock+0x29/0x70
> [18006.978933] [<ffffffff810738ce>] ? try_to_del_timer_sync+0xba/0xc8
> [18006.986452] [<ffffffff8109660d>] ? local_clock+0x41/0x5a
> [18006.992999] [<ffffffff8108683a>] ? wake_up_bit+0x2a/0x2a
> [18006.999542] [<ffffffff810738ce>] ? try_to_del_timer_sync+0xba/0xc8
> [18007.007062] [<ffffffff81073a6f>] ? del_timer_sync+0xbb/0xce
> [18007.013898] [<ffffffff810739b4>] ? process_timeout+0x10/0x10
> [18007.020835] [<ffffffff81237bc1>] kjournald2+0xcf/0x242
> [18007.027187] [<ffffffff8108683a>] ? wake_up_bit+0x2a/0x2a
> [18007.033733] [<ffffffff81237af2>] ? commit_timeout+0x10/0x10
> [18007.040574] [<ffffffff81086384>] kthread+0x95/0x9d
> [18007.046542] [<ffffffff81a61134>] kernel_thread_helper+0x4/0x10
> [18007.053675] [<ffffffff81a591b4>] ? retint_restore_args+0x13/0x13
> [18007.061003] [<ffffffff810862ef>] ? __init_kthread_worker+0x5b/0x5b
> [18007.068521] [<ffffffff81a61130>] ? gs_change+0x13/0x13
> [18007.074878] no locks held by jbd2/sda1-8/51294.
>
> Sometimes I also catch dd/ext4lazyinit/flush all stalling in start_this_handle:
>
> [17985.439567] dd D 0000000000000007 3616 61440 1 0x00000004
> [17985.448088] ffff88080d71b9b8 0000000000000046 ffff88081ec80070 ffff88080d71a000
> [17985.457545] 00000000001d2f00 00000000001d2f00 ffff88081ec80000 00000000001d2f00
> [17985.467168] ffff88080d71bfd8 00000000001d2f00 ffff88080d71bfd8 00000000001d2f00
> [17985.476647] Call Trace:
> [17985.479843] [<ffffffff8103d4af>] ? native_sched_clock+0x29/0x70
> [17985.487025] [<ffffffff81230b9d>] ? start_this_handle+0x357/0x4ed
> [17985.494313] [<ffffffff8109660d>] ? local_clock+0x41/0x5a
> [17985.500815] [<ffffffff810b0ddd>] ? lock_release_holdtime+0xa3/0xac
> [17985.508287] [<ffffffff81230b9d>] ? start_this_handle+0x357/0x4ed
> [17985.515575] [<ffffffff81a57904>] schedule+0x5a/0x5c
> [17985.521588] [<ffffffff81230c39>] start_this_handle+0x3f3/0x4ed
> [17985.528669] [<ffffffff81147820>] ? kmem_cache_free+0xfa/0x13a
> [17985.545142] [<ffffffff8108683a>] ? wake_up_bit+0x2a/0x2a
> [17985.551650] [<ffffffff81230f0e>] jbd2__journal_start+0xb0/0xf6
> [17985.558732] [<ffffffff811f7ad7>] ? ext4_dirty_inode+0x1d/0x4c
> [17985.565716] [<ffffffff81230f67>] jbd2_journal_start+0x13/0x15
> [17985.572703] [<ffffffff8120e3e9>] ext4_journal_start_sb+0x13f/0x157
> [17985.580172] [<ffffffff8109660d>] ? local_clock+0x41/0x5a
> [17985.586680] [<ffffffff811f7ad7>] ext4_dirty_inode+0x1d/0x4c
> [17985.593472] [<ffffffff81176827>] __mark_inode_dirty+0x2e/0x1cc
> [17985.600552] [<ffffffff81168e84>] file_update_time+0xe4/0x106
> [17985.607441] [<ffffffff811079f6>] __generic_file_aio_write+0x254/0x364
> [17985.615202] [<ffffffff81a565da>] ? mutex_lock_nested+0x2e4/0x2f3
> [17985.622488] [<ffffffff81107b50>] ? generic_file_aio_write+0x4a/0xc1
> [17985.630057] [<ffffffff81107b6c>] generic_file_aio_write+0x66/0xc1
> [17985.637442] [<ffffffff811ef72b>] ext4_file_write+0x1f9/0x251
> [17985.644330] [<ffffffff8109660d>] ? local_clock+0x41/0x5a
> [17985.650835] [<ffffffff8118809e>] ? fsnotify+0x222/0x27b
> [17985.657238] [<ffffffff81153612>] do_sync_write+0xce/0x10b
> [17985.663844] [<ffffffff8118809e>] ? fsnotify+0x222/0x27b
> [17985.670243] [<ffffffff81187ef8>] ? fsnotify+0x7c/0x27b
> [17985.676561] [<ffffffff81153dbe>] vfs_write+0xb8/0x157
> [17985.682767] [<ffffffff81154075>] sys_write+0x4d/0x77
> [17985.688878] [<ffffffff81a5fce9>] system_call_fastpath+0x16/0x1b
>
> and jbd2 in
>
> [17983.623657] jbd2/sda1-8 D 0000000000000000 5464 51294 2 0x00000000
> [17983.632173] ffff88040b097c70 0000000000000046 ffff880823032310 ffff88040b096000
> [17983.641640] 00000000001d2f00 00000000001d2f00 ffff8808230322a0 00000000001d2f00
> [17983.651119] ffff88040b097fd8 00000000001d2f00 ffff88040b097fd8 00000000001d2f00
> [17983.660603] Call Trace:
> [17983.663808] [<ffffffff8103d4af>] ? native_sched_clock+0x29/0x70
> [17983.670997] [<ffffffff81232aab>] ? jbd2_journal_commit_transaction+0x1d0/0x1281
> [17983.680124] [<ffffffff8109660d>] ? local_clock+0x41/0x5a
> [17983.686638] [<ffffffff810b0ddd>] ? lock_release_holdtime+0xa3/0xac
> [17983.694108] [<ffffffff81232aab>] ? jbd2_journal_commit_transaction+0x1d0/0x1281
> [17983.703243] [<ffffffff81a57904>] schedule+0x5a/0x5c
> [17983.709262] [<ffffffff81232ab0>] jbd2_journal_commit_transaction+0x1d5/0x1281
> [17983.718195] [<ffffffff8103d4af>] ? native_sched_clock+0x29/0x70
> [17983.725392] [<ffffffff810738ce>] ? try_to_del_timer_sync+0xba/0xc8
> [17983.732867] [<ffffffff8109660d>] ? local_clock+0x41/0x5a
> [17983.739374] [<ffffffff8108683a>] ? wake_up_bit+0x2a/0x2a
> [17983.745864] [<ffffffff810738ce>] ? try_to_del_timer_sync+0xba/0xc8
> [17983.753343] [<ffffffff81073a6f>] ? del_timer_sync+0xbb/0xce
> [17983.760137] [<ffffffff810739b4>] ? process_timeout+0x10/0x10
> [17983.767041] [<ffffffff81237bc1>] kjournald2+0xcf/0x242
> [17983.773361] [<ffffffff8108683a>] ? wake_up_bit+0x2a/0x2a
> [17983.779863] [<ffffffff81237af2>] ? commit_timeout+0x10/0x10
> [17983.786665] [<ffffffff81086384>] kthread+0x95/0x9d
> [17983.792585] [<ffffffff81a61134>] kernel_thread_helper+0x4/0x10
> [17983.799670] [<ffffffff81a591b4>] ? retint_restore_args+0x13/0x13
> [17983.806948] [<ffffffff810862ef>] ? __init_kthread_worker+0x5b/0x5b
>
> Here is the updated patch used in the new tests. It moves
> congestion_wait() out of the page lock and make flush_inode_page() no
> longer wait for memory allocation (looks unnecessary).
Honza
--
Jan Kara <jack@suse.cz>
SUSE Labs, CR
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: reclaim the LRU lists full of dirty/writeback pages
2012-02-14 13:18 ` Wu Fengguang
@ 2012-02-14 13:35 ` Wu Fengguang
2012-02-14 15:51 ` Mel Gorman
2012-02-16 0:00 ` KAMEZAWA Hiroyuki
2 siblings, 0 replies; 33+ messages in thread
From: Wu Fengguang @ 2012-02-14 13:35 UTC (permalink / raw)
To: Mel Gorman
Cc: Greg Thelen, Jan Kara, bsingharora, Hugh Dickins, Michal Hocko,
linux-mm, Ying Han, hannes, KAMEZAWA Hiroyuki, Rik van Riel,
Minchan Kim
[-- Attachment #1: Type: text/plain, Size: 432 bytes --]
> For the single dd inside memcg, ext4 is now working pretty well, with
> least CPU overheads:
Attached are the dd progresses for btrfs, ext4 and xfs in turn, with
btrfs performing the most fluent and fast.
Also attached the "10 memcg each running 1 dd" graph for btrfs. Well
it's unacceptably bumpy because there are no any efforts to make a
round-robin walk between the works from different memcg LRU lists.
Thanks,
Fengguang
[-- Attachment #2: balance_dirty_pages-task-bw.png --]
[-- Type: image/png, Size: 36661 bytes --]
[-- Attachment #3: balance_dirty_pages-task-bw.png --]
[-- Type: image/png, Size: 37858 bytes --]
[-- Attachment #4: balance_dirty_pages-task-bw.png --]
[-- Type: image/png, Size: 38995 bytes --]
[-- Attachment #5: balance_dirty_pages-task-bw.png --]
[-- Type: image/png, Size: 45482 bytes --]
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: reclaim the LRU lists full of dirty/writeback pages
2012-02-14 13:18 ` Wu Fengguang
2012-02-14 13:35 ` Wu Fengguang
@ 2012-02-14 15:51 ` Mel Gorman
2012-02-16 9:50 ` Wu Fengguang
2012-02-16 0:00 ` KAMEZAWA Hiroyuki
2 siblings, 1 reply; 33+ messages in thread
From: Mel Gorman @ 2012-02-14 15:51 UTC (permalink / raw)
To: Wu Fengguang
Cc: Greg Thelen, Jan Kara, bsingharora, Hugh Dickins, Michal Hocko,
linux-mm, Ying Han, hannes, KAMEZAWA Hiroyuki, Rik van Riel,
Minchan Kim
On Tue, Feb 14, 2012 at 09:18:12PM +0800, Wu Fengguang wrote:
> > For the OOM problem, a more reasonable stopgap might be to identify when
> > a process is scanning a memcg at high priority and encountered all
> > PageReclaim with no forward progress and to congestion_wait() if that
> > situation occurs. A preferable way would be to wait until the flusher
> > wakes up a waiter on PageReclaim pages to be written out because we want
> > to keep moving way from congestion_wait() if at all possible.
>
> Good points! Below is the more serious page reclaim changes.
>
> The dirty/writeback pages may often come close to each other in the
> LRU list, so the local test during a 32-page scan may still trigger
> reclaim waits unnecessarily.
Yes, this is particularly the case when writing back to USB. It is not
unusual that all dirty pages under writeback are backed by USB and at the
end of the LRU. Right now what happens is that reclaimers see higher CPU
usage as they scan over these pages uselessly. If the wrong choice is
made on how to throttle, we'll see yet more variants of the "system
responsiveness drops when writing to USB".
> Some global information on the percent
> of dirty/writeback pages in the LRU list may help. Anyway the added
> tests should still be much better than no protection.
>
You can tell how many dirty pages and writeback pages are in the zone
already.
> A global wait queue and reclaim_wait() is introduced. The waiters will
> be wakeup when pages are rotated by end_page_writeback() or lru drain.
>
> I have to say its effectiveness depends on the filesystem... ext4
> and btrfs do fluent IO completions, so reclaim_wait() works pretty
> well:
> dd-14560 [017] .... 1360.894605: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=10000
> dd-14560 [017] .... 1360.904456: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=8000
> dd-14560 [017] .... 1360.908293: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
> dd-14560 [017] .... 1360.923960: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=15000
> dd-14560 [017] .... 1360.927810: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
> dd-14560 [017] .... 1360.931656: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
> dd-14560 [017] .... 1360.943503: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=10000
> dd-14560 [017] .... 1360.953289: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=7000
> dd-14560 [017] .... 1360.957177: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
> dd-14560 [017] .... 1360.972949: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=15000
>
> However XFS does IO completions in very large batches (there may be
> only several big IO completions in one second). So reclaim_wait()
> mostly end up waiting to the full HZ/10 timeout:
>
> dd-4177 [008] .... 866.367661: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> dd-4177 [010] .... 866.567583: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> dd-4177 [012] .... 866.767458: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> dd-4177 [013] .... 866.867419: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> dd-4177 [008] .... 867.167266: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> dd-4177 [010] .... 867.367168: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> dd-4177 [012] .... 867.818950: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> dd-4177 [013] .... 867.918905: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> dd-4177 [013] .... 867.971657: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=52000
> dd-4177 [013] .... 867.971812: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=0
> dd-4177 [008] .... 868.355700: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> dd-4177 [010] .... 868.700515: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
>
And where people will get hit by regressions in this area is writing to
vfat and in more rare cases ntfs on USB stick.
> > Another possibility would be to relook at LRU_IMMEDIATE but right now it
> > requires a page flag and I haven't devised a way around that. Besides,
> > it would only address the problem of PageREclaim pages being encountered,
> > it would not handle the case where a memcg was filled with PageReclaim pages.
>
> I also considered things like LRU_IMMEDIATE, however got no clear idea yet.
> Since the simple "wait on PG_reclaim" approach appears to work for this
> memcg dd case, it effectively disables me to think any further ;-)
>
Test with interactive use while writing heavily to a USB stick.
> For the single dd inside memcg, ext4 is now working pretty well, with
> least CPU overheads:
>
> (running from another test box, so not directly comparable with old tests)
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 0.03 0.00 0.85 5.35 0.00 93.77
>
> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
> sda 0.00 0.00 0.00 112.00 0.00 57348.00 1024.07 81.66 1045.21 8.93 100.00
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 0.00 0.00 0.69 4.07 0.00 95.24
>
> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
> sda 0.00 142.00 0.00 112.00 0.00 56832.00 1014.86 127.94 790.04 8.93 100.00
>
> And xfs a bit less fluent:
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 0.00 0.00 3.79 2.54 0.00 93.68
>
> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
> sda 0.00 0.00 0.00 108.00 0.00 54644.00 1011.93 48.13 1044.83 8.44 91.20
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 0.00 0.00 3.38 3.88 0.00 92.74
>
> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
> sda 0.00 0.00 0.00 105.00 0.00 53156.00 1012.50 128.50 451.90 9.25 97.10
>
> btrfs also looks good:
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 0.00 0.00 8.05 3.85 0.00 88.10
>
> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
> sda 0.00 0.00 0.00 108.00 0.00 53248.00 986.07 88.11 643.99 9.26 100.00
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 0.00 0.00 4.04 2.51 0.00 93.45
>
> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
> sda 0.00 0.00 0.00 112.00 0.00 57344.00 1024.00 91.58 998.41 8.93 100.00
>
> ---
>
> --- linux.orig/include/linux/backing-dev.h 2012-02-14 19:43:06.000000000 +0800
> +++ linux/include/linux/backing-dev.h 2012-02-14 19:49:26.000000000 +0800
> @@ -304,6 +304,8 @@ void clear_bdi_congested(struct backing_
> void set_bdi_congested(struct backing_dev_info *bdi, int sync);
> long congestion_wait(int sync, long timeout);
> long wait_iff_congested(struct zone *zone, int sync, long timeout);
> +long reclaim_wait(long timeout);
> +void reclaim_rotated(void);
>
> static inline bool bdi_cap_writeback_dirty(struct backing_dev_info *bdi)
> {
> --- linux.orig/mm/backing-dev.c 2012-02-14 19:26:15.000000000 +0800
> +++ linux/mm/backing-dev.c 2012-02-14 20:09:45.000000000 +0800
> @@ -873,3 +873,38 @@ out:
> return ret;
> }
> EXPORT_SYMBOL(wait_iff_congested);
> +
> +static DECLARE_WAIT_QUEUE_HEAD(reclaim_wqh);
> +
> +/**
> + * reclaim_wait - wait for some pages being rotated to the LRU tail
> + * @timeout: timeout in jiffies
> + *
> + * Wait until @timeout, or when some (typically PG_reclaim under writeback)
> + * pages rotated to the LRU so that page reclaim can make progress.
> + */
> +long reclaim_wait(long timeout)
> +{
> + long ret;
> + unsigned long start = jiffies;
> + DEFINE_WAIT(wait);
> +
> + prepare_to_wait(&reclaim_wqh, &wait, TASK_KILLABLE);
> + ret = io_schedule_timeout(timeout);
> + finish_wait(&reclaim_wqh, &wait);
> +
> + trace_writeback_reclaim_wait(jiffies_to_usecs(timeout),
> + jiffies_to_usecs(jiffies - start));
> +
> + return ret;
> +}
> +EXPORT_SYMBOL(reclaim_wait);
> +
> +void reclaim_rotated()
> +{
> + wait_queue_head_t *wqh = &reclaim_wqh;
> +
> + if (waitqueue_active(wqh))
> + wake_up(wqh);
> +}
> +
> --- linux.orig/mm/swap.c 2012-02-14 19:40:10.000000000 +0800
> +++ linux/mm/swap.c 2012-02-14 19:45:13.000000000 +0800
> @@ -253,6 +253,7 @@ static void pagevec_move_tail(struct pag
>
> pagevec_lru_move_fn(pvec, pagevec_move_tail_fn, &pgmoved);
> __count_vm_events(PGROTATED, pgmoved);
> + reclaim_rotated();
> }
>
> /*
> --- linux.orig/mm/vmscan.c 2012-02-14 17:53:27.000000000 +0800
> +++ linux/mm/vmscan.c 2012-02-14 19:44:11.000000000 +0800
> @@ -767,7 +767,8 @@ static unsigned long shrink_page_list(st
> struct scan_control *sc,
> int priority,
> unsigned long *ret_nr_dirty,
> - unsigned long *ret_nr_writeback)
> + unsigned long *ret_nr_writeback,
> + unsigned long *ret_nr_pgreclaim)
> {
> LIST_HEAD(ret_pages);
> LIST_HEAD(free_pages);
> @@ -776,6 +777,7 @@ static unsigned long shrink_page_list(st
> unsigned long nr_congested = 0;
> unsigned long nr_reclaimed = 0;
> unsigned long nr_writeback = 0;
> + unsigned long nr_pgreclaim = 0;
>
> cond_resched();
>
> @@ -813,6 +815,10 @@ static unsigned long shrink_page_list(st
>
> if (PageWriteback(page)) {
> nr_writeback++;
> + if (PageReclaim(page))
> + nr_pgreclaim++;
> + else
> + SetPageReclaim(page);
> /*
This check is unexpected. We already SetPageReclaim when queuing pages for
IO from reclaim context and if dirty pages are encountered during the LRU
scan that cannot be queued for IO. How often is it that nr_pgreclaim !=
nr_writeback and by how much do they differ?
> * Synchronous reclaim cannot queue pages for
> * writeback due to the possibility of stack overflow
> @@ -874,12 +880,15 @@ static unsigned long shrink_page_list(st
> nr_dirty++;
>
> /*
> - * Only kswapd can writeback filesystem pages to
> - * avoid risk of stack overflow but do not writeback
> - * unless under significant pressure.
> + * run into the visited page again: we are scanning
> + * faster than the flusher can writeout dirty pages
> */
which in itself is not an abnormal condition. We get into this situation
when writing to USB. Dirty throttling stops too much memory getting dirtied
but that does not mean we should throttle instead of reclaiming clean pages.
That's why I worry that if this is aimed at fixing a memcg problem, it
will have the impact of making interactive performance on normal systems
worse.
> - if (page_is_file_cache(page) &&
> - (!current_is_kswapd() || priority >= DEF_PRIORITY - 2)) {
> + if (page_is_file_cache(page) && PageReclaim(page)) {
> + nr_pgreclaim++;
> + goto keep_locked;
> + }
> + if (page_is_file_cache(page) && mapping &&
> + flush_inode_page(mapping, page, false) >= 0) {
> /*
> * Immediately reclaim when written back.
> * Similar in principal to deactivate_page()
> @@ -1028,6 +1037,7 @@ keep_lumpy:
> count_vm_events(PGACTIVATE, pgactivate);
> *ret_nr_dirty += nr_dirty;
> *ret_nr_writeback += nr_writeback;
> + *ret_nr_pgreclaim += nr_pgreclaim;
> return nr_reclaimed;
> }
>
> @@ -1087,8 +1097,10 @@ int __isolate_lru_page(struct page *page
> */
> if (mode & (ISOLATE_CLEAN|ISOLATE_ASYNC_MIGRATE)) {
> /* All the caller can do on PageWriteback is block */
> - if (PageWriteback(page))
> + if (PageWriteback(page)) {
> + SetPageReclaim(page);
> return ret;
> + }
>
This hunk means that if async compaction (common for THP) encounters a page
under writeback, it will still skip it but mark it for immediate reclaim
after IO completes. This will have the impact that compaction causes an
abnormally high number of pages to be reclaimed.
> if (PageDirty(page)) {
> struct address_space *mapping;
> @@ -1509,6 +1521,7 @@ shrink_inactive_list(unsigned long nr_to
> unsigned long nr_file;
> unsigned long nr_dirty = 0;
> unsigned long nr_writeback = 0;
> + unsigned long nr_pgreclaim = 0;
> isolate_mode_t reclaim_mode = ISOLATE_INACTIVE;
> struct zone *zone = mz->zone;
>
> @@ -1559,13 +1572,13 @@ shrink_inactive_list(unsigned long nr_to
> spin_unlock_irq(&zone->lru_lock);
>
> nr_reclaimed = shrink_page_list(&page_list, mz, sc, priority,
> - &nr_dirty, &nr_writeback);
> + &nr_dirty, &nr_writeback, &nr_pgreclaim);
>
> /* Check if we should syncronously wait for writeback */
> if (should_reclaim_stall(nr_taken, nr_reclaimed, priority, sc)) {
> set_reclaim_mode(priority, sc, true);
> nr_reclaimed += shrink_page_list(&page_list, mz, sc,
> - priority, &nr_dirty, &nr_writeback);
> + priority, &nr_dirty, &nr_writeback, &nr_pgreclaim);
> }
>
> spin_lock_irq(&zone->lru_lock);
> @@ -1608,6 +1621,8 @@ shrink_inactive_list(unsigned long nr_to
> */
> if (nr_writeback && nr_writeback >= (nr_taken >> (DEF_PRIORITY-priority)))
> wait_iff_congested(zone, BLK_RW_ASYNC, HZ/10);
> + if (nr_pgreclaim && nr_pgreclaim >= (nr_taken >> (DEF_PRIORITY-priority)))
> + reclaim_wait(HZ/10);
>
We risk going to sleep too easily when USB-backed pages are at the end of
the LRU list. Note that the nr_writeback check only goes to sleep if it
detects that the underlying storage is also congested. In contrast, it
will take very few PageReclaim pages at teh end of the LRU to cause the
process to sleep when it instead should find clean pages to discard.
If the intention is to avoid memcg going OOM prematurely, the
nr_pgreclaim value needs to be treated at a higher level that records
how many PageReclaim pages were encountered. If no progress was made
because all the pages were PageReclaim, then throttle and return 1 to
the page allocator where it will retry the allocation without going OOM
after some pages have been cleaned and reclaimed.
> trace_mm_vmscan_lru_shrink_inactive(zone->zone_pgdat->node_id,
> zone_idx(zone),
> @@ -2382,8 +2397,6 @@ static unsigned long do_try_to_free_page
> */
> writeback_threshold = sc->nr_to_reclaim + sc->nr_to_reclaim / 2;
> if (total_scanned > writeback_threshold) {
> - wakeup_flusher_threads(laptop_mode ? 0 : total_scanned,
> - WB_REASON_TRY_TO_FREE_PAGES);
> sc->may_writepage = 1;
> }
>
--
Mel Gorman
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: reclaim the LRU lists full of dirty/writeback pages
2012-02-14 13:18 ` Wu Fengguang
2012-02-14 13:35 ` Wu Fengguang
2012-02-14 15:51 ` Mel Gorman
@ 2012-02-16 0:00 ` KAMEZAWA Hiroyuki
2012-02-16 3:04 ` Wu Fengguang
2 siblings, 1 reply; 33+ messages in thread
From: KAMEZAWA Hiroyuki @ 2012-02-16 0:00 UTC (permalink / raw)
To: Wu Fengguang
Cc: Mel Gorman, Greg Thelen, Jan Kara, bsingharora, Hugh Dickins,
Michal Hocko, linux-mm, Ying Han, hannes, Rik van Riel,
Minchan Kim
On Tue, 14 Feb 2012 21:18:12 +0800
Wu Fengguang <fengguang.wu@intel.com> wrote:
>
> --- linux.orig/include/linux/backing-dev.h 2012-02-14 19:43:06.000000000 +0800
> +++ linux/include/linux/backing-dev.h 2012-02-14 19:49:26.000000000 +0800
> @@ -304,6 +304,8 @@ void clear_bdi_congested(struct backing_
> void set_bdi_congested(struct backing_dev_info *bdi, int sync);
> long congestion_wait(int sync, long timeout);
> long wait_iff_congested(struct zone *zone, int sync, long timeout);
> +long reclaim_wait(long timeout);
> +void reclaim_rotated(void);
>
> static inline bool bdi_cap_writeback_dirty(struct backing_dev_info *bdi)
> {
> --- linux.orig/mm/backing-dev.c 2012-02-14 19:26:15.000000000 +0800
> +++ linux/mm/backing-dev.c 2012-02-14 20:09:45.000000000 +0800
> @@ -873,3 +873,38 @@ out:
> return ret;
> }
> EXPORT_SYMBOL(wait_iff_congested);
> +
> +static DECLARE_WAIT_QUEUE_HEAD(reclaim_wqh);
> +
> +/**
> + * reclaim_wait - wait for some pages being rotated to the LRU tail
> + * @timeout: timeout in jiffies
> + *
> + * Wait until @timeout, or when some (typically PG_reclaim under writeback)
> + * pages rotated to the LRU so that page reclaim can make progress.
> + */
> +long reclaim_wait(long timeout)
> +{
> + long ret;
> + unsigned long start = jiffies;
> + DEFINE_WAIT(wait);
> +
> + prepare_to_wait(&reclaim_wqh, &wait, TASK_KILLABLE);
> + ret = io_schedule_timeout(timeout);
> + finish_wait(&reclaim_wqh, &wait);
> +
> + trace_writeback_reclaim_wait(jiffies_to_usecs(timeout),
> + jiffies_to_usecs(jiffies - start));
> +
> + return ret;
> +}
> +EXPORT_SYMBOL(reclaim_wait);
> +
> +void reclaim_rotated()
> +{
> + wait_queue_head_t *wqh = &reclaim_wqh;
> +
> + if (waitqueue_active(wqh))
> + wake_up(wqh);
> +}
> +
Thank you.
I like this approach. A nitpick is that this may wake up all waiters
in the system when a memcg is rotated.
How about wait_event() + condition by bitmap (using per memcg unique IDs.) ?
Thanks,
-Kame
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: reclaim the LRU lists full of dirty/writeback pages
2012-02-16 0:00 ` KAMEZAWA Hiroyuki
@ 2012-02-16 3:04 ` Wu Fengguang
2012-02-16 3:52 ` KAMEZAWA Hiroyuki
0 siblings, 1 reply; 33+ messages in thread
From: Wu Fengguang @ 2012-02-16 3:04 UTC (permalink / raw)
To: KAMEZAWA Hiroyuki
Cc: Mel Gorman, Greg Thelen, Jan Kara, bsingharora, Hugh Dickins,
Michal Hocko, linux-mm, Ying Han, hannes, Rik van Riel,
Minchan Kim
On Thu, Feb 16, 2012 at 09:00:37AM +0900, KAMEZAWA Hiroyuki wrote:
> On Tue, 14 Feb 2012 21:18:12 +0800
> Wu Fengguang <fengguang.wu@intel.com> wrote:
>
> >
> > --- linux.orig/include/linux/backing-dev.h 2012-02-14 19:43:06.000000000 +0800
> > +++ linux/include/linux/backing-dev.h 2012-02-14 19:49:26.000000000 +0800
> > @@ -304,6 +304,8 @@ void clear_bdi_congested(struct backing_
> > void set_bdi_congested(struct backing_dev_info *bdi, int sync);
> > long congestion_wait(int sync, long timeout);
> > long wait_iff_congested(struct zone *zone, int sync, long timeout);
> > +long reclaim_wait(long timeout);
> > +void reclaim_rotated(void);
> >
> > static inline bool bdi_cap_writeback_dirty(struct backing_dev_info *bdi)
> > {
> > --- linux.orig/mm/backing-dev.c 2012-02-14 19:26:15.000000000 +0800
> > +++ linux/mm/backing-dev.c 2012-02-14 20:09:45.000000000 +0800
> > @@ -873,3 +873,38 @@ out:
> > return ret;
> > }
> > EXPORT_SYMBOL(wait_iff_congested);
> > +
> > +static DECLARE_WAIT_QUEUE_HEAD(reclaim_wqh);
> > +
> > +/**
> > + * reclaim_wait - wait for some pages being rotated to the LRU tail
> > + * @timeout: timeout in jiffies
> > + *
> > + * Wait until @timeout, or when some (typically PG_reclaim under writeback)
> > + * pages rotated to the LRU so that page reclaim can make progress.
> > + */
> > +long reclaim_wait(long timeout)
> > +{
> > + long ret;
> > + unsigned long start = jiffies;
> > + DEFINE_WAIT(wait);
> > +
> > + prepare_to_wait(&reclaim_wqh, &wait, TASK_KILLABLE);
> > + ret = io_schedule_timeout(timeout);
> > + finish_wait(&reclaim_wqh, &wait);
> > +
> > + trace_writeback_reclaim_wait(jiffies_to_usecs(timeout),
> > + jiffies_to_usecs(jiffies - start));
> > +
> > + return ret;
> > +}
> > +EXPORT_SYMBOL(reclaim_wait);
> > +
> > +void reclaim_rotated()
> > +{
> > + wait_queue_head_t *wqh = &reclaim_wqh;
> > +
> > + if (waitqueue_active(wqh))
> > + wake_up(wqh);
> > +}
> > +
>
> Thank you.
>
> I like this approach. A nitpick is that this may wake up all waiters
> in the system when a memcg is rotated.
Thank you. It sure helps to start it simple :-)
> How about wait_event() + condition by bitmap (using per memcg unique IDs.) ?
I'm not sure how to manage the bitmap. The idea in my mind is to
- maintain a memcg->pages_rotated counter
- in reclaim_wait(), grab the current ->pages_rotated value before
going to wait, compare it to the new value on every wakeup, and
return to the user when seeing a different ->pages_rotated value.
(this cannot stop waking up multiple tasks in the same memcg...)
Does that sound reasonable?
Thanks,
Fengguang
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: reclaim the LRU lists full of dirty/writeback pages
2012-02-16 3:04 ` Wu Fengguang
@ 2012-02-16 3:52 ` KAMEZAWA Hiroyuki
2012-02-16 4:05 ` Wu Fengguang
0 siblings, 1 reply; 33+ messages in thread
From: KAMEZAWA Hiroyuki @ 2012-02-16 3:52 UTC (permalink / raw)
To: Wu Fengguang
Cc: Mel Gorman, Greg Thelen, Jan Kara, bsingharora, Hugh Dickins,
Michal Hocko, linux-mm, Ying Han, hannes, Rik van Riel,
Minchan Kim
On Thu, 16 Feb 2012 11:04:15 +0800
Wu Fengguang <fengguang.wu@intel.com> wrote:
> On Thu, Feb 16, 2012 at 09:00:37AM +0900, KAMEZAWA Hiroyuki wrote:
> > On Tue, 14 Feb 2012 21:18:12 +0800
> > Wu Fengguang <fengguang.wu@intel.com> wrote:
> >
> > >
> > > --- linux.orig/include/linux/backing-dev.h 2012-02-14 19:43:06.000000000 +0800
> > > +++ linux/include/linux/backing-dev.h 2012-02-14 19:49:26.000000000 +0800
> > > @@ -304,6 +304,8 @@ void clear_bdi_congested(struct backing_
> > > void set_bdi_congested(struct backing_dev_info *bdi, int sync);
> > > long congestion_wait(int sync, long timeout);
> > > long wait_iff_congested(struct zone *zone, int sync, long timeout);
> > > +long reclaim_wait(long timeout);
> > > +void reclaim_rotated(void);
> > >
> > > static inline bool bdi_cap_writeback_dirty(struct backing_dev_info *bdi)
> > > {
> > > --- linux.orig/mm/backing-dev.c 2012-02-14 19:26:15.000000000 +0800
> > > +++ linux/mm/backing-dev.c 2012-02-14 20:09:45.000000000 +0800
> > > @@ -873,3 +873,38 @@ out:
> > > return ret;
> > > }
> > > EXPORT_SYMBOL(wait_iff_congested);
> > > +
> > > +static DECLARE_WAIT_QUEUE_HEAD(reclaim_wqh);
> > > +
> > > +/**
> > > + * reclaim_wait - wait for some pages being rotated to the LRU tail
> > > + * @timeout: timeout in jiffies
> > > + *
> > > + * Wait until @timeout, or when some (typically PG_reclaim under writeback)
> > > + * pages rotated to the LRU so that page reclaim can make progress.
> > > + */
> > > +long reclaim_wait(long timeout)
> > > +{
> > > + long ret;
> > > + unsigned long start = jiffies;
> > > + DEFINE_WAIT(wait);
> > > +
> > > + prepare_to_wait(&reclaim_wqh, &wait, TASK_KILLABLE);
> > > + ret = io_schedule_timeout(timeout);
> > > + finish_wait(&reclaim_wqh, &wait);
> > > +
> > > + trace_writeback_reclaim_wait(jiffies_to_usecs(timeout),
> > > + jiffies_to_usecs(jiffies - start));
> > > +
> > > + return ret;
> > > +}
> > > +EXPORT_SYMBOL(reclaim_wait);
> > > +
> > > +void reclaim_rotated()
> > > +{
> > > + wait_queue_head_t *wqh = &reclaim_wqh;
> > > +
> > > + if (waitqueue_active(wqh))
> > > + wake_up(wqh);
> > > +}
> > > +
> >
> > Thank you.
> >
> > I like this approach. A nitpick is that this may wake up all waiters
> > in the system when a memcg is rotated.
>
> Thank you. It sure helps to start it simple :-)
>
> > How about wait_event() + condition by bitmap (using per memcg unique IDs.) ?
>
> I'm not sure how to manage the bitmap. The idea in my mind is to
>
> - maintain a memcg->pages_rotated counter
>
> - in reclaim_wait(), grab the current ->pages_rotated value before
> going to wait, compare it to the new value on every wakeup, and
> return to the user when seeing a different ->pages_rotated value.
> (this cannot stop waking up multiple tasks in the same memcg...)
>
> Does that sound reasonable?
>
Maybe. But there may be problem in looking up memcg from page at every
rotation. I think it's ok to start with a way ignoring per-memcg status.
Sorry for noise.
Thanks,
-Kame
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: reclaim the LRU lists full of dirty/writeback pages
2012-02-14 13:29 ` Jan Kara
@ 2012-02-16 4:00 ` Wu Fengguang
2012-02-16 12:44 ` Jan Kara
2012-02-17 16:41 ` Wu Fengguang
0 siblings, 2 replies; 33+ messages in thread
From: Wu Fengguang @ 2012-02-16 4:00 UTC (permalink / raw)
To: Jan Kara
Cc: Rik van Riel, Greg Thelen, bsingharora, Hugh Dickins,
Michal Hocko, linux-mm, Mel Gorman, Ying Han, hannes,
KAMEZAWA Hiroyuki, Minchan Kim
On Tue, Feb 14, 2012 at 02:29:50PM +0100, Jan Kara wrote:
> > > I wonder what happens if you run:
> > > mkdir /cgroup/x
> > > echo 100M > /cgroup/x/memory.limit_in_bytes
> > > echo $$ > /cgroup/x/tasks
> > >
> > > for (( i = 0; i < 2; i++ )); do
> > > mkdir /fs/d$i
> > > for (( j = 0; j < 5000; j++ )); do
> > > dd if=/dev/zero of=/fs/d$i/f$j bs=1k count=50
> > > done &
> > > done
> >
> > That's a very good case, thanks!
> >
> > > Because for small files the writearound logic won't help much...
> >
> > Right, it also means the native background work cannot be more I/O
> > efficient than the pageout works, except for the overheads of more
> > work items..
> Yes, that's true.
>
> > > Also the number of work items queued might become interesting.
> >
> > It turns out that the 1024 mempool reservations are not exhausted at
> > all (the below patch as a trace_printk on alloc failure and it didn't
> > trigger at all).
> >
> > Here is the representative iostat lines on XFS (full "iostat -kx 1 20" log attached):
> >
> > avg-cpu: %user %nice %system %iowait %steal %idle
> > 0.80 0.00 6.03 0.03 0.00 93.14
> >
> > Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
> > sda 0.00 205.00 0.00 163.00 0.00 16900.00 207.36 4.09 21.63 1.88 30.70
> >
> > The attached dirtied/written progress graph looks interesting.
> > Although the iostat disk utilization is low, the "dirtied" progress
> > line is pretty straight and there is no single congestion_wait event
> > in the trace log. Which makes me wonder if there are some unknown
> > blocking issues in the way.
> Interesting. I'd also expect we should block in reclaim path. How fast
> can dd threads progress when there is no cgroup involved?
I tried running the dd tasks in global context with
echo $((100<<20)) > /proc/sys/vm/dirty_bytes
and got mostly the same results on XFS:
avg-cpu: %user %nice %system %iowait %steal %idle
0.85 0.00 8.88 0.00 0.00 90.26
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 50.00 0.00 23036.00 921.44 9.59 738.02 7.38 36.90
avg-cpu: %user %nice %system %iowait %steal %idle
0.95 0.00 8.95 0.00 0.00 90.11
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 854.00 0.00 99.00 0.00 19552.00 394.99 34.14 87.98 3.82 37.80
Interestingly, ext4 shows comparable throughput, however is reporting
near 100% disk utilization:
avg-cpu: %user %nice %system %iowait %steal %idle
0.76 0.00 9.02 0.00 0.00 90.23
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 317.00 0.00 20956.00 132.21 28.57 82.71 3.16 100.10
avg-cpu: %user %nice %system %iowait %steal %idle
0.82 0.00 8.95 0.00 0.00 90.23
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 402.00 0.00 24388.00 121.33 21.09 58.55 2.42 97.40
avg-cpu: %user %nice %system %iowait %steal %idle
0.82 0.00 8.99 0.00 0.00 90.19
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 409.00 0.00 21996.00 107.56 15.25 36.74 2.30 94.10
And btrfs shows
avg-cpu: %user %nice %system %iowait %steal %idle
0.76 0.00 23.59 0.00 0.00 75.65
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 801.00 0.00 141.00 0.00 48984.00 694.81 41.08 291.36 6.11 86.20
avg-cpu: %user %nice %system %iowait %steal %idle
0.72 0.00 12.65 0.00 0.00 86.62
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 792.00 0.00 69.00 0.00 15288.00 443.13 22.74 69.35 4.09 28.20
avg-cpu: %user %nice %system %iowait %steal %idle
0.83 0.00 23.11 0.00 0.00 76.06
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 73.00 0.00 33280.00 911.78 22.09 548.58 8.10 59.10
> > > Another common case to test - run 'slapadd' command in each cgroup to
> > > create big LDAP database. That does pretty much random IO on a big mmaped
> > > DB file.
> >
> > I've not used this. Will it need some configuration and data feed?
> > fio looks more handy to me for emulating mmap random IO.
> Yes, fio can generate random mmap IO. It's just that this is a real life
> workload. So it is not completely random, it happens on several files and
> is also interleaved with other memory allocations from DB. I can send you
> the config files and data feed if you are interested.
I'm very interested, thank you!
> > > > +/*
> > > > + * schedule writeback on a range of inode pages.
> > > > + */
> > > > +static struct wb_writeback_work *
> > > > +bdi_flush_inode_range(struct backing_dev_info *bdi,
> > > > + struct inode *inode,
> > > > + pgoff_t offset,
> > > > + pgoff_t len,
> > > > + bool wait)
> > > > +{
> > > > + struct wb_writeback_work *work;
> > > > +
> > > > + if (!igrab(inode))
> > > > + return ERR_PTR(-ENOENT);
> > > One technical note here: If the inode is deleted while it is queued, this
> > > reference will keep it living until flusher thread gets to it. Then when
> > > flusher thread puts its reference, the inode will get deleted in flusher
> > > thread context. I don't see an immediate problem in that but it might be
> > > surprising sometimes. Another problem I see is that if you try to
> > > unmount the filesystem while the work item is queued, you'll get EBUSY for
> > > no apparent reason (for userspace).
> >
> > Yeah, we need to make umount work.
> The positive thing is that if the inode is reaped while the work item is
> queue, we know all that needed to be done is done. So we don't really need
> to pin the inode.
But I do need to make sure the *inode pointer does not point to some
invalid memory at work exec time. Is this possible without raising
->i_count?
> > And I find the pageout works seem to have some problems with ext4.
> > For example, this can be easily triggered with 10 dd tasks running
> > inside the 100MB limited memcg:
> So journal thread is getting stuck while committing transaction. Most
> likely waiting for some dd thread to stop a transaction so that commit can
> proceed. The processes waiting in start_this_handle() are just secondary
> effect resulting from the first problem. It might be interesting to get
> stack traces of all bloked processes when the journal thread is stuck.
For completeness of discussion, citing your conclusion on my private
data feed:
: We enter memcg reclaim from grab_cache_page_write_begin() and are
: waiting in congestion_wait(). Because grab_cache_page_write_begin() is
: called with transaction started, this blocks transaction from
: committing and subsequently blocks all other activity on the
: filesystem. The fact is this isn't new with your patches, just your
: changes or the fact that we are running in a memory constrained cgroup
: make this more visible.
Thanks,
Fengguang
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: reclaim the LRU lists full of dirty/writeback pages
2012-02-16 3:52 ` KAMEZAWA Hiroyuki
@ 2012-02-16 4:05 ` Wu Fengguang
0 siblings, 0 replies; 33+ messages in thread
From: Wu Fengguang @ 2012-02-16 4:05 UTC (permalink / raw)
To: KAMEZAWA Hiroyuki
Cc: Mel Gorman, Greg Thelen, Jan Kara, bsingharora, Hugh Dickins,
Michal Hocko, linux-mm, Ying Han, hannes, Rik van Riel,
Minchan Kim
On Thu, Feb 16, 2012 at 12:52:21PM +0900, KAMEZAWA Hiroyuki wrote:
> On Thu, 16 Feb 2012 11:04:15 +0800
> Wu Fengguang <fengguang.wu@intel.com> wrote:
>
> > On Thu, Feb 16, 2012 at 09:00:37AM +0900, KAMEZAWA Hiroyuki wrote:
> > > On Tue, 14 Feb 2012 21:18:12 +0800
> > > Wu Fengguang <fengguang.wu@intel.com> wrote:
> > >
> > > >
> > > > --- linux.orig/include/linux/backing-dev.h 2012-02-14 19:43:06.000000000 +0800
> > > > +++ linux/include/linux/backing-dev.h 2012-02-14 19:49:26.000000000 +0800
> > > > @@ -304,6 +304,8 @@ void clear_bdi_congested(struct backing_
> > > > void set_bdi_congested(struct backing_dev_info *bdi, int sync);
> > > > long congestion_wait(int sync, long timeout);
> > > > long wait_iff_congested(struct zone *zone, int sync, long timeout);
> > > > +long reclaim_wait(long timeout);
> > > > +void reclaim_rotated(void);
> > > >
> > > > static inline bool bdi_cap_writeback_dirty(struct backing_dev_info *bdi)
> > > > {
> > > > --- linux.orig/mm/backing-dev.c 2012-02-14 19:26:15.000000000 +0800
> > > > +++ linux/mm/backing-dev.c 2012-02-14 20:09:45.000000000 +0800
> > > > @@ -873,3 +873,38 @@ out:
> > > > return ret;
> > > > }
> > > > EXPORT_SYMBOL(wait_iff_congested);
> > > > +
> > > > +static DECLARE_WAIT_QUEUE_HEAD(reclaim_wqh);
> > > > +
> > > > +/**
> > > > + * reclaim_wait - wait for some pages being rotated to the LRU tail
> > > > + * @timeout: timeout in jiffies
> > > > + *
> > > > + * Wait until @timeout, or when some (typically PG_reclaim under writeback)
> > > > + * pages rotated to the LRU so that page reclaim can make progress.
> > > > + */
> > > > +long reclaim_wait(long timeout)
> > > > +{
> > > > + long ret;
> > > > + unsigned long start = jiffies;
> > > > + DEFINE_WAIT(wait);
> > > > +
> > > > + prepare_to_wait(&reclaim_wqh, &wait, TASK_KILLABLE);
> > > > + ret = io_schedule_timeout(timeout);
> > > > + finish_wait(&reclaim_wqh, &wait);
> > > > +
> > > > + trace_writeback_reclaim_wait(jiffies_to_usecs(timeout),
> > > > + jiffies_to_usecs(jiffies - start));
> > > > +
> > > > + return ret;
> > > > +}
> > > > +EXPORT_SYMBOL(reclaim_wait);
> > > > +
> > > > +void reclaim_rotated()
> > > > +{
> > > > + wait_queue_head_t *wqh = &reclaim_wqh;
> > > > +
> > > > + if (waitqueue_active(wqh))
> > > > + wake_up(wqh);
> > > > +}
> > > > +
> > >
> > > Thank you.
> > >
> > > I like this approach. A nitpick is that this may wake up all waiters
> > > in the system when a memcg is rotated.
> >
> > Thank you. It sure helps to start it simple :-)
> >
> > > How about wait_event() + condition by bitmap (using per memcg unique IDs.) ?
> >
> > I'm not sure how to manage the bitmap. The idea in my mind is to
> >
> > - maintain a memcg->pages_rotated counter
> >
> > - in reclaim_wait(), grab the current ->pages_rotated value before
> > going to wait, compare it to the new value on every wakeup, and
> > return to the user when seeing a different ->pages_rotated value.
> > (this cannot stop waking up multiple tasks in the same memcg...)
> >
> > Does that sound reasonable?
> >
>
> Maybe. But there may be problem in looking up memcg from page at every
> rotation. I think it's ok to start with a way ignoring per-memcg status.
> Sorry for noise.
When there comes such a need, we could take some sampled approach: for
each pagevec rotated, only dereference one of its pages and wakeup the
corresponding memcg task(s). That should do well enough in practice
even for random write patterns.
Thanks,
Fengguang
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: reclaim the LRU lists full of dirty/writeback pages
2012-02-14 15:51 ` Mel Gorman
@ 2012-02-16 9:50 ` Wu Fengguang
2012-02-16 17:31 ` Mel Gorman
0 siblings, 1 reply; 33+ messages in thread
From: Wu Fengguang @ 2012-02-16 9:50 UTC (permalink / raw)
To: Mel Gorman
Cc: Greg Thelen, Jan Kara, bsingharora, Hugh Dickins, Michal Hocko,
linux-mm, Ying Han, hannes, KAMEZAWA Hiroyuki, Rik van Riel,
Minchan Kim
On Tue, Feb 14, 2012 at 03:51:24PM +0000, Mel Gorman wrote:
> On Tue, Feb 14, 2012 at 09:18:12PM +0800, Wu Fengguang wrote:
> > > For the OOM problem, a more reasonable stopgap might be to identify when
> > > a process is scanning a memcg at high priority and encountered all
> > > PageReclaim with no forward progress and to congestion_wait() if that
> > > situation occurs. A preferable way would be to wait until the flusher
> > > wakes up a waiter on PageReclaim pages to be written out because we want
> > > to keep moving way from congestion_wait() if at all possible.
> >
> > Good points! Below is the more serious page reclaim changes.
> >
> > The dirty/writeback pages may often come close to each other in the
> > LRU list, so the local test during a 32-page scan may still trigger
> > reclaim waits unnecessarily.
>
> Yes, this is particularly the case when writing back to USB. It is not
> unusual that all dirty pages under writeback are backed by USB and at the
> end of the LRU. Right now what happens is that reclaimers see higher CPU
> usage as they scan over these pages uselessly. If the wrong choice is
> made on how to throttle, we'll see yet more variants of the "system
> responsiveness drops when writing to USB".
Yes, USB is an important case to support. I'd imagine the heavy USB
writes typically happen in desktops and run *outside* of any memcg.
So they'll typically take <= 20% memory in the zone. As long as we
start the PG_reclaim throttling only when above the 20% dirty
threshold (ie. on zone_dirty_ok()), the USB case should be safe.
> > Some global information on the percent
> > of dirty/writeback pages in the LRU list may help. Anyway the added
> > tests should still be much better than no protection.
> >
>
> You can tell how many dirty pages and writeback pages are in the zone
> already.
Right. I changed the test to
+ if (nr_pgreclaim && nr_pgreclaim >= (nr_taken >> (DEF_PRIORITY-priority)) &&
+ (!global_reclaim(sc) || !zone_dirty_ok(zone)))
+ reclaim_wait(HZ/10);
And I'd prefer to use a higher threshold than the default 20% for the
above zone_dirty_ok() test, so that when Johannes' zone dirty
balancing does the job fine, PG_reclaim based page reclaim throttling
won't happen at all.
> > A global wait queue and reclaim_wait() is introduced. The waiters will
> > be wakeup when pages are rotated by end_page_writeback() or lru drain.
> >
> > I have to say its effectiveness depends on the filesystem... ext4
> > and btrfs do fluent IO completions, so reclaim_wait() works pretty
> > well:
> > dd-14560 [017] .... 1360.894605: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=10000
> > dd-14560 [017] .... 1360.904456: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=8000
> > dd-14560 [017] .... 1360.908293: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
> > dd-14560 [017] .... 1360.923960: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=15000
> > dd-14560 [017] .... 1360.927810: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
> > dd-14560 [017] .... 1360.931656: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
> > dd-14560 [017] .... 1360.943503: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=10000
> > dd-14560 [017] .... 1360.953289: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=7000
> > dd-14560 [017] .... 1360.957177: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
> > dd-14560 [017] .... 1360.972949: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=15000
> >
> > However XFS does IO completions in very large batches (there may be
> > only several big IO completions in one second). So reclaim_wait()
> > mostly end up waiting to the full HZ/10 timeout:
> >
> > dd-4177 [008] .... 866.367661: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > dd-4177 [010] .... 866.567583: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > dd-4177 [012] .... 866.767458: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > dd-4177 [013] .... 866.867419: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > dd-4177 [008] .... 867.167266: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > dd-4177 [010] .... 867.367168: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > dd-4177 [012] .... 867.818950: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > dd-4177 [013] .... 867.918905: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > dd-4177 [013] .... 867.971657: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=52000
> > dd-4177 [013] .... 867.971812: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=0
> > dd-4177 [008] .... 868.355700: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > dd-4177 [010] .... 868.700515: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> >
>
> And where people will get hit by regressions in this area is writing to
> vfat and in more rare cases ntfs on USB stick.
vfat IO completions seem to lie somewhere between ext4 and xfs:
<...>-46385 [010] .... 143570.714470: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
<...>-46385 [008] .... 143570.752391: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=12000
<...>-46385 [008] .... 143570.937327: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=52000
<...>-46385 [010] .... 143571.160252: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
<...>-46385 [011] .... 143571.286197: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
<...>-46385 [008] .... 143571.329644: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=15000
<...>-46385 [008] .... 143571.475433: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=18000
<...>-46385 [008] .... 143571.653461: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=52000
<...>-46385 [008] .... 143571.839949: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=56000
<...>-46385 [010] .... 143572.060816: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
<...>-46385 [011] .... 143572.185754: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
<...>-46385 [008] .... 143572.212522: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=1000
<...>-46385 [008] .... 143572.217825: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
<...>-46385 [008] .... 143572.312395: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=91000
<...>-46385 [008] .... 143572.315122: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=1000
<...>-46385 [009] .... 143572.433630: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
<...>-46385 [010] .... 143572.534569: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
and has lower throughput
avg-cpu: %user %nice %system %iowait %steal %idle
0.03 0.00 3.88 2.22 0.00 93.86
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 34.00 20.00 82.00 10.00 34137.50 669.56 8.09 79.34 5.97 60.90
I'm yet to get a USB stick for the vfat-on-USB test.
> > > Another possibility would be to relook at LRU_IMMEDIATE but right now it
> > > requires a page flag and I haven't devised a way around that. Besides,
> > > it would only address the problem of PageREclaim pages being encountered,
> > > it would not handle the case where a memcg was filled with PageReclaim pages.
> >
> > I also considered things like LRU_IMMEDIATE, however got no clear idea yet.
> > Since the simple "wait on PG_reclaim" approach appears to work for this
> > memcg dd case, it effectively disables me to think any further ;-)
> >
>
> Test with interactive use while writing heavily to a USB stick.
Sure.
> > @@ -813,6 +815,10 @@ static unsigned long shrink_page_list(st
> >
> > if (PageWriteback(page)) {
> > nr_writeback++;
> > + if (PageReclaim(page))
> > + nr_pgreclaim++;
> > + else
> > + SetPageReclaim(page);
> > /*
>
> This check is unexpected. We already SetPageReclaim when queuing pages for
> IO from reclaim context and if dirty pages are encountered during the LRU
> scan that cannot be queued for IO. How often is it that nr_pgreclaim !=
> nr_writeback and by how much do they differ?
Quite often, I suspect. The pageout writeback works do 1-8MB write
around which may start I/O a bit earlier than the covered pages are
encountered by page reclaim. ext4 forces 128MB write chunk size, which
further increases the opportunities.
> > * Synchronous reclaim cannot queue pages for
> > * writeback due to the possibility of stack overflow
> > @@ -874,12 +880,15 @@ static unsigned long shrink_page_list(st
> > nr_dirty++;
> >
> > /*
> > - * Only kswapd can writeback filesystem pages to
> > - * avoid risk of stack overflow but do not writeback
> > - * unless under significant pressure.
> > + * run into the visited page again: we are scanning
> > + * faster than the flusher can writeout dirty pages
> > */
>
> which in itself is not an abnormal condition. We get into this situation
> when writing to USB. Dirty throttling stops too much memory getting dirtied
> but that does not mean we should throttle instead of reclaiming clean pages.
>
> That's why I worry that if this is aimed at fixing a memcg problem, it
> will have the impact of making interactive performance on normal systems
> worse.
You are right. This patch only addresses the pageout I/O efficiency
and dirty throttling problems for a fully dirtied LRU. Next step, I'll
think about the interactive performance problem for a less dirtied LRU.
> > @@ -1087,8 +1097,10 @@ int __isolate_lru_page(struct page *page
> > */
> > if (mode & (ISOLATE_CLEAN|ISOLATE_ASYNC_MIGRATE)) {
> > /* All the caller can do on PageWriteback is block */
> > - if (PageWriteback(page))
> > + if (PageWriteback(page)) {
> > + SetPageReclaim(page);
> > return ret;
> > + }
> >
>
> This hunk means that if async compaction (common for THP) encounters a page
> under writeback, it will still skip it but mark it for immediate reclaim
> after IO completes. This will have the impact that compaction causes an
> abnormally high number of pages to be reclaimed.
Sorry I overlooked that, will drop it. isolate_migratepages() walks
by PFN and the opportunistic peak at the writeback pages should not
make it rotated (and disturb its LRU order) on I/O completion.
> > @@ -1608,6 +1621,8 @@ shrink_inactive_list(unsigned long nr_to
> > */
> > if (nr_writeback && nr_writeback >= (nr_taken >> (DEF_PRIORITY-priority)))
> > wait_iff_congested(zone, BLK_RW_ASYNC, HZ/10);
> > + if (nr_pgreclaim && nr_pgreclaim >= (nr_taken >> (DEF_PRIORITY-priority)))
> > + reclaim_wait(HZ/10);
> >
>
> We risk going to sleep too easily when USB-backed pages are at the end of
> the LRU list. Note that the nr_writeback check only goes to sleep if it
> detects that the underlying storage is also congested. In contrast, it
> will take very few PageReclaim pages at teh end of the LRU to cause the
> process to sleep when it instead should find clean pages to discard.
Right.
> If the intention is to avoid memcg going OOM prematurely, the
> nr_pgreclaim value needs to be treated at a higher level that records
> how many PageReclaim pages were encountered. If no progress was made
> because all the pages were PageReclaim, then throttle and return 1 to
> the page allocator where it will retry the allocation without going OOM
> after some pages have been cleaned and reclaimed.
Agreed in general, but changed to this test for now, which is made a
bit more global wise with the use of zone_dirty_ok().
memcg is ignored due to no dirty accounting (Greg has the patch though).
And even zone_dirty_ok() may be inaccurate for the global reclaim, if
some memcgs are skipped by the global reclaim by the memcg soft limit.
But anyway, it's a handy hack for now. I'm looking into some more
radical changes to put most dirty/writeback pages into a standalone
LRU list (in addition to your LRU_IMMEDIATE, which I think is a good
idea) for addressing the clustered way they tend to lie in the
inactive LRU list.
+ if (nr_pgreclaim && nr_pgreclaim >= (nr_taken >> (DEF_PRIORITY-priority)) &&
+ (!global_reclaim(sc) || !zone_dirty_ok(zone)))
+ reclaim_wait(HZ/10);
Thanks,
Fengguang
---
Subject: writeback: introduce the pageout work
Date: Thu Jul 29 14:41:19 CST 2010
This relays file pageout IOs to the flusher threads.
The ultimate target is to gracefully handle the LRU lists full of
dirty/writeback pages.
1) I/O efficiency
The flusher will piggy back the nearby ~10ms worth of dirty pages for I/O.
This takes advantage of the time/spacial locality in most workloads: the
nearby pages of one file are typically populated into the LRU at the same
time, hence will likely be close to each other in the LRU list. Writing
them in one shot helps clean more pages effectively for page reclaim.
2) OOM avoidance and scan rate control
Typically we do LRU scan w/o rate control and quickly get enough clean
pages for the LRU lists not full of dirty pages.
Or we can still get a number of freshly cleaned pages (moved to LRU tail
by end_page_writeback()) when the queued pageout I/O is completed within
tens of milli-seconds.
However if the LRU list is small and full of dirty pages, it can be
quickly fully scanned and go OOM before the flusher manages to clean
enough pages.
A simple yet reliable scheme is employed to avoid OOM and keep scan rate
in sync with the I/O rate:
if (PageReclaim(page))
congestion_wait(HZ/10);
PG_reclaim plays the key role. When dirty pages are encountered, we
queue I/O for it, set PG_reclaim and put it back to the LRU head.
So if PG_reclaim pages are encountered again, it means the dirty page
has not yet been cleaned by the flusher after a full zone scan. It
indicates we are scanning more fast than I/O and shall take a snap.
The runtime behavior on a fully dirtied small LRU list would be:
It will start with a quick scan of the list, queuing all pages for I/O.
Then the scan will be slowed down by the PG_reclaim pages *adaptively*
to match the I/O bandwidth.
3) writeback work coordinations
To avoid memory allocations at page reclaim, a mempool for struct
wb_writeback_work is created.
wakeup_flusher_threads() is removed because it can easily delay the
more oriented pageout works and even exhaust the mempool reservations.
It's also found to not I/O efficient by frequently submitting writeback
works with small ->nr_pages.
Background/periodic works will quit automatically, so as to clean the
pages under reclaim ASAP. However for now the sync work can still block
us for long time.
Jan Kara: limit the search scope. Note that the limited search and work
pool is not a big problem: 1000 IOs under flight are typically more than
enough to saturate the disk. And the overheads of searching in the work
list didn't even show up in the perf report.
4) test case
Run 2 dd tasks in a 100MB memcg (a very handy test case from Greg Thelen):
mkdir /cgroup/x
echo 100M > /cgroup/x/memory.limit_in_bytes
echo $$ > /cgroup/x/tasks
for i in `seq 2`
do
dd if=/dev/zero of=/fs/f$i bs=1k count=1M &
done
Before patch, the dd tasks are quickly OOM killed.
After patch, they run well with reasonably good performance and overheads:
1073741824 bytes (1.1 GB) copied, 22.2196 s, 48.3 MB/s
1073741824 bytes (1.1 GB) copied, 22.4675 s, 47.8 MB/s
iostat -kx 1
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 178.00 0.00 89568.00 1006.38 74.35 417.71 4.80 85.40
sda 0.00 2.00 0.00 191.00 0.00 94428.00 988.77 53.34 219.03 4.34 82.90
sda 0.00 20.00 0.00 196.00 0.00 97712.00 997.06 71.11 337.45 4.77 93.50
sda 0.00 5.00 0.00 175.00 0.00 84648.00 967.41 54.03 316.44 5.06 88.60
sda 0.00 0.00 0.00 186.00 0.00 92432.00 993.89 56.22 267.54 5.38 100.00
sda 0.00 1.00 0.00 183.00 0.00 90156.00 985.31 37.99 325.55 4.33 79.20
sda 0.00 0.00 0.00 175.00 0.00 88692.00 1013.62 48.70 218.43 4.69 82.10
sda 0.00 0.00 0.00 196.00 0.00 97528.00 995.18 43.38 236.87 5.10 100.00
sda 0.00 0.00 0.00 179.00 0.00 88648.00 990.48 45.83 285.43 5.59 100.00
sda 0.00 0.00 0.00 178.00 0.00 88500.00 994.38 28.28 158.89 4.99 88.80
sda 0.00 0.00 0.00 194.00 0.00 95852.00 988.16 32.58 167.39 5.15 100.00
sda 0.00 2.00 0.00 215.00 0.00 105996.00 986.01 41.72 201.43 4.65 100.00
sda 0.00 4.00 0.00 173.00 0.00 84332.00 974.94 50.48 260.23 5.76 99.60
sda 0.00 0.00 0.00 182.00 0.00 90312.00 992.44 36.83 212.07 5.49 100.00
sda 0.00 8.00 0.00 195.00 0.00 95940.50 984.01 50.18 221.06 5.13 100.00
sda 0.00 1.00 0.00 220.00 0.00 108852.00 989.56 40.99 202.68 4.55 100.00
sda 0.00 2.00 0.00 161.00 0.00 80384.00 998.56 37.19 268.49 6.21 100.00
sda 0.00 4.00 0.00 182.00 0.00 90830.00 998.13 50.58 239.77 5.49 100.00
sda 0.00 0.00 0.00 197.00 0.00 94877.00 963.22 36.68 196.79 5.08 100.00
avg-cpu: %user %nice %system %iowait %steal %idle
0.25 0.00 15.08 33.92 0.00 50.75
0.25 0.00 14.54 35.09 0.00 50.13
0.50 0.00 13.57 32.41 0.00 53.52
0.50 0.00 11.28 36.84 0.00 51.38
0.50 0.00 15.75 32.00 0.00 51.75
0.50 0.00 10.50 34.00 0.00 55.00
0.50 0.00 17.63 27.46 0.00 54.41
0.50 0.00 15.08 30.90 0.00 53.52
0.50 0.00 11.28 32.83 0.00 55.39
0.75 0.00 16.79 26.82 0.00 55.64
0.50 0.00 16.08 29.15 0.00 54.27
0.50 0.00 13.50 30.50 0.00 55.50
0.50 0.00 14.32 35.18 0.00 50.00
0.50 0.00 12.06 33.92 0.00 53.52
0.50 0.00 17.29 30.58 0.00 51.63
0.50 0.00 15.08 29.65 0.00 54.77
0.50 0.00 12.53 29.32 0.00 57.64
0.50 0.00 15.29 31.83 0.00 52.38
The global dd numbers for comparison:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 189.00 0.00 95752.00 1013.25 143.09 684.48 5.29 100.00
sda 0.00 0.00 0.00 208.00 0.00 105480.00 1014.23 143.06 733.29 4.81 100.00
sda 0.00 0.00 0.00 161.00 0.00 81924.00 1017.69 141.71 757.79 6.21 100.00
sda 0.00 0.00 0.00 217.00 0.00 109580.00 1009.95 143.09 749.55 4.61 100.10
sda 0.00 0.00 0.00 187.00 0.00 94728.00 1013.13 144.31 773.67 5.35 100.00
sda 0.00 0.00 0.00 189.00 0.00 95752.00 1013.25 144.14 742.00 5.29 100.00
sda 0.00 0.00 0.00 177.00 0.00 90032.00 1017.31 143.32 656.59 5.65 100.00
sda 0.00 0.00 0.00 215.00 0.00 108640.00 1010.60 142.90 817.54 4.65 100.00
sda 0.00 2.00 0.00 166.00 0.00 83858.00 1010.34 143.64 808.61 6.02 100.00
sda 0.00 0.00 0.00 186.00 0.00 92813.00 997.99 141.18 736.95 5.38 100.00
sda 0.00 0.00 0.00 206.00 0.00 104456.00 1014.14 146.27 729.33 4.85 100.00
sda 0.00 0.00 0.00 213.00 0.00 107024.00 1004.92 143.25 705.70 4.69 100.00
sda 0.00 0.00 0.00 188.00 0.00 95748.00 1018.60 141.82 764.78 5.32 100.00
avg-cpu: %user %nice %system %iowait %steal %idle
0.51 0.00 11.22 52.30 0.00 35.97
0.25 0.00 10.15 52.54 0.00 37.06
0.25 0.00 5.01 56.64 0.00 38.10
0.51 0.00 15.15 43.94 0.00 40.40
0.25 0.00 12.12 48.23 0.00 39.39
0.51 0.00 11.20 53.94 0.00 34.35
0.26 0.00 9.72 51.41 0.00 38.62
0.76 0.00 9.62 50.63 0.00 38.99
0.51 0.00 10.46 53.32 0.00 35.71
0.51 0.00 9.41 51.91 0.00 38.17
0.25 0.00 10.69 49.62 0.00 39.44
0.51 0.00 12.21 52.67 0.00 34.61
0.51 0.00 11.45 53.18 0.00 34.86
XXX: commit NFS unstable pages via write_inode()
XXX: the added congestion_wait() may be undesirable in some situations
CC: Jan Kara <jack@suse.cz>
CC: Mel Gorman <mgorman@suse.de>
Acked-by: Rik van Riel <riel@redhat.com>
CC: Greg Thelen <gthelen@google.com>
CC: Minchan Kim <minchan.kim@gmail.com>
Signed-off-by: Wu Fengguang <fengguang.wu@intel.com>
---
fs/fs-writeback.c | 169 ++++++++++++++++++++++++++++-
include/linux/backing-dev.h | 2
include/linux/writeback.h | 4
include/trace/events/writeback.h | 19 ++-
mm/backing-dev.c | 35 ++++++
mm/swap.c | 1
mm/vmscan.c | 32 +++--
7 files changed, 245 insertions(+), 17 deletions(-)
- move congestion_wait() out of the page lock: it's blocking btrfs lock_delalloc_pages()
--- linux.orig/include/linux/backing-dev.h 2012-02-14 20:11:21.000000000 +0800
+++ linux/include/linux/backing-dev.h 2012-02-15 12:34:24.000000000 +0800
@@ -304,6 +304,8 @@ void clear_bdi_congested(struct backing_
void set_bdi_congested(struct backing_dev_info *bdi, int sync);
long congestion_wait(int sync, long timeout);
long wait_iff_congested(struct zone *zone, int sync, long timeout);
+long reclaim_wait(long timeout);
+void reclaim_rotated(void);
static inline bool bdi_cap_writeback_dirty(struct backing_dev_info *bdi)
{
--- linux.orig/mm/backing-dev.c 2012-02-14 20:11:21.000000000 +0800
+++ linux/mm/backing-dev.c 2012-02-15 12:34:19.000000000 +0800
@@ -873,3 +873,38 @@ out:
return ret;
}
EXPORT_SYMBOL(wait_iff_congested);
+
+static DECLARE_WAIT_QUEUE_HEAD(reclaim_wqh);
+
+/**
+ * reclaim_wait - wait for some pages being rotated to the LRU tail
+ * @timeout: timeout in jiffies
+ *
+ * Wait until @timeout, or when some (typically PG_reclaim under writeback)
+ * pages rotated to the LRU so that page reclaim can make progress.
+ */
+long reclaim_wait(long timeout)
+{
+ long ret;
+ unsigned long start = jiffies;
+ DEFINE_WAIT(wait);
+
+ prepare_to_wait(&reclaim_wqh, &wait, TASK_KILLABLE);
+ ret = io_schedule_timeout(timeout);
+ finish_wait(&reclaim_wqh, &wait);
+
+ trace_writeback_reclaim_wait(jiffies_to_usecs(timeout),
+ jiffies_to_usecs(jiffies - start));
+
+ return ret;
+}
+EXPORT_SYMBOL(reclaim_wait);
+
+void reclaim_rotated()
+{
+ wait_queue_head_t *wqh = &reclaim_wqh;
+
+ if (waitqueue_active(wqh))
+ wake_up(wqh);
+}
+
--- linux.orig/mm/swap.c 2012-02-14 20:11:21.000000000 +0800
+++ linux/mm/swap.c 2012-02-15 12:27:35.000000000 +0800
@@ -253,6 +253,7 @@ static void pagevec_move_tail(struct pag
pagevec_lru_move_fn(pvec, pagevec_move_tail_fn, &pgmoved);
__count_vm_events(PGROTATED, pgmoved);
+ reclaim_rotated();
}
/*
--- linux.orig/mm/vmscan.c 2012-02-14 20:11:21.000000000 +0800
+++ linux/mm/vmscan.c 2012-02-16 17:23:17.000000000 +0800
@@ -767,7 +767,8 @@ static unsigned long shrink_page_list(st
struct scan_control *sc,
int priority,
unsigned long *ret_nr_dirty,
- unsigned long *ret_nr_writeback)
+ unsigned long *ret_nr_writeback,
+ unsigned long *ret_nr_pgreclaim)
{
LIST_HEAD(ret_pages);
LIST_HEAD(free_pages);
@@ -776,6 +777,7 @@ static unsigned long shrink_page_list(st
unsigned long nr_congested = 0;
unsigned long nr_reclaimed = 0;
unsigned long nr_writeback = 0;
+ unsigned long nr_pgreclaim = 0;
cond_resched();
@@ -813,6 +815,10 @@ static unsigned long shrink_page_list(st
if (PageWriteback(page)) {
nr_writeback++;
+ if (PageReclaim(page))
+ nr_pgreclaim++;
+ else
+ SetPageReclaim(page);
/*
* Synchronous reclaim cannot queue pages for
* writeback due to the possibility of stack overflow
@@ -874,12 +880,15 @@ static unsigned long shrink_page_list(st
nr_dirty++;
/*
- * Only kswapd can writeback filesystem pages to
- * avoid risk of stack overflow but do not writeback
- * unless under significant pressure.
+ * run into the visited page again: we are scanning
+ * faster than the flusher can writeout dirty pages
*/
- if (page_is_file_cache(page) &&
- (!current_is_kswapd() || priority >= DEF_PRIORITY - 2)) {
+ if (page_is_file_cache(page) && PageReclaim(page)) {
+ nr_pgreclaim++;
+ goto keep_locked;
+ }
+ if (page_is_file_cache(page) && mapping &&
+ flush_inode_page(mapping, page, false) >= 0) {
/*
* Immediately reclaim when written back.
* Similar in principal to deactivate_page()
@@ -1028,6 +1037,7 @@ keep_lumpy:
count_vm_events(PGACTIVATE, pgactivate);
*ret_nr_dirty += nr_dirty;
*ret_nr_writeback += nr_writeback;
+ *ret_nr_pgreclaim += nr_pgreclaim;
return nr_reclaimed;
}
@@ -1509,6 +1519,7 @@ shrink_inactive_list(unsigned long nr_to
unsigned long nr_file;
unsigned long nr_dirty = 0;
unsigned long nr_writeback = 0;
+ unsigned long nr_pgreclaim = 0;
isolate_mode_t reclaim_mode = ISOLATE_INACTIVE;
struct zone *zone = mz->zone;
@@ -1559,13 +1570,13 @@ shrink_inactive_list(unsigned long nr_to
spin_unlock_irq(&zone->lru_lock);
nr_reclaimed = shrink_page_list(&page_list, mz, sc, priority,
- &nr_dirty, &nr_writeback);
+ &nr_dirty, &nr_writeback, &nr_pgreclaim);
/* Check if we should syncronously wait for writeback */
if (should_reclaim_stall(nr_taken, nr_reclaimed, priority, sc)) {
set_reclaim_mode(priority, sc, true);
nr_reclaimed += shrink_page_list(&page_list, mz, sc,
- priority, &nr_dirty, &nr_writeback);
+ priority, &nr_dirty, &nr_writeback, &nr_pgreclaim);
}
spin_lock_irq(&zone->lru_lock);
@@ -1608,6 +1619,9 @@ shrink_inactive_list(unsigned long nr_to
*/
if (nr_writeback && nr_writeback >= (nr_taken >> (DEF_PRIORITY-priority)))
wait_iff_congested(zone, BLK_RW_ASYNC, HZ/10);
+ if (nr_pgreclaim && nr_pgreclaim >= (nr_taken >> (DEF_PRIORITY-priority)) &&
+ (!global_reclaim(sc) || !zone_dirty_ok(zone)))
+ reclaim_wait(HZ/10);
trace_mm_vmscan_lru_shrink_inactive(zone->zone_pgdat->node_id,
zone_idx(zone),
@@ -2382,8 +2396,6 @@ static unsigned long do_try_to_free_page
*/
writeback_threshold = sc->nr_to_reclaim + sc->nr_to_reclaim / 2;
if (total_scanned > writeback_threshold) {
- wakeup_flusher_threads(laptop_mode ? 0 : total_scanned,
- WB_REASON_TRY_TO_FREE_PAGES);
sc->may_writepage = 1;
}
--- linux.orig/fs/fs-writeback.c 2012-02-14 20:11:21.000000000 +0800
+++ linux/fs/fs-writeback.c 2012-02-15 12:27:35.000000000 +0800
@@ -41,6 +41,8 @@ struct wb_writeback_work {
long nr_pages;
struct super_block *sb;
unsigned long *older_than_this;
+ struct inode *inode;
+ pgoff_t offset;
enum writeback_sync_modes sync_mode;
unsigned int tagged_writepages:1;
unsigned int for_kupdate:1;
@@ -65,6 +67,27 @@ struct wb_writeback_work {
*/
int nr_pdflush_threads;
+static mempool_t *wb_work_mempool;
+
+static void *wb_work_alloc(gfp_t gfp_mask, void *pool_data)
+{
+ /*
+ * bdi_flush_inode_range() may be called on page reclaim
+ */
+ if (current->flags & PF_MEMALLOC)
+ return NULL;
+
+ return kmalloc(sizeof(struct wb_writeback_work), gfp_mask);
+}
+
+static __init int wb_work_init(void)
+{
+ wb_work_mempool = mempool_create(1024,
+ wb_work_alloc, mempool_kfree, NULL);
+ return wb_work_mempool ? 0 : -ENOMEM;
+}
+fs_initcall(wb_work_init);
+
/**
* writeback_in_progress - determine whether there is writeback in progress
* @bdi: the device's backing_dev_info structure.
@@ -129,7 +152,7 @@ __bdi_start_writeback(struct backing_dev
* This is WB_SYNC_NONE writeback, so if allocation fails just
* wakeup the thread for old dirty data writeback
*/
- work = kzalloc(sizeof(*work), GFP_ATOMIC);
+ work = mempool_alloc(wb_work_mempool, GFP_NOWAIT);
if (!work) {
if (bdi->wb.task) {
trace_writeback_nowork(bdi);
@@ -138,6 +161,7 @@ __bdi_start_writeback(struct backing_dev
return;
}
+ memset(work, 0, sizeof(*work));
work->sync_mode = WB_SYNC_NONE;
work->nr_pages = nr_pages;
work->range_cyclic = range_cyclic;
@@ -186,6 +210,125 @@ void bdi_start_background_writeback(stru
spin_unlock_bh(&bdi->wb_lock);
}
+static bool extend_writeback_range(struct wb_writeback_work *work,
+ pgoff_t offset,
+ unsigned long write_around_pages)
+{
+ pgoff_t end = work->offset + work->nr_pages;
+
+ if (offset >= work->offset && offset < end)
+ return true;
+
+ /*
+ * for sequential workloads with good locality, include up to 8 times
+ * more data in one chunk
+ */
+ if (work->nr_pages >= 8 * write_around_pages)
+ return false;
+
+ /* the unsigned comparison helps eliminate one compare */
+ if (work->offset - offset < write_around_pages) {
+ work->nr_pages += write_around_pages;
+ work->offset -= write_around_pages;
+ return true;
+ }
+
+ if (offset - end < write_around_pages) {
+ work->nr_pages += write_around_pages;
+ return true;
+ }
+
+ return false;
+}
+
+/*
+ * schedule writeback on a range of inode pages.
+ */
+static struct wb_writeback_work *
+bdi_flush_inode_range(struct backing_dev_info *bdi,
+ struct inode *inode,
+ pgoff_t offset,
+ pgoff_t len,
+ bool wait)
+{
+ struct wb_writeback_work *work;
+
+ if (!igrab(inode))
+ return ERR_PTR(-ENOENT);
+
+ work = mempool_alloc(wb_work_mempool, wait ? GFP_NOIO : GFP_NOWAIT);
+ if (!work) {
+ trace_printk("wb_work_mempool alloc fail\n");
+ return ERR_PTR(-ENOMEM);
+ }
+
+ memset(work, 0, sizeof(*work));
+ work->sync_mode = WB_SYNC_NONE;
+ work->inode = inode;
+ work->offset = offset;
+ work->nr_pages = len;
+ work->reason = WB_REASON_PAGEOUT;
+
+ bdi_queue_work(bdi, work);
+
+ return work;
+}
+
+/*
+ * Called by page reclaim code to flush the dirty page ASAP. Do write-around to
+ * improve IO throughput. The nearby pages will have good chance to reside in
+ * the same LRU list that vmscan is working on, and even close to each other
+ * inside the LRU list in the common case of sequential read/write.
+ *
+ * ret > 0: success, found/reused a previous writeback work
+ * ret = 0: success, allocated/queued a new writeback work
+ * ret < 0: failed
+ */
+long flush_inode_page(struct address_space *mapping,
+ struct page *page,
+ bool wait)
+{
+ struct backing_dev_info *bdi = mapping->backing_dev_info;
+ struct inode *inode = mapping->host;
+ struct wb_writeback_work *work;
+ unsigned long write_around_pages;
+ pgoff_t offset = page->index;
+ int i;
+ long ret = 0;
+
+ if (unlikely(!inode))
+ return -ENOENT;
+
+ /*
+ * piggy back 8-15ms worth of data
+ */
+ write_around_pages = bdi->avg_write_bandwidth + MIN_WRITEBACK_PAGES;
+ write_around_pages = rounddown_pow_of_two(write_around_pages) >> 6;
+
+ i = 1;
+ spin_lock_bh(&bdi->wb_lock);
+ list_for_each_entry_reverse(work, &bdi->work_list, list) {
+ if (work->inode != inode)
+ continue;
+ if (extend_writeback_range(work, offset, write_around_pages)) {
+ ret = i;
+ break;
+ }
+ if (i++ > 100) /* limit search depth */
+ break;
+ }
+ spin_unlock_bh(&bdi->wb_lock);
+
+ if (!ret) {
+ offset = round_down(offset, write_around_pages);
+ work = bdi_flush_inode_range(bdi, inode,
+ offset, write_around_pages, wait);
+ if (IS_ERR(work))
+ ret = PTR_ERR(work);
+ }
+ return ret;
+}
+
/*
* Remove the inode from the writeback list it is on.
*/
@@ -833,6 +976,23 @@ static unsigned long get_nr_dirty_pages(
get_nr_dirty_inodes();
}
+static long wb_flush_inode(struct bdi_writeback *wb,
+ struct wb_writeback_work *work)
+{
+ struct writeback_control wbc = {
+ .sync_mode = WB_SYNC_NONE,
+ .nr_to_write = LONG_MAX,
+ .range_start = work->offset << PAGE_CACHE_SHIFT,
+ .range_end = (work->offset + work->nr_pages - 1)
+ << PAGE_CACHE_SHIFT,
+ };
+
+ do_writepages(work->inode->i_mapping, &wbc);
+ iput(work->inode);
+
+ return LONG_MAX - wbc.nr_to_write;
+}
+
static long wb_check_background_flush(struct bdi_writeback *wb)
{
if (over_bground_thresh(wb->bdi)) {
@@ -905,7 +1065,10 @@ long wb_do_writeback(struct bdi_writebac
trace_writeback_exec(bdi, work);
- wrote += wb_writeback(wb, work);
+ if (work->inode)
+ wrote += wb_flush_inode(wb, work);
+ else
+ wrote += wb_writeback(wb, work);
/*
* Notify the caller of completion if this is a synchronous
@@ -914,7 +1077,7 @@ long wb_do_writeback(struct bdi_writebac
if (work->done)
complete(work->done);
else
- kfree(work);
+ mempool_free(work, wb_work_mempool);
}
/*
--- linux.orig/include/trace/events/writeback.h 2012-02-14 20:11:22.000000000 +0800
+++ linux/include/trace/events/writeback.h 2012-02-15 12:27:35.000000000 +0800
@@ -23,7 +23,7 @@
#define WB_WORK_REASON \
{WB_REASON_BACKGROUND, "background"}, \
- {WB_REASON_TRY_TO_FREE_PAGES, "try_to_free_pages"}, \
+ {WB_REASON_PAGEOUT, "pageout"}, \
{WB_REASON_SYNC, "sync"}, \
{WB_REASON_PERIODIC, "periodic"}, \
{WB_REASON_LAPTOP_TIMER, "laptop_timer"}, \
@@ -45,6 +45,8 @@ DECLARE_EVENT_CLASS(writeback_work_class
__field(int, range_cyclic)
__field(int, for_background)
__field(int, reason)
+ __field(unsigned long, ino)
+ __field(unsigned long, offset)
),
TP_fast_assign(
strncpy(__entry->name, dev_name(bdi->dev), 32);
@@ -55,9 +57,11 @@ DECLARE_EVENT_CLASS(writeback_work_class
__entry->range_cyclic = work->range_cyclic;
__entry->for_background = work->for_background;
__entry->reason = work->reason;
+ __entry->ino = work->inode ? work->inode->i_ino : 0;
+ __entry->offset = work->offset;
),
TP_printk("bdi %s: sb_dev %d:%d nr_pages=%ld sync_mode=%d "
- "kupdate=%d range_cyclic=%d background=%d reason=%s",
+ "kupdate=%d range_cyclic=%d background=%d reason=%s ino=%lu offset=%lu",
__entry->name,
MAJOR(__entry->sb_dev), MINOR(__entry->sb_dev),
__entry->nr_pages,
@@ -65,7 +69,9 @@ DECLARE_EVENT_CLASS(writeback_work_class
__entry->for_kupdate,
__entry->range_cyclic,
__entry->for_background,
- __print_symbolic(__entry->reason, WB_WORK_REASON)
+ __print_symbolic(__entry->reason, WB_WORK_REASON),
+ __entry->ino,
+ __entry->offset
)
);
#define DEFINE_WRITEBACK_WORK_EVENT(name) \
@@ -437,6 +443,13 @@ DEFINE_EVENT(writeback_congest_waited_te
TP_ARGS(usec_timeout, usec_delayed)
);
+DEFINE_EVENT(writeback_congest_waited_template, writeback_reclaim_wait,
+
+ TP_PROTO(unsigned int usec_timeout, unsigned int usec_delayed),
+
+ TP_ARGS(usec_timeout, usec_delayed)
+);
+
DECLARE_EVENT_CLASS(writeback_single_inode_template,
TP_PROTO(struct inode *inode,
--- linux.orig/include/linux/writeback.h 2012-02-14 20:11:21.000000000 +0800
+++ linux/include/linux/writeback.h 2012-02-15 12:27:35.000000000 +0800
@@ -40,7 +40,7 @@ enum writeback_sync_modes {
*/
enum wb_reason {
WB_REASON_BACKGROUND,
- WB_REASON_TRY_TO_FREE_PAGES,
+ WB_REASON_PAGEOUT,
WB_REASON_SYNC,
WB_REASON_PERIODIC,
WB_REASON_LAPTOP_TIMER,
@@ -94,6 +94,8 @@ long writeback_inodes_wb(struct bdi_writ
enum wb_reason reason);
long wb_do_writeback(struct bdi_writeback *wb, int force_wait);
void wakeup_flusher_threads(long nr_pages, enum wb_reason reason);
+long flush_inode_page(struct address_space *mapping, struct page *page,
+ bool wait);
/* writeback.h requires fs.h; it, too, is not included from here. */
static inline void wait_on_inode(struct inode *inode)
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: reclaim the LRU lists full of dirty/writeback pages
2012-02-16 4:00 ` Wu Fengguang
@ 2012-02-16 12:44 ` Jan Kara
2012-02-16 13:32 ` Wu Fengguang
2012-02-17 16:41 ` Wu Fengguang
1 sibling, 1 reply; 33+ messages in thread
From: Jan Kara @ 2012-02-16 12:44 UTC (permalink / raw)
To: Wu Fengguang
Cc: Jan Kara, Rik van Riel, Greg Thelen, bsingharora, Hugh Dickins,
Michal Hocko, linux-mm, Mel Gorman, Ying Han, hannes,
KAMEZAWA Hiroyuki, Minchan Kim
On Thu 16-02-12 12:00:19, Wu Fengguang wrote:
> On Tue, Feb 14, 2012 at 02:29:50PM +0100, Jan Kara wrote:
> > > > I wonder what happens if you run:
> > > > mkdir /cgroup/x
> > > > echo 100M > /cgroup/x/memory.limit_in_bytes
> > > > echo $$ > /cgroup/x/tasks
> > > >
> > > > for (( i = 0; i < 2; i++ )); do
> > > > mkdir /fs/d$i
> > > > for (( j = 0; j < 5000; j++ )); do
> > > > dd if=/dev/zero of=/fs/d$i/f$j bs=1k count=50
> > > > done &
> > > > done
> > >
> > > That's a very good case, thanks!
> > >
> > > > Because for small files the writearound logic won't help much...
> > >
> > > Right, it also means the native background work cannot be more I/O
> > > efficient than the pageout works, except for the overheads of more
> > > work items..
> > Yes, that's true.
> >
> > > > Also the number of work items queued might become interesting.
> > >
> > > It turns out that the 1024 mempool reservations are not exhausted at
> > > all (the below patch as a trace_printk on alloc failure and it didn't
> > > trigger at all).
> > >
> > > Here is the representative iostat lines on XFS (full "iostat -kx 1 20" log attached):
> > >
> > > avg-cpu: %user %nice %system %iowait %steal %idle
> > > 0.80 0.00 6.03 0.03 0.00 93.14
> > >
> > > Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
> > > sda 0.00 205.00 0.00 163.00 0.00 16900.00 207.36 4.09 21.63 1.88 30.70
> > >
> > > The attached dirtied/written progress graph looks interesting.
> > > Although the iostat disk utilization is low, the "dirtied" progress
> > > line is pretty straight and there is no single congestion_wait event
> > > in the trace log. Which makes me wonder if there are some unknown
> > > blocking issues in the way.
> > Interesting. I'd also expect we should block in reclaim path. How fast
> > can dd threads progress when there is no cgroup involved?
>
> I tried running the dd tasks in global context with
>
> echo $((100<<20)) > /proc/sys/vm/dirty_bytes
>
> and got mostly the same results on XFS:
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 0.85 0.00 8.88 0.00 0.00 90.26
>
> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
> sda 0.00 0.00 0.00 50.00 0.00 23036.00 921.44 9.59 738.02 7.38 36.90
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 0.95 0.00 8.95 0.00 0.00 90.11
>
> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
> sda 0.00 854.00 0.00 99.00 0.00 19552.00 394.99 34.14 87.98 3.82 37.80
OK, so it seems that reclaiming pages in memcg reclaim acted as a natural
throttling similar to what balance_dirty_pages() does in the global case.
> Interestingly, ext4 shows comparable throughput, however is reporting
> near 100% disk utilization:
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 0.76 0.00 9.02 0.00 0.00 90.23
>
> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
> sda 0.00 0.00 0.00 317.00 0.00 20956.00 132.21 28.57 82.71 3.16 100.10
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 0.82 0.00 8.95 0.00 0.00 90.23
>
> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
> sda 0.00 0.00 0.00 402.00 0.00 24388.00 121.33 21.09 58.55 2.42 97.40
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 0.82 0.00 8.99 0.00 0.00 90.19
>
> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
> sda 0.00 0.00 0.00 409.00 0.00 21996.00 107.56 15.25 36.74 2.30 94.10
Average request size is smaller so maybe ext4 does more seeking.
> > > > Another common case to test - run 'slapadd' command in each cgroup to
> > > > create big LDAP database. That does pretty much random IO on a big mmaped
> > > > DB file.
> > >
> > > I've not used this. Will it need some configuration and data feed?
> > > fio looks more handy to me for emulating mmap random IO.
> > Yes, fio can generate random mmap IO. It's just that this is a real life
> > workload. So it is not completely random, it happens on several files and
> > is also interleaved with other memory allocations from DB. I can send you
> > the config files and data feed if you are interested.
>
> I'm very interested, thank you!
OK, I'll send it in private email...
> > > > > +/*
> > > > > + * schedule writeback on a range of inode pages.
> > > > > + */
> > > > > +static struct wb_writeback_work *
> > > > > +bdi_flush_inode_range(struct backing_dev_info *bdi,
> > > > > + struct inode *inode,
> > > > > + pgoff_t offset,
> > > > > + pgoff_t len,
> > > > > + bool wait)
> > > > > +{
> > > > > + struct wb_writeback_work *work;
> > > > > +
> > > > > + if (!igrab(inode))
> > > > > + return ERR_PTR(-ENOENT);
> > > > One technical note here: If the inode is deleted while it is queued, this
> > > > reference will keep it living until flusher thread gets to it. Then when
> > > > flusher thread puts its reference, the inode will get deleted in flusher
> > > > thread context. I don't see an immediate problem in that but it might be
> > > > surprising sometimes. Another problem I see is that if you try to
> > > > unmount the filesystem while the work item is queued, you'll get EBUSY for
> > > > no apparent reason (for userspace).
> > >
> > > Yeah, we need to make umount work.
> > The positive thing is that if the inode is reaped while the work item is
> > queue, we know all that needed to be done is done. So we don't really need
> > to pin the inode.
>
> But I do need to make sure the *inode pointer does not point to some
> invalid memory at work exec time. Is this possible without raising
> ->i_count?
I was thinking about it and what should work is that we have inode
reference in work item but in generic_shutdown_super() we go through
the worklist and drop all work items for superblock before calling
evict_inodes()...
Honza
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: reclaim the LRU lists full of dirty/writeback pages
2012-02-16 12:44 ` Jan Kara
@ 2012-02-16 13:32 ` Wu Fengguang
2012-02-16 14:06 ` Wu Fengguang
0 siblings, 1 reply; 33+ messages in thread
From: Wu Fengguang @ 2012-02-16 13:32 UTC (permalink / raw)
To: Jan Kara
Cc: Rik van Riel, Greg Thelen, bsingharora, Hugh Dickins,
Michal Hocko, linux-mm, Mel Gorman, Ying Han, hannes,
KAMEZAWA Hiroyuki, Minchan Kim
On Thu, Feb 16, 2012 at 01:44:45PM +0100, Jan Kara wrote:
> On Thu 16-02-12 12:00:19, Wu Fengguang wrote:
> > On Tue, Feb 14, 2012 at 02:29:50PM +0100, Jan Kara wrote:
> > > > > > +/*
> > > > > > + * schedule writeback on a range of inode pages.
> > > > > > + */
> > > > > > +static struct wb_writeback_work *
> > > > > > +bdi_flush_inode_range(struct backing_dev_info *bdi,
> > > > > > + struct inode *inode,
> > > > > > + pgoff_t offset,
> > > > > > + pgoff_t len,
> > > > > > + bool wait)
> > > > > > +{
> > > > > > + struct wb_writeback_work *work;
> > > > > > +
> > > > > > + if (!igrab(inode))
> > > > > > + return ERR_PTR(-ENOENT);
> > > > > One technical note here: If the inode is deleted while it is queued, this
> > > > > reference will keep it living until flusher thread gets to it. Then when
> > > > > flusher thread puts its reference, the inode will get deleted in flusher
> > > > > thread context. I don't see an immediate problem in that but it might be
> > > > > surprising sometimes. Another problem I see is that if you try to
> > > > > unmount the filesystem while the work item is queued, you'll get EBUSY for
> > > > > no apparent reason (for userspace).
> > > >
> > > > Yeah, we need to make umount work.
> > > The positive thing is that if the inode is reaped while the work item is
> > > queue, we know all that needed to be done is done. So we don't really need
> > > to pin the inode.
> >
> > But I do need to make sure the *inode pointer does not point to some
> > invalid memory at work exec time. Is this possible without raising
> > ->i_count?
> I was thinking about it and what should work is that we have inode
> reference in work item but in generic_shutdown_super() we go through
> the worklist and drop all work items for superblock before calling
> evict_inodes()...
Good point!
This diff removes the works after the sync_filesystem(sb) call. After
which, no more dirty pages are expected on that sb (otherwise the
umount will fail anyway), hence no more pageout works will be queued
for that sb.
+static void wb_free_work(struct wb_writeback_work *work)
+{
+ /*
+ * Notify the caller of completion if this is a synchronous
+ * work item, otherwise just free it.
+ */
+ if (work->done)
+ complete(work->done);
+ else
+ mempool_free(work, wb_work_mempool);
+}
+
+/*
+ * Remove works for @sb; or if (@sb == NULL), remove all works on @bdi.
+ */
+void bdi_remove_works(struct backing_dev_info *bdi, struct super_block *sb)
+{
+ struct inode *inode = mapping->host;
+ struct wb_writeback_work *work;
+
+ spin_lock_bh(&bdi->wb_lock);
+ list_for_each_entry_safe(work, &bdi->work_list, list) {
+ if (work->inode && work->inode->i_sb == sb) {
+ iput(inode);
+ } else if (sb && work->sb != sb)
+ continue;
+
+ list_del_init(&work->list);
+ wb_free_work(work);
+ }
+ spin_unlock_bh(&bdi->wb_lock);
+}
--- linux.orig/fs/super.c 2012-02-16 21:08:09.000000000 +0800
+++ linux/fs/super.c 2012-02-16 21:22:19.000000000 +0800
@@ -389,6 +389,7 @@ void generic_shutdown_super(struct super
fsnotify_unmount_inodes(&sb->s_inodes);
+ bdi_remove_works(sb->s_bdi, sb);
evict_inodes(sb);
if (sop->put_super)
Thanks,
Fengguang
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: reclaim the LRU lists full of dirty/writeback pages
2012-02-16 13:32 ` Wu Fengguang
@ 2012-02-16 14:06 ` Wu Fengguang
0 siblings, 0 replies; 33+ messages in thread
From: Wu Fengguang @ 2012-02-16 14:06 UTC (permalink / raw)
To: Jan Kara
Cc: Rik van Riel, Greg Thelen, bsingharora, Hugh Dickins,
Michal Hocko, linux-mm, Mel Gorman, Ying Han, hannes,
KAMEZAWA Hiroyuki, Minchan Kim
On Thu, Feb 16, 2012 at 09:32:33PM +0800, Wu Fengguang wrote:
> On Thu, Feb 16, 2012 at 01:44:45PM +0100, Jan Kara wrote:
> > On Thu 16-02-12 12:00:19, Wu Fengguang wrote:
> > > On Tue, Feb 14, 2012 at 02:29:50PM +0100, Jan Kara wrote:
>
> > > > > > > +/*
> > > > > > > + * schedule writeback on a range of inode pages.
> > > > > > > + */
> > > > > > > +static struct wb_writeback_work *
> > > > > > > +bdi_flush_inode_range(struct backing_dev_info *bdi,
> > > > > > > + struct inode *inode,
> > > > > > > + pgoff_t offset,
> > > > > > > + pgoff_t len,
> > > > > > > + bool wait)
> > > > > > > +{
> > > > > > > + struct wb_writeback_work *work;
> > > > > > > +
> > > > > > > + if (!igrab(inode))
> > > > > > > + return ERR_PTR(-ENOENT);
> > > > > > One technical note here: If the inode is deleted while it is queued, this
> > > > > > reference will keep it living until flusher thread gets to it. Then when
> > > > > > flusher thread puts its reference, the inode will get deleted in flusher
> > > > > > thread context. I don't see an immediate problem in that but it might be
> > > > > > surprising sometimes. Another problem I see is that if you try to
> > > > > > unmount the filesystem while the work item is queued, you'll get EBUSY for
> > > > > > no apparent reason (for userspace).
> > > > >
> > > > > Yeah, we need to make umount work.
> > > > The positive thing is that if the inode is reaped while the work item is
> > > > queue, we know all that needed to be done is done. So we don't really need
> > > > to pin the inode.
> > >
> > > But I do need to make sure the *inode pointer does not point to some
> > > invalid memory at work exec time. Is this possible without raising
> > > ->i_count?
> > I was thinking about it and what should work is that we have inode
> > reference in work item but in generic_shutdown_super() we go through
> > the worklist and drop all work items for superblock before calling
> > evict_inodes()...
>
> Good point!
>
> This diff removes the works after the sync_filesystem(sb) call. After
> which, no more dirty pages are expected on that sb (otherwise the
> umount will fail anyway), hence no more pageout works will be queued
> for that sb.
>
> +static void wb_free_work(struct wb_writeback_work *work)
> +{
> + /*
> + * Notify the caller of completion if this is a synchronous
> + * work item, otherwise just free it.
> + */
> + if (work->done)
> + complete(work->done);
> + else
> + mempool_free(work, wb_work_mempool);
> +}
> +
> +/*
> + * Remove works for @sb; or if (@sb == NULL), remove all works on @bdi.
> + */
> +void bdi_remove_works(struct backing_dev_info *bdi, struct super_block *sb)
> +{
> + struct inode *inode = mapping->host;
> + struct wb_writeback_work *work;
> +
> + spin_lock_bh(&bdi->wb_lock);
> + list_for_each_entry_safe(work, &bdi->work_list, list) {
> + if (work->inode && work->inode->i_sb == sb) {
> + iput(inode);
> + } else if (sb && work->sb != sb)
> + continue;
> +
> + list_del_init(&work->list);
> + wb_free_work(work);
> + }
> + spin_unlock_bh(&bdi->wb_lock);
> +}
Sorry, this corrected function actually compiles:
+/*
+ * Remove works for @sb; or if (@sb == NULL), remove all works on @bdi.
+ */
+void bdi_remove_works(struct backing_dev_info *bdi, struct super_block *sb)
+{
+ struct wb_writeback_work *work, *tmp;
+ LIST_HEAD(works);
+
+ spin_lock_bh(&bdi->wb_lock);
+ list_for_each_entry_safe(work, tmp, &bdi->work_list, list) {
+ if (sb) {
+ if (work->sb && work->sb != sb)
+ continue;
+ if (work->inode && work->inode->i_sb != sb)
+ continue;
+ }
+ list_move(&work->list, &works);
+ }
+ spin_unlock_bh(&bdi->wb_lock);
+
+ while (!list_empty(&works)) {
+ work = list_entry(works.next,
+ struct wb_writeback_work, list);
+ list_del_init(&work->list);
+ if (work->inode)
+ iput(work->inode);
+ wb_free_work(work);
+ }
+}
> --- linux.orig/fs/super.c 2012-02-16 21:08:09.000000000 +0800
> +++ linux/fs/super.c 2012-02-16 21:22:19.000000000 +0800
> @@ -389,6 +389,7 @@ void generic_shutdown_super(struct super
>
> fsnotify_unmount_inodes(&sb->s_inodes);
>
> + bdi_remove_works(sb->s_bdi, sb);
> evict_inodes(sb);
>
> if (sop->put_super)
>
> Thanks,
> Fengguang
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: reclaim the LRU lists full of dirty/writeback pages
2012-02-16 9:50 ` Wu Fengguang
@ 2012-02-16 17:31 ` Mel Gorman
2012-02-27 14:24 ` Fengguang Wu
0 siblings, 1 reply; 33+ messages in thread
From: Mel Gorman @ 2012-02-16 17:31 UTC (permalink / raw)
To: Wu Fengguang
Cc: Greg Thelen, Jan Kara, bsingharora, Hugh Dickins, Michal Hocko,
linux-mm, Ying Han, hannes, KAMEZAWA Hiroyuki, Rik van Riel,
Minchan Kim
On Thu, Feb 16, 2012 at 05:50:42PM +0800, Wu Fengguang wrote:
> On Tue, Feb 14, 2012 at 03:51:24PM +0000, Mel Gorman wrote:
> > On Tue, Feb 14, 2012 at 09:18:12PM +0800, Wu Fengguang wrote:
> > > > For the OOM problem, a more reasonable stopgap might be to identify when
> > > > a process is scanning a memcg at high priority and encountered all
> > > > PageReclaim with no forward progress and to congestion_wait() if that
> > > > situation occurs. A preferable way would be to wait until the flusher
> > > > wakes up a waiter on PageReclaim pages to be written out because we want
> > > > to keep moving way from congestion_wait() if at all possible.
> > >
> > > Good points! Below is the more serious page reclaim changes.
> > >
> > > The dirty/writeback pages may often come close to each other in the
> > > LRU list, so the local test during a 32-page scan may still trigger
> > > reclaim waits unnecessarily.
> >
> > Yes, this is particularly the case when writing back to USB. It is not
> > unusual that all dirty pages under writeback are backed by USB and at the
> > end of the LRU. Right now what happens is that reclaimers see higher CPU
> > usage as they scan over these pages uselessly. If the wrong choice is
> > made on how to throttle, we'll see yet more variants of the "system
> > responsiveness drops when writing to USB".
>
> Yes, USB is an important case to support. I'd imagine the heavy USB
> writes typically happen in desktops and run *outside* of any memcg.
I would expect it's common that USB writes are outside a memcg.
> So they'll typically take <= 20% memory in the zone. As long as we
> start the PG_reclaim throttling only when above the 20% dirty
> threshold (ie. on zone_dirty_ok()), the USB case should be safe.
>
It's not just the USB writer, it's unrelated process that are allocating
memory at the same time the writing happens. What we want to avoid is a
situation where something like firefox or evolution or even gnome-terminal
is performing a small read and gets either
a) started for IO bandwidth and stalls (not the focus here obviously)
b) enter page reclaim, finds PG_reclaim pages from the USB write and stalls
It's (b) we need to watch out for. I accept that this patch is heading
in the right direction and that the tracepoint can be used to identify
processes get throttled unfairly. Before merging, it'd be nice to hear of
such a test and include details in the changelog similar to the test case
in https://bugzilla.kernel.org/show_bug.cgi?id=31142 (a bug that lasted a
*long* time as it turned out, fixes merged for 3.3 with sync-light migration).
> > > Some global information on the percent
> > > of dirty/writeback pages in the LRU list may help. Anyway the added
> > > tests should still be much better than no protection.
> > >
> >
> > You can tell how many dirty pages and writeback pages are in the zone
> > already.
>
> Right. I changed the test to
>
> + if (nr_pgreclaim && nr_pgreclaim >= (nr_taken >> (DEF_PRIORITY-priority)) &&
> + (!global_reclaim(sc) || !zone_dirty_ok(zone)))
> + reclaim_wait(HZ/10);
>
> And I'd prefer to use a higher threshold than the default 20% for the
> above zone_dirty_ok() test, so that when Johannes' zone dirty
> balancing does the job fine, PG_reclaim based page reclaim throttling
> won't happen at all.
>
We'd also need to watch that we do not get throttled on small zones
like ZONE_DMA (which shouldn't happen but still). To detect this if
it happens, please consider including node and zone information in the
writeback_reclaim_wait tracepoint. The memcg people might want to be
able to see the memcg which I guess could be available
node=[NID|memcg]
NID if zone >=0
memcg if zone == -1
Which is hacky but avoids creating two tracepoints.
> > > A global wait queue and reclaim_wait() is introduced. The waiters will
> > > be wakeup when pages are rotated by end_page_writeback() or lru drain.
> > >
> > > I have to say its effectiveness depends on the filesystem... ext4
> > > and btrfs do fluent IO completions, so reclaim_wait() works pretty
> > > well:
> > > dd-14560 [017] .... 1360.894605: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=10000
> > > dd-14560 [017] .... 1360.904456: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=8000
> > > dd-14560 [017] .... 1360.908293: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
> > > dd-14560 [017] .... 1360.923960: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=15000
> > > dd-14560 [017] .... 1360.927810: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
> > > dd-14560 [017] .... 1360.931656: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
> > > dd-14560 [017] .... 1360.943503: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=10000
> > > dd-14560 [017] .... 1360.953289: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=7000
> > > dd-14560 [017] .... 1360.957177: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
> > > dd-14560 [017] .... 1360.972949: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=15000
> > >
> > > However XFS does IO completions in very large batches (there may be
> > > only several big IO completions in one second). So reclaim_wait()
> > > mostly end up waiting to the full HZ/10 timeout:
> > >
> > > dd-4177 [008] .... 866.367661: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > > dd-4177 [010] .... 866.567583: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > > dd-4177 [012] .... 866.767458: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > > dd-4177 [013] .... 866.867419: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > > dd-4177 [008] .... 867.167266: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > > dd-4177 [010] .... 867.367168: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > > dd-4177 [012] .... 867.818950: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > > dd-4177 [013] .... 867.918905: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > > dd-4177 [013] .... 867.971657: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=52000
> > > dd-4177 [013] .... 867.971812: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=0
> > > dd-4177 [008] .... 868.355700: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > > dd-4177 [010] .... 868.700515: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > >
> >
> > And where people will get hit by regressions in this area is writing to
> > vfat and in more rare cases ntfs on USB stick.
>
> vfat IO completions seem to lie somewhere between ext4 and xfs:
>
> <...>-46385 [010] .... 143570.714470: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> <...>-46385 [008] .... 143570.752391: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=12000
> <...>-46385 [008] .... 143570.937327: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=52000
> <...>-46385 [010] .... 143571.160252: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> <...>-46385 [011] .... 143571.286197: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> <...>-46385 [008] .... 143571.329644: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=15000
> <...>-46385 [008] .... 143571.475433: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=18000
> <...>-46385 [008] .... 143571.653461: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=52000
> <...>-46385 [008] .... 143571.839949: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=56000
> <...>-46385 [010] .... 143572.060816: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> <...>-46385 [011] .... 143572.185754: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> <...>-46385 [008] .... 143572.212522: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=1000
> <...>-46385 [008] .... 143572.217825: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
> <...>-46385 [008] .... 143572.312395: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=91000
> <...>-46385 [008] .... 143572.315122: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=1000
> <...>-46385 [009] .... 143572.433630: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> <...>-46385 [010] .... 143572.534569: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
>
Ok. It's interesting to note that we are stalling a lot there - roughly
30ms every second. As long as it's the writer, that's fine. If it's
firefox, it will create bug reports :)
> > > <SNIP>
> > > @@ -813,6 +815,10 @@ static unsigned long shrink_page_list(st
> > >
> > > if (PageWriteback(page)) {
> > > nr_writeback++;
> > > + if (PageReclaim(page))
> > > + nr_pgreclaim++;
> > > + else
> > > + SetPageReclaim(page);
> > > /*
> >
> > This check is unexpected. We already SetPageReclaim when queuing pages for
> > IO from reclaim context and if dirty pages are encountered during the LRU
> > scan that cannot be queued for IO. How often is it that nr_pgreclaim !=
> > nr_writeback and by how much do they differ?
>
> Quite often, I suspect. The pageout writeback works do 1-8MB write
> around which may start I/O a bit earlier than the covered pages are
> encountered by page reclaim. ext4 forces 128MB write chunk size, which
> further increases the opportunities.
>
Ok, thanks for the clarification. Stick a wee comment on it please.
> > > * Synchronous reclaim cannot queue pages for
> > > * writeback due to the possibility of stack overflow
> > > @@ -874,12 +880,15 @@ static unsigned long shrink_page_list(st
> > > nr_dirty++;
> > >
> > > /*
> > > - * Only kswapd can writeback filesystem pages to
> > > - * avoid risk of stack overflow but do not writeback
> > > - * unless under significant pressure.
> > > + * run into the visited page again: we are scanning
> > > + * faster than the flusher can writeout dirty pages
> > > */
> >
> > which in itself is not an abnormal condition. We get into this situation
> > when writing to USB. Dirty throttling stops too much memory getting dirtied
> > but that does not mean we should throttle instead of reclaiming clean pages.
> >
> > That's why I worry that if this is aimed at fixing a memcg problem, it
> > will have the impact of making interactive performance on normal systems
> > worse.
>
> You are right. This patch only addresses the pageout I/O efficiency
> and dirty throttling problems for a fully dirtied LRU. Next step, I'll
> think about the interactive performance problem for a less dirtied LRU.
>
Ok, thanks.
> > <SNIP>
> >
> > If the intention is to avoid memcg going OOM prematurely, the
> > nr_pgreclaim value needs to be treated at a higher level that records
> > how many PageReclaim pages were encountered. If no progress was made
> > because all the pages were PageReclaim, then throttle and return 1 to
> > the page allocator where it will retry the allocation without going OOM
> > after some pages have been cleaned and reclaimed.
>
> Agreed in general, but changed to this test for now, which is made a
> bit more global wise with the use of zone_dirty_ok().
>
Ok, sure. We know what to look out for and where unrelated regressions
might get introduced.
> memcg is ignored due to no dirty accounting (Greg has the patch though).
> And even zone_dirty_ok() may be inaccurate for the global reclaim, if
> some memcgs are skipped by the global reclaim by the memcg soft limit.
>
> But anyway, it's a handy hack for now. I'm looking into some more
> radical changes to put most dirty/writeback pages into a standalone
> LRU list (in addition to your LRU_IMMEDIATE, which I think is a good
> idea) for addressing the clustered way they tend to lie in the
> inactive LRU list.
>
> + if (nr_pgreclaim && nr_pgreclaim >= (nr_taken >> (DEF_PRIORITY-priority)) &&
> + (!global_reclaim(sc) || !zone_dirty_ok(zone)))
> + reclaim_wait(HZ/10);
>
This should make it harder to get stalled. Your tracepoint should help
us catch if it happens unnecessarily.
> ---
> Subject: writeback: introduce the pageout work
> Date: Thu Jul 29 14:41:19 CST 2010
>
> This relays file pageout IOs to the flusher threads.
>
> The ultimate target is to gracefully handle the LRU lists full of
> dirty/writeback pages.
>
It would be worth mentioning in the changelog that this is much more
important now that page reclaim generally does not writeout filesystem-backed
pages.
> 1) I/O efficiency
>
> The flusher will piggy back the nearby ~10ms worth of dirty pages for I/O.
>
> This takes advantage of the time/spacial locality in most workloads: the
> nearby pages of one file are typically populated into the LRU at the same
> time, hence will likely be close to each other in the LRU list. Writing
> them in one shot helps clean more pages effectively for page reclaim.
>
> 2) OOM avoidance and scan rate control
>
> Typically we do LRU scan w/o rate control and quickly get enough clean
> pages for the LRU lists not full of dirty pages.
>
> Or we can still get a number of freshly cleaned pages (moved to LRU tail
> by end_page_writeback()) when the queued pageout I/O is completed within
> tens of milli-seconds.
>
> However if the LRU list is small and full of dirty pages, it can be
> quickly fully scanned and go OOM before the flusher manages to clean
> enough pages.
>
It's worth pointing out here that generally this does not happen for global
reclaim which does dirty throttling but happens easily with memcg LRUs.
> A simple yet reliable scheme is employed to avoid OOM and keep scan rate
> in sync with the I/O rate:
>
> if (PageReclaim(page))
> congestion_wait(HZ/10);
>
This comment is stale now.
> PG_reclaim plays the key role. When dirty pages are encountered, we
> queue I/O for it,
This is misleading. The process that encounters the dirty page does
not queue the page for IO unless it is kswapd scanning at high priority
(currently anyway, you patch changes it). The process that finds the page
queues work for flusher threads that will queue the actual I/O for it at
some unknown time in the future.
> set PG_reclaim and put it back to the LRU head.
> So if PG_reclaim pages are encountered again, it means the dirty page
> has not yet been cleaned by the flusher after a full zone scan. It
> indicates we are scanning more fast than I/O and shall take a snap.
>
This is also slightly misleading because the page can be encountered
after rescanning the inactive list, not necessarily a full zone scan but
it's a minor point.
> The runtime behavior on a fully dirtied small LRU list would be:
> It will start with a quick scan of the list, queuing all pages for I/O.
> Then the scan will be slowed down by the PG_reclaim pages *adaptively*
> to match the I/O bandwidth.
>
> 3) writeback work coordinations
>
> To avoid memory allocations at page reclaim, a mempool for struct
> wb_writeback_work is created.
>
> wakeup_flusher_threads() is removed because it can easily delay the
> more oriented pageout works and even exhaust the mempool reservations.
> It's also found to not I/O efficient by frequently submitting writeback
> works with small ->nr_pages.
>
> Background/periodic works will quit automatically, so as to clean the
> pages under reclaim ASAP. However for now the sync work can still block
> us for long time.
>
> Jan Kara: limit the search scope. Note that the limited search and work
> pool is not a big problem: 1000 IOs under flight are typically more than
> enough to saturate the disk. And the overheads of searching in the work
> list didn't even show up in the perf report.
>
> 4) test case
>
> Run 2 dd tasks in a 100MB memcg (a very handy test case from Greg Thelen):
>
> mkdir /cgroup/x
> echo 100M > /cgroup/x/memory.limit_in_bytes
> echo $$ > /cgroup/x/tasks
>
> for i in `seq 2`
> do
> dd if=/dev/zero of=/fs/f$i bs=1k count=1M &
> done
>
> Before patch, the dd tasks are quickly OOM killed.
> After patch, they run well with reasonably good performance and overheads:
>
> 1073741824 bytes (1.1 GB) copied, 22.2196 s, 48.3 MB/s
> 1073741824 bytes (1.1 GB) copied, 22.4675 s, 47.8 MB/s
>
> iostat -kx 1
>
> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
> sda 0.00 0.00 0.00 178.00 0.00 89568.00 1006.38 74.35 417.71 4.80 85.40
> sda 0.00 2.00 0.00 191.00 0.00 94428.00 988.77 53.34 219.03 4.34 82.90
> sda 0.00 20.00 0.00 196.00 0.00 97712.00 997.06 71.11 337.45 4.77 93.50
> sda 0.00 5.00 0.00 175.00 0.00 84648.00 967.41 54.03 316.44 5.06 88.60
> sda 0.00 0.00 0.00 186.00 0.00 92432.00 993.89 56.22 267.54 5.38 100.00
> sda 0.00 1.00 0.00 183.00 0.00 90156.00 985.31 37.99 325.55 4.33 79.20
> sda 0.00 0.00 0.00 175.00 0.00 88692.00 1013.62 48.70 218.43 4.69 82.10
> sda 0.00 0.00 0.00 196.00 0.00 97528.00 995.18 43.38 236.87 5.10 100.00
> sda 0.00 0.00 0.00 179.00 0.00 88648.00 990.48 45.83 285.43 5.59 100.00
> sda 0.00 0.00 0.00 178.00 0.00 88500.00 994.38 28.28 158.89 4.99 88.80
> sda 0.00 0.00 0.00 194.00 0.00 95852.00 988.16 32.58 167.39 5.15 100.00
> sda 0.00 2.00 0.00 215.00 0.00 105996.00 986.01 41.72 201.43 4.65 100.00
> sda 0.00 4.00 0.00 173.00 0.00 84332.00 974.94 50.48 260.23 5.76 99.60
> sda 0.00 0.00 0.00 182.00 0.00 90312.00 992.44 36.83 212.07 5.49 100.00
> sda 0.00 8.00 0.00 195.00 0.00 95940.50 984.01 50.18 221.06 5.13 100.00
> sda 0.00 1.00 0.00 220.00 0.00 108852.00 989.56 40.99 202.68 4.55 100.00
> sda 0.00 2.00 0.00 161.00 0.00 80384.00 998.56 37.19 268.49 6.21 100.00
> sda 0.00 4.00 0.00 182.00 0.00 90830.00 998.13 50.58 239.77 5.49 100.00
> sda 0.00 0.00 0.00 197.00 0.00 94877.00 963.22 36.68 196.79 5.08 100.00
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 0.25 0.00 15.08 33.92 0.00 50.75
> 0.25 0.00 14.54 35.09 0.00 50.13
> 0.50 0.00 13.57 32.41 0.00 53.52
> 0.50 0.00 11.28 36.84 0.00 51.38
> 0.50 0.00 15.75 32.00 0.00 51.75
> 0.50 0.00 10.50 34.00 0.00 55.00
> 0.50 0.00 17.63 27.46 0.00 54.41
> 0.50 0.00 15.08 30.90 0.00 53.52
> 0.50 0.00 11.28 32.83 0.00 55.39
> 0.75 0.00 16.79 26.82 0.00 55.64
> 0.50 0.00 16.08 29.15 0.00 54.27
> 0.50 0.00 13.50 30.50 0.00 55.50
> 0.50 0.00 14.32 35.18 0.00 50.00
> 0.50 0.00 12.06 33.92 0.00 53.52
> 0.50 0.00 17.29 30.58 0.00 51.63
> 0.50 0.00 15.08 29.65 0.00 54.77
> 0.50 0.00 12.53 29.32 0.00 57.64
> 0.50 0.00 15.29 31.83 0.00 52.38
>
> The global dd numbers for comparison:
>
> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
> sda 0.00 0.00 0.00 189.00 0.00 95752.00 1013.25 143.09 684.48 5.29 100.00
> sda 0.00 0.00 0.00 208.00 0.00 105480.00 1014.23 143.06 733.29 4.81 100.00
> sda 0.00 0.00 0.00 161.00 0.00 81924.00 1017.69 141.71 757.79 6.21 100.00
> sda 0.00 0.00 0.00 217.00 0.00 109580.00 1009.95 143.09 749.55 4.61 100.10
> sda 0.00 0.00 0.00 187.00 0.00 94728.00 1013.13 144.31 773.67 5.35 100.00
> sda 0.00 0.00 0.00 189.00 0.00 95752.00 1013.25 144.14 742.00 5.29 100.00
> sda 0.00 0.00 0.00 177.00 0.00 90032.00 1017.31 143.32 656.59 5.65 100.00
> sda 0.00 0.00 0.00 215.00 0.00 108640.00 1010.60 142.90 817.54 4.65 100.00
> sda 0.00 2.00 0.00 166.00 0.00 83858.00 1010.34 143.64 808.61 6.02 100.00
> sda 0.00 0.00 0.00 186.00 0.00 92813.00 997.99 141.18 736.95 5.38 100.00
> sda 0.00 0.00 0.00 206.00 0.00 104456.00 1014.14 146.27 729.33 4.85 100.00
> sda 0.00 0.00 0.00 213.00 0.00 107024.00 1004.92 143.25 705.70 4.69 100.00
> sda 0.00 0.00 0.00 188.00 0.00 95748.00 1018.60 141.82 764.78 5.32 100.00
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 0.51 0.00 11.22 52.30 0.00 35.97
> 0.25 0.00 10.15 52.54 0.00 37.06
> 0.25 0.00 5.01 56.64 0.00 38.10
> 0.51 0.00 15.15 43.94 0.00 40.40
> 0.25 0.00 12.12 48.23 0.00 39.39
> 0.51 0.00 11.20 53.94 0.00 34.35
> 0.26 0.00 9.72 51.41 0.00 38.62
> 0.76 0.00 9.62 50.63 0.00 38.99
> 0.51 0.00 10.46 53.32 0.00 35.71
> 0.51 0.00 9.41 51.91 0.00 38.17
> 0.25 0.00 10.69 49.62 0.00 39.44
> 0.51 0.00 12.21 52.67 0.00 34.61
> 0.51 0.00 11.45 53.18 0.00 34.86
>
> XXX: commit NFS unstable pages via write_inode()
> XXX: the added congestion_wait() may be undesirable in some situations
>
This second XXX may also not be redundant.
> CC: Jan Kara <jack@suse.cz>
> CC: Mel Gorman <mgorman@suse.de>
> Acked-by: Rik van Riel <riel@redhat.com>
> CC: Greg Thelen <gthelen@google.com>
> CC: Minchan Kim <minchan.kim@gmail.com>
> Signed-off-by: Wu Fengguang <fengguang.wu@intel.com>
> ---
> fs/fs-writeback.c | 169 ++++++++++++++++++++++++++++-
> include/linux/backing-dev.h | 2
> include/linux/writeback.h | 4
> include/trace/events/writeback.h | 19 ++-
> mm/backing-dev.c | 35 ++++++
> mm/swap.c | 1
> mm/vmscan.c | 32 +++--
> 7 files changed, 245 insertions(+), 17 deletions(-)
>
> - move congestion_wait() out of the page lock: it's blocking btrfs lock_delalloc_pages()
>
> --- linux.orig/include/linux/backing-dev.h 2012-02-14 20:11:21.000000000 +0800
> +++ linux/include/linux/backing-dev.h 2012-02-15 12:34:24.000000000 +0800
> @@ -304,6 +304,8 @@ void clear_bdi_congested(struct backing_
> void set_bdi_congested(struct backing_dev_info *bdi, int sync);
> long congestion_wait(int sync, long timeout);
> long wait_iff_congested(struct zone *zone, int sync, long timeout);
> +long reclaim_wait(long timeout);
> +void reclaim_rotated(void);
>
> static inline bool bdi_cap_writeback_dirty(struct backing_dev_info *bdi)
> {
> --- linux.orig/mm/backing-dev.c 2012-02-14 20:11:21.000000000 +0800
> +++ linux/mm/backing-dev.c 2012-02-15 12:34:19.000000000 +0800
> @@ -873,3 +873,38 @@ out:
> return ret;
> }
> EXPORT_SYMBOL(wait_iff_congested);
> +
> +static DECLARE_WAIT_QUEUE_HEAD(reclaim_wqh);
> +
Should this be declared on a per-NUMA node basis to avoid throttling on one
node being woken up by activity on an unrelated node? reclaim_rorated()
is called from a context that has a page so looking up the waitqueue would
be easy. Grep for place that initialise kswapd_wait and the initialisation
code will be easier although watch that if a node is hot-removed that
the queue is woken.
> +/**
> + * reclaim_wait - wait for some pages being rotated to the LRU tail
> + * @timeout: timeout in jiffies
> + *
> + * Wait until @timeout, or when some (typically PG_reclaim under writeback)
> + * pages rotated to the LRU so that page reclaim can make progress.
> + */
> +long reclaim_wait(long timeout)
> +{
> + long ret;
> + unsigned long start = jiffies;
> + DEFINE_WAIT(wait);
> +
> + prepare_to_wait(&reclaim_wqh, &wait, TASK_KILLABLE);
> + ret = io_schedule_timeout(timeout);
> + finish_wait(&reclaim_wqh, &wait);
> +
> + trace_writeback_reclaim_wait(jiffies_to_usecs(timeout),
> + jiffies_to_usecs(jiffies - start));
> +
> + return ret;
> +}
> +EXPORT_SYMBOL(reclaim_wait);
> +
Why do we export this? Only vmscan.c is calling it and I'm scratching my
head trying to figure out why a kernel module would want to call it.
> +void reclaim_rotated()
> +{
style nit
void reclaim_rotated(void)
> + wait_queue_head_t *wqh = &reclaim_wqh;
> +
> + if (waitqueue_active(wqh))
> + wake_up(wqh);
> +}
> +
> --- linux.orig/mm/swap.c 2012-02-14 20:11:21.000000000 +0800
> +++ linux/mm/swap.c 2012-02-15 12:27:35.000000000 +0800
> @@ -253,6 +253,7 @@ static void pagevec_move_tail(struct pag
>
> pagevec_lru_move_fn(pvec, pagevec_move_tail_fn, &pgmoved);
> __count_vm_events(PGROTATED, pgmoved);
> + reclaim_rotated();
> }
>
> /*
> --- linux.orig/mm/vmscan.c 2012-02-14 20:11:21.000000000 +0800
> +++ linux/mm/vmscan.c 2012-02-16 17:23:17.000000000 +0800
> @@ -767,7 +767,8 @@ static unsigned long shrink_page_list(st
> struct scan_control *sc,
> int priority,
> unsigned long *ret_nr_dirty,
> - unsigned long *ret_nr_writeback)
> + unsigned long *ret_nr_writeback,
> + unsigned long *ret_nr_pgreclaim)
> {
> LIST_HEAD(ret_pages);
> LIST_HEAD(free_pages);
> @@ -776,6 +777,7 @@ static unsigned long shrink_page_list(st
> unsigned long nr_congested = 0;
> unsigned long nr_reclaimed = 0;
> unsigned long nr_writeback = 0;
> + unsigned long nr_pgreclaim = 0;
>
> cond_resched();
>
> @@ -813,6 +815,10 @@ static unsigned long shrink_page_list(st
>
> if (PageWriteback(page)) {
> nr_writeback++;
> + if (PageReclaim(page))
> + nr_pgreclaim++;
> + else
> + SetPageReclaim(page);
> /*
> * Synchronous reclaim cannot queue pages for
> * writeback due to the possibility of stack overflow
> @@ -874,12 +880,15 @@ static unsigned long shrink_page_list(st
> nr_dirty++;
>
> /*
> - * Only kswapd can writeback filesystem pages to
> - * avoid risk of stack overflow but do not writeback
> - * unless under significant pressure.
> + * run into the visited page again: we are scanning
> + * faster than the flusher can writeout dirty pages
> */
> - if (page_is_file_cache(page) &&
> - (!current_is_kswapd() || priority >= DEF_PRIORITY - 2)) {
> + if (page_is_file_cache(page) && PageReclaim(page)) {
> + nr_pgreclaim++;
> + goto keep_locked;
> + }
This change means that kswapd is no longer doing any writeback from page
reclaim. Was that intended because it's not discussed in the changelog. I
know writeback from kswapd is poor in terms of IO performance but it's a
last resort for freeing a page when reclaim is in trouble. If we are to
disable it and depend 100% on the flusher threads, it should be in its
own patch for bisection reasons if nothing else.
> + if (page_is_file_cache(page) && mapping &&
> + flush_inode_page(mapping, page, false) >= 0) {
> /*
> * Immediately reclaim when written back.
> * Similar in principal to deactivate_page()
> @@ -1028,6 +1037,7 @@ keep_lumpy:
> count_vm_events(PGACTIVATE, pgactivate);
> *ret_nr_dirty += nr_dirty;
> *ret_nr_writeback += nr_writeback;
> + *ret_nr_pgreclaim += nr_pgreclaim;
> return nr_reclaimed;
> }
>
> @@ -1509,6 +1519,7 @@ shrink_inactive_list(unsigned long nr_to
> unsigned long nr_file;
> unsigned long nr_dirty = 0;
> unsigned long nr_writeback = 0;
> + unsigned long nr_pgreclaim = 0;
> isolate_mode_t reclaim_mode = ISOLATE_INACTIVE;
> struct zone *zone = mz->zone;
>
> @@ -1559,13 +1570,13 @@ shrink_inactive_list(unsigned long nr_to
> spin_unlock_irq(&zone->lru_lock);
>
> nr_reclaimed = shrink_page_list(&page_list, mz, sc, priority,
> - &nr_dirty, &nr_writeback);
> + &nr_dirty, &nr_writeback, &nr_pgreclaim);
>
> /* Check if we should syncronously wait for writeback */
> if (should_reclaim_stall(nr_taken, nr_reclaimed, priority, sc)) {
> set_reclaim_mode(priority, sc, true);
> nr_reclaimed += shrink_page_list(&page_list, mz, sc,
> - priority, &nr_dirty, &nr_writeback);
> + priority, &nr_dirty, &nr_writeback, &nr_pgreclaim);
> }
>
> spin_lock_irq(&zone->lru_lock);
> @@ -1608,6 +1619,9 @@ shrink_inactive_list(unsigned long nr_to
> */
> if (nr_writeback && nr_writeback >= (nr_taken >> (DEF_PRIORITY-priority)))
> wait_iff_congested(zone, BLK_RW_ASYNC, HZ/10);
> + if (nr_pgreclaim && nr_pgreclaim >= (nr_taken >> (DEF_PRIORITY-priority)) &&
> + (!global_reclaim(sc) || !zone_dirty_ok(zone)))
> + reclaim_wait(HZ/10);
>
I prefer this but it would be nice if there was a comment explaining it
or at least expand the comment explaining how nr_writeback can lead to
wait_iff_congested() being called.
> trace_mm_vmscan_lru_shrink_inactive(zone->zone_pgdat->node_id,
> zone_idx(zone),
> @@ -2382,8 +2396,6 @@ static unsigned long do_try_to_free_page
> */
> writeback_threshold = sc->nr_to_reclaim + sc->nr_to_reclaim / 2;
> if (total_scanned > writeback_threshold) {
> - wakeup_flusher_threads(laptop_mode ? 0 : total_scanned,
> - WB_REASON_TRY_TO_FREE_PAGES);
> sc->may_writepage = 1;
> }
>
> --- linux.orig/fs/fs-writeback.c 2012-02-14 20:11:21.000000000 +0800
> +++ linux/fs/fs-writeback.c 2012-02-15 12:27:35.000000000 +0800
> @@ -41,6 +41,8 @@ struct wb_writeback_work {
> long nr_pages;
> struct super_block *sb;
> unsigned long *older_than_this;
> + struct inode *inode;
> + pgoff_t offset;
> enum writeback_sync_modes sync_mode;
> unsigned int tagged_writepages:1;
> unsigned int for_kupdate:1;
> @@ -65,6 +67,27 @@ struct wb_writeback_work {
> */
> int nr_pdflush_threads;
>
> +static mempool_t *wb_work_mempool;
> +
> +static void *wb_work_alloc(gfp_t gfp_mask, void *pool_data)
> +{
> + /*
> + * bdi_flush_inode_range() may be called on page reclaim
> + */
> + if (current->flags & PF_MEMALLOC)
> + return NULL;
> +
This check is why I worry about kswapd being unable to write pages at
all. If the mempool is depleted for whatever reason, reclaim has no way
of telling the flushers what work is needed or waking them. Potentially,
we could be waiting a long time for pending flusher work to complete to
free up a slot. I recognise it may not be bad in practice because the
pool is large and other work will be completing but it's why kswapd not
writing back pages should be in its own patch.
> + return kmalloc(sizeof(struct wb_writeback_work), gfp_mask);
> +}
> +
> +static __init int wb_work_init(void)
> +{
> + wb_work_mempool = mempool_create(1024,
> + wb_work_alloc, mempool_kfree, NULL);
> + return wb_work_mempool ? 0 : -ENOMEM;
> +}
> +fs_initcall(wb_work_init);
> +
> /**
> * writeback_in_progress - determine whether there is writeback in progress
> * @bdi: the device's backing_dev_info structure.
> @@ -129,7 +152,7 @@ __bdi_start_writeback(struct backing_dev
> * This is WB_SYNC_NONE writeback, so if allocation fails just
> * wakeup the thread for old dirty data writeback
> */
> - work = kzalloc(sizeof(*work), GFP_ATOMIC);
> + work = mempool_alloc(wb_work_mempool, GFP_NOWAIT);
> if (!work) {
> if (bdi->wb.task) {
> trace_writeback_nowork(bdi);
> @@ -138,6 +161,7 @@ __bdi_start_writeback(struct backing_dev
> return;
> }
>
> + memset(work, 0, sizeof(*work));
> work->sync_mode = WB_SYNC_NONE;
> work->nr_pages = nr_pages;
> work->range_cyclic = range_cyclic;
> @@ -186,6 +210,125 @@ void bdi_start_background_writeback(stru
> spin_unlock_bh(&bdi->wb_lock);
> }
>
> +static bool extend_writeback_range(struct wb_writeback_work *work,
> + pgoff_t offset,
> + unsigned long write_around_pages)
> +{
comment on what this function is for and what the return values mean.
"returns true if the wb_writeback_work now encompasses the request"
or something
> + pgoff_t end = work->offset + work->nr_pages;
> +
> + if (offset >= work->offset && offset < end)
> + return true;
> +
This does not ensure that the full span of
offset -> offset+write_around_pages is encompassed by work. All it
checks is that the start of the requested range is going to be handled.
I guess it's ok because the page reclaims cares about is covered and
avoids a situation where too much IO is being queued. It's unclear if
this is what you intended though because you check for too much IO being
queued in the next block.
> + /*
> + * for sequential workloads with good locality, include up to 8 times
> + * more data in one chunk
> + */
> + if (work->nr_pages >= 8 * write_around_pages)
> + return false;
> +
> + /* the unsigned comparison helps eliminate one compare */
> + if (work->offset - offset < write_around_pages) {
> + work->nr_pages += write_around_pages;
> + work->offset -= write_around_pages;
> + return true;
> + }
> +
> + if (offset - end < write_around_pages) {
> + work->nr_pages += write_around_pages;
> + return true;
> + }
> +
> + return false;
> +}
> +
> +/*
> + * schedule writeback on a range of inode pages.
> + */
> +static struct wb_writeback_work *
> +bdi_flush_inode_range(struct backing_dev_info *bdi,
> + struct inode *inode,
> + pgoff_t offset,
> + pgoff_t len,
> + bool wait)
> +{
> + struct wb_writeback_work *work;
> +
> + if (!igrab(inode))
> + return ERR_PTR(-ENOENT);
> +
Explain why the igrab is necessary. I think it's because we are calling
this from page reclaim context and the only thing pinning the
address_space is the page lock . If I'm right, it should be made clear
in the comment for bdi_flush_inode_range that this should only be called
from page reclaim context. Maybe even VM_BUG_ON if
!(current->flags & PF_MEMALLOC)?
> + work = mempool_alloc(wb_work_mempool, wait ? GFP_NOIO : GFP_NOWAIT);
> + if (!work) {
> + trace_printk("wb_work_mempool alloc fail\n");
> + return ERR_PTR(-ENOMEM);
> + }
> +
> + memset(work, 0, sizeof(*work));
> + work->sync_mode = WB_SYNC_NONE;
> + work->inode = inode;
> + work->offset = offset;
> + work->nr_pages = len;
> + work->reason = WB_REASON_PAGEOUT;
> +
> + bdi_queue_work(bdi, work);
> +
> + return work;
> +}
> +
> +/*
> + * Called by page reclaim code to flush the dirty page ASAP. Do write-around to
> + * improve IO throughput. The nearby pages will have good chance to reside in
> + * the same LRU list that vmscan is working on, and even close to each other
> + * inside the LRU list in the common case of sequential read/write.
> + *
> + * ret > 0: success, found/reused a previous writeback work
> + * ret = 0: success, allocated/queued a new writeback work
> + * ret < 0: failed
> + */
> +long flush_inode_page(struct address_space *mapping,
> + struct page *page,
> + bool wait)
> +{
> + struct backing_dev_info *bdi = mapping->backing_dev_info;
> + struct inode *inode = mapping->host;
> + struct wb_writeback_work *work;
> + unsigned long write_around_pages;
> + pgoff_t offset = page->index;
> + int i;
> + long ret = 0;
> +
> + if (unlikely(!inode))
> + return -ENOENT;
> +
> + /*
> + * piggy back 8-15ms worth of data
> + */
> + write_around_pages = bdi->avg_write_bandwidth + MIN_WRITEBACK_PAGES;
> + write_around_pages = rounddown_pow_of_two(write_around_pages) >> 6;
> +
> + i = 1;
> + spin_lock_bh(&bdi->wb_lock);
> + list_for_each_entry_reverse(work, &bdi->work_list, list) {
> + if (work->inode != inode)
> + continue;
> + if (extend_writeback_range(work, offset, write_around_pages)) {
> + ret = i;
> + break;
> + }
> + if (i++ > 100) /* limit search depth */
> + break;
No harm to move Jan's comment on depth limit search to here adding why
100 is as good as number as any to use.
> + }
> + spin_unlock_bh(&bdi->wb_lock);
> +
> + if (!ret) {
> + offset = round_down(offset, write_around_pages);
> + work = bdi_flush_inode_range(bdi, inode,
> + offset, write_around_pages, wait);
> + if (IS_ERR(work))
> + ret = PTR_ERR(work);
> + }
> + return ret;
> +}
> +
> /*
> * Remove the inode from the writeback list it is on.
> */
> @@ -833,6 +976,23 @@ static unsigned long get_nr_dirty_pages(
> get_nr_dirty_inodes();
> }
>
> +static long wb_flush_inode(struct bdi_writeback *wb,
> + struct wb_writeback_work *work)
> +{
> + struct writeback_control wbc = {
> + .sync_mode = WB_SYNC_NONE,
> + .nr_to_write = LONG_MAX,
> + .range_start = work->offset << PAGE_CACHE_SHIFT,
> + .range_end = (work->offset + work->nr_pages - 1)
> + << PAGE_CACHE_SHIFT,
> + };
> +
> + do_writepages(work->inode->i_mapping, &wbc);
> + iput(work->inode);
> +
> + return LONG_MAX - wbc.nr_to_write;
> +}
> +
> static long wb_check_background_flush(struct bdi_writeback *wb)
> {
> if (over_bground_thresh(wb->bdi)) {
> @@ -905,7 +1065,10 @@ long wb_do_writeback(struct bdi_writebac
>
> trace_writeback_exec(bdi, work);
>
> - wrote += wb_writeback(wb, work);
> + if (work->inode)
> + wrote += wb_flush_inode(wb, work);
> + else
> + wrote += wb_writeback(wb, work);
>
> /*
> * Notify the caller of completion if this is a synchronous
> @@ -914,7 +1077,7 @@ long wb_do_writeback(struct bdi_writebac
> if (work->done)
> complete(work->done);
> else
> - kfree(work);
> + mempool_free(work, wb_work_mempool);
> }
>
> /*
> --- linux.orig/include/trace/events/writeback.h 2012-02-14 20:11:22.000000000 +0800
> +++ linux/include/trace/events/writeback.h 2012-02-15 12:27:35.000000000 +0800
> @@ -23,7 +23,7 @@
>
> #define WB_WORK_REASON \
> {WB_REASON_BACKGROUND, "background"}, \
> - {WB_REASON_TRY_TO_FREE_PAGES, "try_to_free_pages"}, \
> + {WB_REASON_PAGEOUT, "pageout"}, \
> {WB_REASON_SYNC, "sync"}, \
> {WB_REASON_PERIODIC, "periodic"}, \
> {WB_REASON_LAPTOP_TIMER, "laptop_timer"}, \
> @@ -45,6 +45,8 @@ DECLARE_EVENT_CLASS(writeback_work_class
> __field(int, range_cyclic)
> __field(int, for_background)
> __field(int, reason)
> + __field(unsigned long, ino)
> + __field(unsigned long, offset)
> ),
> TP_fast_assign(
> strncpy(__entry->name, dev_name(bdi->dev), 32);
> @@ -55,9 +57,11 @@ DECLARE_EVENT_CLASS(writeback_work_class
> __entry->range_cyclic = work->range_cyclic;
> __entry->for_background = work->for_background;
> __entry->reason = work->reason;
> + __entry->ino = work->inode ? work->inode->i_ino : 0;
> + __entry->offset = work->offset;
> ),
> TP_printk("bdi %s: sb_dev %d:%d nr_pages=%ld sync_mode=%d "
> - "kupdate=%d range_cyclic=%d background=%d reason=%s",
> + "kupdate=%d range_cyclic=%d background=%d reason=%s ino=%lu offset=%lu",
> __entry->name,
> MAJOR(__entry->sb_dev), MINOR(__entry->sb_dev),
> __entry->nr_pages,
> @@ -65,7 +69,9 @@ DECLARE_EVENT_CLASS(writeback_work_class
> __entry->for_kupdate,
> __entry->range_cyclic,
> __entry->for_background,
> - __print_symbolic(__entry->reason, WB_WORK_REASON)
> + __print_symbolic(__entry->reason, WB_WORK_REASON),
> + __entry->ino,
> + __entry->offset
> )
> );
> #define DEFINE_WRITEBACK_WORK_EVENT(name) \
> @@ -437,6 +443,13 @@ DEFINE_EVENT(writeback_congest_waited_te
> TP_ARGS(usec_timeout, usec_delayed)
> );
>
> +DEFINE_EVENT(writeback_congest_waited_template, writeback_reclaim_wait,
> +
> + TP_PROTO(unsigned int usec_timeout, unsigned int usec_delayed),
> +
> + TP_ARGS(usec_timeout, usec_delayed)
> +);
> +
> DECLARE_EVENT_CLASS(writeback_single_inode_template,
>
> TP_PROTO(struct inode *inode,
> --- linux.orig/include/linux/writeback.h 2012-02-14 20:11:21.000000000 +0800
> +++ linux/include/linux/writeback.h 2012-02-15 12:27:35.000000000 +0800
> @@ -40,7 +40,7 @@ enum writeback_sync_modes {
> */
> enum wb_reason {
> WB_REASON_BACKGROUND,
> - WB_REASON_TRY_TO_FREE_PAGES,
> + WB_REASON_PAGEOUT,
> WB_REASON_SYNC,
> WB_REASON_PERIODIC,
> WB_REASON_LAPTOP_TIMER,
> @@ -94,6 +94,8 @@ long writeback_inodes_wb(struct bdi_writ
> enum wb_reason reason);
> long wb_do_writeback(struct bdi_writeback *wb, int force_wait);
> void wakeup_flusher_threads(long nr_pages, enum wb_reason reason);
> +long flush_inode_page(struct address_space *mapping, struct page *page,
> + bool wait);
>
> /* writeback.h requires fs.h; it, too, is not included from here. */
> static inline void wait_on_inode(struct inode *inode)
--
Mel Gorman
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: reclaim the LRU lists full of dirty/writeback pages
2012-02-16 4:00 ` Wu Fengguang
2012-02-16 12:44 ` Jan Kara
@ 2012-02-17 16:41 ` Wu Fengguang
2012-02-20 14:00 ` Jan Kara
1 sibling, 1 reply; 33+ messages in thread
From: Wu Fengguang @ 2012-02-17 16:41 UTC (permalink / raw)
To: Jan Kara
Cc: Rik van Riel, Greg Thelen, bsingharora, Hugh Dickins,
Michal Hocko, linux-mm, Mel Gorman, Ying Han, hannes,
KAMEZAWA Hiroyuki, Minchan Kim
> > > And I find the pageout works seem to have some problems with ext4.
> > > For example, this can be easily triggered with 10 dd tasks running
> > > inside the 100MB limited memcg:
> > So journal thread is getting stuck while committing transaction. Most
> > likely waiting for some dd thread to stop a transaction so that commit can
> > proceed. The processes waiting in start_this_handle() are just secondary
> > effect resulting from the first problem. It might be interesting to get
> > stack traces of all bloked processes when the journal thread is stuck.
>
> For completeness of discussion, citing your conclusion on my private
> data feed:
>
> : We enter memcg reclaim from grab_cache_page_write_begin() and are
> : waiting in congestion_wait(). Because grab_cache_page_write_begin() is
> : called with transaction started, this blocks transaction from
> : committing and subsequently blocks all other activity on the
> : filesystem. The fact is this isn't new with your patches, just your
> : changes or the fact that we are running in a memory constrained cgroup
> : make this more visible.
Maybe I'm missing some deep FS restrictions, but can this page
allocation (and the one in ext4_write_begin) be moved before
ext4_journal_start()? So that the page reclaim can throttle the
__GFP_WRITE allocations at will.
---
fs/ext4/inode.c | 22 +++++++++++-----------
1 file changed, 11 insertions(+), 11 deletions(-)
--- linux.orig/fs/ext4/inode.c 2012-02-18 00:10:27.000000000 +0800
+++ linux/fs/ext4/inode.c 2012-02-18 00:31:19.000000000 +0800
@@ -2398,38 +2398,38 @@ static int ext4_da_write_begin(struct fi
if (ext4_nonda_switch(inode->i_sb)) {
*fsdata = (void *)FALL_BACK_TO_NONDELALLOC;
return ext4_write_begin(file, mapping, pos,
len, flags, pagep, fsdata);
}
*fsdata = (void *)0;
trace_ext4_da_write_begin(inode, pos, len, flags);
retry:
+ page = grab_cache_page_write_begin(mapping, index, flags);
+ if (!page) {
+ ret = -ENOMEM;
+ goto out;
+ }
+ *pagep = page;
+
/*
* With delayed allocation, we don't log the i_disksize update
* if there is delayed block allocation. But we still need
* to journalling the i_disksize update if writes to the end
* of file which has an already mapped buffer.
*/
handle = ext4_journal_start(inode, 1);
if (IS_ERR(handle)) {
ret = PTR_ERR(handle);
+ unlock_page(page);
+ page_cache_release(page);
+ if (pos + len > inode->i_size)
+ truncate_inode_pages(inode->i_mapping, inode->i_size);
goto out;
}
- /* We cannot recurse into the filesystem as the transaction is already
- * started */
- flags |= AOP_FLAG_NOFS;
-
- page = grab_cache_page_write_begin(mapping, index, flags);
- if (!page) {
- ext4_journal_stop(handle);
- ret = -ENOMEM;
- goto out;
- }
- *pagep = page;
ret = __block_write_begin(page, pos, len, ext4_da_get_block_prep);
if (ret < 0) {
unlock_page(page);
ext4_journal_stop(handle);
page_cache_release(page);
/*
* block_write_begin may have instantiated a few blocks
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: reclaim the LRU lists full of dirty/writeback pages
2012-02-17 16:41 ` Wu Fengguang
@ 2012-02-20 14:00 ` Jan Kara
0 siblings, 0 replies; 33+ messages in thread
From: Jan Kara @ 2012-02-20 14:00 UTC (permalink / raw)
To: Wu Fengguang
Cc: Jan Kara, Rik van Riel, Greg Thelen, bsingharora, Hugh Dickins,
Michal Hocko, linux-mm, Mel Gorman, Ying Han, hannes,
KAMEZAWA Hiroyuki, Minchan Kim
On Sat 18-02-12 00:41:33, Wu Fengguang wrote:
> > > > And I find the pageout works seem to have some problems with ext4.
> > > > For example, this can be easily triggered with 10 dd tasks running
> > > > inside the 100MB limited memcg:
> > > So journal thread is getting stuck while committing transaction. Most
> > > likely waiting for some dd thread to stop a transaction so that commit can
> > > proceed. The processes waiting in start_this_handle() are just secondary
> > > effect resulting from the first problem. It might be interesting to get
> > > stack traces of all bloked processes when the journal thread is stuck.
> >
> > For completeness of discussion, citing your conclusion on my private
> > data feed:
> >
> > : We enter memcg reclaim from grab_cache_page_write_begin() and are
> > : waiting in congestion_wait(). Because grab_cache_page_write_begin() is
> > : called with transaction started, this blocks transaction from
> > : committing and subsequently blocks all other activity on the
> > : filesystem. The fact is this isn't new with your patches, just your
> > : changes or the fact that we are running in a memory constrained cgroup
> > : make this more visible.
>
> Maybe I'm missing some deep FS restrictions, but can this page
> allocation (and the one in ext4_write_begin) be moved before
> ext4_journal_start()? So that the page reclaim can throttle the
> __GFP_WRITE allocations at will.
You are missing the fact that in this way, things would deadlock quickly.
Lock ordering of ext4 is 'transaction start' -> 'page lock' (so that
writepages may be efficient) and so you cannot really have it differently
here - you can think of transaction start - transaction end pair as a
lock - unlock for lock ordering purposes. You could play some tricks like
allocating the page without locking, starting a transaction, locking a page
and checking that everything is OK but it isn't really nice and when memory
pressure is big, the gain is questionable.
Honza
> ---
> fs/ext4/inode.c | 22 +++++++++++-----------
> 1 file changed, 11 insertions(+), 11 deletions(-)
>
> --- linux.orig/fs/ext4/inode.c 2012-02-18 00:10:27.000000000 +0800
> +++ linux/fs/ext4/inode.c 2012-02-18 00:31:19.000000000 +0800
> @@ -2398,38 +2398,38 @@ static int ext4_da_write_begin(struct fi
> if (ext4_nonda_switch(inode->i_sb)) {
> *fsdata = (void *)FALL_BACK_TO_NONDELALLOC;
> return ext4_write_begin(file, mapping, pos,
> len, flags, pagep, fsdata);
> }
> *fsdata = (void *)0;
> trace_ext4_da_write_begin(inode, pos, len, flags);
> retry:
> + page = grab_cache_page_write_begin(mapping, index, flags);
> + if (!page) {
> + ret = -ENOMEM;
> + goto out;
> + }
> + *pagep = page;
> +
> /*
> * With delayed allocation, we don't log the i_disksize update
> * if there is delayed block allocation. But we still need
> * to journalling the i_disksize update if writes to the end
> * of file which has an already mapped buffer.
> */
> handle = ext4_journal_start(inode, 1);
> if (IS_ERR(handle)) {
> ret = PTR_ERR(handle);
> + unlock_page(page);
> + page_cache_release(page);
> + if (pos + len > inode->i_size)
> + truncate_inode_pages(inode->i_mapping, inode->i_size);
> goto out;
> }
> - /* We cannot recurse into the filesystem as the transaction is already
> - * started */
> - flags |= AOP_FLAG_NOFS;
> -
> - page = grab_cache_page_write_begin(mapping, index, flags);
> - if (!page) {
> - ext4_journal_stop(handle);
> - ret = -ENOMEM;
> - goto out;
> - }
> - *pagep = page;
>
> ret = __block_write_begin(page, pos, len, ext4_da_get_block_prep);
> if (ret < 0) {
> unlock_page(page);
> ext4_journal_stop(handle);
> page_cache_release(page);
> /*
> * block_write_begin may have instantiated a few blocks
--
Jan Kara <jack@suse.cz>
SUSE Labs, CR
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: reclaim the LRU lists full of dirty/writeback pages
2012-02-16 17:31 ` Mel Gorman
@ 2012-02-27 14:24 ` Fengguang Wu
0 siblings, 0 replies; 33+ messages in thread
From: Fengguang Wu @ 2012-02-27 14:24 UTC (permalink / raw)
To: Mel Gorman
Cc: Greg Thelen, Jan Kara, bsingharora, Hugh Dickins, Michal Hocko,
linux-mm, Ying Han, hannes, KAMEZAWA Hiroyuki, Rik van Riel,
Minchan Kim
Mel,
Thanks a lot for the in depth review!
On Thu, Feb 16, 2012 at 05:31:11PM +0000, Mel Gorman wrote:
> On Thu, Feb 16, 2012 at 05:50:42PM +0800, Wu Fengguang wrote:
> > On Tue, Feb 14, 2012 at 03:51:24PM +0000, Mel Gorman wrote:
> > > On Tue, Feb 14, 2012 at 09:18:12PM +0800, Wu Fengguang wrote:
> > > > > For the OOM problem, a more reasonable stopgap might be to identify when
> > > > > a process is scanning a memcg at high priority and encountered all
> > > > > PageReclaim with no forward progress and to congestion_wait() if that
> > > > > situation occurs. A preferable way would be to wait until the flusher
> > > > > wakes up a waiter on PageReclaim pages to be written out because we want
> > > > > to keep moving way from congestion_wait() if at all possible.
> > > >
> > > > Good points! Below is the more serious page reclaim changes.
> > > >
> > > > The dirty/writeback pages may often come close to each other in the
> > > > LRU list, so the local test during a 32-page scan may still trigger
> > > > reclaim waits unnecessarily.
> > >
> > > Yes, this is particularly the case when writing back to USB. It is not
> > > unusual that all dirty pages under writeback are backed by USB and at the
> > > end of the LRU. Right now what happens is that reclaimers see higher CPU
> > > usage as they scan over these pages uselessly. If the wrong choice is
> > > made on how to throttle, we'll see yet more variants of the "system
> > > responsiveness drops when writing to USB".
> >
> > Yes, USB is an important case to support. I'd imagine the heavy USB
> > writes typically happen in desktops and run *outside* of any memcg.
>
> I would expect it's common that USB writes are outside a memcg.
>
> > So they'll typically take <= 20% memory in the zone. As long as we
> > start the PG_reclaim throttling only when above the 20% dirty
> > threshold (ie. on zone_dirty_ok()), the USB case should be safe.
> >
>
> It's not just the USB writer, it's unrelated process that are allocating
> memory at the same time the writing happens. What we want to avoid is a
> situation where something like firefox or evolution or even gnome-terminal
> is performing a small read and gets either
>
> a) started for IO bandwidth and stalls (not the focus here obviously)
> b) enter page reclaim, finds PG_reclaim pages from the USB write and stalls
>
> It's (b) we need to watch out for. I accept that this patch is heading
> in the right direction and that the tracepoint can be used to identify
> processes get throttled unfairly. Before merging, it'd be nice to hear of
> such a test and include details in the changelog similar to the test case
> in https://bugzilla.kernel.org/show_bug.cgi?id=31142 (a bug that lasted a
> *long* time as it turned out, fixes merged for 3.3 with sync-light migration).
To avoid PG_reclaim waiting for unrelated page allocators, I improved
the patch to
a) distinguish dirtier tasks and unrelated clean tasks by testing
1) whether __GFP_WRITE is set
2) whether current->nr_dirtied changed recently
b) put dirtier tasks to wait at lower dirty fill levels (~50%) and
clean tasks at much higher threshold (80%).
To test the "desktop responsiveness on dd to USB" case, I run one dd
to USB (6-7MB/s) and another dd from a huge sparse file (857 MB/s) on
kernel 3.3.0-rc4 plus the to-be-posted dirty reclaim patch. The result
is perfect, I see no single waits of any kind:
$ grep -r . /debug/vm
/debug/vm/nr_reclaim_throttle_clean:0
/debug/vm/nr_reclaim_throttle_kswapd:0
/debug/vm/nr_reclaim_throttle_recent_write:0
/debug/vm/nr_reclaim_throttle_write:0
/debug/vm/nr_congestion_wait:0
/debug/vm/nr_reclaim_wait_congested:0
/debug/vm/nr_reclaim_wait_writeback:0
/debug/vm/nr_migrate_wait_writeback:0
The above numbers are collected by a debug patch that accounts the
wait_on_page_writeback(), congestion_wait(), wait_iff_congested(), and
reclaim_wait() sleeps.
I went further to currently do
- startx
- start a number of applications and switching through all windows in a loop
- run "usemem 6G" (it's a 8G memory box)
- swapon
All goes pretty smooth until swap is turned on. Upon swapon, the
system immediately becomes unusably slow. The wait numbers go up to
/debug/vm/nr_reclaim_throttle_clean:0
/debug/vm/nr_reclaim_throttle_kswapd:1771
/debug/vm/nr_reclaim_throttle_recent_write:0
/debug/vm/nr_reclaim_throttle_write:0
/debug/vm/nr_congestion_wait:33
/debug/vm/nr_reclaim_wait_congested:8626
/debug/vm/nr_reclaim_wait_writeback:0
/debug/vm/nr_migrate_wait_writeback:0
There are over 8k wait_iff_congested() sleeps and 1.7k reclaim_wait() sleeps
for kswapd. I'm a bit surprised to see nr_reclaim_throttle_write still
remains 0 while nr_reclaim_throttle_kswapd goes up..
The congestion_wait traces indicate kswapd run into trouble around some time
and triggered a series of congestion_wait()s. This should in turn decrease its
scan priority and triggered the reclaim_wait()s for kswapd.
<...>-903 [011] .... 1230.617682: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
<...>-903 [010] .... 1230.905387: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
<...>-903 [000] .... 1231.016542: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
<...>-903 [010] .... 1231.042926: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
<...>-903 [010] .... 1231.131773: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
<...>-903 [000] .... 1231.143408: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
<...>-903 [010] .... 1231.165119: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
<...>-903 [010] .... 1231.170291: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
<...>-903 [010] .... 1231.273022: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
<...>-903 [004] .... 1231.359771: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
<...>-903 [005] .... 1231.457118: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
<...>-903 [011] .... 1231.514525: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
<...>-903 [011] .... 1231.515736: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
<...>-903 [002] .... 1231.556657: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
<...>-903 [002] .... 1231.591155: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
<...>-903 [002] .... 1231.592753: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
<...>-903 [000] .... 1231.597581: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
<...>-903 [002] .... 1231.614586: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
<...>-903 [000] .... 1231.639371: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
<...>-903 [003] .... 1231.653001: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
<...>-903 [003] .... 1231.666135: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
<...>-903 [003] .... 1231.676705: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
<...>-903 [004] .... 1231.715875: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
<...>-903 [004] .... 1231.717754: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
<...>-903 [004] .... 1231.718269: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
<...>-903 [004] .... 1231.796658: congestion_wait: balance_pgdat+0x477/0x58c kswapd+0x309/0x322
The wait_iff_congested() sleeps w/ swap may be explained by clustered
PG_writeback pages and clustered PG_dirty pages. When there are 32 such pages
come back-to-back in the LRU, wait_iff_congested() can easily be
triggered into sleeps. IMHO the conditions wait_iff_congested() based
on is still kind of "local information" which is not very reliable.
The sleeps are not triggered w/o swap because there are now almost no
pageout() calls for file pages, hence no chances to increase
nr_congested and set ZONE_CONGESTED.
> > > > Some global information on the percent
> > > > of dirty/writeback pages in the LRU list may help. Anyway the added
> > > > tests should still be much better than no protection.
> > > >
> > >
> > > You can tell how many dirty pages and writeback pages are in the zone
> > > already.
> >
> > Right. I changed the test to
> >
> > + if (nr_pgreclaim && nr_pgreclaim >= (nr_taken >> (DEF_PRIORITY-priority)) &&
> > + (!global_reclaim(sc) || !zone_dirty_ok(zone)))
> > + reclaim_wait(HZ/10);
> >
> > And I'd prefer to use a higher threshold than the default 20% for the
> > above zone_dirty_ok() test, so that when Johannes' zone dirty
> > balancing does the job fine, PG_reclaim based page reclaim throttling
> > won't happen at all.
> >
>
> We'd also need to watch that we do not get throttled on small zones
> like ZONE_DMA (which shouldn't happen but still). To detect this if
> it happens, please consider including node and zone information in the
> writeback_reclaim_wait tracepoint. The memcg people might want to be
> able to see the memcg which I guess could be available
>
> node=[NID|memcg]
> NID if zone >=0
> memcg if zone == -1
>
> Which is hacky but avoids creating two tracepoints.
I changed it to mm_vmscan_reclaim_wait w/ output format
+ TP_printk("usec_timeout=%u usec_delayed=%u memcg=%u node=%u zone=%u",
Since it's kind of rate limited code path, there is no need to compact
the output. Actually memcg reclaims do have the node/zone ids which
might be helpful information.
> > > > A global wait queue and reclaim_wait() is introduced. The waiters will
> > > > be wakeup when pages are rotated by end_page_writeback() or lru drain.
> > > >
> > > > I have to say its effectiveness depends on the filesystem... ext4
> > > > and btrfs do fluent IO completions, so reclaim_wait() works pretty
> > > > well:
> > > > dd-14560 [017] .... 1360.894605: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=10000
> > > > dd-14560 [017] .... 1360.904456: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=8000
> > > > dd-14560 [017] .... 1360.908293: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
> > > > dd-14560 [017] .... 1360.923960: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=15000
> > > > dd-14560 [017] .... 1360.927810: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
> > > > dd-14560 [017] .... 1360.931656: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
> > > > dd-14560 [017] .... 1360.943503: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=10000
> > > > dd-14560 [017] .... 1360.953289: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=7000
> > > > dd-14560 [017] .... 1360.957177: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
> > > > dd-14560 [017] .... 1360.972949: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=15000
> > > >
> > > > However XFS does IO completions in very large batches (there may be
> > > > only several big IO completions in one second). So reclaim_wait()
> > > > mostly end up waiting to the full HZ/10 timeout:
> > > >
> > > > dd-4177 [008] .... 866.367661: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > > > dd-4177 [010] .... 866.567583: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > > > dd-4177 [012] .... 866.767458: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > > > dd-4177 [013] .... 866.867419: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > > > dd-4177 [008] .... 867.167266: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > > > dd-4177 [010] .... 867.367168: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > > > dd-4177 [012] .... 867.818950: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > > > dd-4177 [013] .... 867.918905: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > > > dd-4177 [013] .... 867.971657: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=52000
> > > > dd-4177 [013] .... 867.971812: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=0
> > > > dd-4177 [008] .... 868.355700: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > > > dd-4177 [010] .... 868.700515: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > > >
> > >
> > > And where people will get hit by regressions in this area is writing to
> > > vfat and in more rare cases ntfs on USB stick.
> >
> > vfat IO completions seem to lie somewhere between ext4 and xfs:
> >
> > <...>-46385 [010] .... 143570.714470: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > <...>-46385 [008] .... 143570.752391: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=12000
> > <...>-46385 [008] .... 143570.937327: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=52000
> > <...>-46385 [010] .... 143571.160252: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > <...>-46385 [011] .... 143571.286197: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > <...>-46385 [008] .... 143571.329644: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=15000
> > <...>-46385 [008] .... 143571.475433: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=18000
> > <...>-46385 [008] .... 143571.653461: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=52000
> > <...>-46385 [008] .... 143571.839949: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=56000
> > <...>-46385 [010] .... 143572.060816: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > <...>-46385 [011] .... 143572.185754: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > <...>-46385 [008] .... 143572.212522: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=1000
> > <...>-46385 [008] .... 143572.217825: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=2000
> > <...>-46385 [008] .... 143572.312395: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=91000
> > <...>-46385 [008] .... 143572.315122: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=1000
> > <...>-46385 [009] .... 143572.433630: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> > <...>-46385 [010] .... 143572.534569: writeback_reclaim_wait: usec_timeout=100000 usec_delayed=100000
> >
>
> Ok. It's interesting to note that we are stalling a lot there - roughly
> 30ms every second. As long as it's the writer, that's fine. If it's
> firefox, it will create bug reports :)
Good news is, we can reasonably tell firefox from heavy writers with
the help of __GFP_WRITE/->nr_dirtied indicators :-)
> > > > <SNIP>
> > > > @@ -813,6 +815,10 @@ static unsigned long shrink_page_list(st
> > > >
> > > > if (PageWriteback(page)) {
> > > > nr_writeback++;
> > > > + if (PageReclaim(page))
> > > > + nr_pgreclaim++;
> > > > + else
> > > > + SetPageReclaim(page);
> > > > /*
> > >
> > > This check is unexpected. We already SetPageReclaim when queuing pages for
> > > IO from reclaim context and if dirty pages are encountered during the LRU
> > > scan that cannot be queued for IO. How often is it that nr_pgreclaim !=
> > > nr_writeback and by how much do they differ?
> >
> > Quite often, I suspect. The pageout writeback works do 1-8MB write
> > around which may start I/O a bit earlier than the covered pages are
> > encountered by page reclaim. ext4 forces 128MB write chunk size, which
> > further increases the opportunities.
> >
>
> Ok, thanks for the clarification. Stick a wee comment on it please.
OK, added a simple comment:
if (PageWriteback(page)) {
nr_writeback++;
/*
+ * The pageout works do write around which may put
+ * close-to-LRU-tail pages to writeback a bit earlier.
+ */
+ if (PageReclaim(page))
+ nr_pgreclaim++;
+ else
+ SetPageReclaim(page);
> > > > * Synchronous reclaim cannot queue pages for
> > > > * writeback due to the possibility of stack overflow
> > > > @@ -874,12 +880,15 @@ static unsigned long shrink_page_list(st
> > > > nr_dirty++;
> > > >
> > > > /*
> > > > - * Only kswapd can writeback filesystem pages to
> > > > - * avoid risk of stack overflow but do not writeback
> > > > - * unless under significant pressure.
> > > > + * run into the visited page again: we are scanning
> > > > + * faster than the flusher can writeout dirty pages
> > > > */
> > >
> > > which in itself is not an abnormal condition. We get into this situation
> > > when writing to USB. Dirty throttling stops too much memory getting dirtied
> > > but that does not mean we should throttle instead of reclaiming clean pages.
> > >
> > > That's why I worry that if this is aimed at fixing a memcg problem, it
> > > will have the impact of making interactive performance on normal systems
> > > worse.
> >
> > You are right. This patch only addresses the pageout I/O efficiency
> > and dirty throttling problems for a fully dirtied LRU. Next step, I'll
> > think about the interactive performance problem for a less dirtied LRU.
> >
>
> Ok, thanks.
>
> > > <SNIP>
> > >
> > > If the intention is to avoid memcg going OOM prematurely, the
> > > nr_pgreclaim value needs to be treated at a higher level that records
> > > how many PageReclaim pages were encountered. If no progress was made
> > > because all the pages were PageReclaim, then throttle and return 1 to
> > > the page allocator where it will retry the allocation without going OOM
> > > after some pages have been cleaned and reclaimed.
> >
> > Agreed in general, but changed to this test for now, which is made a
> > bit more global wise with the use of zone_dirty_ok().
> >
>
> Ok, sure. We know what to look out for and where unrelated regressions
> might get introduced.
>
> > memcg is ignored due to no dirty accounting (Greg has the patch though).
> > And even zone_dirty_ok() may be inaccurate for the global reclaim, if
> > some memcgs are skipped by the global reclaim by the memcg soft limit.
> >
> > But anyway, it's a handy hack for now. I'm looking into some more
> > radical changes to put most dirty/writeback pages into a standalone
> > LRU list (in addition to your LRU_IMMEDIATE, which I think is a good
> > idea) for addressing the clustered way they tend to lie in the
> > inactive LRU list.
> >
> > + if (nr_pgreclaim && nr_pgreclaim >= (nr_taken >> (DEF_PRIORITY-priority)) &&
> > + (!global_reclaim(sc) || !zone_dirty_ok(zone)))
> > + reclaim_wait(HZ/10);
> >
>
> This should make it harder to get stalled. Your tracepoint should help
> us catch if it happens unnecessarily.
Yeah ;) FYI the above zone_dirty_ok() etc. tests are further expanded to
+ /*
+ * If reclaimed any pages, it's safe from busy scanning. Otherwise when
+ * encountered PG_reclaim pages or writeback work queue congested,
+ * consider I/O throttling. Try to throttle only the dirtier tasks by
+ * honouring higher throttle thresholds to kswapd and other clean tasks.
+ */
+ if (!nr_reclaimed && nr_pgreclaim &&
+ should_throttle_dirty(mz, sc, priority))
+ reclaim_wait(mz, HZ/10);
I choose to not trust the exact number of nr_pgreclaim because it's
kind of *unreliable* local information. Instead, the reclaim priority
looks a better indicator of a long contiguous run of hard-to-reclaim
pages which will be taken into account in should_throttle_dirty().
> > ---
> > Subject: writeback: introduce the pageout work
> > Date: Thu Jul 29 14:41:19 CST 2010
> >
> > This relays file pageout IOs to the flusher threads.
> >
> > The ultimate target is to gracefully handle the LRU lists full of
> > dirty/writeback pages.
> >
>
> It would be worth mentioning in the changelog that this is much more
> important now that page reclaim generally does not writeout filesystem-backed
> pages.
OK. Good point.
> > 1) I/O efficiency
> >
> > The flusher will piggy back the nearby ~10ms worth of dirty pages for I/O.
> >
> > This takes advantage of the time/spacial locality in most workloads: the
> > nearby pages of one file are typically populated into the LRU at the same
> > time, hence will likely be close to each other in the LRU list. Writing
> > them in one shot helps clean more pages effectively for page reclaim.
> >
> > 2) OOM avoidance and scan rate control
> >
> > Typically we do LRU scan w/o rate control and quickly get enough clean
> > pages for the LRU lists not full of dirty pages.
> >
> > Or we can still get a number of freshly cleaned pages (moved to LRU tail
> > by end_page_writeback()) when the queued pageout I/O is completed within
> > tens of milli-seconds.
> >
> > However if the LRU list is small and full of dirty pages, it can be
> > quickly fully scanned and go OOM before the flusher manages to clean
> > enough pages.
> >
>
> It's worth pointing out here that generally this does not happen for global
> reclaim which does dirty throttling but happens easily with memcg LRUs.
Done.
> > A simple yet reliable scheme is employed to avoid OOM and keep scan rate
> > in sync with the I/O rate:
> >
> > if (PageReclaim(page))
> > congestion_wait(HZ/10);
> >
>
> This comment is stale now.
Yeah, will update to
if (encountered PG_reclaim pages)
do some throttle wait
followed by introduction of the basic idea as well as further
considerations for interactive performance.
> > PG_reclaim plays the key role. When dirty pages are encountered, we
> > queue I/O for it,
>
> This is misleading. The process that encounters the dirty page does
> not queue the page for IO unless it is kswapd scanning at high priority
> (currently anyway, you patch changes it). The process that finds the page
> queues work for flusher threads that will queue the actual I/O for it at
> some unknown time in the future.
You are right. Will change to "queue pageout writeback work for it".
> > set PG_reclaim and put it back to the LRU head.
> > So if PG_reclaim pages are encountered again, it means the dirty page
> > has not yet been cleaned by the flusher after a full zone scan. It
> > indicates we are scanning more fast than I/O and shall take a snap.
> >
>
> This is also slightly misleading because the page can be encountered
> after rescanning the inactive list, not necessarily a full zone scan but
> it's a minor point.
Yeah, "inactive list" is more precise. Updated.
> > The runtime behavior on a fully dirtied small LRU list would be:
> > It will start with a quick scan of the list, queuing all pages for I/O.
> > Then the scan will be slowed down by the PG_reclaim pages *adaptively*
> > to match the I/O bandwidth.
> >
> > 3) writeback work coordinations
> >
> > To avoid memory allocations at page reclaim, a mempool for struct
> > wb_writeback_work is created.
> >
> > wakeup_flusher_threads() is removed because it can easily delay the
> > more oriented pageout works and even exhaust the mempool reservations.
> > It's also found to not I/O efficient by frequently submitting writeback
> > works with small ->nr_pages.
> >
> > Background/periodic works will quit automatically, so as to clean the
> > pages under reclaim ASAP. However for now the sync work can still block
> > us for long time.
> >
> > Jan Kara: limit the search scope. Note that the limited search and work
> > pool is not a big problem: 1000 IOs under flight are typically more than
> > enough to saturate the disk. And the overheads of searching in the work
> > list didn't even show up in the perf report.
> >
> > 4) test case
> >
> > Run 2 dd tasks in a 100MB memcg (a very handy test case from Greg Thelen):
> >
> > mkdir /cgroup/x
> > echo 100M > /cgroup/x/memory.limit_in_bytes
> > echo $$ > /cgroup/x/tasks
> >
> > for i in `seq 2`
> > do
> > dd if=/dev/zero of=/fs/f$i bs=1k count=1M &
> > done
> >
> > Before patch, the dd tasks are quickly OOM killed.
> > After patch, they run well with reasonably good performance and overheads:
> >
> > 1073741824 bytes (1.1 GB) copied, 22.2196 s, 48.3 MB/s
> > 1073741824 bytes (1.1 GB) copied, 22.4675 s, 47.8 MB/s
> >
> > iostat -kx 1
> >
> > Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
> > sda 0.00 0.00 0.00 178.00 0.00 89568.00 1006.38 74.35 417.71 4.80 85.40
> > sda 0.00 2.00 0.00 191.00 0.00 94428.00 988.77 53.34 219.03 4.34 82.90
> > sda 0.00 20.00 0.00 196.00 0.00 97712.00 997.06 71.11 337.45 4.77 93.50
> > sda 0.00 5.00 0.00 175.00 0.00 84648.00 967.41 54.03 316.44 5.06 88.60
> > sda 0.00 0.00 0.00 186.00 0.00 92432.00 993.89 56.22 267.54 5.38 100.00
> > sda 0.00 1.00 0.00 183.00 0.00 90156.00 985.31 37.99 325.55 4.33 79.20
> > sda 0.00 0.00 0.00 175.00 0.00 88692.00 1013.62 48.70 218.43 4.69 82.10
> > sda 0.00 0.00 0.00 196.00 0.00 97528.00 995.18 43.38 236.87 5.10 100.00
> > sda 0.00 0.00 0.00 179.00 0.00 88648.00 990.48 45.83 285.43 5.59 100.00
> > sda 0.00 0.00 0.00 178.00 0.00 88500.00 994.38 28.28 158.89 4.99 88.80
> > sda 0.00 0.00 0.00 194.00 0.00 95852.00 988.16 32.58 167.39 5.15 100.00
> > sda 0.00 2.00 0.00 215.00 0.00 105996.00 986.01 41.72 201.43 4.65 100.00
> > sda 0.00 4.00 0.00 173.00 0.00 84332.00 974.94 50.48 260.23 5.76 99.60
> > sda 0.00 0.00 0.00 182.00 0.00 90312.00 992.44 36.83 212.07 5.49 100.00
> > sda 0.00 8.00 0.00 195.00 0.00 95940.50 984.01 50.18 221.06 5.13 100.00
> > sda 0.00 1.00 0.00 220.00 0.00 108852.00 989.56 40.99 202.68 4.55 100.00
> > sda 0.00 2.00 0.00 161.00 0.00 80384.00 998.56 37.19 268.49 6.21 100.00
> > sda 0.00 4.00 0.00 182.00 0.00 90830.00 998.13 50.58 239.77 5.49 100.00
> > sda 0.00 0.00 0.00 197.00 0.00 94877.00 963.22 36.68 196.79 5.08 100.00
> >
> > avg-cpu: %user %nice %system %iowait %steal %idle
> > 0.25 0.00 15.08 33.92 0.00 50.75
> > 0.25 0.00 14.54 35.09 0.00 50.13
> > 0.50 0.00 13.57 32.41 0.00 53.52
> > 0.50 0.00 11.28 36.84 0.00 51.38
> > 0.50 0.00 15.75 32.00 0.00 51.75
> > 0.50 0.00 10.50 34.00 0.00 55.00
> > 0.50 0.00 17.63 27.46 0.00 54.41
> > 0.50 0.00 15.08 30.90 0.00 53.52
> > 0.50 0.00 11.28 32.83 0.00 55.39
> > 0.75 0.00 16.79 26.82 0.00 55.64
> > 0.50 0.00 16.08 29.15 0.00 54.27
> > 0.50 0.00 13.50 30.50 0.00 55.50
> > 0.50 0.00 14.32 35.18 0.00 50.00
> > 0.50 0.00 12.06 33.92 0.00 53.52
> > 0.50 0.00 17.29 30.58 0.00 51.63
> > 0.50 0.00 15.08 29.65 0.00 54.77
> > 0.50 0.00 12.53 29.32 0.00 57.64
> > 0.50 0.00 15.29 31.83 0.00 52.38
> >
> > The global dd numbers for comparison:
> >
> > Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
> > sda 0.00 0.00 0.00 189.00 0.00 95752.00 1013.25 143.09 684.48 5.29 100.00
> > sda 0.00 0.00 0.00 208.00 0.00 105480.00 1014.23 143.06 733.29 4.81 100.00
> > sda 0.00 0.00 0.00 161.00 0.00 81924.00 1017.69 141.71 757.79 6.21 100.00
> > sda 0.00 0.00 0.00 217.00 0.00 109580.00 1009.95 143.09 749.55 4.61 100.10
> > sda 0.00 0.00 0.00 187.00 0.00 94728.00 1013.13 144.31 773.67 5.35 100.00
> > sda 0.00 0.00 0.00 189.00 0.00 95752.00 1013.25 144.14 742.00 5.29 100.00
> > sda 0.00 0.00 0.00 177.00 0.00 90032.00 1017.31 143.32 656.59 5.65 100.00
> > sda 0.00 0.00 0.00 215.00 0.00 108640.00 1010.60 142.90 817.54 4.65 100.00
> > sda 0.00 2.00 0.00 166.00 0.00 83858.00 1010.34 143.64 808.61 6.02 100.00
> > sda 0.00 0.00 0.00 186.00 0.00 92813.00 997.99 141.18 736.95 5.38 100.00
> > sda 0.00 0.00 0.00 206.00 0.00 104456.00 1014.14 146.27 729.33 4.85 100.00
> > sda 0.00 0.00 0.00 213.00 0.00 107024.00 1004.92 143.25 705.70 4.69 100.00
> > sda 0.00 0.00 0.00 188.00 0.00 95748.00 1018.60 141.82 764.78 5.32 100.00
> >
> > avg-cpu: %user %nice %system %iowait %steal %idle
> > 0.51 0.00 11.22 52.30 0.00 35.97
> > 0.25 0.00 10.15 52.54 0.00 37.06
> > 0.25 0.00 5.01 56.64 0.00 38.10
> > 0.51 0.00 15.15 43.94 0.00 40.40
> > 0.25 0.00 12.12 48.23 0.00 39.39
> > 0.51 0.00 11.20 53.94 0.00 34.35
> > 0.26 0.00 9.72 51.41 0.00 38.62
> > 0.76 0.00 9.62 50.63 0.00 38.99
> > 0.51 0.00 10.46 53.32 0.00 35.71
> > 0.51 0.00 9.41 51.91 0.00 38.17
> > 0.25 0.00 10.69 49.62 0.00 39.44
> > 0.51 0.00 12.21 52.67 0.00 34.61
> > 0.51 0.00 11.45 53.18 0.00 34.86
> >
> > XXX: commit NFS unstable pages via write_inode()
> > XXX: the added congestion_wait() may be undesirable in some situations
> >
>
> This second XXX may also not be redundant.
Sure. Removed.
> > CC: Jan Kara <jack@suse.cz>
> > CC: Mel Gorman <mgorman@suse.de>
> > Acked-by: Rik van Riel <riel@redhat.com>
> > CC: Greg Thelen <gthelen@google.com>
> > CC: Minchan Kim <minchan.kim@gmail.com>
> > Signed-off-by: Wu Fengguang <fengguang.wu@intel.com>
> > ---
> > fs/fs-writeback.c | 169 ++++++++++++++++++++++++++++-
> > include/linux/backing-dev.h | 2
> > include/linux/writeback.h | 4
> > include/trace/events/writeback.h | 19 ++-
> > mm/backing-dev.c | 35 ++++++
> > mm/swap.c | 1
> > mm/vmscan.c | 32 +++--
> > 7 files changed, 245 insertions(+), 17 deletions(-)
> >
> > - move congestion_wait() out of the page lock: it's blocking btrfs lock_delalloc_pages()
> >
> > --- linux.orig/include/linux/backing-dev.h 2012-02-14 20:11:21.000000000 +0800
> > +++ linux/include/linux/backing-dev.h 2012-02-15 12:34:24.000000000 +0800
> > @@ -304,6 +304,8 @@ void clear_bdi_congested(struct backing_
> > void set_bdi_congested(struct backing_dev_info *bdi, int sync);
> > long congestion_wait(int sync, long timeout);
> > long wait_iff_congested(struct zone *zone, int sync, long timeout);
> > +long reclaim_wait(long timeout);
> > +void reclaim_rotated(void);
> >
> > static inline bool bdi_cap_writeback_dirty(struct backing_dev_info *bdi)
> > {
> > --- linux.orig/mm/backing-dev.c 2012-02-14 20:11:21.000000000 +0800
> > +++ linux/mm/backing-dev.c 2012-02-15 12:34:19.000000000 +0800
> > @@ -873,3 +873,38 @@ out:
> > return ret;
> > }
> > EXPORT_SYMBOL(wait_iff_congested);
> > +
> > +static DECLARE_WAIT_QUEUE_HEAD(reclaim_wqh);
> > +
>
> Should this be declared on a per-NUMA node basis to avoid throttling on one
> node being woken up by activity on an unrelated node? reclaim_rorated()
> is called from a context that has a page so looking up the waitqueue would
> be easy. Grep for place that initialise kswapd_wait and the initialisation
> code will be easier although watch that if a node is hot-removed that
> the queue is woken.
OK I've changed reclaim_wqh to per-NUMA node wqh. It may be further
expanded to per-memcg wqh in similar way.
But note that it will now only wake the node corresponding to one
random page in the pagevec. It should work well enough in practice,
though.
> > +/**
> > + * reclaim_wait - wait for some pages being rotated to the LRU tail
> > + * @timeout: timeout in jiffies
> > + *
> > + * Wait until @timeout, or when some (typically PG_reclaim under writeback)
> > + * pages rotated to the LRU so that page reclaim can make progress.
> > + */
> > +long reclaim_wait(long timeout)
> > +{
> > + long ret;
> > + unsigned long start = jiffies;
> > + DEFINE_WAIT(wait);
> > +
> > + prepare_to_wait(&reclaim_wqh, &wait, TASK_KILLABLE);
> > + ret = io_schedule_timeout(timeout);
> > + finish_wait(&reclaim_wqh, &wait);
> > +
> > + trace_writeback_reclaim_wait(jiffies_to_usecs(timeout),
> > + jiffies_to_usecs(jiffies - start));
> > +
> > + return ret;
> > +}
> > +EXPORT_SYMBOL(reclaim_wait);
> > +
>
> Why do we export this? Only vmscan.c is calling it and I'm scratching my
> head trying to figure out why a kernel module would want to call it.
Err a typical copy&paste error... Removed this and further moved
reclaim_wait()/reclaim_rotated() to vmscan.c.
> > +void reclaim_rotated()
> > +{
>
> style nit
>
> void reclaim_rotated(void)
Updated, to...
void reclaim_rotated(struct page *page)
which will wakeup tasks from the wqh
&NODE_DATA(page_to_nid(page))->reclaim_wait
> > + wait_queue_head_t *wqh = &reclaim_wqh;
> > +
> > + if (waitqueue_active(wqh))
> > + wake_up(wqh);
> > +}
> > +
> > --- linux.orig/mm/swap.c 2012-02-14 20:11:21.000000000 +0800
> > +++ linux/mm/swap.c 2012-02-15 12:27:35.000000000 +0800
> > @@ -253,6 +253,7 @@ static void pagevec_move_tail(struct pag
> >
> > pagevec_lru_move_fn(pvec, pagevec_move_tail_fn, &pgmoved);
> > __count_vm_events(PGROTATED, pgmoved);
> > + reclaim_rotated();
> > }
> >
> > /*
> > --- linux.orig/mm/vmscan.c 2012-02-14 20:11:21.000000000 +0800
> > +++ linux/mm/vmscan.c 2012-02-16 17:23:17.000000000 +0800
> > @@ -767,7 +767,8 @@ static unsigned long shrink_page_list(st
> > struct scan_control *sc,
> > int priority,
> > unsigned long *ret_nr_dirty,
> > - unsigned long *ret_nr_writeback)
> > + unsigned long *ret_nr_writeback,
> > + unsigned long *ret_nr_pgreclaim)
> > {
> > LIST_HEAD(ret_pages);
> > LIST_HEAD(free_pages);
> > @@ -776,6 +777,7 @@ static unsigned long shrink_page_list(st
> > unsigned long nr_congested = 0;
> > unsigned long nr_reclaimed = 0;
> > unsigned long nr_writeback = 0;
> > + unsigned long nr_pgreclaim = 0;
> >
> > cond_resched();
> >
> > @@ -813,6 +815,10 @@ static unsigned long shrink_page_list(st
> >
> > if (PageWriteback(page)) {
> > nr_writeback++;
> > + if (PageReclaim(page))
> > + nr_pgreclaim++;
> > + else
> > + SetPageReclaim(page);
> > /*
> > * Synchronous reclaim cannot queue pages for
> > * writeback due to the possibility of stack overflow
> > @@ -874,12 +880,15 @@ static unsigned long shrink_page_list(st
> > nr_dirty++;
> >
> > /*
> > - * Only kswapd can writeback filesystem pages to
> > - * avoid risk of stack overflow but do not writeback
> > - * unless under significant pressure.
> > + * run into the visited page again: we are scanning
> > + * faster than the flusher can writeout dirty pages
> > */
> > - if (page_is_file_cache(page) &&
> > - (!current_is_kswapd() || priority >= DEF_PRIORITY - 2)) {
> > + if (page_is_file_cache(page) && PageReclaim(page)) {
> > + nr_pgreclaim++;
> > + goto keep_locked;
> > + }
>
> This change means that kswapd is no longer doing any writeback from page
> reclaim.
Not really skipping all pageout()s. Here we skip PG_reclaim dirty
pages because there are already pageout works queued for them.
> Was that intended because it's not discussed in the changelog. I
> know writeback from kswapd is poor in terms of IO performance but it's a
> last resort for freeing a page when reclaim is in trouble. If we are to
> disable it and depend 100% on the flusher threads, it should be in its
> own patch for bisection reasons if nothing else.
I'm actually totally with your points. The patch will still default
to pageout() if failed to queue pageout work for the page.
But yes, there is the possibility of pageout works delayed by some
sync works for a dozen of seconds. In this case the sync work may also
hit some PG_reclaim pages and do the clean-and-rotate job for us. But
it would certainly be much better to have some kind of guarantee...
> > + if (page_is_file_cache(page) && mapping &&
> > + flush_inode_page(mapping, page, false) >= 0) {
> > /*
> > * Immediately reclaim when written back.
> > * Similar in principal to deactivate_page()
> > @@ -1028,6 +1037,7 @@ keep_lumpy:
> > count_vm_events(PGACTIVATE, pgactivate);
> > *ret_nr_dirty += nr_dirty;
> > *ret_nr_writeback += nr_writeback;
> > + *ret_nr_pgreclaim += nr_pgreclaim;
> > return nr_reclaimed;
> > }
> >
> > @@ -1509,6 +1519,7 @@ shrink_inactive_list(unsigned long nr_to
> > unsigned long nr_file;
> > unsigned long nr_dirty = 0;
> > unsigned long nr_writeback = 0;
> > + unsigned long nr_pgreclaim = 0;
> > isolate_mode_t reclaim_mode = ISOLATE_INACTIVE;
> > struct zone *zone = mz->zone;
> >
> > @@ -1559,13 +1570,13 @@ shrink_inactive_list(unsigned long nr_to
> > spin_unlock_irq(&zone->lru_lock);
> >
> > nr_reclaimed = shrink_page_list(&page_list, mz, sc, priority,
> > - &nr_dirty, &nr_writeback);
> > + &nr_dirty, &nr_writeback, &nr_pgreclaim);
> >
> > /* Check if we should syncronously wait for writeback */
> > if (should_reclaim_stall(nr_taken, nr_reclaimed, priority, sc)) {
> > set_reclaim_mode(priority, sc, true);
> > nr_reclaimed += shrink_page_list(&page_list, mz, sc,
> > - priority, &nr_dirty, &nr_writeback);
> > + priority, &nr_dirty, &nr_writeback, &nr_pgreclaim);
> > }
> >
> > spin_lock_irq(&zone->lru_lock);
> > @@ -1608,6 +1619,9 @@ shrink_inactive_list(unsigned long nr_to
> > */
> > if (nr_writeback && nr_writeback >= (nr_taken >> (DEF_PRIORITY-priority)))
> > wait_iff_congested(zone, BLK_RW_ASYNC, HZ/10);
> > + if (nr_pgreclaim && nr_pgreclaim >= (nr_taken >> (DEF_PRIORITY-priority)) &&
> > + (!global_reclaim(sc) || !zone_dirty_ok(zone)))
> > + reclaim_wait(HZ/10);
> >
>
> I prefer this but it would be nice if there was a comment explaining it
> or at least expand the comment explaining how nr_writeback can lead to
> wait_iff_congested() being called.
Sure, done.
> > trace_mm_vmscan_lru_shrink_inactive(zone->zone_pgdat->node_id,
> > zone_idx(zone),
> > @@ -2382,8 +2396,6 @@ static unsigned long do_try_to_free_page
> > */
> > writeback_threshold = sc->nr_to_reclaim + sc->nr_to_reclaim / 2;
> > if (total_scanned > writeback_threshold) {
> > - wakeup_flusher_threads(laptop_mode ? 0 : total_scanned,
> > - WB_REASON_TRY_TO_FREE_PAGES);
> > sc->may_writepage = 1;
> > }
> >
> > --- linux.orig/fs/fs-writeback.c 2012-02-14 20:11:21.000000000 +0800
> > +++ linux/fs/fs-writeback.c 2012-02-15 12:27:35.000000000 +0800
> > @@ -41,6 +41,8 @@ struct wb_writeback_work {
> > long nr_pages;
> > struct super_block *sb;
> > unsigned long *older_than_this;
> > + struct inode *inode;
> > + pgoff_t offset;
> > enum writeback_sync_modes sync_mode;
> > unsigned int tagged_writepages:1;
> > unsigned int for_kupdate:1;
> > @@ -65,6 +67,27 @@ struct wb_writeback_work {
> > */
> > int nr_pdflush_threads;
> >
> > +static mempool_t *wb_work_mempool;
> > +
> > +static void *wb_work_alloc(gfp_t gfp_mask, void *pool_data)
> > +{
> > + /*
> > + * bdi_flush_inode_range() may be called on page reclaim
> > + */
> > + if (current->flags & PF_MEMALLOC)
> > + return NULL;
> > +
>
> This check is why I worry about kswapd being unable to write pages at
> all. If the mempool is depleted for whatever reason, reclaim has no way
> of telling the flushers what work is needed or waking them. Potentially,
> we could be waiting a long time for pending flusher work to complete to
> free up a slot. I recognise it may not be bad in practice because the
> pool is large and other work will be completing but it's why kswapd not
> writing back pages should be in its own patch.
See above, the patch still falls back to pageout() if we fail to queue
the work. The main problem is, it's possible for the *queued* work to
be delayed for long time by some other writeback works. It should be
trivial to make pageout works the No.1 priority for the flushers, but
then that may risk delaying sync() works for long time...
> > + return kmalloc(sizeof(struct wb_writeback_work), gfp_mask);
> > +}
> > +
> > +static __init int wb_work_init(void)
> > +{
> > + wb_work_mempool = mempool_create(1024,
> > + wb_work_alloc, mempool_kfree, NULL);
> > + return wb_work_mempool ? 0 : -ENOMEM;
> > +}
> > +fs_initcall(wb_work_init);
> > +
> > /**
> > * writeback_in_progress - determine whether there is writeback in progress
> > * @bdi: the device's backing_dev_info structure.
> > @@ -129,7 +152,7 @@ __bdi_start_writeback(struct backing_dev
> > * This is WB_SYNC_NONE writeback, so if allocation fails just
> > * wakeup the thread for old dirty data writeback
> > */
> > - work = kzalloc(sizeof(*work), GFP_ATOMIC);
> > + work = mempool_alloc(wb_work_mempool, GFP_NOWAIT);
> > if (!work) {
> > if (bdi->wb.task) {
> > trace_writeback_nowork(bdi);
> > @@ -138,6 +161,7 @@ __bdi_start_writeback(struct backing_dev
> > return;
> > }
> >
> > + memset(work, 0, sizeof(*work));
> > work->sync_mode = WB_SYNC_NONE;
> > work->nr_pages = nr_pages;
> > work->range_cyclic = range_cyclic;
> > @@ -186,6 +210,125 @@ void bdi_start_background_writeback(stru
> > spin_unlock_bh(&bdi->wb_lock);
> > }
> >
> > +static bool extend_writeback_range(struct wb_writeback_work *work,
> > + pgoff_t offset,
> > + unsigned long write_around_pages)
> > +{
>
> comment on what this function is for and what the return values mean.
>
> "returns true if the wb_writeback_work now encompasses the request"
>
> or something
Here it goes:
/*
* Check if @work already covers @offset, or try to extend it to cover @offset.
* Returns true if the wb_writeback_work now encompasses the requested offset.
*/
> > + pgoff_t end = work->offset + work->nr_pages;
> > +
> > + if (offset >= work->offset && offset < end)
> > + return true;
> > +
>
> This does not ensure that the full span of
> offset -> offset+write_around_pages is encompassed by work. All it
> checks is that the start of the requested range is going to be handled.
Oh this function does not work as you expected.
I'd better change the "write_around_pages" parameter name to "unit".
Every pageout work starts with [begin, begin+unit] and each time it's
extended, the added length will be N*unit.
extend_writeback_range() only aims to extend the work to encompass the
single page at @offset. "unit" merely serves as the extending unit.
> I guess it's ok because the page reclaims cares about is covered and
> avoids a situation where too much IO is being queued. It's unclear if
> this is what you intended though because you check for too much IO being
> queued in the next block.
Yeah both the number of pending pageout works and the chunk size of
each work will be limited.
> > + /*
> > + * for sequential workloads with good locality, include up to 8 times
> > + * more data in one chunk
> > + */
> > + if (work->nr_pages >= 8 * write_around_pages)
> > + return false;
> > +
> > + /* the unsigned comparison helps eliminate one compare */
> > + if (work->offset - offset < write_around_pages) {
> > + work->nr_pages += write_around_pages;
> > + work->offset -= write_around_pages;
> > + return true;
> > + }
> > +
> > + if (offset - end < write_around_pages) {
> > + work->nr_pages += write_around_pages;
> > + return true;
> > + }
> > +
> > + return false;
> > +}
> > +
> > +/*
> > + * schedule writeback on a range of inode pages.
> > + */
> > +static struct wb_writeback_work *
> > +bdi_flush_inode_range(struct backing_dev_info *bdi,
> > + struct inode *inode,
> > + pgoff_t offset,
> > + pgoff_t len,
> > + bool wait)
> > +{
> > + struct wb_writeback_work *work;
> > +
> > + if (!igrab(inode))
> > + return ERR_PTR(-ENOENT);
> > +
>
> Explain why the igrab is necessary. I think it's because we are calling
> this from page reclaim context and the only thing pinning the
> address_space is the page lock . If I'm right, it should be made clear
> in the comment for bdi_flush_inode_range that this should only be called
> from page reclaim context. Maybe even VM_BUG_ON if
> !(current->flags & PF_MEMALLOC)?
OK, I added this comment.
/*
* Grab the inode until work execution time. We are calling this from
* page reclaim context and the only thing pinning the address_space
* for the moment is the page lock.
*/
Maybe not VM_BUG_ON, since it won't hurt that much if called by
someone else ;)
> > + work = mempool_alloc(wb_work_mempool, wait ? GFP_NOIO : GFP_NOWAIT);
> > + if (!work) {
> > + trace_printk("wb_work_mempool alloc fail\n");
> > + return ERR_PTR(-ENOMEM);
> > + }
> > +
> > + memset(work, 0, sizeof(*work));
> > + work->sync_mode = WB_SYNC_NONE;
> > + work->inode = inode;
> > + work->offset = offset;
> > + work->nr_pages = len;
> > + work->reason = WB_REASON_PAGEOUT;
> > +
> > + bdi_queue_work(bdi, work);
> > +
> > + return work;
> > +}
> > +
> > +/*
> > + * Called by page reclaim code to flush the dirty page ASAP. Do write-around to
> > + * improve IO throughput. The nearby pages will have good chance to reside in
> > + * the same LRU list that vmscan is working on, and even close to each other
> > + * inside the LRU list in the common case of sequential read/write.
> > + *
> > + * ret > 0: success, found/reused a previous writeback work
> > + * ret = 0: success, allocated/queued a new writeback work
> > + * ret < 0: failed
> > + */
> > +long flush_inode_page(struct address_space *mapping,
> > + struct page *page,
> > + bool wait)
> > +{
> > + struct backing_dev_info *bdi = mapping->backing_dev_info;
> > + struct inode *inode = mapping->host;
> > + struct wb_writeback_work *work;
> > + unsigned long write_around_pages;
> > + pgoff_t offset = page->index;
> > + int i;
> > + long ret = 0;
> > +
> > + if (unlikely(!inode))
> > + return -ENOENT;
> > +
> > + /*
> > + * piggy back 8-15ms worth of data
> > + */
> > + write_around_pages = bdi->avg_write_bandwidth + MIN_WRITEBACK_PAGES;
> > + write_around_pages = rounddown_pow_of_two(write_around_pages) >> 6;
> > +
> > + i = 1;
> > + spin_lock_bh(&bdi->wb_lock);
> > + list_for_each_entry_reverse(work, &bdi->work_list, list) {
> > + if (work->inode != inode)
> > + continue;
> > + if (extend_writeback_range(work, offset, write_around_pages)) {
> > + ret = i;
> > + break;
> > + }
> > + if (i++ > 100) /* limit search depth */
> > + break;
>
> No harm to move Jan's comment on depth limit search to here adding why
> 100 is as good as number as any to use.
Good idea. Here is the refined work pool throttling scheme:
/*
* Tailored for vmscan which may submit lots of pageout works. The page reclaim
* will try to slow down the pageout work submission rate when the queue size
* grows to LOTS_OF_WRITEBACK_WORKS. flush_inode_page() will accordingly limit
* its search depth to (2 * LOTS_OF_WRITEBACK_WORKS).
*
* Note that the limited search and work pool is not a big problem: 1024 IOs
* under flight are typically more than enough to saturate the disk. And the
* overheads of searching in the work list didn't even show up in perf report.
*/
#define WB_WORK_MEMPOOL_SIZE 1024
#define LOTS_OF_WRITEBACK_WORKS (WB_WORK_MEMPOOL_SIZE / 8)
Thanks,
Fengguang
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 33+ messages in thread
end of thread, other threads:[~2012-02-27 14:29 UTC | newest]
Thread overview: 33+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-02-08 7:55 memcg writeback (was Re: [Lsf-pc] [LSF/MM TOPIC] memcg topics.) Greg Thelen
2012-02-08 9:31 ` Wu Fengguang
2012-02-08 20:54 ` Ying Han
2012-02-09 13:50 ` Wu Fengguang
2012-02-13 18:40 ` Ying Han
2012-02-10 5:51 ` Greg Thelen
2012-02-10 5:52 ` Greg Thelen
2012-02-10 9:20 ` Wu Fengguang
2012-02-10 11:47 ` Wu Fengguang
2012-02-11 12:44 ` reclaim the LRU lists full of dirty/writeback pages Wu Fengguang
2012-02-11 14:55 ` Rik van Riel
2012-02-12 3:10 ` Wu Fengguang
2012-02-12 6:45 ` Wu Fengguang
2012-02-13 15:43 ` Jan Kara
2012-02-14 10:03 ` Wu Fengguang
2012-02-14 13:29 ` Jan Kara
2012-02-16 4:00 ` Wu Fengguang
2012-02-16 12:44 ` Jan Kara
2012-02-16 13:32 ` Wu Fengguang
2012-02-16 14:06 ` Wu Fengguang
2012-02-17 16:41 ` Wu Fengguang
2012-02-20 14:00 ` Jan Kara
2012-02-14 10:19 ` Mel Gorman
2012-02-14 13:18 ` Wu Fengguang
2012-02-14 13:35 ` Wu Fengguang
2012-02-14 15:51 ` Mel Gorman
2012-02-16 9:50 ` Wu Fengguang
2012-02-16 17:31 ` Mel Gorman
2012-02-27 14:24 ` Fengguang Wu
2012-02-16 0:00 ` KAMEZAWA Hiroyuki
2012-02-16 3:04 ` Wu Fengguang
2012-02-16 3:52 ` KAMEZAWA Hiroyuki
2012-02-16 4:05 ` Wu Fengguang
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.