re: mmap over nfs leads to excessive system load

re: mmap over nfs leads to excessive system load

[Kenny Simpson]

I ran the same test again against 2.6.15-rc, and got pretty much the same thing.  It starts nice
and fast (30+MB/s, but drops down to under 10MB/s with the system time pegging one CPU).

Here is the oprofile result:

CPU: P4 / Xeon with 2 hyper-threads, speed 2658.47 MHz (estimated)
Counted GLOBAL_POWER_EVENTS events (time during which processor is not stopped) with a unit mask
of 0x01 (mandatory) count 100000
samples  %        symbol name
412585   14.6687  find_get_pages_tag
343898   12.2267  mpage_writepages
290144   10.3155  release_pages
288631   10.2617  unlock_page
286181   10.1746  pci_conf1_write
267619    9.5147  clear_page_dirty_for_io
128128    4.5554  __lookup_tag
120895    4.2982  page_waitqueue
52739     1.8750  _spin_lock_irqsave
43623     1.5509  skb_copy_bits
30157     1.0722  __wake_up_bit
29973     1.0656  _read_lock_irqsave


-Kenny



		
__________________________________ 
Yahoo! FareChase: Search multiple travel sites in one click.
http://farechase.yahoo.com

Re: mmap over nfs leads to excessive system load

[William Lee Irwin III]

On Tue, Nov 15, 2005 at 03:47:30PM -0800, Kenny Simpson wrote:
> CPU: P4 / Xeon with 2 hyper-threads, speed 2658.47 MHz (estimated)
> Counted GLOBAL_POWER_EVENTS events (time during which processor is not stopped) with a unit mask
> of 0x01 (mandatory) count 100000
> samples  %        symbol name
> 412585   14.6687  find_get_pages_tag
> 343898   12.2267  mpage_writepages
> 290144   10.3155  release_pages
> 288631   10.2617  unlock_page
> 286181   10.1746  pci_conf1_write
> 267619    9.5147  clear_page_dirty_for_io
> 128128    4.5554  __lookup_tag
> 120895    4.2982  page_waitqueue
> 52739     1.8750  _spin_lock_irqsave
> 43623     1.5509  skb_copy_bits
> 30157     1.0722  __wake_up_bit
> 29973     1.0656  _read_lock_irqsave

67%, or 2/3 of the samples, are in the top 6 functions. Have you tried
instruction-level profiling? It would be interesting to see what
codepaths within the functions are the largest offenders.


-- wli

Re: mmap over nfs leads to excessive system load

[Kenny Simpson]

--- William Lee Irwin III <wli@holomorphy.com> wrote:
> 67%, or 2/3 of the samples, are in the top 6 functions. Have you tried
> instruction-level profiling? It would be interesting to see what
> codepaths within the functions are the largest offenders.
> 
> 
> -- wli
> 
I'm a little new to oprofile, but I'm willing to try any configuration or set of flags that could
be useful.
Are you referring to the -d option in opreport?
 --details / -d
              Show per-instruction details for all selected symbols.

I'll give it a go when I get back to work.

-Kenny



	
		
__________________________________ 
Yahoo! Mail - PC Magazine Editors' Choice 2005 
http://mail.yahoo.com

Re: mmap over nfs leads to excessive system load

[Andrew Morton]

Kenny Simpson <theonetruekenny@yahoo.com> wrote:
>
> I ran the same test again against 2.6.15-rc, and got pretty much the same thing.  It starts nice
> and fast (30+MB/s, but drops down to under 10MB/s with the system time pegging one CPU).
> 
> Here is the oprofile result:
> 
> CPU: P4 / Xeon with 2 hyper-threads, speed 2658.47 MHz (estimated)
> Counted GLOBAL_POWER_EVENTS events (time during which processor is not stopped) with a unit mask
> of 0x01 (mandatory) count 100000
> samples  %        symbol name
> 412585   14.6687  find_get_pages_tag
> 343898   12.2267  mpage_writepages
> 290144   10.3155  release_pages
> 288631   10.2617  unlock_page
> 286181   10.1746  pci_conf1_write
> 267619    9.5147  clear_page_dirty_for_io
> 128128    4.5554  __lookup_tag
> 120895    4.2982  page_waitqueue
> 52739     1.8750  _spin_lock_irqsave
> 43623     1.5509  skb_copy_bits
> 30157     1.0722  __wake_up_bit
> 29973     1.0656  _read_lock_irqsave
> 

Your application walks the file in 2MB hunks, doing ftruncate() each time
to expand the file by another 2MB.

nfs_setattr() implements the truncate.  It syncs the whole file, using
filemap_write_and_wait() (that seems a bit suboptimal.  All we're doing is
increasing i_size??)

So filemap_write_and_wait() has to write 2MB's worth of pages.  Problem is,
_all_ the pages, even the 99% which are clean are tagged as dirty in the
pagecache radix tree.  So find_get_pages_tag() ends up visiting each page
in the file, and blows much CPU doing so.

The writeout happens in mpage_writepages(), which uses
clear_page_dirty_for_io() to clear PG_dirty.  But it doesn't clear the
dirty tag in the radix tree.  It relies upon the filesystem to do the right
thing later on.  Which is all very unpleasant, sorry.  See the explanatory
comment over clear_page_dirty_for_io().

nfs_writepage() doesn't do any of the things which that comment says it
should, hence the radix tree tags are getting out of sync, hence this
problem.

NFS does strange, incomprehensible-to-little-akpms things in its writeout
path.  Ideally, it should run set_page_writeback() prior to unlocking the
page and end_page_writeback() when I/O completes.  That'll keep the VM
happier while fixing this performance glitch.

Re: mmap over nfs leads to excessive system load

[Trond Myklebust]

On Tue, 2005-11-15 at 23:45 -0800, Andrew Morton wrote:

> So filemap_write_and_wait() has to write 2MB's worth of pages.  Problem is,
> _all_ the pages, even the 99% which are clean are tagged as dirty in the
> pagecache radix tree.  So find_get_pages_tag() ends up visiting each page
> in the file, and blows much CPU doing so.
> 
> The writeout happens in mpage_writepages(), which uses
> clear_page_dirty_for_io() to clear PG_dirty.  But it doesn't clear the
> dirty tag in the radix tree.  It relies upon the filesystem to do the right
> thing later on.  Which is all very unpleasant, sorry.  See the explanatory
> comment over clear_page_dirty_for_io().

> nfs_writepage() doesn't do any of the things which that comment says it
> should, hence the radix tree tags are getting out of sync, hence this
> problem.
> 
> NFS does strange, incomprehensible-to-little-akpms things in its writeout
> path.  Ideally, it should run set_page_writeback() prior to unlocking the
> page and end_page_writeback() when I/O completes.  That'll keep the VM
> happier while fixing this performance glitch.

Actually that will screw over performance even further by forcing us to
send out loads of little RPC requests to write 4k pages instead of
allowing us to gather those writes into 32k (or larger) chunks.

Anyhow, does the following patch help?

Cheers,
  Trond
------
NFS: resync to yet more writepage() changes...

 Ensure that we call clear_page_dirty() for pages that have been written
 via writepage().

 Signed-off-by: Trond Myklebust <Trond.Myklebust@netapp.com>
---

 fs/nfs/write.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/fs/nfs/write.c b/fs/nfs/write.c
index 8f71e76..ea77da5 100644
--- a/fs/nfs/write.c
+++ b/fs/nfs/write.c
@@ -213,6 +213,7 @@ static int nfs_writepage_sync(struct nfs
 	} while (count);
 	/* Update file length */
 	nfs_grow_file(page, offset, written);
+	clear_page_dirty(page);
 	/* Set the PG_uptodate flag? */
 	nfs_mark_uptodate(page, offset, written);
 
@@ -238,6 +239,7 @@ static int nfs_writepage_async(struct nf
 		goto out;
 	/* Update file length */
 	nfs_grow_file(page, offset, count);
+	clear_page_dirty(page);
 	/* Set the PG_uptodate flag? */
 	nfs_mark_uptodate(page, offset, count);
 	nfs_unlock_request(req);

Re: mmap over nfs leads to excessive system load

[Kenny Simpson]

--- Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> Anyhow, does the following patch help?

Unfortunately, not:

samples  %        symbol name
545009   15.2546  find_get_pages_tag
450595   12.6120  mpage_writepages
383196   10.7255  release_pages
381479   10.6775  unlock_page
351513    9.8387  clear_page_dirty_for_io
317784    8.8947  pci_conf1_write
167918    4.7000  __lookup_tag
160701    4.4980  page_waitqueue
59142     1.6554  _spin_lock_irqsave
47655     1.3338  skb_copy_bits
39136     1.0954  __wake_up_bit
38143     1.0676  _read_lock_irqsave


With reducing the window size to 32k, things aren't much different:
samples  %        symbol name
474589   21.2001  find_get_pages_tag
370512   16.5509  mpage_writepages
310556   13.8727  release_pages
302571   13.5160  unlock_page
286541   12.7999  clear_page_dirty_for_io
119717    5.3478  page_waitqueue
109920    4.9102  __lookup_tag
33313     1.4881  pci_conf1_write
29198     1.3043  __wake_up_bit
27075     1.2095  _read_lock_irqsave
25009     1.1172  _read_unlock_irq

... except the performance is much worse than with the 2M buffer (hence the 2M choice).  With the
smaller buffer, the throughput starts at 8M/sec and quickly drops to 1M/sec.

-Kenny



		
__________________________________ 
Yahoo! FareChase: Search multiple travel sites in one click.
http://farechase.yahoo.com

Re: mmap over nfs leads to excessive system load

[Trond Myklebust]

On Wed, 2005-11-16 at 07:01 -0800, Kenny Simpson wrote:
> --- Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> > Anyhow, does the following patch help?
> 
> Unfortunately, not:
> 
> samples  %        symbol name
> 545009   15.2546  find_get_pages_tag

Argh... I totally missed the point there with the last patch. We should
be resyncing the page tag with the value of the PG_dirty flag...

OK, please back out the patch that I sent you, and try this one instead.

Cheers,
 Trond

------
NFS: resync to yet more writepage() changes...

 Ensure that we call clear_page_dirty() for pages that have been written
 via writepage().

 Signed-off-by: Trond Myklebust <Trond.Myklebust@netapp.com>
---

 fs/nfs/write.c      |    2 ++
 include/linux/mm.h  |    1 +
 mm/page-writeback.c |   20 ++++++++++++++++++++
 3 files changed, 23 insertions(+), 0 deletions(-)

diff --git a/fs/nfs/write.c b/fs/nfs/write.c
index 8f71e76..61ec355 100644
--- a/fs/nfs/write.c
+++ b/fs/nfs/write.c
@@ -213,6 +213,7 @@ static int nfs_writepage_sync(struct nfs
 	} while (count);
 	/* Update file length */
 	nfs_grow_file(page, offset, written);
+	clear_page_dirty_tag(page);
 	/* Set the PG_uptodate flag? */
 	nfs_mark_uptodate(page, offset, written);
 
@@ -238,6 +239,7 @@ static int nfs_writepage_async(struct nf
 		goto out;
 	/* Update file length */
 	nfs_grow_file(page, offset, count);
+	clear_page_dirty_tag(page);
 	/* Set the PG_uptodate flag? */
 	nfs_mark_uptodate(page, offset, count);
 	nfs_unlock_request(req);
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 1013a42..cb1cfe1 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -730,6 +730,7 @@ int redirty_page_for_writepage(struct wr
 int FASTCALL(set_page_dirty(struct page *page));
 int set_page_dirty_lock(struct page *page);
 int clear_page_dirty_for_io(struct page *page);
+int clear_page_dirty_tag(struct page *page);
 
 extern unsigned long do_mremap(unsigned long addr,
 			       unsigned long old_len, unsigned long new_len,
diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index 74138c9..65c58fa 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -751,6 +751,26 @@ int clear_page_dirty_for_io(struct page 
 	return TestClearPageDirty(page);
 }
 
+/*
+ * Clears the page dirty tag. See comment in clear_page_dirty_for_io()
+ */
+int clear_page_dirty_tag(struct page *page)
+{
+	struct address_space *mapping = page_mapping(page);
+
+	if (mapping) {
+		unsigned long flags;
+
+		write_lock_irqsave(&mapping->tree_lock, flags);
+		if (!PageDirty(page))
+			radix_tree_tag_clear(&mapping->page_tree,
+						page_index(page),
+						PAGECACHE_TAG_DIRTY);
+		write_unlock_irqrestore(&mapping->tree_lock, flags);
+	}
+}
+EXPORT_SYMBOL(clear_page_dirty_tag);
+
 int test_clear_page_writeback(struct page *page)
 {
 	struct address_space *mapping = page_mapping(page);

Re: mmap over nfs leads to excessive system load

[Andrew Morton]

Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
>
> On Wed, 2005-11-16 at 07:01 -0800, Kenny Simpson wrote:
> > --- Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> > > Anyhow, does the following patch help?
> > 
> > Unfortunately, not:
> > 
> > samples  %        symbol name
> > 545009   15.2546  find_get_pages_tag
> 
> Argh... I totally missed the point there with the last patch. We should
> be resyncing the page tag with the value of the PG_dirty flag...
> 
> OK, please back out the patch that I sent you, and try this one instead.
> 
> ...
> diff --git a/fs/nfs/write.c b/fs/nfs/write.c
> index 8f71e76..61ec355 100644
> --- a/fs/nfs/write.c
> +++ b/fs/nfs/write.c
> @@ -213,6 +213,7 @@ static int nfs_writepage_sync(struct nfs
>  	} while (count);
>  	/* Update file length */
>  	nfs_grow_file(page, offset, written);
> +	clear_page_dirty_tag(page);
>  	/* Set the PG_uptodate flag? */
>  	nfs_mark_uptodate(page, offset, written);
>
> ....  
> +int clear_page_dirty_tag(struct page *page)
> +{
> +	struct address_space *mapping = page_mapping(page);
> +
> +	if (mapping) {
> +		unsigned long flags;
> +
> +		write_lock_irqsave(&mapping->tree_lock, flags);
> +		if (!PageDirty(page))
> +			radix_tree_tag_clear(&mapping->page_tree,
> +						page_index(page),
> +						PAGECACHE_TAG_DIRTY);
> +		write_unlock_irqrestore(&mapping->tree_lock, flags);
> +	}
> +}

That will fix it, but the PageWriteback accounting is still wrong.

Is it not possible to use set_page_writeback()/end_page_writeback()?

Are these pages marked "unstable" at this time?

Re: mmap over nfs leads to excessive system load

[Trond Myklebust]

On Wed, 2005-11-16 at 10:00 -0800, Andrew Morton wrote:

> That will fix it, but the PageWriteback accounting is still wrong.
> 
> Is it not possible to use set_page_writeback()/end_page_writeback()?

Not really. The pages aren't flushed at this time. We the point is to
gather several pages and coalesce them into one over-the-wire RPC call.
That means we cannot really do it from inside ->writepage().

We do start the actual RPC calls in ->writepages(), though.

> Are these pages marked "unstable" at this time?

No. "unstable" means that the RPC call to send the pages to the server
has completed, but the pages have not been flushed to disk by the
server. In this case we haven't even sent the pages to the server.

Instead the pages are accounted for in nr_dirty, and are tracked by the
internal NFS 'dirty request' lists. We also mark the inode as being
dirty in order to ensure that pdflush will kick off the actual RPC calls
if nobody else does so first.

Cheers,
  Trond

Re: mmap over nfs leads to excessive system load

[Christoph Hellwig]

On Wed, Nov 16, 2005 at 01:34:22PM -0500, Trond Myklebust wrote:
> Not really. The pages aren't flushed at this time. We the point is to
> gather several pages and coalesce them into one over-the-wire RPC call.
> That means we cannot really do it from inside ->writepage().
> 
> We do start the actual RPC calls in ->writepages(), though.

This is a problem we have in various filesystems.  Except for really
bad OOM situations the filesystem should never get a writeout request
for a single file.  We should really stop having ->writepage called by
the VM and move this kind of batching code into the VM.  I'm runnin into
similar issues for XFS and unwritten/delayed extent conversion once again.

Re: mmap over nfs leads to excessive system load

[Kenny Simpson]

--- Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> OK, please back out the patch that I sent you, and try this one instead.

THAT'S IT!

Very nice..  30MB+/sec sustained for several minutes..
only 25% system CPU and the new profile is:

samples  %        symbol name
1047754  32.9054  pci_conf1_write
193876    6.0888  _spin_lock_irqsave
152897    4.8018  skb_copy_bits
74745     2.3474  _spin_lock
73273     2.3012  __copy_from_user_ll
69273     2.1756  __lookup_tag
65084     2.0440  _spin_unlock_irqrestore
43803     1.3757  sub_preempt_count
32047     1.0065  tcp_v4_rcv
30895     0.9703  schedule
26161     0.8216  kfree

Thank you!

-Kenny



		
__________________________________ 
Yahoo! FareChase: Search multiple travel sites in one click.
http://farechase.yahoo.com

Re: mmap over nfs leads to excessive system load

[Kenny Simpson]

--- Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> OK, please back out the patch that I sent you, and try this one instead.

With jumbo frames, the profile is even happier:
(throughput is a little higher and CPU usage is a little lower too)

samples  %        symbol name
74463    12.1129  skb_copy_bits
30351     4.9372  __lookup_tag
24520     3.9887  _spin_lock
20353     3.3108  _spin_lock_irqsave
19306     3.1405  __copy_from_user_ll
15393     2.5040  __copy_user_zeroing_intel
10014     1.6290  isolate_lru_pages
9002      1.4644  sub_preempt_count
7997      1.3009  debug_smp_processor_id
7691      1.2511  schedule
6999      1.1385  shrink_list
6699      1.0897  tcp_sendmsg
6669      1.0848  radix_tree_delete
6532      1.0626  _write_lock_irqsave
6413      1.0432  __mod_page_state
6170      1.0037  acpi_safe_halt

Again... this is excellent.
So will this make 2.6.16? or can this be called a bug fix for 2.6.15?

-Kenny



		
__________________________________ 
Yahoo! FareChase: Search multiple travel sites in one click.
http://farechase.yahoo.com

Re: mmap over nfs leads to excessive system load

[Andrew Morton]

Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
>
> On Wed, 2005-11-16 at 10:00 -0800, Andrew Morton wrote:
> 
> > That will fix it, but the PageWriteback accounting is still wrong.
> > 
> > Is it not possible to use set_page_writeback()/end_page_writeback()?
> 
> Not really. The pages aren't flushed at this time. We the point is to
> gather several pages and coalesce them into one over-the-wire RPC call.
> That means we cannot really do it from inside ->writepage().
> 

I still don't get it.

Once nfs_writepage() has been called, the page is conceptually "under
writeback", yes?  In that, at some point in the future, it will be written
to backing store.

Hence it's perfectly appropriate to run set_page_writepage() within
nfs_writepage().  It's a matter of finding the right place for the
end_page_writeback().

Re: mmap over nfs leads to excessive system load

[Trond Myklebust]

On Wed, 2005-11-16 at 11:09 -0800, Andrew Morton wrote:
> Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> >
> > On Wed, 2005-11-16 at 10:00 -0800, Andrew Morton wrote:
> > 
> > > That will fix it, but the PageWriteback accounting is still wrong.
> > > 
> > > Is it not possible to use set_page_writeback()/end_page_writeback()?
> > 
> > Not really. The pages aren't flushed at this time. We the point is to
> > gather several pages and coalesce them into one over-the-wire RPC call.
> > That means we cannot really do it from inside ->writepage().
> > 
> 
> I still don't get it.
> 
> Once nfs_writepage() has been called, the page is conceptually "under
> writeback", yes?  In that, at some point in the future, it will be written
> to backing store.
> 
> Hence it's perfectly appropriate to run set_page_writepage() within
> nfs_writepage().  It's a matter of finding the right place for the
> end_page_writeback().

The point is that the process of flushing has not been started at that
time, so anybody that calls wait_on_page_writeback() immediately after
calling writepage() may end up waiting for a very long time indeed
(probably until the next pdflush).

Cheers,
  Trond

Re: mmap over nfs leads to excessive system load

[Kenny Simpson]

I tried the same test, but instead of ftruncate64, I simply did a pwrite64 to get the file
extended... and got 40M+ with mostly outbound traffic, and much less CPU usage..... 

Unfortunately, once my test file hit 4295065601, Bad Things (TM) started to happen.  The system
time went to 100% of a CPU, and the nfs traffic on that mount stopped.

I got an oprofile of the spinning system:
samples  %        symbol name
301039   27.9748  zap_pte_range
156234   14.5184  unmap_vmas
111760   10.3856  __bitmap_weight
103624    9.6295  _spin_lock
97063     9.0198  unmap_page_range
67011     6.2272  unmap_mapping_range
59382     5.5182  sub_preempt_count
51258     4.7633  zap_page_range
25235     2.3450  page_address
16768     1.5582  unmap_mapping_range_vma
13257     1.2319  debug_smp_processor_id
11594     1.0774  add_preempt_count

I also seem unable to kill the test process.

Any ideas?  (2**32 file size issue somewhere?)

-Kenny



	
		
__________________________________ 
Yahoo! Mail - PC Magazine Editors' Choice 2005 
http://mail.yahoo.com

Re: mmap over nfs leads to excessive system load

[Kenny Simpson]

--- Kenny Simpson <theonetruekenny@yahoo.com> wrote:
> I also seem unable to kill the test process.

Root is also unable to kill the process (even -9).
top shows its status a R.

-Kenny



	
		
__________________________________ 
Yahoo! Mail - PC Magazine Editors' Choice 2005 
http://mail.yahoo.com

Re: mmap over nfs leads to excessive system load

[Trond Myklebust]

On Wed, 2005-11-16 at 12:56 -0800, Kenny Simpson wrote:
> I tried the same test, but instead of ftruncate64, I simply did a pwrite64 to get the file
> extended... and got 40M+ with mostly outbound traffic, and much less CPU usage..... 
> 
> Unfortunately, once my test file hit 4295065601, Bad Things (TM) started to happen.  The system
> time went to 100% of a CPU, and the nfs traffic on that mount stopped.
> 
> I got an oprofile of the spinning system:
> samples  %        symbol name
> 301039   27.9748  zap_pte_range
> 156234   14.5184  unmap_vmas
> 111760   10.3856  __bitmap_weight
> 103624    9.6295  _spin_lock
> 97063     9.0198  unmap_page_range
> 67011     6.2272  unmap_mapping_range
> 59382     5.5182  sub_preempt_count
> 51258     4.7633  zap_page_range
> 25235     2.3450  page_address
> 16768     1.5582  unmap_mapping_range_vma
> 13257     1.2319  debug_smp_processor_id
> 11594     1.0774  add_preempt_count
> 
> I also seem unable to kill the test process.
> 
> Any ideas?  (2**32 file size issue somewhere?)

Is this NFSv2?

Cheers,
  Trond

Re: mmap over nfs leads to excessive system load

[Kenny Simpson]

--- Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> 
> Is this NFSv2?
> 
> Cheers,
>   Trond
> 

Not according to mount(1): 
 (rw,vers=3,tcp,rsize=32768,wsize=32768,hard,intr,addr=x.x.x.x)

-Kenny





		
__________________________________ 
Yahoo! FareChase: Search multiple travel sites in one click.
http://farechase.yahoo.com

Re: mmap over nfs leads to excessive system load

[Andrew Morton]

Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
>
> On Wed, 2005-11-16 at 11:09 -0800, Andrew Morton wrote:
> > Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> > >
> > > On Wed, 2005-11-16 at 10:00 -0800, Andrew Morton wrote:
> > > 
> > > > That will fix it, but the PageWriteback accounting is still wrong.
> > > > 
> > > > Is it not possible to use set_page_writeback()/end_page_writeback()?
> > > 
> > > Not really. The pages aren't flushed at this time. We the point is to
> > > gather several pages and coalesce them into one over-the-wire RPC call.
> > > That means we cannot really do it from inside ->writepage().
> > > 
> > 
> > I still don't get it.
> > 
> > Once nfs_writepage() has been called, the page is conceptually "under
> > writeback", yes?  In that, at some point in the future, it will be written
> > to backing store.
> > 
> > Hence it's perfectly appropriate to run set_page_writepage() within
> > nfs_writepage().  It's a matter of finding the right place for the
> > end_page_writeback().
> 
> The point is that the process of flushing has not been started at that
> time, so anybody that calls wait_on_page_writeback() immediately after
> calling writepage() may end up waiting for a very long time indeed
> (probably until the next pdflush).

But block-backed filesytems have the same concern: we don't want to do a
whole bunch of 4k I/Os.  Hence the writepages() interface, which is the
appropriate place to be building up these large I/Os.

NFS does nfw_writepages->mpage_writepages->nfs_writepage and to build the
large I/Os it leaves the I/O pending on return from nfs_writepage().  It
appears to flush any pending pages on the exit path from nfs_writepages().

If that's a correct reading then there doesn't appear to be any way in
which there's dangling I/O left to do after nfs_writepages() completes.

If there _is_ dandling I/O left over then that's problematic, and probably
doesn't buy us much in the way of performance benefit.

Re: mmap over nfs leads to excessive system load

[Kenny Simpson]

--- Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> 
> Is this NFSv2?
> 
> Cheers,
>   Trond
> 
This is reproducible with O_DIRECT, but not without.

The profile looks the same:
samples  %        symbol name
647042   28.4114  zap_pte_range
572195   25.1249  unmap_mapping_range
324291   14.2395  _spin_lock
139259    6.1148  __bitmap_weight
137048    6.0177  zap_page_range
104614    4.5936  unmap_mapping_range_vma
63406     2.7841  debug_smp_processor_id
48906     2.1474  sub_preempt_count
46090     2.0238  unmap_vmas
27966     1.2280  add_preempt_count
23224     1.0198  invalidate_inode_pages2_range
21676     0.9518  unmap_page_range
17825     0.7827  _spin_unlock

I've had mixed results with a local ext3 file with the same test.  One run had a 37 second delay
while crossing 4GB, another happily went by without incident.

-Kenny



	
		
__________________________________ 
Yahoo! Mail - PC Magazine Editors' Choice 2005 
http://mail.yahoo.com

Re: mmap over nfs leads to excessive system load

[Trond Myklebust]

On Wed, 2005-11-16 at 13:31 -0800, Andrew Morton wrote:
> But block-backed filesytems have the same concern: we don't want to do a
> whole bunch of 4k I/Os.  Hence the writepages() interface, which is the
> appropriate place to be building up these large I/Os.
> 
> NFS does nfw_writepages->mpage_writepages->nfs_writepage and to build the
> large I/Os it leaves the I/O pending on return from nfs_writepage().  It
> appears to flush any pending pages on the exit path from nfs_writepages().
> 
> If that's a correct reading then there doesn't appear to be any way in
> which there's dangling I/O left to do after nfs_writepages() completes.

Agreed. AFAICS, nfs_writepages should be quite OK, however writepage()
on its own _is_ problematic.

Look at the usage in write_one_page(), which calls directly down to
->writepage(), and then immediately does a wait_on_page_writeback().

How is the filesystem supposed to distinguish between the cases
"VM->writepage()", and "VM->writepages->mpage_writepages->writepage()"?

Cheers,
  Trond

Re: mmap over nfs leads to excessive system load

[Trond Myklebust]

On Wed, 2005-11-16 at 13:41 -0800, Kenny Simpson wrote:
> --- Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> > 
> > Is this NFSv2?
> > 
> > Cheers,
> >   Trond
> > 
> This is reproducible with O_DIRECT, but not without.

I'm getting lost here. Please could you spell out the testcases that are
not working.

Are you saying that the combination mmap() + pwrite64() fails on
O_DIRECT, but works on ordinary open, and that mmap() + ftruncate64()
always works?

Cheers,
  Trond

Re: mmap over nfs leads to excessive system load

[Kenny Simpson]

--- Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> I'm getting lost here. Please could you spell out the testcases that are
> not working.
> 
> Are you saying that the combination mmap() + pwrite64() fails on
> O_DIRECT, but works on ordinary open, and that mmap() + ftruncate64()
> always works?
> 
> Cheers,
>   Trond
> 
ftruncate64 works with O_DIRECT
ftruncate64 works w/o O_DIRECT

pwrite64 FAILS with O_DIRECT at ~4GB
pwrite64 works w/o O_DIRECT.

I am re-running these tests to confirm (could take a minute).

All opens are with O_RDWR | O_CREAT | O_LARGEFILE.
All test over GbE w/ jumbo frames (8160 mtu) to a netapp filer (via x-over cable).

-Kenny



		
__________________________________ 
Yahoo! FareChase: Search multiple travel sites in one click.
http://farechase.yahoo.com

Re: mmap over nfs leads to excessive system load

[Andrew Morton]

Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
>
> On Wed, 2005-11-16 at 13:31 -0800, Andrew Morton wrote:
> > But block-backed filesytems have the same concern: we don't want to do a
> > whole bunch of 4k I/Os.  Hence the writepages() interface, which is the
> > appropriate place to be building up these large I/Os.
> > 
> > NFS does nfw_writepages->mpage_writepages->nfs_writepage and to build the
> > large I/Os it leaves the I/O pending on return from nfs_writepage().  It
> > appears to flush any pending pages on the exit path from nfs_writepages().
> > 
> > If that's a correct reading then there doesn't appear to be any way in
> > which there's dangling I/O left to do after nfs_writepages() completes.
> 
> Agreed. AFAICS, nfs_writepages should be quite OK, however writepage()
> on its own _is_ problematic.
> 
> Look at the usage in write_one_page(), which calls directly down to
> ->writepage(), and then immediately does a wait_on_page_writeback().
> 
> How is the filesystem supposed to distinguish between the cases
> "VM->writepage()", and "VM->writepages->mpage_writepages->writepage()"?
> 

Via the writeback_control, hopefully.

For write_one_page(), sync_mode==WB_SYNC_ALL, so NFS should start the I/O
immediately (it appears to not do so).

For vmscan->writepage, wbc->for_reclaim is set, so we know that the IO
should be pushed immediately.  nfs_writepage() seems to dtrt here.

With the proposed changes, we don't need that iput() in nfs_writepage(). 
That worries me because I recall from a couple of years back that there are
really subtle races with doing iput() on the vmscan->writepage() path. 
Cannot remember what they were though...

Re: mmap over nfs leads to excessive system load

[Trond Myklebust]

On Wed, 2005-11-16 at 14:10 -0800, Andrew Morton wrote:

> > How is the filesystem supposed to distinguish between the cases
> > "VM->writepage()", and "VM->writepages->mpage_writepages->writepage()"?
> > 
> 
> Via the writeback_control, hopefully.
> 
> For write_one_page(), sync_mode==WB_SYNC_ALL, so NFS should start the I/O
> immediately (it appears to not do so).

Sorry, but so does filemap_fdatawrite(). WB_SYNC_ALL clearly does not
discriminate between a writepages() and a single writepage() situation,
whatever the original intention was.

> For vmscan->writepage, wbc->for_reclaim is set, so we know that the IO
> should be pushed immediately.  nfs_writepage() seems to dtrt here.
> 
> With the proposed changes, we don't need that iput() in nfs_writepage(). 
> That worries me because I recall from a couple of years back that there are
> really subtle races with doing iput() on the vmscan->writepage() path. 
> Cannot remember what they were though...

Possibly to do with block filesystems that may trigger ->writepage()
while inside iput_final()? NFS can't do that.

Cheers,
  Trond

Re: mmap over nfs leads to excessive system load

[Trond Myklebust]

On Wed, 2005-11-16 at 17:23 -0500, Trond Myklebust wrote:
> On Wed, 2005-11-16 at 14:10 -0800, Andrew Morton wrote:
> 
> > > How is the filesystem supposed to distinguish between the cases
> > > "VM->writepage()", and "VM->writepages->mpage_writepages->writepage()"?
> > > 
> > 
> > Via the writeback_control, hopefully.
> > 
> > For write_one_page(), sync_mode==WB_SYNC_ALL, so NFS should start the I/O
> > immediately (it appears to not do so).
> 
> Sorry, but so does filemap_fdatawrite(). WB_SYNC_ALL clearly does not
> discriminate between a writepages() and a single writepage() situation,
> whatever the original intention was.

IMHO, the correct way to distinguish between the two would be to use the
wbc->nr_to_write field. If all the instances of writepage() were to set
that field to '1', then the filesystems could do the right thing.

As it is, you have shrink_list() that sets it to the value
"SWAP_CLUSTER_MAX" for no apparent reason...

Cheers,
  Trond

Re: mmap over nfs leads to excessive system load

[Kenny Simpson]

--- Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> I'm getting lost here. Please could you spell out the testcases that are
> not working.

I've redone my test cases and have confirmed that O_DIRECT with pwrite64 triggers the bad
condition.

The cases that are fine are:
  pwrite64
  ftruncate with O_DIRECT
  ftruncate

Also, when the system is in this state, if I try to 'ls' the file,
the 'ls' process becomes stuck in state D in sync_page.  stracing the 'ls'
shows it is in a call to stat64.

-Kenny



		
__________________________________ 
Yahoo! FareChase: Search multiple travel sites in one click.
http://farechase.yahoo.com

Re: mmap over nfs leads to excessive system load

[Andrew Morton]

Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
>
> On Wed, 2005-11-16 at 14:10 -0800, Andrew Morton wrote:
> 
> > > How is the filesystem supposed to distinguish between the cases
> > > "VM->writepage()", and "VM->writepages->mpage_writepages->writepage()"?
> > > 
> > 
> > Via the writeback_control, hopefully.
> > 
> > For write_one_page(), sync_mode==WB_SYNC_ALL, so NFS should start the I/O
> > immediately (it appears to not do so).
> 
> Sorry, but so does filemap_fdatawrite(). WB_SYNC_ALL clearly does not
> discriminate between a writepages() and a single writepage() situation,
> whatever the original intention was.

Could peek at wbc->nr_pages, or add another boolean to writeback_control
for this.

diff -puN include/linux/writeback.h~writeback_control-flag-writepages include/linux/writeback.h
--- devel/include/linux/writeback.h~writeback_control-flag-writepages	2005-11-16 14:43:52.000000000 -0800
+++ devel-akpm/include/linux/writeback.h	2005-11-16 14:43:52.000000000 -0800
@@ -53,10 +53,11 @@ struct writeback_control {
 	loff_t start;
 	loff_t end;
 
-	unsigned nonblocking:1;			/* Don't get stuck on request queues */
-	unsigned encountered_congestion:1;	/* An output: a queue is full */
-	unsigned for_kupdate:1;			/* A kupdate writeback */
-	unsigned for_reclaim:1;			/* Invoked from the page allocator */
+	unsigned nonblocking:1;		/* Don't get stuck on request queues */
+	unsigned encountered_congestion:1; /* An output: a queue is full */
+	unsigned for_kupdate:1;		/* A kupdate writeback */
+	unsigned for_reclaim:1;		/* Invoked from the page allocator */
+	unsigned for_writepages:1;	/* This is a writepages() call */
 };
 
 /*
diff -puN mm/page-writeback.c~writeback_control-flag-writepages mm/page-writeback.c
--- devel/mm/page-writeback.c~writeback_control-flag-writepages	2005-11-16 14:43:52.000000000 -0800
+++ devel-akpm/mm/page-writeback.c	2005-11-16 14:43:52.000000000 -0800
@@ -550,11 +550,17 @@ void __init page_writeback_init(void)
 
 int do_writepages(struct address_space *mapping, struct writeback_control *wbc)
 {
+	int ret;
+
 	if (wbc->nr_to_write <= 0)
 		return 0;
+	wbc->for_writepages = 1;
 	if (mapping->a_ops->writepages)
-		return mapping->a_ops->writepages(mapping, wbc);
-	return generic_writepages(mapping, wbc);
+		ret =  mapping->a_ops->writepages(mapping, wbc);
+	else
+		ret = generic_writepages(mapping, wbc);
+	wbc->for_writepages = 0;
+	return ret;
 }
 
 /**
_

> > For vmscan->writepage, wbc->for_reclaim is set, so we know that the IO
> > should be pushed immediately.  nfs_writepage() seems to dtrt here.
> > 
> > With the proposed changes, we don't need that iput() in nfs_writepage(). 
> > That worries me because I recall from a couple of years back that there are
> > really subtle races with doing iput() on the vmscan->writepage() path. 
> > Cannot remember what they were though...
> 
> Possibly to do with block filesystems that may trigger ->writepage()
> while inside iput_final()? NFS can't do that.

iput_final() can call truncate_inode_pages - maybe it was a deadlock, but
I'm fairly sure it was a race.

Re: mmap over nfs leads to excessive system load

[Andrew Morton]

Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
>
> On Wed, 2005-11-16 at 17:23 -0500, Trond Myklebust wrote:
> > On Wed, 2005-11-16 at 14:10 -0800, Andrew Morton wrote:
> > 
> > > > How is the filesystem supposed to distinguish between the cases
> > > > "VM->writepage()", and "VM->writepages->mpage_writepages->writepage()"?
> > > > 
> > > 
> > > Via the writeback_control, hopefully.
> > > 
> > > For write_one_page(), sync_mode==WB_SYNC_ALL, so NFS should start the I/O
> > > immediately (it appears to not do so).
> > 
> > Sorry, but so does filemap_fdatawrite(). WB_SYNC_ALL clearly does not
> > discriminate between a writepages() and a single writepage() situation,
> > whatever the original intention was.
> 
> IMHO, the correct way to distinguish between the two would be to use the
> wbc->nr_to_write field. If all the instances of writepage() were to set
> that field to '1', then the filesystems could do the right thing.

yes, except ->writepages is supposed to decrement nr_to_write as it proceeds,
so it'll end up at `1' by accident on the last go around the loop.

I think a separate boolean is better - it's just a single bit.

> As it is, you have shrink_list() that sets it to the value
> "SWAP_CLUSTER_MAX" for no apparent reason...

How weird.  That's presumably wrong, but I'd need to check the changelogs
to doublecheck.  ugh, 264 of them.

Re: mmap over nfs leads to excessive system load

[Trond Myklebust]

On Wed, 2005-11-16 at 14:39 -0800, Kenny Simpson wrote:
> --- Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> > I'm getting lost here. Please could you spell out the testcases that are
> > not working.
> 
> I've redone my test cases and have confirmed that O_DIRECT with pwrite64 triggers the bad
> condition.
> 
> The cases that are fine are:
>   pwrite64
>   ftruncate with O_DIRECT
>   ftruncate
> 
> Also, when the system is in this state, if I try to 'ls' the file,
> the 'ls' process becomes stuck in state D in sync_page.  stracing the 'ls'
> shows it is in a call to stat64.
> 
> -Kenny

Chuck, can you take a look at this?

Kenny is seeing what a hang when using pwrite64() on an O_DIRECT file
and the file size exceeds 4Gb. Server is a NetApp filer w/ NFSv3.

I had a quick look at nfs_file_direct_write(), and among other things,
it would appear that it is not doing any of the usual overflow checks on
*pos and the count size (see generic_write_checks()). In particular,
checks are missing against overflow vs. MAX_NON_LFS if O_LARGEFILE is
not set (and also against overflow vs. s_maxbytes, but that is less
relevant here).

Cheers,
  Trond

Re: mmap over nfs leads to excessive system load

[Trond Myklebust]

On Wed, 2005-11-16 at 14:44 -0800, Andrew Morton wrote:

> Could peek at wbc->nr_pages, or add another boolean to writeback_control
> for this.
> 
> diff -puN include/linux/writeback.h~writeback_control-flag-writepages include/linux/writeback.h
> --- devel/include/linux/writeback.h~writeback_control-flag-writepages	2005-11-16 14:43:52.000000000 -0800
> +++ devel-akpm/include/linux/writeback.h	2005-11-16 14:43:52.000000000 -0800
> @@ -53,10 +53,11 @@ struct writeback_control {
>  	loff_t start;
>  	loff_t end;
>  
> -	unsigned nonblocking:1;			/* Don't get stuck on request queues */
> -	unsigned encountered_congestion:1;	/* An output: a queue is full */
> -	unsigned for_kupdate:1;			/* A kupdate writeback */
> -	unsigned for_reclaim:1;			/* Invoked from the page allocator */
> +	unsigned nonblocking:1;		/* Don't get stuck on request queues */
> +	unsigned encountered_congestion:1; /* An output: a queue is full */
> +	unsigned for_kupdate:1;		/* A kupdate writeback */
> +	unsigned for_reclaim:1;		/* Invoked from the page allocator */
> +	unsigned for_writepages:1;	/* This is a writepages() call */
>  };
>  
>  /*
> diff -puN mm/page-writeback.c~writeback_control-flag-writepages mm/page-writeback.c
> --- devel/mm/page-writeback.c~writeback_control-flag-writepages	2005-11-16 14:43:52.000000000 -0800
> +++ devel-akpm/mm/page-writeback.c	2005-11-16 14:43:52.000000000 -0800
> @@ -550,11 +550,17 @@ void __init page_writeback_init(void)
>  
>  int do_writepages(struct address_space *mapping, struct writeback_control *wbc)
>  {
> +	int ret;
> +
>  	if (wbc->nr_to_write <= 0)
>  		return 0;
> +	wbc->for_writepages = 1;
>  	if (mapping->a_ops->writepages)
> -		return mapping->a_ops->writepages(mapping, wbc);
> -	return generic_writepages(mapping, wbc);
> +		ret =  mapping->a_ops->writepages(mapping, wbc);
> +	else
> +		ret = generic_writepages(mapping, wbc);
> +	wbc->for_writepages = 0;
> +	return ret;
>  }

That would work...

> > > For vmscan->writepage, wbc->for_reclaim is set, so we know that the IO
> > > should be pushed immediately.  nfs_writepage() seems to dtrt here.
> > > 
> > > With the proposed changes, we don't need that iput() in nfs_writepage(). 
> > > That worries me because I recall from a couple of years back that there are
> > > really subtle races with doing iput() on the vmscan->writepage() path. 
> > > Cannot remember what they were though...
> > 
> > Possibly to do with block filesystems that may trigger ->writepage()
> > while inside iput_final()? NFS can't do that.
> 
> iput_final() can call truncate_inode_pages - maybe it was a deadlock, but
> I'm fairly sure it was a race.

Doesn't matter. There can be no dirty pages when NFS hits iput_final().
We make sure that we flush them into the filesystem accounting before we
release the file descriptor, then we make sure that we don't release the
dentry before the inode has been synced up.

Cheers,
  Trond

Re: mmap over nfs leads to excessive system load

[Trond Myklebust]

On Wed, 2005-11-16 at 14:44 -0800, Andrew Morton wrote:
> diff -puN include/linux/writeback.h~writeback_control-flag-writepages include/linux/writeback.h
> --- devel/include/linux/writeback.h~writeback_control-flag-writepages	2005-11-16 14:43:52.000000000 -0800
> +++ devel-akpm/include/linux/writeback.h	2005-11-16 14:43:52.000000000 -0800
> @@ -53,10 +53,11 @@ struct writeback_control {
>  	loff_t start;
>  	loff_t end;
>  
> -	unsigned nonblocking:1;			/* Don't get stuck on request queues */
> -	unsigned encountered_congestion:1;	/* An output: a queue is full */
> -	unsigned for_kupdate:1;			/* A kupdate writeback */
> -	unsigned for_reclaim:1;			/* Invoked from the page allocator */
> +	unsigned nonblocking:1;		/* Don't get stuck on request queues */
> +	unsigned encountered_congestion:1; /* An output: a queue is full */
> +	unsigned for_kupdate:1;		/* A kupdate writeback */
> +	unsigned for_reclaim:1;		/* Invoked from the page allocator */
> +	unsigned for_writepages:1;	/* This is a writepages() call */
>  };
>  
>  /*
> diff -puN mm/page-writeback.c~writeback_control-flag-writepages mm/page-writeback.c
> --- devel/mm/page-writeback.c~writeback_control-flag-writepages	2005-11-16 14:43:52.000000000 -0800
> +++ devel-akpm/mm/page-writeback.c	2005-11-16 14:43:52.000000000 -0800
> @@ -550,11 +550,17 @@ void __init page_writeback_init(void)
>  
>  int do_writepages(struct address_space *mapping, struct writeback_control *wbc)
>  {
> +	int ret;
> +
>  	if (wbc->nr_to_write <= 0)
>  		return 0;
> +	wbc->for_writepages = 1;
>  	if (mapping->a_ops->writepages)
> -		return mapping->a_ops->writepages(mapping, wbc);
> -	return generic_writepages(mapping, wbc);
> +		ret =  mapping->a_ops->writepages(mapping, wbc);
> +	else
> +		ret = generic_writepages(mapping, wbc);
> +	wbc->for_writepages = 0;
> +	return ret;
>  }

The accompanying NFS patch makes use of this in order to figure out when
to flush the data correctly.
-------------
NFS: Work correctly with single-page ->writepage() calls

 Ensure that we use set_page_writeback() in the appropriate places
 to help the VM in keeping its page radix_tree in sync.
 Ensure that we always initiate flushing of data before we exit
 a single-page ->writepage() call.

 Signed-off-by: Trond Myklebust <Trond.Myklebust@netapp.com>
---

 fs/nfs/write.c |   22 +++++++++-------------
 1 files changed, 9 insertions(+), 13 deletions(-)

diff --git a/fs/nfs/write.c b/fs/nfs/write.c
index 8f71e76..95d00f9 100644
--- a/fs/nfs/write.c
+++ b/fs/nfs/write.c
@@ -189,6 +189,7 @@ static int nfs_writepage_sync(struct nfs
 		(long long)NFS_FILEID(inode),
 		count, (long long)(page_offset(page) + offset));
 
+	set_page_writeback(page);
 	nfs_begin_data_update(inode);
 	do {
 		if (count < wsize)
@@ -221,6 +222,7 @@ static int nfs_writepage_sync(struct nfs
 
 io_error:
 	nfs_end_data_update(inode);
+	end_page_writeback(page);
 	nfs_writedata_free(wdata);
 	return written ? written : result;
 }
@@ -230,19 +232,16 @@ static int nfs_writepage_async(struct nf
 		unsigned int offset, unsigned int count)
 {
 	struct nfs_page	*req;
-	int		status;
 
 	req = nfs_update_request(ctx, inode, page, offset, count);
-	status = (IS_ERR(req)) ? PTR_ERR(req) : 0;
-	if (status < 0)
-		goto out;
+	if (IS_ERR(req))
+		return PTR_ERR(req);
 	/* Update file length */
 	nfs_grow_file(page, offset, count);
 	/* Set the PG_uptodate flag? */
 	nfs_mark_uptodate(page, offset, count);
 	nfs_unlock_request(req);
- out:
-	return status;
+	return 0;
 }
 
 static int wb_priority(struct writeback_control *wbc)
@@ -302,11 +301,8 @@ do_it:
 	lock_kernel();
 	if (!IS_SYNC(inode) && inode_referenced) {
 		err = nfs_writepage_async(ctx, inode, page, 0, offset);
-		if (err >= 0) {
-			err = 0;
-			if (wbc->for_reclaim)
-				nfs_flush_inode(inode, 0, 0, FLUSH_STABLE);
-		}
+		if (!wbc->for_writepages)
+			nfs_flush_inode(inode, 0, 0, wb_priority(wbc));
 	} else {
 		err = nfs_writepage_sync(ctx, inode, page, 0,
 						offset, priority);
@@ -929,7 +925,7 @@ static int nfs_flush_multi(struct list_h
 	atomic_set(&req->wb_complete, requests);
 
 	ClearPageError(page);
-	SetPageWriteback(page);
+	set_page_writeback(page);
 	offset = 0;
 	nbytes = req->wb_bytes;
 	do {
@@ -992,7 +988,7 @@ static int nfs_flush_one(struct list_hea
 		nfs_list_remove_request(req);
 		nfs_list_add_request(req, &data->pages);
 		ClearPageError(req->wb_page);
-		SetPageWriteback(req->wb_page);
+		set_page_writeback(req->wb_page);
 		*pages++ = req->wb_page;
 		count += req->wb_bytes;
 	}

Re: mmap over nfs leads to excessive system load

[Andrew Morton]

Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
>
> The accompanying NFS patch makes use of this in order to figure out when
>  to flush the data correctly.

OK.  So with that patch, nfs_writepages() may still leave I/O pending,
uninitiated, yes?

I don't understand why NFS hasn't been BUGging as it stands at present.  It
has several end_page_writeback() calls but no set_page_writeback()s. 
end_page_writeback() or rotate_reclaimable_page() will go BUG if the page
wasn't PageWriteback().

Re: mmap over nfs leads to excessive system load

[Trond Myklebust]

On Wed, 2005-11-16 at 16:25 -0800, Andrew Morton wrote:
> Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> >
> > The accompanying NFS patch makes use of this in order to figure out when
> >  to flush the data correctly.
> 
> OK.  So with that patch, nfs_writepages() may still leave I/O pending,
> uninitiated, yes?
> 
> I don't understand why NFS hasn't been BUGging as it stands at present.  It
> has several end_page_writeback() calls but no set_page_writeback()s. 
> end_page_writeback() or rotate_reclaimable_page() will go BUG if the page
> wasn't PageWriteback().

It does have SetPageWriteback() calls in the asynchronous writeback
path. As you can see from the patch I just sent, I only needed to
replace them with set_page_writebacks().

Cheers,
 Trond

Re: mmap over nfs leads to excessive system load

[Andrew Morton]

Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
>
> On Wed, 2005-11-16 at 16:25 -0800, Andrew Morton wrote:
> > Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> > >
> > > The accompanying NFS patch makes use of this in order to figure out when
> > >  to flush the data correctly.
> > 
> > OK.  So with that patch, nfs_writepages() may still leave I/O pending,
> > uninitiated, yes?

This?

I don't know if it'll be a problem.  One factor is that when the VFS is
doing an fsync() or whatever, it will fail to notice these left-over pages
are "dirty", so it won't launch writepage() against them.

But if they are marked PageWriteback(), sync will notice them on the second
pass and will wait upon them, which apparently could mean a stall until
pdflush kicks off the I/O?

If they're not marked PageDirty() or PageWriteback(), the VFS will miss
them altogether during the sync.  But perhaps NFS's own page tracking will
flush them and wait upon the result?

> > I don't understand why NFS hasn't been BUGging as it stands at present.  It
> > has several end_page_writeback() calls but no set_page_writeback()s. 
> > end_page_writeback() or rotate_reclaimable_page() will go BUG if the page
> > wasn't PageWriteback().
> 
> It does have SetPageWriteback() calls in the asynchronous writeback
> path. As you can see from the patch I just sent, I only needed to
> replace them with set_page_writebacks().

Ah, OK.   Things are improved.

Re: mmap over nfs leads to excessive system load

[Trond Myklebust]

On Wed, 2005-11-16 at 16:38 -0800, Andrew Morton wrote:

> I don't know if it'll be a problem.  One factor is that when the VFS is
> doing an fsync() or whatever, it will fail to notice these left-over pages
> are "dirty", so it won't launch writepage() against them.

That doesn't matter. They are being tracked by the NFS client. We don't
want anyone to call writepage() against them again because that will
cause them to be written out twice.

> But if they are marked PageWriteback(), sync will notice them on the second
> pass and will wait upon them, which apparently could mean a stall until
> pdflush kicks off the I/O?
> 
> If they're not marked PageDirty() or PageWriteback(), the VFS will miss
> them altogether during the sync.  But perhaps NFS's own page tracking will
> flush them and wait upon the result?

Yes. There is no chance of data loss (unless someone physically pulls
the plug on the client - there's no protecting against that).

Note that writepages() will normally end up calling nfs_flush_inode().

It will only fail to do so if

  - generic_writepages() returns an error
or
  - there is write congestion, and wbc->nonblocking is set.

Cheers,
  Trond

Re: mmap over nfs leads to excessive system load

[Nikita Danilov]

Christoph Hellwig writes:
 > On Wed, Nov 16, 2005 at 01:34:22PM -0500, Trond Myklebust wrote:
 > > Not really. The pages aren't flushed at this time. We the point is to
 > > gather several pages and coalesce them into one over-the-wire RPC call.
 > > That means we cannot really do it from inside ->writepage().
 > > 
 > > We do start the actual RPC calls in ->writepages(), though.
 > 
 > This is a problem we have in various filesystems.  Except for really
 > bad OOM situations the filesystem should never get a writeout request
 > for a single file.  We should really stop having ->writepage called by
 > the VM and move this kind of batching code into the VM.  I'm runnin into
 > similar issues for XFS and unwritten/delayed extent conversion once again.

Simplistic version if such batching is implemented in the patch below
(also available at

http://linuxhacker.ru/~nikita/patches/2.6.15-rc1/05-cluster-pageout.patch

it depends on page_referenced-move-dirty patch from the same place)

This version pokes into address_space radix tree to find a cluster of
pages suitable for page-out and then calls ->writepage() on pages in
that cluster in the proper order. This relies on the underlying layer
(e.g., block device) to perform request coalescing.

My earlier attempts to do this through ->writepages() were all racy,
because at some point ->writepages() has to release a lock at the
original page around which the cluster is built, and that lock is the
only thing that protects inode/address_space from the destruction. As
was already noted by Andrew, one cannot use igrab/iput in the VM scanner
to deal with that.

I still think it's possible to do higher layer batching, but that would
require more extensive changes to both VM scanner and ->writepages().

 > 

Nikita.
--
Implement pageout clustering at the VM level.

With this patch VM scanner calls pageout_cluster() instead of
->writepage(). pageout_cluster() tries to find a group of dirty pages around
target page, called "pivot" page of the cluster. If group of suitable size is
found, ->writepages() is called for it, otherwise, page_cluster() falls back
to ->writepage().

This is supposed to help in work-loads with significant page-out of
file-system pages from tail of the inactive list (for example, heavy dirtying
through mmap), because file system usually writes multiple pages more
efficiently. Should also be advantageous for file-systems doing delayed
allocation, as in this case they will allocate whole extents at once.

Few points:

 - swap-cache pages are not clustered (although they can be, but by
   page->private rather than page->index)

 - only kswapd does clustering, because direct reclaim path should be low
   latency.

 - Original version of this patch added new fields to struct writeback_control
   and expected ->writepages() to interpret them. This led to hard-to-fix races
   against inode reclamation. Current version simply calls ->writepage() in the
   "correct" order, i.e., in the order of increasing page indices.

Signed-off-by: Nikita Danilov <nikita@clusterfs.com>


 mm/shmem.c  |   14 ++++++-
 mm/vmscan.c |  112 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 124 insertions(+), 2 deletions(-)

diff -puN mm/vmscan.c~cluster-pageout mm/vmscan.c
--- git-linux/mm/vmscan.c~cluster-pageout	2005-11-13 21:25:15.000000000 +0300
+++ git-linux-nikita/mm/vmscan.c	2005-11-13 21:25:15.000000000 +0300
@@ -360,6 +360,116 @@ static void send_page_to_kpgout(struct p
 	spin_unlock(&kpgout_queue_lock);
 }
 
+enum {
+	PAGE_CLUSTER_WING = 16,
+	PAGE_CLUSTER_SIZE = 2 * PAGE_CLUSTER_WING,
+};
+
+static int page_fits_cluster(struct address_space *mapping, struct page *page)
+{
+	int result;
+
+	if (page != NULL && !PageActive(page) && !TestSetPageLocked(page)) {
+		/*
+		 * unlock ->tree_lock to avoid lock inversion with
+		 * ->i_mmap_lock in page_referenced().
+		 */
+		read_unlock_irq(&mapping->tree_lock);
+		result =
+			/* try_to_unmap(page) == SWAP_SUCCESS && */
+			PageDirty(page) && !PageWriteback(page) &&
+			!page_referenced(page, 1,
+					 page_zone(page)->temp_priority <= 0,
+					 1);
+		if (result == 0)
+			unlock_page(page);
+		read_lock_irq(&mapping->tree_lock);
+	} else
+		result = 0;
+	return result;
+}
+
+static void call_writepage(struct page *page, struct address_space *mapping,
+			   struct writeback_control *wbc)
+{
+	if (clear_page_dirty_for_io(page)) {
+		int result;
+
+		BUG_ON(!PageLocked(page));
+		BUG_ON(PageWriteback(page));
+
+		result = mapping->a_ops->writepage(page, wbc);
+		if (result == WRITEPAGE_ACTIVATE)
+			unlock_page(page);
+	}
+}
+
+int __pageout_cluster(struct page *page, struct address_space *mapping,
+		      struct writeback_control *wbc)
+{
+	int result;
+	int used;
+
+	pgoff_t punct;
+	pgoff_t start;
+	pgoff_t end;
+
+	struct page *pages_out[PAGE_CLUSTER_WING];
+	struct page *scan;
+
+	BUG_ON(PageAnon(page));
+
+	punct = page->index;
+	read_lock_irq(&mapping->tree_lock);
+	for (start = punct - 1, used = 0;
+	     start < punct && punct - start <= PAGE_CLUSTER_WING; start --) {
+		scan = radix_tree_lookup(&mapping->page_tree, start);
+		if (!page_fits_cluster(mapping, scan))
+			/*
+			 * no suitable page, stop cluster at this point
+			 */
+			break;
+		pages_out[used ++] = scan;
+		if ((start % PAGE_CLUSTER_SIZE) == 0)
+			/*
+			 * we reached aligned page.
+			 */
+			break;
+	}
+	read_unlock_irq(&mapping->tree_lock);
+
+	while (used > 0)
+		call_writepage(pages_out[--used], mapping, wbc);
+
+	result = mapping->a_ops->writepage(page, wbc);
+
+	for (end = punct + 1;
+	     end > punct && end - start < PAGE_CLUSTER_SIZE; ++ end) {
+		int enough;
+
+		/*
+		 * XXX nikita: consider find_get_pages_tag()
+		 */
+		read_lock_irq(&mapping->tree_lock);
+		scan = radix_tree_lookup(&mapping->page_tree, end);
+		enough = !page_fits_cluster(mapping, scan);
+		read_unlock_irq(&mapping->tree_lock);
+		if (enough)
+			break;
+		call_writepage(scan, mapping, wbc);
+	}
+	return result;
+}
+
+static int pageout_cluster(struct page *page, struct address_space *mapping,
+			   struct writeback_control *wbc)
+{
+	if (PageSwapCache(page)	|| !current_is_kswapd())
+		return mapping->a_ops->writepage(page, wbc);
+	else
+		return __pageout_cluster(page, mapping, wbc);
+}
+
 /*
  * Called by shrink_list() for each dirty page. Calls ->writepage().
  */
@@ -445,7 +555,7 @@ static pageout_t pageout(struct page *pa
 
 		ClearPageSkipped(page);
 		SetPageReclaim(page);
-		res = mapping->a_ops->writepage(page, &wbc);
+		res = pageout_cluster(page, mapping, &wbc);
 
 		if (res < 0)
 			handle_write_error(mapping, page, res);
diff -puN include/linux/writeback.h~cluster-pageout include/linux/writeback.h
diff -puN fs/mpage.c~cluster-pageout fs/mpage.c
diff -puN mm/shmem.c~cluster-pageout mm/shmem.c
--- git-linux/mm/shmem.c~cluster-pageout	2005-11-13 21:25:15.000000000 +0300
+++ git-linux-nikita/mm/shmem.c	2005-11-13 21:25:15.000000000 +0300
@@ -45,6 +45,7 @@
 #include <linux/swapops.h>
 #include <linux/mempolicy.h>
 #include <linux/namei.h>
+#include <linux/rmap.h>
 #include <asm/uaccess.h>
 #include <asm/div64.h>
 #include <asm/pgtable.h>
@@ -813,7 +814,18 @@ static int shmem_writepage(struct page *
 	struct inode *inode;
 
 	BUG_ON(!PageLocked(page));
-	BUG_ON(page_mapped(page));
+
+	/*
+	 * If shmem_writepage() is called on mapped page, a problem arises for
+	 * a tmpfs file mapped shared into different mms. Viz. shmem_writepage
+	 * changes the tmpfs-file identity of the page to swap identity: so if
+	 * it's unmapped later, the instances would then become private (to be
+	 * COWed) instead of shared.
+	 *
+	 * Just unmap page.
+	 */
+	if (page_mapped(page) && try_to_unmap(page) != SWAP_SUCCESS)
+		goto redirty;
 
 	mapping = page->mapping;
 	index = page->index;

_

Re: mmap over nfs leads to excessive system load

[Chuck Lever]

[-- Attachment #1: Type: text/plain, Size: 1222 bytes --]

Trond Myklebust wrote:
> On Wed, 2005-11-16 at 14:39 -0800, Kenny Simpson wrote:
> 
>>--- Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
>>
>>>I'm getting lost here. Please could you spell out the testcases that are
>>>not working.
>>
>>I've redone my test cases and have confirmed that O_DIRECT with pwrite64 triggers the bad
>>condition.
>>
>>The cases that are fine are:
>>  pwrite64
>>  ftruncate with O_DIRECT
>>  ftruncate
>>
>>Also, when the system is in this state, if I try to 'ls' the file,
>>the 'ls' process becomes stuck in state D in sync_page.  stracing the 'ls'
>>shows it is in a call to stat64.
>>
>>-Kenny
> 
> 
> Chuck, can you take a look at this?
> 
> Kenny is seeing a hang when using pwrite64() on an O_DIRECT file
> and the file size exceeds 4Gb. Server is a NetApp filer w/ NFSv3.
> 
> I had a quick look at nfs_file_direct_write(), and among other things,
> it would appear that it is not doing any of the usual overflow checks on
> *pos and the count size (see generic_write_checks()). In particular,
> checks are missing against overflow vs. MAX_NON_LFS if O_LARGEFILE is
> not set (and also against overflow vs. s_maxbytes, but that is less
> relevant here).

'uname -a' on the client?


[-- Attachment #2: cel.vcf --]
[-- Type: text/x-vcard, Size: 439 bytes --]

begin:vcard
fn:Chuck Lever
n:Lever;Charles
org:Network Appliance, Incorporated;Linux NFS Client Development
adr:535 West William Street, Suite 3100;;Center for Information Technology Integration;Ann Arbor;MI;48103-4943;USA
email;internet:cel@citi.umich.edu
title:Member of Technical Staff
tel;work:+1 734 763 4415
tel;fax:+1 734 763 4434
tel;home:+1 734 668 1089
x-mozilla-html:FALSE
url:http://www.monkey.org/~cel/
version:2.1
end:vcard

Re: mmap over nfs leads to excessive system load

[Kenny Simpson]

--- Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> Chuck, can you take a look at this?
> 
> Kenny is seeing what a hang when using pwrite64() on an O_DIRECT file
> and the file size exceeds 4Gb. Server is a NetApp filer w/ NFSv3.
> 
> I had a quick look at nfs_file_direct_write(), and among other things,
> it would appear that it is not doing any of the usual overflow checks on
> *pos and the count size (see generic_write_checks()). In particular,
> checks are missing against overflow vs. MAX_NON_LFS if O_LARGEFILE is
> not set (and also against overflow vs. s_maxbytes, but that is less
> relevant here).
> 
> Cheers,
>   Trond

I tried the same test, but starting closer to 4GB... here is the final lines from strace:
remap_file_pages(0xb7b55000, 2097152, PROT_NONE, 1047544, MAP_SHARED) = 0
pwrite(3, "\0", 1, 8564768768)          = 1
remap_file_pages(0xb7b55000, 2097152, PROT_NONE, 1048056, MAP_SHARED) = 0
pwrite(3, "\0", 1, 8566865920)          = 1
remap_file_pages(0xb7b55000, 2097152, PROT_NONE, 1048568, MAP_SHARED) = 0
pwrite(3, "\0", 1, 8568963072

The pwrite never returns.
So it seems to be a problem NOT with an absolute 4GB, but with a total of 4GB having been written.

Here are the first few lines from the strace to show all the options being used:
open("/mnt/bar", O_RDWR|O_CREAT|O_DIRECT|O_LARGEFILE, 0644) = 3
pwrite(3, "\0", 1, 4280287232)          = 1
mmap2(NULL, 2097152, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0xff000) = 0xb7b8e000
pwrite(3, "\0", 1, 4282384384)          = 1
remap_file_pages(0xb7b8e000, 2097152, PROT_NONE, 2552, MAP_SHARED) = 0
pwrite(3, "\0", 1, 4284481536)          = 1
remap_file_pages(0xb7b8e000, 2097152, PROT_NONE, 3064, MAP_SHARED) = 0

/mnt is an nfs mount over GbE w/ jumbo frames (8160 mtu) cross-over directly to a netapp filer.

The mount options are: (from /proc/mounts)
 /mnt nfs rw,v3,rsize=32768,wsize=32768,hard,intr,lock,proto=tcp,addr=x.x.x.x 0 0

The card is an Intel e1000 - default module options (NAPI-enabled)
on a 64-bit PCIX 100MHz.
Kernel is 2.6.15-rc w/ Trond's nfs patch.
Machine is a 2x Pentium 4 Xeon 2.66GHz (HT enabled), w/ 2GB ram and 4GB swap.

vmstat shows:
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 2  0      0 1336864 123212 203936    0    0     0    20 1111  1129  1 26 73  0
 1  0      0 1336608 123212 203936    0    0     0     0 1078  1076  1 25 74  0
 1  0      0 1336864 123212 203936    0    0     0     0 1077  1087  1 26 73  0

the sy of 25 is one virtual CPU with 100% system.

Oprofile shows time being spent:

samples  %        symbol name
303102   42.4732  zap_pte_range
133702   18.7355  _spin_lock
61145     8.5682  __bitmap_weight
43169     6.0492  page_address
42196     5.9129  unmap_vmas
30132     4.2224  unmap_page_range

-Kenny



	
		
__________________________________ 
Yahoo! Mail - PC Magazine Editors' Choice 2005 
http://mail.yahoo.com

Re: mmap over nfs leads to excessive system load

[Kenny Simpson]

--- Chuck Lever <cel@citi.umich.edu> wrote:
> 
> 'uname -a' on the client?

Linux tux6127 2.6.15-rc1 #6 SMP PREEMPT Wed Nov 16 14:47:14 EST 2005 i686 GNU/Linux

I also sent the .config on a previous posting.  I can send it again if you'd like.

-Kenny



		
__________________________________ 
Yahoo! FareChase: Search multiple travel sites in one click.
http://farechase.yahoo.com

Re: mmap over nfs leads to excessive system load

[Chuck Lever]

[-- Attachment #1: Type: text/plain, Size: 480 bytes --]

Trond Myklebust wrote:
> I had a quick look at nfs_file_direct_write(), and among other things,
> it would appear that it is not doing any of the usual overflow checks on
> *pos and the count size (see generic_write_checks()). In particular,
> checks are missing against overflow vs. MAX_NON_LFS if O_LARGEFILE is
> not set (and also against overflow vs. s_maxbytes, but that is less
> relevant here).

the architecture is to allow the NFS protocol and server to do these checks.

[-- Attachment #2: cel.vcf --]
[-- Type: text/x-vcard, Size: 439 bytes --]

begin:vcard
fn:Chuck Lever
n:Lever;Charles
org:Network Appliance, Incorporated;Linux NFS Client Development
adr:535 West William Street, Suite 3100;;Center for Information Technology Integration;Ann Arbor;MI;48103-4943;USA
email;internet:cel@citi.umich.edu
title:Member of Technical Staff
tel;work:+1 734 763 4415
tel;fax:+1 734 763 4434
tel;home:+1 734 668 1089
x-mozilla-html:FALSE
url:http://www.monkey.org/~cel/
version:2.1
end:vcard

Re: mmap over nfs leads to excessive system load

[Trond Myklebust]

On Thu, 2005-11-17 at 12:02 -0500, Chuck Lever wrote:
> Trond Myklebust wrote:
> > I had a quick look at nfs_file_direct_write(), and among other things,
> > it would appear that it is not doing any of the usual overflow checks on
> > *pos and the count size (see generic_write_checks()). In particular,
> > checks are missing against overflow vs. MAX_NON_LFS if O_LARGEFILE is
> > not set (and also against overflow vs. s_maxbytes, but that is less
> > relevant here).
> 
> the architecture is to allow the NFS protocol and server to do these checks.

No it isn't.

The NFS protocol has no clue as to whether or not you opened the file
using O_LARGEFILE. For NFSv2, we do _not_ want file pointers to wrap
once they hit the 32-bit boundary.

The protocol and server cannot be involved in any of those checks. They
must be done on the client.

Cheers,
  Trond

Re: mmap over nfs leads to excessive system load

[Andrew Morton]

Kenny Simpson <theonetruekenny@yahoo.com> wrote:
>
> The pwrite never returns.
> So it seems to be a problem NOT with an absolute 4GB, but with a total of 4GB having been written.

Could you send the test app please?  (Apologies if you've already done so
and I missed it).

Re: mmap over nfs leads to excessive system load

[Kenny Simpson]

[-- Attachment #1: Type: text/plain, Size: 394 bytes --]

--- Andrew Morton <akpm@osdl.org> wrote:
> Could you send the test app please?  (Apologies if you've already done so
> and I missed it).
> 

Here it is again... this one skips to just under 4GB before starting.
run with "writetest -m <filename>" for the mmap test.

-Kenny



		
__________________________________ 
Start your day with Yahoo! - Make it your home page! 
http://www.yahoo.com/r/hs

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 3360621278-writetest.cpp --]
[-- Type: text/x-c++src; name="writetest.cpp", Size: 3615 bytes --]

// test the write throughput of a sliding mmap window vs simple FILE*

#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <signal.h>
#include <string.h>
#include <unistd.h>
#include <time.h>
#include <errno.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <sys/time.h>

int total_bytes = 0;
int last_bytes = 0;

struct timeval last_time;

void timeout(int)
{
  struct timeval now;
  gettimeofday(&now, 0);

  double time_diff = (now.tv_sec - last_time.tv_sec) + ((now.tv_usec - last_time.tv_usec) / 1000000.);
  last_time = now;

  int new_bytes = total_bytes;
  int diff = new_bytes - last_bytes;
  printf("wrote %dk %dM bytes in %f seconds -> %fM/sec\n", diff / 1024, diff / (1024 * 1024), time_diff, (diff / (time_diff * 1000000.)));
  last_bytes = new_bytes;
}


// tests O_DIRECT + pwrite = fail
//       O_DIRECT + truncate = happy
//                  truncate = happy

//                  pwrite = happy
//       O_DIRECT + truncate = happy
//                  truncate = happy
//       O_DIRECT + pwrite = fail

void do_mapwrite(int fd)
{
  // lets write as fast as we can....
  int const window_size = 2 * 1024 * 1024; // 2 * 1024 * 1024; // 16k
  int const window_pages = window_size / 4096;

  int file_page_offset = 0;
  long long file_size = window_size;


  // fast-forward... by 2046 windows
  file_size += 2046u * (2 * 1024 * 1024);
  file_page_offset += 2046;


  //ftruncate64(fd, file_size);
  printf("pwrite to %llx  %llu\n", file_size, file_size);
  pwrite64(fd, "", 1, file_size);

  char* mapping_start = static_cast<char*>(mmap64(0, window_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, file_size - window_size));


  // scribble into buffer and walk the window
  for (;;) {
    memset(mapping_start, 0, window_size);

    // grow file
    file_size += window_size;

    //ftruncate64(fd, file_size);
    pwrite64(fd, "", 1, file_size);

    file_page_offset += window_pages;
    //munmap(mapping_start, window_size);
    //mapping_start = static_cast<char*>(mmap64(mapping_start, window_size, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_FIXED, fd, file_size - window_size));
    while (remap_file_pages(mapping_start, window_size, 0, file_page_offset, MAP_SHARED /*| MAP_NONBLOCK*/) <0)
      perror("remap_file_pages");

    total_bytes += window_size;
  }
}



void do_filewrite(int fd)
{
  FILE* ptr = fdopen(fd, "w+");

  int line_len = 80;
  char* buf = (char*)malloc(line_len);
  memset(buf, 0, line_len);

  for (;;) {
    fwrite(buf, line_len, 1, ptr);
    total_bytes += line_len;
  }
}

void do_syswrite(int fd)
{
  int line_len = 64 * 1024;
  char* buf = (char*)malloc(line_len);
  memset(buf, 0, line_len);

  for (;;) {
    write(fd, buf, line_len);
    total_bytes += line_len;
  }
}


int main(int argc, char* argv[])
{
  if (argc != 3) {
    printf("usage: %s -[mfw] <filename>\n", argv[0]);
    return 0;
  }

  if ((argv[1][0] != '-') || 
      ((argv[1][1] != 'm') &&
       (argv[1][1] != 'w') &&
       (argv[1][1] != 'f'))) {
    printf("usage: %s -[mfw] <filename>\n", argv[0]);
    return 0;
  }

  int fd = open(argv[2], O_RDWR | O_CREAT | O_LARGEFILE | O_DIRECT, 0644);
  if (fd < 0) {
    perror("open");
    return 0;
  }

  // start the clock...
  signal(SIGALRM, timeout);
  {
    struct itimerval itv;
    itv.it_interval.tv_sec = 1;
    itv.it_interval.tv_usec = 0;
    itv.it_value = itv.it_interval;

    setitimer(ITIMER_REAL, &itv, 0);
  }

  switch (argv[1][1]) {
  case 'm': do_mapwrite(fd); break;
  case 'f': do_filewrite(fd); break;
  case 'w': do_syswrite(fd); break;
  }

  return 0;
}

Re: mmap over nfs leads to excessive system load

[Kenny Simpson]

[-- Attachment #1: Type: text/plain, Size: 662 bytes --]

Same test now done w/ rc1-mm2...
similar result.

However, instead of pegging 1 virtual cpu in system time, two virtual CPUs on the same core now
share the 100% load (so each shows 50% load and 50% idle).

Oprofile shows:

5610635  66.9136  zap_pte_range
711147    8.4813  _raw_spin_trylock
498282    5.9426  unmap_mapping_range
196369    2.3419  add_preempt_count
126243    1.5056  unmap_page_range
111791    1.3332  __bitmap_weight
96284     1.1483  page_address
88998     1.0614  _raw_spin_unlock

And I'm attaching the sysrq dump

-Kenny


		
__________________________________ 
Start your day with Yahoo! - Make it your home page! 
http://www.yahoo.com/r/hs

[-- Attachment #2: 3794922530-sysrq --]
[-- Type: application/octet-stream, Size: 54031 bytes --]

Nov 18 11:53:23 tux6127 kernel: [ 1309.383101] SysRq : Show State
Nov 18 11:53:23 tux6127 kernel: [ 1309.383112] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.383113]                                                sibling
Nov 18 11:53:23 tux6127 kernel: [ 1309.383116]   task             PC      pid father child younger older
Nov 18 11:53:23 tux6127 kernel: [ 1309.383121] init          S 00000001   716     1      0     2               (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.383153] c2985ea4 c2985e94 00000004 00000001 c0143c42 c04bf900 c04bf000 c2985e54 
Nov 18 11:53:23 tux6127 kernel: [ 1309.383187]        c02a6df8 c01275c3 c281ffe0 c281ffe0 00000286 c2985e6c c042b140 c281ffe0 
Nov 18 11:53:23 tux6127 kernel: [ 1309.383218]        c281ffe0 c2985eb8 00000286 c2985eb8 c2985e80 c281f460 00000001 00003f1f 
Nov 18 11:53:23 tux6127 kernel: [ 1309.383245] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.383249]  [<c042a555>] schedule_timeout+0x54/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.383268]  [<c0173514>] do_select+0x15e/0x27a
Nov 18 11:53:23 tux6127 kernel: [ 1309.383283]  [<c01738e2>] sys_select+0x27b/0x44f
Nov 18 11:53:23 tux6127 kernel: [ 1309.383298]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.383314] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.383320] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.383327] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.383333] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.383339] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.383349] .....[<c042a555>] ..   ( <= schedule_timeout+0x54/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.383361] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.383373] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.383383] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.383387] migration/0   S 00000002  3364     2      1             3       (L-TLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.383415] c299cf94 c299cf84 00000004 00000002 ffffe658 ffffffff c28174b8 0000654e 
Nov 18 11:53:23 tux6127 kernel: [ 1309.383440]        00000000 c281f460 00000001 00000001 0000007d c281f4b4 c281fe00 00000001 
Nov 18 11:53:23 tux6127 kernel: [ 1309.383465]        c299cf5c c042b48a c281f460 c299cf94 c0118006 c2817460 00000000 00000c53 
Nov 18 11:53:23 tux6127 kernel: [ 1309.383493] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.383499]  [<c0119a1b>] migration_thread+0x81/0x10d
Nov 18 11:53:23 tux6127 kernel: [ 1309.383517]  [<c0132844>] kthread+0xb7/0xbc
Nov 18 11:53:23 tux6127 kernel: [ 1309.383531]  [<c0101181>] kernel_thread_helper+0x5/0xb
Nov 18 11:53:23 tux6127 kernel: [ 1309.383543] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.383548] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.383553] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.383558] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.383563] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.383574] .....[<c0119a1b>] ..   ( <= migration_thread+0x81/0x10d)
Nov 18 11:53:23 tux6127 kernel: [ 1309.383584] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.383595] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.383606] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.383611] ksoftirqd/0   S 00000002  3580     3      1             4     2 (L-TLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.383642] c29a0f98 c29a0f88 00000004 00000002 c05b3000 c29a0f68 c29a0f38 c012381f 
Nov 18 11:53:23 tux6127 kernel: [ 1309.383668]        c29a0f40 c29a0f60 c2a759d0 c0118a46 ffffffff c29a0000 00000000 000000b1 
Nov 18 11:53:23 tux6127 kernel: [ 1309.383697]        c05a6380 c29a0000 c05a6380 c29a0fb4 c010390e c2817460 00000000 0000012d 
Nov 18 11:53:23 tux6127 kernel: [ 1309.383730] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.383735]  [<c0123d02>] ksoftirqd+0xde/0x106
Nov 18 11:53:23 tux6127 kernel: [ 1309.383746]  [<c0132844>] kthread+0xb7/0xbc
Nov 18 11:53:23 tux6127 kernel: [ 1309.383760]  [<c0101181>] kernel_thread_helper+0x5/0xb
Nov 18 11:53:23 tux6127 kernel: [ 1309.383774] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.383779] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.383784] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.383789] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.383795] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.383806] .....[<c0123d02>] ..   ( <= ksoftirqd+0xde/0x106)
Nov 18 11:53:23 tux6127 kernel: [ 1309.383816] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.383830] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.383838] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.383842] watchdog/0    S 00000002  3304     4      1             5     3 (L-TLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.383872] c29a4f44 c29a4f30 00000004 00000002 00000286 c29a4eec c02a6df8 00000000 
Nov 18 11:53:23 tux6127 kernel: [ 1309.383897]        c29899d0 c01275c3 c2817fe0 c2817fe0 00000286 c29a4f0c c042b140 f1e7f9d0 
Nov 18 11:53:23 tux6127 kernel: [ 1309.383929]        0000002d c29a4f58 f1e7f9d0 00000000 c2817e00 c2817460 00000000 00000ff3 
Nov 18 11:53:23 tux6127 kernel: [ 1309.383959] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.383964]  [<c042a555>] schedule_timeout+0x54/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.383977]  [<c042a5c0>] schedule_timeout_interruptible+0x1a/0x1c
Nov 18 11:53:23 tux6127 kernel: [ 1309.383990]  [<c0128356>] msleep_interruptible+0x3a/0x48
Nov 18 11:53:23 tux6127 kernel: [ 1309.384007]  [<c013d982>] watchdo900 000000d0 dc505ea4 00000130 0001a90e 00000000 f24909d0 
Nov 18 11:53:23 tux6127 kernel: [ 1309.396571]        c0132be1 ef7ec1e8 f24909d0 00000000 c282fe00 c282f460 00000003 00008795 
Nov 18 11:53:23 tux6127 kernel: [ 1309.396581] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.396583]  [<c042a5a4>] schedule_timeout+0xa3/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.396589]  [<c0173514>] do_select+0x15e/0x27a
Nov 18 11:53:23 tux6127 kernel: [ 1309.396594]  [<c01738e2>] sys_select+0x27b/0x44f
Nov 18 11:53:23 tux6127 kernel: [ 1309.396599]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.396603] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.396605] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.396607] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.396609] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.396612] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.396616] .....[<c042a5a4>] ..   ( <= schedule_timeout+0xa3/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.396621] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.396625] .....[<c0428f50>] ..   ( <= schedule+0x1f8/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.396629] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.396631] kwin          S 00000003   784  4084   4059          4092  4075 (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.396641] f7f71ea4 f7f71e94 00000004 00000003 000200d0 00000002 00000001 00000002 
Nov 18 11:53:23 tux6127 kernel: [ 1309.396650]        c18ff820 c04bf900 000000d0 f2e659d0 f7f71e8c c0143c97 f7f71e70 c02a6df8 
Nov 18 11:53:23 tux6127 kernel: [ 1309.396661]        c0132be1 effd600c f2e60e04 00000286 00000286 c282f460 00000003 00004001 
Nov 18 11:53:23 tux6127 kernel: [ 1309.396671] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.396673]  [<c042a5a4>] schedule_timeout+0xa3/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.396679]  [<c0173514>] do_select+0x15e/0x27a
Nov 18 11:53:23 tux6127 kernel: [ 1309.396684]  [<c01738e2>] sys_select+0x27b/0x44f
Nov 18 11:53:23 tux6127 kernel: [ 1309.396689]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.396693] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.396695] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.396697] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.396700] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.396702] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.396706] .....[<c042a5a4>] ..   ( <= schedule_timeout+0xa3/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.396711] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.396715] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.396719] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.396721] khotkeys      S 00000003   784  4087      1          4089  4083 (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.396731] f5ac2ea4 f5ac2e94 00000004 00000003 000200d0 00000002 00000001 00000002 
Nov 18 11:53:23 tux6127 kernel: [ 1309.396740]        c19300c0 c04bf900 000000d0 f24909d0 f5ac2e8c c0143c97 f5ac2e70 c02a6df8 
Nov 18 11:53:23 tux6127 kernel: [ 1309.396750]        c0132be1 f5ac2e7c c02a6df8 c0132be1 f100407c c282f460 00000003 0000b903 
Nov 18 11:53:23 tux6127 kernel: [ 1309.396761] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.396763]  [<c042a5a4>] schedule_timeout+0xa3/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.396768]  [<c0173514>] do_select+0x15e/0x27a
Nov 18 11:53:23 tux6127 kernel: [ 1309.396773]  [<c01738e2>] sys_select+0x27b/0x44f
Nov 18 11:53:23 tux6127 kernel: [ 1309.396778]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.396782] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.396785] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.396787] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.396789] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.396791] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.396795] .....[<c042a5a4>] ..   ( <= schedule_timeout+0xa3/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.396800] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.396804] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.396808] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.396810] kdesktop      S 00000004   700  4089      1  4122    4091  4087 (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.396821] f2b36ea4 f2b36e90 00000004 00000004 00000286 f2402dec f2b36ed4 f2b05e0c 
Nov 18 11:53:23 tux6127 kernel: [ 1309.396830]        f2e659d0 c01275c3 c2827fe0 c2827fe0 00000286 f2b36e6c c042b140 f7b319d0 
Nov 18 11:53:23 tux6127 kernel: [ 1309.396841]        c2827fe0 f2b36eb8 f7b319d0 00000000 c2827e00 c2827460 00000002 000090b5 
Nov 18 11:53:23 tux6127 kernel: [ 1309.396851] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.396853]  [<c042a555>] schedule_timeout+0x54/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.396858]  [<c0173514>] do_select+0x15e/0x27a
Nov 18 11:53:23 tux6127 kernel: [ 1309.396863]  [<c01738e2>] sys_select+0x27b/0x44f
Nov 18 11:53:23 tux6127 kernel: [ 1309.396868]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.396873] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.396875] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.396877] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.396879] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.396882] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.396886] .....[<c042a555>] ..   ( <= schedule_timeout+0x54/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.396890] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.396895] .....[<c0428f50>] ..   ( <= schedule+0x1f8/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.396899] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.396900] kicker        S 00000004   784  4091      1          4094  4089 (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.396911] f6ba9ea4 f6ba9e90 00000004 00000004 000200d0 00000002 00000001 f4d72e0c 
Nov 18 11:53:23 tux6127 kernel: [ 1309.396920]        f24909d0 c01275c3 c2827fe0 c2827fe0 00000286 f6ba9e6c c042b140 f7b319d0 
Nov 18 11:53:23 tux6127 kernel: [ 1309.396930]        c2827fe0 f6ba9eb8 f7b319d0 00000000 c2827e00 c2827460 00000002 00007bc5 
Nov 18 11:53:23 tux6127 kernel: [ 1309.396940] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.396943]  [<c042a555>] schedule_timeout+0x54/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.396948]  [<c0173514>] do_select+0x15e/0x27a
Nov 18 11:53:23 tux6127 kernel: [ 1309.396953]  [<c01738e2>] sys_select+0x27b/0x44f
Nov 18 11:53:23 tux6127 kernel: [ 1309.396957]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.396962] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.396964] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.396966] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.396968] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.396971] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.396975] .....[<c042a555>] ..   ( <= schedule_timeout+0x54/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.396979] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.396983] .....[<c0428f50>] ..   ( <= schedule+0x1f8/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.396987] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.396989] kio_file      S 00000001   784  4092   4059          4096  4084 (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397000] f5fecea4 f5fece94 00000004 00000001 000200d0 00000002 00000001 00000002 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397009]        c1972ae0 c04bf900 c29899d0 f736a9d0 f5fece8c c0143c97 f5fece70 c02a6df8 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397019]        c0132be1 f263a00c f25e2e04 00000286 00000286 c281f460 00000001 000001f5 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397030] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.397032]  [<c042a5a4>] schedule_timeout+0xa3/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.397037]  [<c0173514>] do_select+0x15e/0x27a
Nov 18 11:53:23 tux6127 kernel: [ 1309.397043]  [<c01738e2>] sys_select+0x27b/0x44f
Nov 18 11:53:23 tux6127 kernel: [ 1309.397047]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.397052] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.397054] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.397056] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.397058] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.397061] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.397065] .....[<c042a5a4>] ..   ( <= schedule_timeout+0xa3/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397069] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.397073] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397077] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397079] klipper       S 00000002   764  4094      1          4099  4091 (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397090] f2b54ea4 f2b54e94 00000004 00000002 000200d0 00000002 00000001 00000002 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397099]        c18d3600 c04bf900 000000d0 f26c09d0 f2b54e8c c0143c97 f2b54e70 c02a6df8 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397109]        c0132be1 ef12000c f26d4e04 00000286 00000286 c2817460 00000000 0000fcc7 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397119] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.397122]  [<c042a5a4>] schedule_timeout+0xa3/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.397127]  [<c0173514>] do_select+0x15e/0x27a
Nov 18 11:53:23 tux6127 kernel: [ 1309.397132]  [<c01738e2>] sys_select+0x27b/0x44f
Nov 18 11:53:23 tux6127 kernel: [ 1309.397137]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.397141] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.397143] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.397145] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.397147] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.397150] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.397154] .....[<c042a5a4>] ..   ( <= schedule_timeout+0xa3/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397158] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.397163] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397167] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397169] evolution-ala S 00000003   772  4096   4059          4097  4092 (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397179] f266df00 f266def0 00000004 00000003 c04bf900 000000d0 f266deac c02a6df8 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397188]        c0132be1 ef69f07c f26f6e04 00000286 00000286 f26f6e04 f266dec4 c042b524 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397199]        f26f6e04 ef69f07c f266ded8 c0132bff ef69f000 c282f460 00000003 000181d5 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397209] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.397211]  [<c042a5a4>] schedule_timeout+0xa3/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.397217]  [<c0173be8>] do_poll+0x9a/0xb9
Nov 18 11:53:23 tux6127 kernel: [ 1309.397221]  [<c0173d69>] sys_poll+0x162/0x22d
Nov 18 11:53:23 tux6127 kernel: [ 1309.397226]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.397231] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.397233] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.397235] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.397237] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.397240] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.397244] .....[<c042a5a4>] ..   ( <= schedule_timeout+0xa3/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397248] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.397253] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397256] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397259] evolution-ala S 00000003  3140  4106   4059                4097 (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397270] f2648f00 f2648ef0 00000004 00000003 c04bf900 000000d0 f2648eac c02a6df8 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397279]        c0132be1 f1d060ec f21fae04 00000286 00000286 f21fae04 f2648ec4 c042b524 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397290]        f21fae04 f1d060ec f2648ed8 c0132bff f1d06000 c282f460 00000003 000040c6 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397301] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.397303]  [<c042a5a4>] schedule_timeout+0xa3/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.397308]  [<c0173be8>] do_poll+0x9a/0xb9
Nov 18 11:53:23 tux6127 kernel: [ 1309.397313]  [<c0173d69>] sys_poll+0x162/0x22d
Nov 18 11:53:23 tux6127 kernel: [ 1309.397318]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.397323] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.397325] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.397327] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.397329] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.397331] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.397336] .....[<c042a5a4>] ..   ( <= schedule_timeout+0xa3/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397340] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.397344] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397348] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397350] gkrellm       S 00000003   772  4097   4059          4106  4096 (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397360] f6bb9f00 f6bb9eec 00000004 00000003 c04bf900 000000d0 f6bb9eac 00000000 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397369]        c29909d0 c01275c3 c282ffe0 c282ffe0 00000286 f6bb9ec8 c042b140 f7b319d0 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397379]        00000045 f6bb9f14 f7b319d0 00000000 c282fe00 c282f460 00000003 00014414 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397390] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.397392]  [<c042a555>] schedule_timeout+0x54/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.397397]  [<c0173be8>] do_poll+0x9a/0xb9
Nov 18 11:53:23 tux6127 kernel: [ 1309.397402]  [<c0173d69>] sys_poll+0x162/0x22d
Nov 18 11:53:23 tux6127 kernel: [ 1309.397407]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.397411] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.397414] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.397416] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.397418] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.397420] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.397424] .....[<c042a555>] ..   ( <= schedule_timeout+0x54/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397429] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.397435] .....[<c0428f50>] ..   ( <= schedule+0x1f8/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397439] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397441] gconfd-2      S 00000002   752  4099      1          4102  4094 (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397452] f6a83f00 f6a83ef0 00000004 00000002 c04bf900 000000d0 f6a83eac f6a83eb0 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397461]        c02a6df8 c01275c3 c2817fe0 c2817fe0 00000286 f6a83ec8 c042b140 c2817fe0 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397471]        c2817fe0 f6a83f14 00000286 f6a83f14 f6a83edc c2817460 00000000 00001d41 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397482] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.397484]  [<c042a555>] schedule_timeout+0x54/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.397489]  [<c0173be8>] do_poll+0x9a/0xb9
Nov 18 11:53:23 tux6127 kernel: [ 1309.397494]  [<c0173d69>] sys_poll+0x162/0x22d
Nov 18 11:53:23 tux6127 kernel: [ 1309.397499]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.397504] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.397506] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.397508] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.397510] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.397513] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.397517] .....[<c042a555>] ..   ( <= schedule_timeout+0x54/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397522] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.397526] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397530] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397532] bonobo-activa S 00000002  1768  4102      1          4110  4099 (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397543] f6a57f00 f6a57ef0 00000004 00000002 c04bf900 000000d0 f6a57eac c02a6df8 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397553]        c0132be1 f1c270b4 f1cfee04 00000286 00000286 f1cfee04 f6a57ec4 c042b524 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397563]        f1cfee04 f1c270b4 f6a57ed8 c0132bff f1c27000 c2817460 00000000 00017814 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397573] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.397576]  [<c042a5a4>] schedule_timeout+0xa3/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.397581]  [<c0173be8>] do_poll+0x9a/0xb9
Nov 18 11:53:23 tux6127 kernel: [ 1309.397585]  [<c0173d69>] sys_poll+0x162/0x22d
Nov 18 11:53:23 tux6127 kernel: [ 1309.397590]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.397595] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.397597] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.397599] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.397601] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.397604] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.397608] .....[<c042a5a4>] ..   ( <= schedule_timeout+0xa3/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397612] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.397617] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397621] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397623] evolution-dat S 00000004   812  4110      1          4111  4102 (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397633] f2aaff00 f2aafef0 00000004 00000004 c04bf900 000000d0 f2b039d0 f2aafed4 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397642]        c0143c97 f2aafeb8 c02a6df8 c0132be1 f1d34044 f2e37df8 00000286 00000286 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397652]        f2e37df8 f2aafed0 c042b524 f2e37df8 f1d34044 c2827460 00000002 00000e63 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397663] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.397665]  [<c042a5a4>] schedule_timeout+0xa3/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.397670]  [<c0173be8>] do_poll+0x9a/0xb9
Nov 18 11:53:23 tux6127 kernel: [ 1309.397675]  [<c0173d69>] sys_poll+0x162/0x22d
Nov 18 11:53:23 tux6127 kernel: [ 1309.397680]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.397684] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.397686] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.397689] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.397691] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.397693] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.397697] .....[<c042a5a4>] ..   ( <= schedule_timeout+0xa3/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397702] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.397706] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397710] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397712] evolution-dat S 00000004  3004  4111      1          4118  4110 (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397724] f6508f00 f6508ef0 00000004 00000004 c04bf900 000000d0 f6508eac c02a6df8 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397733]        c0132be1 f1d490ec c29909d0 00000286 00000286 f2046e04 f6508ec4 c042b524 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397743]        f2046e04 f1d490ec f6508ed8 c0132bff f1d49000 c2827460 00000002 000004a6 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397754] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.397756]  [<c042a5a4>] schedule_timeout+0xa3/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.397762]  [<c0173be8>] do_poll+0x9a/0xb9
Nov 18 11:53:23 tux6127 kernel: [ 1309.397766]  [<c0173d69>] sys_poll+0x162/0x22d
Nov 18 11:53:23 tux6127 kernel: [ 1309.397771]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.397776] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.397779] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.397782] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.397785] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.397788] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.397792] .....[<c042a5a4>] ..   ( <= schedule_timeout+0xa3/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397796] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.397800] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397804] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397806] evolution-dat S 00000003  3048  4118      1          4701  4111 (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397818] f256bf00 f256bef0 00000004 00000003 c04bf900 000000d0 f20749d0 f256bed4 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397827]        c0143c97 f256beb8 c02a6df8 c0132be1 f221d00c f2048df8 00000286 00000286 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397837]        f2048df8 f256bed0 c042b524 f2048df8 f221d00c c282f460 00000003 00004383 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397848] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.397850]  [<c042a5a4>] schedule_timeout+0xa3/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.397856]  [<c0173be8>] do_poll+0x9a/0xb9
Nov 18 11:53:23 tux6127 kernel: [ 1309.397860]  [<c0173d69>] sys_poll+0x162/0x22d
Nov 18 11:53:23 tux6127 kernel: [ 1309.397865]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.397870] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.397872] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.397874] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.397876] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.397879] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.397883] .....[<c042a5a4>] ..   ( <= schedule_timeout+0xa3/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397887] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.397892] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397895] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397897] xterm         S 00000003   724  4122   4089  4123    4135       (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397908] f2a86ea4 f2a86e94 00000004 00000003 c02e0004 f46c87f8 c04cf964 f2a86e54 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397917]        c02a6df8 c01275c3 c282ffe0 c282ffe0 00000286 f2a86e6c c042b140 c282ffe0 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397928]        c282ffe0 f2a86eb8 00000286 f2a86eb8 f2a86e80 c282f460 00000003 000013c2 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397938] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.397940]  [<c042a555>] schedule_timeout+0x54/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.397945]  [<c0173514>] do_select+0x15e/0x27a
Nov 18 11:53:23 tux6127 kernel: [ 1309.397951]  [<c01738e2>] sys_select+0x27b/0x44f
Nov 18 11:53:23 tux6127 kernel: [ 1309.397955]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.397960] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.397962] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.397964] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.397966] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.397969] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.397973] .....[<c042a555>] ..   ( <= schedule_timeout+0x54/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397978] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.397982] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397986] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.397988] bash          S 00000002   688  4123   4122  4238               (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.397998] f26e5f08 f26e5ef8 00000004 00000002 00000003 c25a6230 c04bf928 000200d2 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398007]        f58739d0 f26e5ee4 f26e5ec4 f26e5ec8 c04cb6dc c04cb6c0 f26e5ed4 c02a6df8 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398018]        f26e5edc c02a6df8 c0132be1 f26e5f54 f5e4fe7c c2817460 00000000 000005d8 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398028] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.398030]  [<c0121f28>] do_wait+0x193/0x37f
Nov 18 11:53:23 tux6127 kernel: [ 1309.398035]  [<c01221d4>] sys_wait4+0x3e/0x42
Nov 18 11:53:23 tux6127 kernel: [ 1309.398040]  [<c01221ff>] sys_waitpid+0x27/0x2b
Nov 18 11:53:23 tux6127 kernel: [ 1309.398045]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.398049] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.398051] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.398053] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.398056] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.398058] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.398062] .....[<c0121f28>] ..   ( <= do_wait+0x193/0x37f)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398066] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.398070] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398074] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398076] xterm         S 00000001   724  4135   4089  4136    4241  4122 (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398087] f2227ea4 f2227e90 00000004 00000001 c02e0004 f1dc77f8 c04cf964 00000000 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398096]        c04b9ae0 c01275c3 c281ffe0 c281ffe0 00000286 f2227e6c c042b140 f1e7f9d0 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398107]        00000013 f2227eb8 f1e7f9d0 00000000 c281fe00 c281f460 00000001 0001075f 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398117] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.398119]  [<c042a555>] schedule_timeout+0x54/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.398124]  [<c0173514>] do_select+0x15e/0x27a
Nov 18 11:53:23 tux6127 kernel: [ 1309.398129]  [<c01738e2>] sys_select+0x27b/0x44f
Nov 18 11:53:23 tux6127 kernel: [ 1309.398134]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.398138] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.398140] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.398142] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.398144] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.398147] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.398151] .....[<c042a555>] ..   ( <= schedule_timeout+0x54/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398155] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.398160] .....[<c0428f50>] ..   ( <= schedule+0x1f8/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398164] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398166] bash          S 00000003   724  4136   4135                     (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398176] f646ce78 f646ce68 00000004 00000003 f1dc77f8 f646ce20 f646ce24 c02a6df8 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398185]        c02e0004 f646ce30 c02a6df8 c02e0120 f2bde804 c04cf964 00000286 00000286 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398196]        00000286 f646ce48 c042b524 c04cf964 f2bde804 c282f460 00000003 0000c649 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398206] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.398208]  [<c042a5a4>] schedule_timeout+0xa3/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.398214]  [<c02e63e6>] read_chan+0x4b7/0x5d8
Nov 18 11:53:23 tux6127 kernel: [ 1309.398220]  [<c02e1066>] tty_read+0xcc/0xd0
Nov 18 11:53:23 tux6127 kernel: [ 1309.398225]  [<c015f81f>] vfs_read+0xa6/0x17d
Nov 18 11:53:23 tux6127 kernel: [ 1309.398230]  [<c015fba5>] sys_read+0x4b/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.398235]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.398239] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.398241] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.398243] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.398245] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.398248] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.398252] .....[<c042a5a4>] ..   ( <= schedule_timeout+0xa3/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398256] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.398261] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398265] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398266] writetest     R running   776  4238   4123                     (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398276] xterm         S 00000003   784  4241   4089  4242    4798  4135 (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398287] f45f4ea4 f45f4e94 00000004 00000003 c02e0004 f1d2b7f8 c04cf964 f45f4e54 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398296]        c02a6df8 c01275c3 c282ffe0 c282ffe0 00000286 f45f4e6c c042b140 c282ffe0 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398307]        c282ffe0 f45f4eb8 00000286 f45f4eb8 f45f4e80 c282f460 00000003 00000de7 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398317] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.398319]  [<c042a555>] schedule_timeout+0x54/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.398324]  [<c0173514>] do_select+0x15e/0x27a
Nov 18 11:53:23 tux6127 kernel: [ 1309.398330]  [<c01738e2>] sys_select+0x27b/0x44f
Nov 18 11:53:23 tux6127 kernel: [ 1309.398334]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.398339] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.398341] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.398343] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.398345] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.398348] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.398352] .....[<c042a555>] ..   ( <= schedule_timeout+0x54/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398356] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.398361] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398364] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398366] bash          S 00000001   784  4242   4241                     (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398376] f1a06e78 f1a06e68 00000004 00000001 f1d2b7f8 f1a06e20 f1a06e24 c02a6df8 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398386]        c02e0004 f1a06e30 c02a6df8 c02e0120 f1703804 c04cf964 00000286 00000286 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398396]        00000286 f1a06e48 c042b524 c04cf964 f1703804 c281f460 00000001 000101a8 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398407] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.398409]  [<c042a5a4>] schedule_timeout+0xa3/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.398414]  [<c02e63e6>] read_chan+0x4b7/0x5d8
Nov 18 11:53:23 tux6127 kernel: [ 1309.398420]  [<c02e1066>] tty_read+0xcc/0xd0
Nov 18 11:53:23 tux6127 kernel: [ 1309.398424]  [<c015f81f>] vfs_read+0xa6/0x17d
Nov 18 11:53:23 tux6127 kernel: [ 1309.398431]  [<c015fba5>] sys_read+0x4b/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.398437]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.398442] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.398444] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.398446] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.398448] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.398450] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.398455] .....[<c042a5a4>] ..   ( <= schedule_timeout+0xa3/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398459] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.398465] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398469] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398472] oprofiled     S 00000001  1396  4701      1                4118 (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398482] f0a08f04 f0a08ef4 00000004 00000001 00030002 f0a08f80 c012293c f09a5ec8 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398492]        f0a08f5c 00030002 c04c0b84 00000000 c01187e8 c04c0b84 00000003 f0a08ed4 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398502]        c02a6df8 c0132ce2 f0a08f38 f8b99b04 00000286 c281f460 00000001 000736eb 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398512] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.398515]  [<f8b935b1>] event_buffer_read+0x157/0x15c [oprofile]
Nov 18 11:53:23 tux6127 kernel: [ 1309.398526]  [<c015f81f>] vfs_read+0xa6/0x17d
Nov 18 11:53:23 tux6127 kernel: [ 1309.398531]  [<c015fba5>] sys_read+0x4b/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.398536]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.398541] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.398543] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.398545] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.398547] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.398549] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.398554] .....[<f8b935b1>] ..   ( <= event_buffer_read+0x157/0x15c [oprofile])
Nov 18 11:53:23 tux6127 kernel: [ 1309.398562] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.398566] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398570] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398572] xterm         S 00000002   724  4798   4089  4799    4809  4241 (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398583] ef65aea4 ef65ae94 00000004 00000002 c02e0004 f1dc67f8 c04cf964 ef65ae54 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398592]        c02a6df8 c01275c3 c2817fe0 c2817fe0 00000286 ef65ae6c c042b140 c2817fe0 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398602]        c2817fe0 ef65aeb8 00000286 ef65aeb8 ef65ae80 c2817460 00000000 000013fd 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398613] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.398615]  [<c042a555>] schedule_timeout+0x54/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.398620]  [<c0173514>] do_select+0x15e/0x27a
Nov 18 11:53:23 tux6127 kernel: [ 1309.398625]  [<c01738e2>] sys_select+0x27b/0x44f
Nov 18 11:53:23 tux6127 kernel: [ 1309.398630]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.398634] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.398636] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.398638] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.398641] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.398643] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.398648] .....[<c042a555>] ..   ( <= schedule_timeout+0x54/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398652] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.398656] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398660] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398662] bash          S 000000DF   724  4799   4798                     (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398673] efbf3e78 f272a9d0 c281f4b4 000000df 00000001 00000001 00000082 f1e28e0c 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398682]        f1e7f9d0 00000001 efbf3e30 4d1775c4 000000df 00001add 00000000 f272a9d0 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398692]        0000003f ef6bf044 f272a9d0 00000000 c281fe00 c281f460 00000001 00000c2a 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398702] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.398704]  [<c042a5a4>] schedule_timeout+0xa3/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.398709]  [<c02e63e6>] read_chan+0x4b7/0x5d8
Nov 18 11:53:23 tux6127 kernel: [ 1309.398715]  [<c02e1066>] tty_read+0xcc/0xd0
Nov 18 11:53:23 tux6127 kernel: [ 1309.398719]  [<c015f81f>] vfs_read+0xa6/0x17d
Nov 18 11:53:23 tux6127 kernel: [ 1309.398724]  [<c015fba5>] sys_read+0x4b/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.398729]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.398734] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.398736] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.398738] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.398740] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.398742] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.398746] .....[<c042a5a4>] ..   ( <= schedule_timeout+0xa3/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398751] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.398755] .....[<c0428f50>] ..   ( <= schedule+0x1f8/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398759] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398761] xterm         S 00000004   724  4809   4089  4810          4798 (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398771] ef70cea4 ef70ce90 00000004 00000004 c02e0004 f17847f8 c04cf964 f2b05e0c 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398780]        f2e659d0 c01275c3 c2827fe0 c2827fe0 00000286 ef70ce6c c042b140 f1e7f9d0 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398791]        c2827fe0 ef70ceb8 f1e7f9d0 00000000 c2827e00 c2827460 00000002 00003c7b 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398801] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.398803]  [<c042a555>] schedule_timeout+0x54/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.398808]  [<c0173514>] do_select+0x15e/0x27a
Nov 18 11:53:23 tux6127 kernel: [ 1309.398813]  [<c01738e2>] sys_select+0x27b/0x44f
Nov 18 11:53:23 tux6127 kernel: [ 1309.398818]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.398823] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.398825] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.398827] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.398829] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.398831] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.398835] .....[<c042a555>] ..   ( <= schedule_timeout+0x54/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398840] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.398844] .....[<c0428f50>] ..   ( <= schedule+0x1f8/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398848] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398850] bash          S 00000002   724  4810   4809  4820               (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398860] ef683f08 ef683ef8 00000004 00000002 00000003 c25fae60 c04bf928 000200d2 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398869]        ef7699d0 ef683ee4 ef683ec4 c02a6df8 c014cab3 c25b1930 ef683ed4 c02a6df8 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398880]        ef683edc c02a6df8 c0132be1 ef683f54 ef7efe7c c2817460 00000000 000059b4 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398890] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.398893]  [<c0121f28>] do_wait+0x193/0x37f
Nov 18 11:53:23 tux6127 kernel: [ 1309.398898]  [<c01221d4>] sys_wait4+0x3e/0x42
Nov 18 11:53:23 tux6127 kernel: [ 1309.398902]  [<c01221ff>] sys_waitpid+0x27/0x2b
Nov 18 11:53:23 tux6127 kernel: [ 1309.398907]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.398911] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.398913] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.398916] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.398918] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.398920] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.398924] .....[<c0121f28>] ..   ( <= do_wait+0x193/0x37f)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398928] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.398933] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398937] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398939] firefox-bin   S 00000002   512  4820   4810          4851       (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.398949] ef682f00 ef682ef0 00000004 00000002 ef682ec8 c0143c97 ef682eac c02a6df8 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398958]        c0132be1 ef682eb8 c04b9ae0 c0132be1 efb930d0 ef7b0df8 00000286 00000286 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398969]        ef7b0df8 ef682ed0 c042b524 ef7b0df8 efb930d0 c2817460 00000000 000002b1 
Nov 18 11:53:23 tux6127 kernel: [ 1309.398979] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.398981]  [<c042a5a4>] schedule_timeout+0xa3/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.398986]  [<c0173be8>] do_poll+0x9a/0xb9
Nov 18 11:53:23 tux6127 kernel: [ 1309.398991]  [<c0173d69>] sys_poll+0x162/0x22d
Nov 18 11:53:23 tux6127 kernel: [ 1309.398996]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.399001] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.399003] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.399005] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.399007] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.399009] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.399014] .....[<c042a5a4>] ..   ( <= schedule_timeout+0xa3/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.399018] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.399022] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.399026] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.399028] firefox-bin   S 00000004   960  4851   4810          4853  4820 (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.399039] ef79df00 ef79def0 00000004 00000004 ef79dea4 c02a6df8 c0132be1 ef06f028 
Nov 18 11:53:23 tux6127 kernel: [ 1309.399048]        eed24e04 00000286 00000286 eed24e04 ef79debc c042b524 eed24e04 ef06f028 
Nov 18 11:53:23 tux6127 kernel: [ 1309.399058]        ef79ded0 c0132bff ef06f000 ef6f21e4 ef79df98 c2827460 00000002 00001391 
Nov 18 11:53:23 tux6127 kernel: [ 1309.399069] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.399071]  [<c042a5a4>] schedule_timeout+0xa3/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.399076]  [<c0173be8>] do_poll+0x9a/0xb9
Nov 18 11:53:23 tux6127 kernel: [ 1309.399081]  [<c0173d69>] sys_poll+0x162/0x22d
Nov 18 11:53:23 tux6127 kernel: [ 1309.399086]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.399090] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.399093] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.399095] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.399097] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.399099] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.399103] .....[<c042a5a4>] ..   ( <= schedule_timeout+0xa3/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.399108] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.399112] .....[<c04297ca>] ..   ( <= schedule+0xa72/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.399116] 
Nov 18 11:53:23 tux6127 kernel: [ 1309.399118] firefox-bin   S 00000004  3256  4853   4810                4851 (NOTLB)
Nov 18 11:53:23 tux6127 kernel: [ 1309.399130] ef70be6c ef70be58 00000004 00000004 ef70be10 ef70be14 c02a6df8 f4ed7e0c 
Nov 18 11:53:23 tux6127 kernel: [ 1309.399140]        f7b319d0 c01275c3 c2827fe0 c2827fe0 00000286 ef70be34 c042b140 f2e659d0 
Nov 18 11:53:23 tux6127 kernel: [ 1309.399150]        00000022 ef70be80 f2e659d0 00000000 c2827e00 c2827460 00000002 000016a1 
Nov 18 11:53:23 tux6127 kernel: [ 1309.399160] Call Trace:
Nov 18 11:53:23 tux6127 kernel: [ 1309.399163]  [<c042a555>] schedule_timeout+0x54/0xa5
Nov 18 11:53:23 tux6127 kernel: [ 1309.399168]  [<c013707f>] futex_wait+0x217/0x2a4
Nov 18 11:53:23 tux6127 kernel: [ 1309.399174]  [<c0137394>] do_futex+0x42/0xaf
Nov 18 11:53:23 tux6127 kernel: [ 1309.399178]  [<c01374ec>] sys_futex+0xeb/0xf7
Nov 18 11:53:23 tux6127 kernel: [ 1309.399183]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 11:53:23 tux6127 kernel: [ 1309.399188] ---------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.399190] | preempt count: 00000002 ]
Nov 18 11:53:23 tux6127 kernel: [ 1309.399192] | 2 level deep critical section nesting:
Nov 18 11:53:23 tux6127 kernel: [ 1309.399194] ----------------------------------------
Nov 18 11:53:23 tux6127 kernel: [ 1309.399196] .. [<c0428d9e>] .... schedule+0x46/0xdf3
Nov 18 11:53:23 tux6127 kernel: [ 1309.399201] .....[<c042a555>] ..   ( <= schedule_timeout+0x54/0xa5)
Nov 18 11:53:23 tux6127 kernel: [ 1309.399205] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 11:53:23 tux6127 kernel: [ 1309.399209] .....[<c0428f50>] ..   ( <= schedule+0x1f8/0xdf3)
Nov 18 11:53:23 tux6127 kernel: [ 1309.399213] 

Re: mmap over nfs leads to excessive system load

[Kenny Simpson]

[-- Attachment #1: Type: text/plain, Size: 247 bytes --]

Instead of sysrq with 't', here is the sysrq with 'p'... it agrees with oprofile

Has anyone else been able to repoduce this?

-Kenny



	
		
__________________________________ 
Yahoo! Mail - PC Magazine Editors' Choice 2005 
http://mail.yahoo.com

[-- Attachment #2: 3794922530-sysrq --]
[-- Type: application/octet-stream, Size: 11561 bytes --]

Nov 18 12:17:10 tux6127 kernel: [ 2735.531683] SysRq : Show Regs
Nov 18 12:17:10 tux6127 kernel: [ 2735.531777] 
Nov 18 12:17:10 tux6127 kernel: [ 2735.531825] Pid: 0, comm:              swapper
Nov 18 12:17:10 tux6127 kernel: [ 2735.531876] EIP: 0060:[<c02d6299>] CPU: 0
Nov 18 12:17:10 tux6127 kernel: [ 2735.531934] EIP is at acpi_safe_halt+0x26/0x32
Nov 18 12:17:10 tux6127 kernel: [ 2735.531986]  EFLAGS: 00000246    Tainted: P       (2.6.15-rc1-mm2)
Nov 18 12:17:10 tux6127 kernel: [ 2735.532040] EAX: 00000000 EBX: c02d62a5 ECX: 00000001 EDX: c056b000
Nov 18 12:17:10 tux6127 kernel: [ 2735.532094] ESI: f7ab9c6c EDI: f7ab9bf8 EBP: c056bf88 DS: 007b ES: 007b
Nov 18 12:17:10 tux6127 kernel: [ 2735.532185] CR0: 8005003b CR2: b7f33e64 CR3: 364a5000 CR4: 00000640
Nov 18 12:17:10 tux6127 kernel: [ 2735.532239] ---------------------------
Nov 18 12:17:10 tux6127 kernel: [ 2735.532289] | preempt count: 00010001 ]
Nov 18 12:17:10 tux6127 kernel: [ 2735.532340] | 1 level deep critical section nesting:
Nov 18 12:17:10 tux6127 kernel: [ 2735.532391] ----------------------------------------
Nov 18 12:17:10 tux6127 kernel: [ 2735.532443] .. [<c0100dc6>] .... cpu_idle+0x3f/0x9f
Nov 18 12:17:10 tux6127 kernel: [ 2735.532533] .....[<c01002dd>] ..   ( <= _stext+0x45/0x49)
Nov 18 12:17:10 tux6127 kernel: [ 2735.532623] 
Nov 18 12:17:10 tux6127 kernel: [ 2735.532670]  ----------- IPI show regs -----------
Nov 18 12:17:10 tux6127 kernel: [ 2735.532766] Pid: 0, comm:              swapper
Nov 18 12:17:10 tux6127 kernel: [ 2735.532772] EIP: 0060:[<c02d6299>] CPU: 1
Nov 18 12:17:10 tux6127 kernel: [ 2735.532777] EIP is at acpi_safe_halt+0x26/0x32
Nov 18 12:17:10 tux6127 kernel: [ 2735.532780]  EFLAGS: 00000246    Tainted: P       (2.6.15-rc1-mm2)
Nov 18 12:17:10 tux6127 kernel: [ 2735.532784] EAX: 00000000 EBX: c02d62a5 ECX: 00000001 EDX: c298a000
Nov 18 12:17:10 tux6127 kernel: [ 2735.532787] ESI: f7abec6c EDI: f7abebf8 EBP: c298af5c DS: 007b ES: 007b
Nov 18 12:17:10 tux6127 kernel: [ 2735.532798] CR0: 8005003b CR2: 0804d054 CR3: 31a28000 CR4: 00000640
Nov 18 12:17:10 tux6127 kernel: [ 2735.532801]  [<c0101155>] show_regs+0x146/0x16d
Nov 18 12:17:10 tux6127 kernel: [ 2735.532806]  [<c0112afd>] smp_show_regs+0x3a/0x57
Nov 18 12:17:10 tux6127 kernel: [ 2735.532814]  [<c0110a58>] smp_nmi_callback+0x69/0x7d
Nov 18 12:17:10 tux6127 kernel: [ 2735.532821]  [<c0104cae>] do_nmi+0x4d/0x6b
Nov 18 12:17:10 tux6127 kernel: [ 2735.532827]  [<c0103b0e>] nmi_stack_correct+0x1d/0x22
Nov 18 12:17:10 tux6127 kernel: [ 2735.532832]  [<c02d63c9>] acpi_processor_idle+0x124/0x2c4
Nov 18 12:17:10 tux6127 kernel: [ 2735.532839]  [<c0100e10>] cpu_idle+0x89/0x9f
Nov 18 12:17:10 tux6127 kernel: [ 2735.532848]  [<c0111032>] start_secondary+0x17f/0x2ef
Nov 18 12:17:10 tux6127 kernel: [ 2735.532854]  [<00000000>] 0x0
Nov 18 12:17:10 tux6127 kernel: [ 2735.532865]  [<c298afb4>] 0xc298afb4
Nov 18 12:17:10 tux6127 kernel: [ 2735.532871] ---------------------------
Nov 18 12:17:10 tux6127 kernel: [ 2735.532874] | preempt count: 00020002 ]
Nov 18 12:17:10 tux6127 kernel: [ 2735.532876] | 2 level deep critical section nesting:
Nov 18 12:17:10 tux6127 kernel: [ 2735.532878] ----------------------------------------
Nov 18 12:17:10 tux6127 kernel: [ 2735.532881] .. [<c0100dc6>] .... cpu_idle+0x3f/0x9f
Nov 18 12:17:10 tux6127 kernel: [ 2735.532885] .....[<c0111032>] ..   ( <= start_secondary+0x17f/0x2ef)
Nov 18 12:17:10 tux6127 kernel: [ 2735.532892] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 12:17:10 tux6127 kernel: [ 2735.532904] .....[<c0112ae9>] ..   ( <= smp_show_regs+0x26/0x57)
Nov 18 12:17:10 tux6127 kernel: [ 2735.532909] 
Nov 18 12:17:10 tux6127 kernel: [ 2735.532911]  ----------- IPI show regs -----------
Nov 18 12:17:10 tux6127 kernel: [ 2735.532918] Pid: 0, comm:              swapper
Nov 18 12:17:10 tux6127 kernel: [ 2735.532923] EIP: 0060:[<c02d6299>] CPU: 3
Nov 18 12:17:10 tux6127 kernel: [ 2735.532931] EIP is at acpi_safe_halt+0x26/0x32
Nov 18 12:17:10 tux6127 kernel: [ 2735.532935]  EFLAGS: 00000246    Tainted: P       (2.6.15-rc1-mm2)
Nov 18 12:17:10 tux6127 kernel: [ 2735.532939] EAX: 00000000 EBX: c02d62a5 ECX: 00000001 EDX: c2997000
Nov 18 12:17:10 tux6127 kernel: [ 2735.532943] ESI: f7abfc6c EDI: f7abfbf8 EBP: c2997f5c DS: 007b ES: 007b
Nov 18 12:17:10 tux6127 kernel: [ 2735.532948] CR0: 8005003b CR2: b7ef9310 CR3: 31a28000 CR4: 00000640
Nov 18 12:17:10 tux6127 kernel: [ 2735.532952]  [<c0101155>] show_regs+0x146/0x16d
Nov 18 12:17:10 tux6127 kernel: [ 2735.532959]  [<c0112afd>] smp_show_regs+0x3a/0x57
Nov 18 12:17:10 tux6127 kernel: [ 2735.532967]  [<c0110a58>] smp_nmi_callback+0x69/0x7d
Nov 18 12:17:10 tux6127 kernel: [ 2735.532972]  [<c0104cae>] do_nmi+0x4d/0x6b
Nov 18 12:17:10 tux6127 kernel: [ 2735.532978]  [<c0103b0e>] nmi_stack_correct+0x1d/0x22
Nov 18 12:17:10 tux6127 kernel: [ 2735.532983]  [<c02d63c9>] acpi_processor_idle+0x124/0x2c4
Nov 18 12:17:10 tux6127 kernel: [ 2735.532989]  [<c0100e10>] cpu_idle+0x89/0x9f
Nov 18 12:17:10 tux6127 kernel: [ 2735.532994]  [<c0111032>] start_secondary+0x17f/0x2ef
Nov 18 12:17:10 tux6127 kernel: [ 2735.532999]  [<00000000>] 0x0
Nov 18 12:17:10 tux6127 kernel: [ 2735.533005]  [<c2997fb4>] 0xc2997fb4
Nov 18 12:17:10 tux6127 kernel: [ 2735.533009] ---------------------------
Nov 18 12:17:10 tux6127 kernel: [ 2735.533012] | preempt count: 00020002 ]
Nov 18 12:17:10 tux6127 kernel: [ 2735.533015] | 2 level deep critical section nesting:
Nov 18 12:17:10 tux6127 kernel: [ 2735.533017] ----------------------------------------
Nov 18 12:17:10 tux6127 kernel: [ 2735.533020] .. [<c0100dc6>] .... cpu_idle+0x3f/0x9f
Nov 18 12:17:10 tux6127 kernel: [ 2735.533025] .....[<c0111032>] ..   ( <= start_secondary+0x17f/0x2ef)
Nov 18 12:17:10 tux6127 kernel: [ 2735.533031] .. [<c042b0ef>] .... _spin_lock+0x6c/0x90
Nov 18 12:17:10 tux6127 kernel: [ 2735.533037] .....[<c0112ae9>] ..   ( <= smp_show_regs+0x26/0x57)
Nov 18 12:17:10 tux6127 kernel: [ 2735.533042] 
Nov 18 12:17:10 tux6127 kernel: [ 2735.533045] ----------- IPI show regs -----------<3>Debug: sleeping function called from invalid context at drivers/char/vt.c:2843
Nov 18 12:17:10 tux6127 kernel: [ 2735.533054] 
Nov 18 12:17:10 tux6127 kernel: [ 2735.533056] in_atomic():1, irqs_disabled():1
Nov 18 12:17:10 tux6127 kernel: [ 2735.533060] Pid: 4238, comm:            writetest
Nov 18 12:17:10 tux6127 kernel: [ 2735.533063]  [<c0103df0>] EIP: 0060:[<c02a6841>] CPU: 2
Nov 18 12:17:10 tux6127 kernel: [ 2735.533068] dump_stack+0x1e/0x22
Nov 18 12:17:10 tux6127 kernel: [ 2735.533072] EIP is at _raw_spin_trylock+0xb/0x2d
Nov 18 12:17:10 tux6127 kernel: [ 2735.533076]  [<c011b38e>]  EFLAGS: 00000246    Tainted: P       (2.6.15-rc1-mm2)
Nov 18 12:17:10 tux6127 kernel: [ 2735.533083] EAX: 00000001 EBX: c19589ec ECX: f73ad000 EDX: f1e7f9d0
Nov 18 12:17:10 tux6127 kernel: [ 2735.533087] __might_sleep+0xa7/0xafESI: f73ad000 EDI: b7c00000 EBP: f73adbf4
Nov 18 12:17:10 tux6127 kernel: [ 2735.533095]  DS: 007b ES: 007b
Nov 18 12:17:10 tux6127 kernel: [ 2735.533098]  [<c02f179d>] CR0: 8005003b CR2: b7fb8000 CR3: 31a28000 CR4: 00000640
Nov 18 12:17:10 tux6127 kernel: [ 2735.533104] do_unblank_screen+0x150/0x155 [<c0101155>] 
Nov 18 12:17:10 tux6127 kernel: [ 2735.533109]  [<c02f17b4>] show_regs+0x146/0x16d
Nov 18 12:17:10 tux6127 kernel: [ 2735.533114] unblank_screen+0x12/0x16 [<c0112afd>] 
Nov 18 12:17:10 tux6127 kernel: [ 2735.533119]  [<c0114cb0>] smp_show_regs+0x3a/0x57
Nov 18 12:17:10 tux6127 kernel: [ 2735.533123] bust_spinlocks+0x2c/0x54 [<c0110a58>] 
Nov 18 12:17:10 tux6127 kernel: [ 2735.533129]  [<c0112b13>] smp_nmi_callback+0x69/0x7d
Nov 18 12:17:10 tux6127 kernel: [ 2735.533134] smp_show_regs+0x50/0x57 [<c0104cae>] 
Nov 18 12:17:10 tux6127 kernel: [ 2735.533139]  [<c0110a58>] do_nmi+0x4d/0x6b
Nov 18 12:17:10 tux6127 kernel: [ 2735.533144] smp_nmi_callback+0x69/0x7d [<c0103b0e>] 
Nov 18 12:17:10 tux6127 kernel: [ 2735.533149]  [<c0104cae>] nmi_stack_correct+0x1d/0x22do_nmi+0x4d/0x6b
Nov 18 12:17:10 tux6127 kernel: [ 2735.533154] 
Nov 18 12:17:10 tux6127 kernel: [ 2735.533156]  [<c042b0aa>]  [<c0103b0e>] _spin_lock+0x27/0x90nmi_stack_correct+0x1d/0x22
Nov 18 12:17:10 tux6127 kernel: [ 2735.533163] 
Nov 18 12:17:10 tux6127 kernel: [ 2735.533166]  [<c014bb29>]  [<c02d63c9>] zap_pte_range+0x5b/0x2f7acpi_processor_idle+0x124/0x2c4
Nov 18 12:17:10 tux6127 kernel: [ 2735.533175] 
Nov 18 12:17:10 tux6127 kernel: [ 2735.533178]  [<c014be76>]  [<c0100e10>] unmap_page_range+0xb1/0x117cpu_idle+0x89/0x9f
Nov 18 12:17:10 tux6127 kernel: [ 2735.533186] 
Nov 18 12:17:10 tux6127 kernel: [ 2735.533188]  [<c014bfb8>]  [<c0111032>] unmap_vmas+0xdc/0x221
Nov 18 12:17:10 tux6127 kernel: [ 2735.533193] start_secondary+0x17f/0x2ef [<c014c1a3>] 
Nov 18 12:17:10 tux6127 kernel: [ 2735.533199]  [<00000000>] zap_page_range+0xa6/0x1100x0
Nov 18 12:17:10 tux6127 kernel: [ 2735.533204] 
Nov 18 12:17:10 tux6127 kernel: [ 2735.533207]  [<c014cd62>]  [<c2997fb4>] unmap_mapping_range_vma+0x46/0xbb0xc2997fb4
Nov 18 12:17:10 tux6127 kernel: [ 2735.533215] 
Nov 18 12:17:10 tux6127 kernel: [ 2735.533218]  [<c014cf0a>] ---------------------------
Nov 18 12:17:10 tux6127 kernel: [ 2735.533222] | preempt count: 00020001 ]
Nov 18 12:17:10 tux6127 kernel: [ 2735.533226] unmap_mapping_range+0x133/0x1ed| 1 level deep critical section nesting:
Nov 18 12:17:10 tux6127 kernel: [ 2735.533232] 
Nov 18 12:17:10 tux6127 kernel: [ 2735.533234] ----------------------------------------
Nov 18 12:17:10 tux6127 kernel: [ 2735.533237]  [<c0147dc2>] .. [<c0100dc6>] .... invalidate_inode_pages2_range+0x1f4/0x22acpu_idle+0x3f/0x9f
Nov 18 12:17:10 tux6127 kernel: [ 2735.533245] 
Nov 18 12:17:10 tux6127 kernel: [ 2735.533247] .....[<c0111032>] ..   ( <=  [<c0147e19>] start_secondary+0x17f/0x2ef)
Nov 18 12:17:10 tux6127 kernel: [ 2735.533254] invalidate_inode_pages2+0x21/0x28
Nov 18 12:17:10 tux6127 kernel: [ 2735.533258] 
Nov 18 12:17:10 tux6127 kernel: [ 2735.533260]   [<c01eba54>] nfs_file_direct_write+0x1c3/0x1ff
Nov 18 12:17:10 tux6127 kernel: [ 2735.533271]  [<c01ca23e>] nfs_file_write+0x113/0x115
Nov 18 12:17:10 tux6127 kernel: [ 2735.533276]  [<c015f99d>] do_sync_write+0xa7/0xe7
Nov 18 12:17:10 tux6127 kernel: [ 2735.533282]  [<c015fa83>] vfs_write+0xa6/0x17d
Nov 18 12:17:10 tux6127 kernel: [ 2735.533287]  [<c015fd42>] sys_pwrite64+0x7d/0x81
Nov 18 12:17:10 tux6127 kernel: [ 2735.533291]  [<c0102e6f>] sysenter_past_esp+0x54/0x75
Nov 18 12:17:10 tux6127 kernel: [ 2735.533295] ---------------------------
Nov 18 12:17:10 tux6127 kernel: [ 2735.533297] | preempt count: 00020004 ]
Nov 18 12:17:10 tux6127 kernel: [ 2735.533299] | 4 level deep critical section nesting:
Nov 18 12:17:10 tux6127 kernel: [ 2735.533300] ----------------------------------------
Nov 18 12:17:10 tux6127 kernel: [ 2735.533303] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 12:17:10 tux6127 kernel: [ 2735.533307] .....[<c014ce6e>] ..   ( <= unmap_mapping_range+0x97/0x1ed)
Nov 18 12:17:10 tux6127 kernel: [ 2735.533311] .. [<c014c127>] .... zap_page_range+0x2a/0x110
Nov 18 12:17:10 tux6127 kernel: [ 2735.533315] .....[<c014cd62>] ..   ( <= unmap_mapping_range_vma+0x46/0xbb)
Nov 18 12:17:10 tux6127 kernel: [ 2735.533319] .. [<c042b09b>] .... _spin_lock+0x18/0x90
Nov 18 12:17:10 tux6127 kernel: [ 2735.533322] .....[<c014bb29>] ..   ( <= zap_pte_range+0x5b/0x2f7)
Nov 18 12:17:10 tux6127 kernel: [ 2735.533326] .. [<c042b0ef>] .... _spin_lock+0x6c/0x90
Nov 18 12:17:10 tux6127 kernel: [ 2735.533330] .....[<c0112ae9>] ..   ( <= smp_show_regs+0x26/0x57)
Nov 18 12:17:10 tux6127 kernel: [ 2735.533334] 

Re: mmap over nfs leads to excessive system load

[Kenny Simpson]

Yet another data point:

Under 2.6.8-2 (debain sarge kernel), the test does not cause a spin.
Instead, the file extension via pwrite does not allow the new pages to be usable by
remap_file_pages.
However, munmap/mmap are happy to use pages intoduces by the pwrite...
and happily writes more than 4GB.

-Kenny



	
		
__________________________________ 
Yahoo! Mail - PC Magazine Editors' Choice 2005 
http://mail.yahoo.com

Re: mmap over nfs leads to excessive system load

[Kenny Simpson]

--- Andrew Morton <akpm@osdl.org> wrote:
> Could you send the test app please?  (Apologies if you've already done so
> and I missed it).

Here is an strace from a reduced test:
open("/mnt/bar", O_RDWR|O_CREAT|O_DIRECT|O_LARGEFILE, 0644) = 3
pwrite(3, "\0", 1, 4292870144)          = 1
mmap2(NULL, 2097152, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0xffc00) = 0xb7ace000
pwrite(3, "\0", 1, 4294967296)          = 1
munmap(0xb7ace000, 2097152)             = 0
mmap2(0xb7ace000, 2097152, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_FIXED, 3, 0xffe00) = 0xb7ace000
pwrite(3, "\0", 1, 4297064448)          = 1
munmap(0xb7ace000, 2097152)             = 0
mmap2(0xb7ace000, 2097152, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_FIXED, 3, 0x100000) = 0xb7ace000
pwrite(3, "\0", 1, 4299161600)          = 1
munmap(0xb7ace000, 2097152)             = 0
mmap2(0xb7ace000, 2097152, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_FIXED, 3, 0x100200) = 0xb7ace000
pwrite(3, "\0", 1, 4301258752


The final pwrite never returns.
This is with 2.6.15-rc1 + Trond's NFS patch (-mm2 would die on strace).

The only change from the previous test is that this one uses munmap/mmap64 instead of
remap_file_pages.

showPc shows:

SysRq : Show Regs

Pid: 4271, comm:            writetest
EIP: 0060:[<c029cd2b>] CPU: 0
EIP is at prio_tree_first+0x26/0xb8
 EFLAGS: 00000292    Tainted: P       (2.6.15-rc1)
EAX: 00000000 EBX: f6765d94 ECX: f6765d94 EDX: 00000200
ESI: 00000000 EDI: f5da475c EBP: f6765db4 DS: 007b ES: 007b
CR0: 8005003b CR2: b7590000 CR3: 36b5d000 CR4: 000006d0
 [<c029ce5e>] prio_tree_next+0xa1/0xa3
 [<c014822a>] vma_prio_tree_next+0x27/0x51
 [<c014b1dc>] unmap_mapping_range+0x18b/0x210
 [<c0120d06>] __do_softirq+0x6a/0xd1
 [<c0103a24>] apic_timer_interrupt+0x1c/0x24
 [<c0146577>] invalidate_inode_pages2_range+0x215/0x24c
 [<c01465cd>] invalidate_inode_pages2+0x1f/0x26
 [<c01e4031>] nfs_file_direct_write+0x1e1/0x21a
 [<c0159ea4>] do_sync_write+0xc7/0x10d
 [<c012ff32>] autoremove_wake_function+0x0/0x57
 [<c0159f92>] vfs_write+0xa8/0x177
 [<c015a275>] sys_pwrite64+0x88/0x8c
 [<c0102f51>] syscall_call+0x7/0xb


-Kenny



		
__________________________________ 
Yahoo! FareChase: Search multiple travel sites in one click.
http://farechase.yahoo.com

infinite loop? with mmap, nfs, pwrite, O_DIRECT

[Kenny Simpson]

I have a smaller test case (4 system calls, and a memset), that causes the test case to hang in an
unkillable state*, and makes the system load consume an entire CPU.

*the process is killable if run under strace, but the system load does not drop when the strace is
killed.

Pass this the name of a target file on an NFS mount.

(tested to fail on 2.6.15-rc1).

-Kenny


Here is the test:

#define _GNU_SOURCE

#include <fcntl.h>
#include <stdio.h>
#include <strind.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <unistd.h>

int main(int argc, char* argv[])
{
  if (argc != 2) {
    printf("usage: %0 <filename>\n", argv[0]);
    return 0;
  }

  {
    int fd = open(argv[1], O_RDWR | O_CREAT | O_LARGEFILE | O_DIRECT, 0644);
    if (fd < 0) {
      perror("open");
      return 0;
    }

    int window_size = 2 * 1024 * 1024;
    long long file_size = window_size;

    /* fast-forward */
    file_size += 2047u * (2 * 1024 * 1024);
    file_size += window_size + window_size;

    /* grow file */
    pwrite64(fd, "", 1, file_size);

    {
      char* mapping_start = (char*)mmap64(0, window_size,
                                          PROT_READ | PROT_WRITE,
                                          MAP_SHARED,
                                          fd, file_size - window_size);

      /* test only fails with this: */
      memset(mapping_start, 0, window_size);
    }

    /* grow file */
    file_size += window_size;

    /* this never returns */
    pwrite64(fd, "", 1, file_size);
  }
  return 0;
}



		
__________________________________ 
Yahoo! FareChase: Search multiple travel sites in one click.
http://farechase.yahoo.com

Re: infinite loop? with mmap, nfs, pwrite, O_DIRECT

[Chuck Lever]

[-- Attachment #1: Type: text/plain, Size: 560 bytes --]

Kenny Simpson wrote:
> I have a smaller test case (4 system calls, and a memset), that causes the test case to hang in an
> unkillable state*, and makes the system load consume an entire CPU.
> 
> *the process is killable if run under strace, but the system load does not drop when the strace is
> killed.
> 
> Pass this the name of a target file on an NFS mount.
> 
> (tested to fail on 2.6.15-rc1).

kenny-

i'm assuming that because you copied trond, this is only reproducible on 
NFS.  have you tried this test on other local and remote file system types?

[-- Attachment #2: cel.vcf --]
[-- Type: text/x-vcard, Size: 439 bytes --]

begin:vcard
fn:Chuck Lever
n:Lever;Charles
org:Network Appliance, Incorporated;Linux NFS Client Development
adr:535 West William Street, Suite 3100;;Center for Information Technology Integration;Ann Arbor;MI;48103-4943;USA
email;internet:cel@citi.umich.edu
title:Member of Technical Staff
tel;work:+1 734 763 4415
tel;fax:+1 734 763 4434
tel;home:+1 734 668 1089
x-mozilla-html:FALSE
url:http://www.monkey.org/~cel/
version:2.1
end:vcard

Re: infinite loop? with mmap, nfs, pwrite, O_DIRECT

[Kenny Simpson]

--- Chuck Lever <cel@citi.umich.edu> wrote:
> kenny-
> 
> i'm assuming that because you copied trond, this is only reproducible on 
> NFS.  have you tried this test on other local and remote file system types?

Yes, this only applies to NFS.
ext3 doesn't let you use pwrite with O_DIRECT, nor does NFS from 2.6.8.
These are the only 2 filesystem types to which I have access.

For ext3, using ftruncate works just fine for extending the file, but on NFS, ftruncate causes the
non-existent pages to be read in.

-Kenny



	
		
__________________________________ 
Yahoo! Mail - PC Magazine Editors' Choice 2005 
http://mail.yahoo.com

Re: infinite loop? with mmap, nfs, pwrite, O_DIRECT

[Kenny Simpson]

> I have a smaller test case (4 system calls, and a memset), that causes the test case to hang in
an
> unkillable state*, and makes the system load consume an entire CPU.

Problem still exists in -rc2, but OProfile shows slightly different results:
samples  %        symbol name
2919823  86.8716  unmap_mapping_range
163379    4.8609  _raw_spin_trylock
36453     1.0846  prio_tree_first

-Kenny



		
__________________________________ 
Start your day with Yahoo! - Make it your home page! 
http://www.yahoo.com/r/hs

Re: infinite loop? with mmap, nfs, pwrite, O_DIRECT

[Andrew Morton]

Kenny Simpson <theonetruekenny@yahoo.com> wrote:
>
> ext3 doesn't let you use pwrite with O_DIRECT

ext3 does permit that.  See odwrite.c from
http://www.zip.com.au/~akpm/linux/patches/stuff/ext3-tools.tar.gz

Re: infinite loop? with mmap, nfs, pwrite, O_DIRECT

[Kenny Simpson]

With gentle beating by a clue-stick from Andrew.. I can run the same test on ext3...

ext3 is happy...

open("/data/foo", O_RDWR|O_CREAT|O_TRUNC|O_DIRECT|O_LARGEFILE, 0666) = 3
pwrite(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096, 4299161600) = 4096
mmap2(NULL, 2097152, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0x100200) = 0xb7c7f000
pwrite(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096, 4301258752) = 4096
exit_group(0)                           = ?


but NFS is still unhappy....

open("/mnt/bar", O_RDWR|O_CREAT|O_TRUNC|O_DIRECT|O_LARGEFILE, 0666) = 3
pwrite(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096, 4299161600) = 4096
mmap2(NULL, 2097152, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0x100200) = 0xb7bc2000
pwrite(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096, 4301258752  <never
returns>


-Kenny



	
		
__________________________________ 
Yahoo! Mail - PC Magazine Editors' Choice 2005 
http://mail.yahoo.com

Re: infinite loop? with mmap, nfs, pwrite, O_DIRECT

[Kenny Simpson]

Another data point:  In 2.6.8, the test works fine (just like on ext3).
Any suggestions as to where to start poking, or shall I just do a binary search?

-Kenny



	
		
__________________________________ 
Yahoo! Mail - PC Magazine Editors' Choice 2005 
http://mail.yahoo.com

Re: infinite loop? with mmap, nfs, pwrite, O_DIRECT

[Trond Myklebust]

On Mon, 2005-11-21 at 13:39 -0800, Kenny Simpson wrote:
> With gentle beating by a clue-stick from Andrew.. I can run the same test on ext3...
> 
> ext3 is happy...
> 
> open("/data/foo", O_RDWR|O_CREAT|O_TRUNC|O_DIRECT|O_LARGEFILE, 0666) = 3
> pwrite(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096, 4299161600) = 4096
> mmap2(NULL, 2097152, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0x100200) = 0xb7c7f000
> pwrite(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096, 4301258752) = 4096
> exit_group(0)                           = ?
> 
> 
> but NFS is still unhappy....
> 
> open("/mnt/bar", O_RDWR|O_CREAT|O_TRUNC|O_DIRECT|O_LARGEFILE, 0666) = 3
> pwrite(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096, 4299161600) = 4096
> mmap2(NULL, 2097152, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0x100200) = 0xb7bc2000
> pwrite(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096, 4301258752  <never
> returns>

Ah... It is the pwrite() _after_ the call to mmap() that fails....

OK, does the following patch fix it?

Cheers,
  Trond
-------------
NFS: O_DIRECT cannot call invalidate_inode_pages2().

 Anything that calls lock_page() should be avoided in O_DIRECT, however
 we should be able to call invalidate_inode_pages() since that doesn't
 wait on the page lock.

 Signed-off-by: Trond Myklebust <Trond.Myklebust@netapp.com>
---

 fs/nfs/direct.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/fs/nfs/direct.c b/fs/nfs/direct.c
index a2d2814..ef299f8 100644
--- a/fs/nfs/direct.c
+++ b/fs/nfs/direct.c
@@ -774,7 +774,7 @@ nfs_file_direct_write(struct kiocb *iocb
 
 	retval = nfs_direct_write(inode, ctx, &iov, pos, 1);
 	if (mapping->nrpages)
-		invalidate_inode_pages2(mapping);
+		invalidate_inode_pages(mapping);
 	if (retval > 0)
 		*ppos = pos + retval;
 

Re: infinite loop? with mmap, nfs, pwrite, O_DIRECT

[Kenny Simpson]

--- Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> Ah... It is the pwrite() _after_ the call to mmap() that fails....
> 
> OK, does the following patch fix it?

YES!

open("/mnt/bar", O_RDWR|O_CREAT|O_TRUNC|O_DIRECT|O_LARGEFILE, 0666) = 3
pwrite(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096, 4299161600) = 4096
mmap2(NULL, 2097152, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0x100200) = 0xb7c26000
pwrite(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096, 4301258752) = 4096
exit_group(0)                           = ?

I'll re-run my original test(s) tomorrow.
Thanks again!

-Kenny



		
__________________________________ 
Yahoo! FareChase: Search multiple travel sites in one click.
http://farechase.yahoo.com

Re: infinite loop? with mmap, nfs, pwrite, O_DIRECT

[Andrew Morton]

Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
>
>  Anything that calls lock_page() should be avoided in O_DIRECT,

Why?

And it's still doing lock_page():

	nfs_file_direct_write()
	->filemap_fdatawrite()
	  ->do_writepages()
	    ->nfs_writepages()
	      ->generic_writepages()
	        ->mpage_writepages()
	          ->lock_page()

> however
>   we should be able to call invalidate_inode_pages() since that doesn't
>   wait on the page lock.

invalidate_inode_pages2() is better.  And using generic_file_direct_IO() is
better still, since it handles mmap coherency and only work upon that part
of the file which is actually undergoing IO.

Re: infinite loop? with mmap, nfs, pwrite, O_DIRECT

[Trond Myklebust]

On Mon, 2005-11-21 at 15:34 -0800, Andrew Morton wrote:
> Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> >
> >  Anything that calls lock_page() should be avoided in O_DIRECT,
> 
> Why?
> 
> And it's still doing lock_page():
> 
> 	nfs_file_direct_write()
> 	->filemap_fdatawrite()
> 	  ->do_writepages()
> 	    ->nfs_writepages()
> 	      ->generic_writepages()
> 	        ->mpage_writepages()
> 	          ->lock_page()

True.

> > however
> >   we should be able to call invalidate_inode_pages() since that doesn't
> >   wait on the page lock.
> 
> invalidate_inode_pages2() is better.


> And using generic_file_direct_IO() is
> better still, since it handles mmap coherency and only work upon that part
> of the file which is actually undergoing IO.

Unlike local filesystems, we don't want to have to take the i_sem in any
of the direct IO paths. The latter is just a liability as far as
applications are concerned: it doesn't offer any protection for local
data (there _is_ no local data to protect), but gets seriously in the
way of write parallelism.

The only difference I can see between the two paths is the call to
unmap_mapping_range(). What effect would that have?

Cheers,
  Trond

Re: infinite loop? with mmap, nfs, pwrite, O_DIRECT

[Andrew Morton]

Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
>
> The only difference I can see between the two paths is the call to
>  unmap_mapping_range(). What effect would that have?

It shoots down any mapped pagecache over the affected file region.  Because
the direct-io write is about to make that pagecache out-of-date.  If the
application tries to use that data again it'll get a major fault and will
re-read the file contents.

Re: infinite loop? with mmap, nfs, pwrite, O_DIRECT

[Trond Myklebust]

On Mon, 2005-11-21 at 16:09 -0800, Andrew Morton wrote:
> Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> >
> > The only difference I can see between the two paths is the call to
> >  unmap_mapping_range(). What effect would that have?
> 
> It shoots down any mapped pagecache over the affected file region.  Because
> the direct-io write is about to make that pagecache out-of-date.  If the
> application tries to use that data again it'll get a major fault and will
> re-read the file contents.

I assume then, that this couldn't be the cause of the
invalidate_inode_pages() failing to complete? Unless there is some
method to prevent applications from faulting in the page while we're
inside generic_file_direct_IO(), then the same race would be able to
occur there.

Cheers,
  Trond

Re: infinite loop? with mmap, nfs, pwrite, O_DIRECT

[Trond Myklebust]

On Mon, 2005-11-21 at 19:18 -0500, Trond Myklebust wrote:
> On Mon, 2005-11-21 at 16:09 -0800, Andrew Morton wrote:
> > Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> > >
> > > The only difference I can see between the two paths is the call to
> > >  unmap_mapping_range(). What effect would that have?
> > 
> > It shoots down any mapped pagecache over the affected file region.  Because
> > the direct-io write is about to make that pagecache out-of-date.  If the
> > application tries to use that data again it'll get a major fault and will
> > re-read the file contents.
> 
> I assume then, that this couldn't be the cause of the
> invalidate_inode_pages() failing to complete? Unless there is some
  ^^^^^^^^^^^^^^^^^^^^^^^^ invalidate_inode_pages2(), sorry....

> method to prevent applications from faulting in the page while we're
> inside generic_file_direct_IO(), then the same race would be able to
> occur there.

Cheers,
  Trond

Re: infinite loop? with mmap, nfs, pwrite, O_DIRECT

[Andrew Morton]

Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
>
> On Mon, 2005-11-21 at 16:09 -0800, Andrew Morton wrote:
> > Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> > >
> > > The only difference I can see between the two paths is the call to
> > >  unmap_mapping_range(). What effect would that have?
> > 
> > It shoots down any mapped pagecache over the affected file region.  Because
> > the direct-io write is about to make that pagecache out-of-date.  If the
> > application tries to use that data again it'll get a major fault and will
> > re-read the file contents.
> 
> I assume then, that this couldn't be the cause of the
> invalidate_inode_pages() failing to complete?

It sounds unlikely.  This hang is associated with crossing the 2G boundary
isn't it?

I don't think we've seen a sysrq-T trace from the hang?

> Unless there is some
> method to prevent applications from faulting in the page while we're
> inside generic_file_direct_IO(), then the same race would be able to
> occur there.

Yes, there are still windows.

Another thing the unmap_mapping_range() does is to push pte-dirty bits into
the software-dirty flags, so the modified data does get written.  If we
didn't do this, a page which was dirtied via mmap before the direct-io
write would get written back _after_ the direct-io write, arguably causing
corruption.

Re: infinite loop? with mmap, nfs, pwrite, O_DIRECT

[Trond Myklebust]

On Mon, 2005-11-21 at 16:28 -0800, Andrew Morton wrote:
> Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> >
> > On Mon, 2005-11-21 at 16:09 -0800, Andrew Morton wrote:
> > > Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> > > >
> > > > The only difference I can see between the two paths is the call to
> > > >  unmap_mapping_range(). What effect would that have?
> > > 
> > > It shoots down any mapped pagecache over the affected file region.  Because
> > > the direct-io write is about to make that pagecache out-of-date.  If the
> > > application tries to use that data again it'll get a major fault and will
> > > re-read the file contents.
> > 
> > I assume then, that this couldn't be the cause of the
> > invalidate_inode_pages() failing to complete?
> 
> It sounds unlikely.  This hang is associated with crossing the 2G boundary
> isn't it?
> 
> I don't think we've seen a sysrq-T trace from the hang?

Kenny sent us this trace. The thing appears to be hanging in
unmap_mapping_range() as called by invalidate_inode_pages2()

Pid: 4271, comm:            writetest
EIP: 0060:[<c029cd2b>] CPU: 0
EIP is at prio_tree_first+0x26/0xb8
 EFLAGS: 00000292    Tainted: P       (2.6.15-rc1)
EAX: 00000000 EBX: f6765d94 ECX: f6765d94 EDX: 00000200
ESI: 00000000 EDI: f5da475c EBP: f6765db4 DS: 007b ES: 007b
CR0: 8005003b CR2: b7590000 CR3: 36b5d000 CR4: 000006d0
 [<c029ce5e>] prio_tree_next+0xa1/0xa3
 [<c014822a>] vma_prio_tree_next+0x27/0x51
 [<c014b1dc>] unmap_mapping_range+0x18b/0x210
 [<c0120d06>] __do_softirq+0x6a/0xd1
 [<c0103a24>] apic_timer_interrupt+0x1c/0x24
 [<c0146577>] invalidate_inode_pages2_range+0x215/0x24c
 [<c01465cd>] invalidate_inode_pages2+0x1f/0x26
 [<c01e4031>] nfs_file_direct_write+0x1e1/0x21a
 [<c0159ea4>] do_sync_write+0xc7/0x10d
 [<c012ff32>] autoremove_wake_function+0x0/0x57
 [<c0159f92>] vfs_write+0xa8/0x177
 [<c015a275>] sys_pwrite64+0x88/0x8c
 [<c0102f51>] syscall_call+0x7/0xb

> > Unless there is some
> > method to prevent applications from faulting in the page while we're
> > inside generic_file_direct_IO(), then the same race would be able to
> > occur there.
> 
> Yes, there are still windows.
> 
> Another thing the unmap_mapping_range() does is to push pte-dirty bits into
> the software-dirty flags, so the modified data does get written.  If we
> didn't do this, a page which was dirtied via mmap before the direct-io
> write would get written back _after_ the direct-io write, arguably causing
> corruption.

As far as we're concerned, anybody using direct-io is responsible for
enforcing their own ordering. The pagecache is one thing, but in NFS,
direct IO is mainly used in situations where several clients are writing
to the same file. There is no way to ensure fully safe mmap() semantics.

Cheers,
  Trond

nfs unhappiness with memory pressure

[Kenny Simpson]

[-- Attachment #1: Type: text/plain, Size: 3042 bytes --]

Hi again... I'm still doing nfs tests.

With 2.6.15-rc3-mm1, a simple program can bring the system to a halt (as it can with previous
kernels).

I ran the test in single user mode, and copied the following output from sysrq-m, sysrq-t by
hand...

sysrq-t:
writetest:
  io_schedule
  sync_page
  __wait_on_bit_lock
  __lock_page
  filemap_nopage
  do_no_page
  __handle_mm_fault
  error_code

rpciod/0:
  io_schedule_timeout
  blk_congestion_wait
  throttle_vm_writeout
  shrink_zone
  shrink_caches
  try_to_free_pages
  __alloc_pages
  tcp_sendmsg
  inet_sendmsg
  sock_sendmsg
  kernel_sendmsg
  sock_no_sendpage
  xs_tcp_send_request
  xprt_transmit
  call_transmit
  __rpc_execute
  rpc_async_schedule
  worker_thread
  kthread
  kernel_thread_helper

sysrq-m:
Mem-Info:
  DMA per-cpu:
  cpu 0  hot: high 12  batch 2  used 0
  cpu 0 cold:       4        1       0
  cpu 1  hot:      12        2       1
  cpu 1 cold:       4        1       0
  cpu 2  hot:      12        2       0
  cpu 2 cold:       4        1       0
  cpu 3  hot:      12        2       0
  cpu 3 cold:       4        1       0

  DMA32 per-cpu: empty

  Normal per-cpu:
  cpu 0  hot: high 384  batch 64  used  1
  cpu 0 cold:      128        32        0
  cpu 1  hot:      384        64       97
  cpu 1 cold:      128        32        0
  cpu 2  hot:      384        64       63
  cpu 2 cold:      128        32       32
  cpu 3  hot:      384        64       47
  cpu 3 cold:      128        32        0

  Highmem per-cpu:
  cpu 0:  hot:  high 384  batch 64  used 0
  cpu 0: cold:       128        32       0
  cpu 1:  hot:  high 384  batch 64  used 0
  cpu 1: cold:       128        32       0
  cpu 2:  hot:  high 384  batch 64  used 0
  cpu 2: cold:       128        32       0
  cpu 3:  hot:  high 384  batch 64  used 0
  cpu 3: cold:       128        32       0

  free pages: 14088kB (6000kB HighMem)
  Active: 453253  inactive: 43719  dirty: 149725  writeback: 310580
  unstable: 0  free: 3502  slab: 14870  mapped: 1230

  524149 pages of RAM
  294773 pages of HIGHMEM
  6137 reserved pages
  311956 pages shared
  0 pages swap cache
  149725 pages dirty
  310580 pages writeback
  1230 pages mapped
  14870 pages slab
  16 pages pagetables

This is the same test program as before.
It simply opens a file O_RDWR | O_CREAT | O_TRUNC | O_LARGEFILE,
  grows the file by doing a pwrite64 of 1 byte,
  maps the end of the file with mmap64(PROT_READ | PROT_WRITE, MAP_SHARED)
  touches all the bytes by doing a memset
  grows the file some more
  unmaps, maps the new region, touches memory, ....

Once all the free memory on the system is used, no new processes can start, and the system is
effectively hung.  Only sysrq and vt switching function (unless running X).

Any further info I could provide?  Any ideas?  Patches to try out?

thanks,
-Kenny

Here is the test program again (run as writetest -m <file-on-nfs>)


		
__________________________________ 
Yahoo! Music Unlimited 
Access over 1 million songs. Try it free. 
http://music.yahoo.com/unlimited/

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 3360621278-writetest.cpp --]
[-- Type: text/x-c++src; name="writetest.cpp", Size: 3822 bytes --]

// test the write throughput of a sliding mmap window vs simple FILE*

#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <signal.h>
#include <string.h>
#include <unistd.h>
#include <time.h>
#include <errno.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <sys/time.h>

int total_bytes = 0;
int last_bytes = 0;

struct timeval last_time;

void timeout(int)
{
  struct timeval now;
  gettimeofday(&now, 0);

  double time_diff = (now.tv_sec - last_time.tv_sec) + ((now.tv_usec - last_time.tv_usec) / 1000000.);
  last_time = now;

  int new_bytes = total_bytes;
  int diff = new_bytes - last_bytes;
  printf("wrote %dk %dM bytes in %f seconds -> %fM/sec\n", diff / 1024, diff / (1024 * 1024), time_diff, (diff / (time_diff * 1000000.)));
  last_bytes = new_bytes;
}


// tests O_DIRECT + pwrite = fail
//       O_DIRECT + truncate = happy
//                  truncate = happy

//                  pwrite = happy
//       O_DIRECT + truncate = happy
//                  truncate = happy
//       O_DIRECT + pwrite = fail

void do_mapwrite(int fd)
{
  // lets write as fast as we can....
  int const window_size = 2 * 1024 * 1024; // 2 * 1024 * 1024; // 16k
  int const window_pages = window_size / 4096;

  int file_page_offset = 0;
  long long file_size = window_size;


  // fast-forward... by 2046 windows
  file_size += 2047u * (2 * 1024 * 1024);
  file_page_offset += 2047;


  //ftruncate64(fd, file_size);
  printf("pwrite to %llx  %llu\n", file_size, file_size);
  pwrite64(fd, "", 1, file_size);

  char* mapping_start = static_cast<char*>(mmap64(0, window_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, file_size - window_size));


  // scribble into buffer and walk the window
  for (;;) {
    memset(mapping_start, 0, window_size);

    // grow file
    file_size += window_size;

    //ftruncate64(fd, file_size);
    pwrite64(fd, "", 1, file_size);

    file_page_offset += window_pages;


#if 1
    munmap(mapping_start, window_size);

    mapping_start = static_cast<char*>(mmap64(mapping_start, window_size, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_FIXED, fd, file_size - window_size));
    if (mapping_start == (char*)MAP_FAILED) {
      perror("mmap");
      return;
    }
#else

    while (remap_file_pages(mapping_start, window_size, 0, file_page_offset, MAP_SHARED /*| MAP_NONBLOCK*/) <0)
      perror("remap_file_pages");
#endif
    //ftruncate64(fd, file_size);

    total_bytes += window_size;
  }
}



void do_filewrite(int fd)
{
  FILE* ptr = fdopen(fd, "w+");

  int line_len = 80;
  char* buf = (char*)malloc(line_len);
  memset(buf, 0, line_len);

  for (;;) {
    fwrite(buf, line_len, 1, ptr);
    total_bytes += line_len;
  }
}

void do_syswrite(int fd)
{
  unsigned int line_len = 64 * 1024;
  char* buf = (char*)(((long)malloc(line_len * 2) + (line_len - 1)) & -line_len);
  memset(buf, 0, line_len);

  for (;;) {
    write(fd, buf, line_len);
    total_bytes += line_len;
  }
}


int main(int argc, char* argv[])
{
  if (argc != 3) {
    printf("usage: %s -[mfw] <filename>\n", argv[0]);
    return 0;
  }

  if ((argv[1][0] != '-') || 
      ((argv[1][1] != 'm') &&
       (argv[1][1] != 'w') &&
       (argv[1][1] != 'f'))) {
    printf("usage: %s -[mfw] <filename>\n", argv[0]);
    return 0;
  }

  int fd = open(argv[2], O_RDWR | O_CREAT | O_TRUNC | O_LARGEFILE /*| O_DIRECT*/, 0644);
  if (fd < 0) {
    perror("open");
    return 0;
  }

  // start the clock...
  signal(SIGALRM, timeout);
  {
    struct itimerval itv;
    itv.it_interval.tv_sec = 1;
    itv.it_interval.tv_usec = 0;
    itv.it_value = itv.it_interval;

    setitimer(ITIMER_REAL, &itv, 0);
  }

  switch (argv[1][1]) {
  case 'm': do_mapwrite(fd); break;
  case 'f': do_filewrite(fd); break;
  case 'w': do_syswrite(fd); break;
  }

  return 0;
}

Re: nfs unhappiness with memory pressure

[Keith Mannthey]

You are still reporing free pages.  Do you seen the OOM killer killing
processes?

How big is the file you are doing your test on?  How big is your
filesize var when the box hangs?

If you run this test without nfs (on a local file system) do you end
up in this low memory state as well or only over a nfs mount?

thanks,
 Keith

Re: nfs unhappiness with memory pressure

[Kenny Simpson]

--- Keith Mannthey wrote:
> You are still reporing free pages.  Do you seen the OOM killer killing
> processes?

I did not see anything being killed from the run under single user mode.
However, the sysrq does scroll past, so I only see the final 60 lines.
Is there a fool-proof way to check?

> 
> How big is the file you are doing your test on?  How big is your
> filesize var when the box hangs?

The File starts empty (the program creates it).  The box hangs when the file is 5.9GB
(6308233217), or at least this is the file size when the box comes back.

> 
> If you run this test without nfs (on a local file system) do you end
> up in this low memory state as well or only over a nfs mount?

I only see problems when running on nfs.

Other details:
nfs options:
rw,v3,rsize=32768,wsize=32768,hard,intr,lock,proto=tcp,addr=x.x.x.x

This is going over a dedicated Gb x-over cable to a clustered NetApp.

-Kenny



	
		
__________________________________ 
Yahoo! Mail - PC Magazine Editors' Choice 2005 
http://mail.yahoo.com

Re: nfs unhappiness with memory pressure

[Kenny Simpson]

--- Keith Mannthey <kmannth@gmail.com> wrote:
> You are still reporing free pages.  Do you seen the OOM killer killing
> processes?

Running the test with /proc/sys/vm/overcommit_memory = 2, I get a similar 
result.  It still hangs after about 5.9GB, but it starts trying to write 
out the file sooner.
Here is the stack trace I have for the process (again, by hand, what didn't scroll by as nothing
makes it to logs...)

writetest:
  schedule_timeout
  io_schedule_timeout
  blk_congestion_wait
  throttle_vm_writeout
  shrink_zone
  shrink_caches
  try_to_free_pages
  __alloc_pages
    -> (up to here it matches the previous run's stack from rpciod/0)
  kmem_getpages
  cache_grow
  cache_alloc_refill
  kmem_cache_alloc
  mempool_alloc_slab
  mempool_alloc
  nfs_flush_one
  nfs_flush_list
  nfs_flush_inode
  nfs_write_pages
  do_writepages
  __filemap_fdatawrite_range
  filemap_fdatawrite
  filemap_write_and_wait
  nfs_revalidate_mapping
  nfs_file_write
  do_sync_write
  vfs_write
  sys_pwrite64

The memory dump showed there was memory still available, with no swap in use.

-Kenny



	
		
__________________________________ 
Yahoo! Mail - PC Magazine Editors' Choice 2005 
http://mail.yahoo.com

Re: nfs unhappiness with memory pressure

[Kenny Simpson]

Tested with rc5 - same results.  It was suggested that I run slabtop when the system freezed, so
here is that info: (again, by hand, I'm getting another machine to use either netconsole or a
serial cable).

 Active / Total Objects (% used)    : 478764 / 485383 (98.6%)
 Active / Total Slabs (% used)      : 14618 / 14635 (99.9%)
 Active / Total Caches (% used)     : 79 / 138 (57.2%)
 Active / Total Size (% used)       : 56663.79K / 57566.41K (98.4%)
 Minimum / Average / Maximum Object : 0.01K / 0.12K / 128.00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME                   
403088 403088 100%    0.06K   6832       59     27328K nfs_page
 30380  30380 100%    0.50K   4340        7     17360K nfs_write_data
 15134  15134 100%    0.27K   1081       14      4324K radix_tree_node
...


The other thing is that the stack trace showsd slabtop as being halted in throttle_vm_writeout
while allocating memory, and the writetest was halted waiting to allocate memory.

I'll get more detailed stack traces once I get the second machine set up.

-Kenny



		
__________________________________ 
Start your day with Yahoo! - Make it your home page! 
http://www.yahoo.com/r/hs

Re: nfs unhappiness with memory pressure

[Kenny Simpson]

[-- Attachment #1: Type: text/plain, Size: 548 bytes --]

> Tested with rc5 - same results.
Again, this time with serial console help:
The resulting file only grew to 1.9G (1971322881) bytes, so I don't think this is a 32-bit issue.

I'm attaching the result of sysrq-p, t, m
This was a run with writetest and slabtop.

Please let me know if anyone else is able to reproduce this behavior, or if there is some other
information I can/should be providing.

thanks,
-Kenny


		
__________________________________________ 
Yahoo! DSL – Something to write home about. 
Just $16.99/mo. or less. 
dsl.yahoo.com 

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 3165733884-sysrq.log --]
[-- Type: text/x-log; name="sysrq.log", Size: 58092 bytes --]

[  120.076119] SysRq : Show Regs
[  120.079286] 
[  120.080871] Pid: 0, comm:              swapper
[  120.085527] EIP: 0060:[<c0100d4f>] CPU: 2
[  120.089736] EIP is at default_idle+0x36/0x56
[  120.094216]  EFLAGS: 00000246    Not tainted  (2.6.15-rc5+nfs+k2)
[  120.100590] EAX: 00000000 EBX: c2336000 ECX: c221e2e0 EDX: c2336000
[  120.107148] ESI: c053c380 EDI: c053c300 EBP: c2336f84 DS: 007b ES: 007b
[  120.114124] CR0: 8005003b CR2: b7ac4000 CR3: 00540000 CR4: 000006d0
[  122.975684] SysRq : Show State
[  122.978947] 
[  122.978948]                                                sibling
[  122.987009]   task             PC      pid father child younger older
[  122.993755] init          D 00000001     0     1      0     2               (NOTLB)
[  123.002080] c2331ce8 c2331cd8 00000004 00000001 c019af87 c2331c9c c015c8ad f7b5d214 
[  123.010304]        00000000 c2330550 00016380 00000000 00000202 c2217fe0 c2331cfc 00000202 
[  123.019433]        c2217fe0 c2331cfc c2217560 00000001 0000018a a263b5ce 0000001c c2330550 
[  123.028567] Call Trace:
[  123.031381]  [<c04081f1>] schedule_timeout+0x54/0xa5
[  123.036682]  [<c040817e>] io_schedule_timeout+0x29/0x33
[  123.042249]  [<c0287ce8>] blk_congestion_wait+0x70/0x85
[  123.047817]  [<c0141365>] throttle_vm_writeout+0x63/0x6d
[  123.053477]  [<c014712d>] shrink_zone+0xe0/0xfa
[  123.058315]  [<c01471b4>] shrink_caches+0x6d/0x6f
[  123.063333]  [<c0147286>] try_to_free_pages+0xd0/0x1b5
[  123.068806]  [<c013fa4b>] __alloc_pages+0x135/0x2e8
[  123.074008]  [<c013fc2a>] __get_free_pages+0x2c/0x4f
[  123.079302]  [<c016c541>] __pollwait+0x3a/0xae
[  123.084053]  [<c0165ff6>] pipe_poll+0x36/0xad
[  123.088714]  [<c016c8b1>] do_select+0x217/0x26e
[  123.093557]  [<c016cba9>] sys_select+0x26a/0x3f7
[  123.098487]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  123.103874] migration/0   S 00000002     0     2      1             3       (L-TLB)
[  123.112195] c2339f94 c2339f84 00000004 00000002 00000082 00000001 c2339f44 c2339f50 
[  123.120420]        c0115555 c2337a70 00000003 c2227560 c2339f60 c01103a3 00000008 000000fc 
[  123.129552]        f722c030 f6ba2f34 c220f560 00000000 0000118f 6b37bab1 00000014 c048db40 
[  123.138685] Call Trace:
[  123.141500]  [<c0118d0c>] migration_thread+0x81/0x10d
[  123.146886]  [<c012fe68>] kthread+0xb7/0xbc
[  123.151367]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  123.156844] ksoftirqd/0   S 00000002     0     3      1             4     2 (L-TLB)
[  123.165164] c233af98 c233af88 00000004 00000002 00000515 000e3c82 00000000 c233ba70 
[  123.173391]        c2337030 c2357a70 00000001 c2331f5c c2330a70 ffffffff c2337030 c233af6c 
[  123.182526]        00000000 00000000 c220f560 00000000 000000c6 1038e436 00000012 c048db40 
[  123.191661] Call Trace:
[  123.194477]  [<c0121786>] ksoftirqd+0xd6/0xf5
[  123.199134]  [<c012fe68>] kthread+0xb7/0xbc
[  123.203611]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  123.209081] migration/1   S 00000001     0     4      1             5     3 (L-TLB)
[  123.217402] c233cf94 c233cf80 00000004 00000001 00000082 00000001 c233cf44 00000000 
[  123.225576]        c048db40 c2337a70 00000003 c2227560 f7d6f030 0000000b 00000008 f7d6f030 
[  123.234602]        00000000 c2217ec0 c2217560 00000001 000012a8 2929c41c 0000000b f7d6f030 
[  123.243625] Call Trace:
[  123.246408]  [<c0118d0c>] migration_thread+0x81/0x10d
[  123.251728]  [<c012fe68>] kthread+0xb7/0xbc
[  123.256154]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  123.261563] ksoftirqd/1   S 00000001     0     5      1             6     4 (L-TLB)
[  123.269790] c2345f98 c233b550 c2217560 00000001 000000ca 001817b5 00000000 c233b550 
[  123.277916]        c233b550 c233b67c 00000001 c2331f28 c2345000 ffffffff c233b550 c2345f6c 
[  123.286939]        c04079e4 c22175a8 c2217560 00000001 0000004c 00181a67 00000000 c2330550 
[  123.295968] Call Trace:
[  123.298754]  [<c0121786>] ksoftirqd+0xd6/0xf5
[  123.303418]  [<c012fe68>] kthread+0xb7/0xbc
[  123.307875]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  123.313324] migration/2   S 00000004     0     6      1             7     5 (L-TLB)
[  123.321646] c2346f94 c2346f84 00000004 00000004 00000082 c2330550 c2217a20 c2346f50 
[  123.329872]        c0115555 c2330550 00000001 c2217560 c2346f60 c01103a3 00000002 000000fc 
[  123.339004]        f7d6f030 f78cef34 c221f560 00000002 00000f44 9852669c 00000009 c2330030 
[  123.348135] Call Trace:
[  123.350952]  [<c0118d0c>] migration_thread+0x81/0x10d
[  123.356336]  [<c012fe68>] kthread+0xb7/0xbc
[  123.360819]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  123.366293] ksoftirqd/2   S 00000004     0     7      1             8     6 (L-TLB)
[  123.374618] c234df98 c234df88 00000004 00000004 00000149 00278084 00000000 c234ca70 
[  123.382841]        c234ca70 c2357550 00000001 c2331f28 c234d000 ffffffff c234ca70 c234df6c 
[  123.391976]        c04079e4 c221f5a8 c221f560 00000002 00000098 137fa71a 0000001a c2330030 
[  123.401100] Call Trace:
[  123.403916]  [<c0121786>] ksoftirqd+0xd6/0xf5
[  123.408579]  [<c012fe68>] kthread+0xb7/0xbc
[  123.413058]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  123.418535] migration/3   S 00000003     0     8      1             9     7 (L-TLB)
[  123.426857] c234ef94 c234ef80 00000004 00000003 00000082 f7238a70 c221f5a8 00000000 
[  123.435078]        c2330030 c2330550 00000001 c2217560 f7f82a70 00000025 00000002 f7f82a70 
[  123.444209]        00000000 c2227ec0 c2227560 00000003 00000ef9 f0522840 00000012 f7f82a70 
[  123.453337] Call Trace:
[  123.456154]  [<c0118d0c>] migration_thread+0x81/0x10d
[  123.461537]  [<c012fe68>] kthread+0xb7/0xbc
[  123.466015]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  123.471488] ksoftirqd/3   S 00000003     0     9      1            10     8 (L-TLB)
[  123.479807] c2353f98 c2353f88 00000004 00000003 000000c6 0036a2e2 00000000 c234c030 
[  123.488026]        c234c030 c2361550 00000001 c2331f28 c2353000 ffffffff c234c030 c2353f6c 
[  123.497155]        c04079e4 c22275a8 c2227560 00000003 000000cd f1a98ae3 00000013 c2337a70 
[  123.506287] Call Trace:
[  123.509103]  [<c0121786>] ksoftirqd+0xd6/0xf5
[  123.513763]  [<c012fe68>] kthread+0xb7/0xbc
[  123.518242]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  123.523712] events/0      S 00000002     0    10      1            11     9 (L-TLB)
[  123.532030] c2358f34 c2358f24 00000004 00000002 c220ffe0 00000202 c2358ef0 c0124bab 
[  123.540248]        c220ffe0 f78ed398 00000202 00000002 f78ed380 f78ed000 c2358f00 c0124c50 
[  123.549377]        c23489b0 c23489a8 c220f560 00000000 00003df1 a5143020 0000001c c048db40 
[  123.558505] Call Trace:
[  123.561321]  [<c012bd90>] worker_thread+0x207/0x225
[  123.566522]  [<c012fe68>] kthread+0xb7/0xbc
[  123.571001]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  123.576469] events/1      S 00000001     0    11      1            12    10 (L-TLB)
[  123.584785] c236af34 c236af24 00000004 00000001 00000202 00000001 c2354540 000007d0 
[  123.593003]        c236aef4 c012bb5c c22191b8 fffd5c23 00000000 c2300780 c22191a4 c236aefc 
[  123.602128]        c2348930 c2348928 c2217560 00000001 000005e9 532e09a0 0000001c c2330550 
[  123.611254] Call Trace:
[  123.614070]  [<c012bd90>] worker_thread+0x207/0x225
[  123.619272]  [<c012fe68>] kthread+0xb7/0xbc
[  123.623755]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  123.629234] events/2      R running     0    12      1            13    11 (L-TLB)
[  123.637467] events/3      S 00000003     0    13      1            14    12 (L-TLB)
[  123.645791] c236cf34 c236cf24 00000004 00000003 00000202 00000001 c2354540 000007d0 
[  123.654015]        c236cef4 c012bb5c c22291b8 fffd5c29 00000000 c2300780 c22291a4 c236cefc 
[  123.663148]        c2348830 c2348828 c2227560 00000003 00000b30 caa6cff9 0000001c c2337a70 
[  123.672282] Call Trace:
[  123.675098]  [<c012bd90>] worker_thread+0x207/0x225
[  123.680301]  [<c012fe68>] kthread+0xb7/0xbc
[  123.684784]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  123.690260] khelper       S 00000004     0    14      1            15    13 (L-TLB)
[  123.698583] c235af34 c235af20 00000004 00000004 00000000 c235aef4 c0117b5d f7ff7080 
[  123.706806]        f7b08030 00000000 00000000 f73ded58 f789d030 00000177 00000202 f789d030 
[  123.715940]        00000000 c221fec0 c221f560 00000002 0000062d 6fbb089f 00000006 f789d030 
[  123.725069] Call Trace:
[  123.727883]  [<c012bd90>] worker_thread+0x207/0x225
[  123.733078]  [<c012fe68>] kthread+0xb7/0xbc
[  123.737556]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  123.743029] kthread       S 00000002     0    15      1    20     144    14 (L-TLB)
[  123.751352] c2389f34 c2389f24 00000004 00000002 c2389ef0 c0117b5d f70c3e88 00000003 
[  123.759572]        00000000 00000000 f70c3f28 f70c3f1c f70c3f20 00000202 c2389f18 c0117cbb 
[  123.769504]        c2380b30 c2380b28 c220f560 00000000 000001d5 6f016241 00000005 c048db40 
[  123.778639] Call Trace:
[  123.781455]  [<c012bd90>] worker_thread+0x207/0x225
[  123.786659]  [<c012fe68>] kthread+0xb7/0xbc
[  123.791139]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  123.796607] kblockd/0     S 00000002     0    20     15            21       (L-TLB)
[  123.804919] c238bf34 c238bf24 00000004 00000002 c238bed0 c0408b7a c238bef8 c02fc8e6 
[  123.813132]        c0579414 f71ef07c c238bee8 00000000 c23eef80 f7f464f4 c23eef80 f7f4659c 
[  123.822258]        c23eefb0 c23eefa8 c220f560 00000000 00000e60 68d99a17 00000013 c048db40 
[  123.831387] Call Trace:
[  123.834198]  [<c012bd90>] worker_thread+0x207/0x225
[  123.839400]  [<c012fe68>] kthread+0xb7/0xbc
[  123.843879]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  123.849349] kblockd/1     S 00000001     0    21     15            22    20 (L-TLB)
[  123.857668] f7d58f34 f7d58f24 00000004 00000001 f78e239c 00a066dd 00000000 00000202 
[  123.865893]        00000001 f78e239c f7d58ee8 c0408b7a f7d58f10 c02fc8e6 c0579414 f78e239c 
[  123.874976]        c23eef30 c23eef28 c2217560 00000001 00000f03 f486b89c 00000009 c2330550 
[  123.884000] Call Trace:
[  123.886827]  [<c012bd90>] worker_thread+0x207/0x225
[  123.891972]  [<c012fe68>] kthread+0xb7/0xbc
[  123.896404]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  123.901821] kblockd/2     S 00000004     0    22     15            23    21 (L-TLB)
[  123.910144] c238ff34 c238ff24 00000004 00000004 f71f143c 00c6c6b5 00000000 00000202 
[  123.918277]        00000001 f71f143c c238fee8 c0408b7a c238ff10 c02fc8e6 c0579414 f71f143c 
[  123.927407]        c23eeeb0 c23eeea8 c221f560 00000002 00000dd9 2cbae8ae 00000006 c2330030 
[  123.936535] Call Trace:
[  123.939353]  [<c012bd90>] worker_thread+0x207/0x225
[  123.944554]  [<c012fe68>] kthread+0xb7/0xbc
[  123.949035]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  123.954508] kblockd/3     S 00000003     0    23     15            24    22 (L-TLB)
[  123.962832] f7d59f34 f7d59f20 00000004 00000003 f7fe357c 00089ebf 00000000 00000000 
[  123.971055]        c2330030 f7fe357c f7d59ee8 c0408b7a f7a9aa70 00000025 c0579414 f7a9aa70 
[  123.980186]        00000000 c2227ec0 c2227560 00000003 00000c68 613d0fa5 00000006 f7a9aa70 
[  123.989313] Call Trace:
[  123.992129]  [<c012bd90>] worker_thread+0x207/0x225
[  123.997333]  [<c012fe68>] kthread+0xb7/0xbc
[  124.001814]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  124.007291] kacpid        S 00000002     0    24     15           142    23 (L-TLB)
[  124.015608] f7d78f34 f7d78f24 00000004 00000002 00000000 00000001 00000100 00000000 
[  124.023835]        00000080 00000080 00000080 c05566ac c05566ac c05566a0 c2217ec0 c2217ec0 
[  124.032968]        00000000 00000202 c220f560 00000000 000001d4 027fa787 00000000 c048db40 
[  124.042099] Call Trace:
[  124.044917]  [<c012bd90>] worker_thread+0x207/0x225
[  124.050124]  [<c012fe68>] kthread+0xb7/0xbc
[  124.054602]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  124.060076] pdflush       S 00000002     0   142     15           143    24 (L-TLB)
[  124.068398] f7dadf68 f7dadf58 00000004 00000002 f7dadf34 c2361030 00000000 c220fec0 
[  124.076622]        c220f560 00000000 0000010c 2c569860 00000000 c053d600 00000286 f7dadf80 
[  124.085747]        c0118b63 c0503fac c220f560 00000000 000000ee 2c56c43d 00000000 c048db40 
[  124.094881] Call Trace:
[  124.097698]  [<c0141dd8>] __pdflush+0x84/0x199
[  124.102450]  [<c0141f2d>] pdflush+0x40/0x47
[  124.106936]  [<c012fe68>] kthread+0xb7/0xbc
[  124.111416]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  124.116889] pdflush       D 00000002     0   143     15           145   142 (L-TLB)
[  124.125217] f7d5dcc0 f7d5dcb0 00000004 00000002 f7dd75c0 f7a9b01c c216d1e0 00000000 
[  124.133437]        f7210680 00000001 f7d5dcbc c0288612 f7210680 00000008 00000000 00000000 
[  124.142564]        f7d5dca8 c015e326 c220f560 00000000 0000039c 8d3687f9 0000001b c048db40 
[  124.151693] Call Trace:
[  124.154508]  [<c040814b>] io_schedule+0x26/0x30
[  124.159344]  [<c013ab27>] sync_page+0x39/0x4e
[  124.164005]  [<c0408439>] __wait_on_bit_lock+0x58/0x67
[  124.169479]  [<c013b343>] __lock_page+0x84/0x8c
[  124.174320]  [<c017de94>] mpage_writepages+0x34b/0x399
[  124.179795]  [<c01c89c5>] nfs_writepages+0x2e/0x11e
[  124.184996]  [<c01416da>] do_writepages+0x26/0x44
[  124.190016]  [<c017c052>] __sync_single_inode+0x60/0x1f1
[  124.195674]  [<c017c275>] __writeback_single_inode+0x92/0x195
[  124.201779]  [<c017c535>] sync_sb_inodes+0x1bd/0x2c2
[  124.207074]  [<c017c714>] writeback_inodes+0xda/0xe6
[  124.212369]  [<c01414c9>] wb_kupdate+0x84/0xf6
[  124.217125]  [<c0141e15>] __pdflush+0xc1/0x199
[  124.221874]  [<c0141f2d>] pdflush+0x40/0x47
[  124.226358]  [<c012fe68>] kthread+0xb7/0xbc
[  124.230837]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  124.236306] kswapd0       D 00000002     0   144      1           804    15 (L-TLB)
[  124.244628] f7de7c94 f7de7c84 00000004 00000002 00011200 00000000 f72e86d0 00000000 
[  124.252851]        00000003 00011200 f7e3f780 0000001b 00011200 f7de7c7c c0143d92 f7e31880 
[  124.261985]        00011200 00000000 c220f560 00000000 001a2eae beaf71d2 00000016 c048db40 
[  124.271116] Call Trace:
[  124.273931]  [<c040814b>] io_schedule+0x26/0x30
[  124.278774]  [<c013e464>] mempool_alloc+0xcf/0xd1
[  124.283796]  [<c01c9a4f>] nfs_flush_one+0x54/0x17f
[  124.288906]  [<c01c9bcf>] nfs_flush_list+0x55/0xa3
[  124.294020]  [<c01ca516>] nfs_flush_inode+0x82/0xad
[  124.299223]  [<c01c8992>] nfs_writepage+0x1da/0x1df
[  124.304426]  [<c01464f6>] pageout+0xb5/0x137
[  124.308998]  [<c0146762>] shrink_list+0x1ea/0x40c
[  124.314020]  [<c0146b20>] shrink_cache+0xf8/0x281
[  124.319043]  [<c0147118>] shrink_zone+0xcb/0xfa
[  124.323885]  [<c01475f1>] balance_pgdat+0x286/0x37e
[  124.329088]  [<c01477ce>] kswapd+0xe5/0x10f
[  124.333574]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  124.339051] aio/0         S 00000002     0   145     15           146   143 (L-TLB)
[  124.347373] f7d67f34 f7d67f20 00000004 00000002 00000000 00000001 00000100 00000000 
[  124.355598]        00000080 00000080 00000080 c05566a0 c2330a70 c05566ac c220fec0 c2330a70 
[  124.364673]        00000000 c220fec0 c220f560 00000000 0000010b 2c9f99da 00000000 c2330a70 
[  124.373704] Call Trace:
[  124.376487]  [<c012bd90>] worker_thread+0x207/0x225
[  124.381628]  [<c012fe68>] kthread+0xb7/0xbc
[  124.386054]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  124.391469] aio/1         S 00000001     0   146     15           147   145 (L-TLB)
[  124.399790] f7e2cf34 f7e2cf20 00000004 00000001 00000000 00000001 00000100 00000000 
[  124.407927]        00000080 00000000 00000000 c05566ac c2330a70 00000000 c2217ec0 c2330a70 
[  124.416965]        00000000 c2217ec0 c2217560 00000001 000000ef 2c9fff06 00000000 c2330a70 
[  124.425999] Call Trace:
[  124.428788]  [<c012bd90>] worker_thread+0x207/0x225
[  124.433964]  [<c012fe68>] kthread+0xb7/0xbc
[  124.438444]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  124.443919] aio/2         S 00000004     0   147     15           148   146 (L-TLB)
[  124.452238] f7d68f34 f7d68f24 00000004 00000004 00000000 00000001 00000100 00000000 
[  124.460458]        00000080 00000080 00000080 c05566a0 c05566a0 c05566ac c220fec0 c220fec0 
[  124.469590]        00000000 00000202 c221f560 00000002 000001ae 2ca066f2 00000000 c2330030 
[  124.478722] Call Trace:
[  124.481537]  [<c012bd90>] worker_thread+0x207/0x225
[  124.486740]  [<c012fe68>] kthread+0xb7/0xbc
[  124.491221]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  124.496696] aio/3         S 00000003     0   148     15           149   147 (L-TLB)
[  124.505020] f7e2ef34 f7e2ef24 00000004 00000003 00000000 00000001 00000100 00000080 
[  124.513241]        00000080 00000000 00000000 c05566ac c05566ac 00000000 c2217ec0 c2217ec0 
[  124.522376]        00000000 00000202 c2227560 00000003 00000178 2ca0c2f3 00000000 c2337a70 
[  124.531506] Call Trace:
[  124.534324]  [<c012bd90>] worker_thread+0x207/0x225
[  124.539532]  [<c012fe68>] kthread+0xb7/0xbc
[  124.544011]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  124.549465] xfslogd/0     S 00000002     0   149     15           150   148 (L-TLB)
[  124.557695] f7d69f34 f7d69f20 00000004 00000002 00000000 00000001 00000100 00000000 
[  124.565916]        00000100 00000080 00000080 c05566a0 c2330a70 c05566ac c220fec0 c2330a70 
[  124.575858]        00000000 c220fec0 c220f560 00000000 000000eb 2e439ab2 00000000 c2330a70 
[  124.584995] Call Trace:
[  124.587810]  [<c012bd90>] worker_thread+0x207/0x225
[  124.593010]  [<c012fe68>] kthread+0xb7/0xbc
[  124.597494]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  124.602968] xfslogd/1     S 00000001     0   150     15           151   149 (L-TLB)
[  124.611294] f7e65f34 f7e65f20 00000004 00000001 00000000 00000001 00000100 00000000 
[  124.619515]        00000100 00000080 00000080 c05566ac c2330a70 c05566a0 c2217ec0 c2330a70 
[  124.628650]        00000000 c2217ec0 c2217560 00000001 000000e3 2e4402c7 00000000 c2330a70 
[  124.637780] Call Trace:
[  124.640598]  [<c012bd90>] worker_thread+0x207/0x225
[  124.645801]  [<c012fe68>] kthread+0xb7/0xbc
[  124.650282]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  124.655758] xfslogd/2     S 00000004     0   151     15           152   150 (L-TLB)
[  124.664084] f7d6cf34 f7d6cf24 00000004 00000004 00000000 00000001 00000100 00000080 
[  124.672309]        00000100 00000080 00000080 c05566a0 c05566a0 c05566ac c220fec0 c220fec0 
[  124.681439]        00000000 00000202 c221f560 00000002 00000141 2e44713f 00000000 c2330030 
[  124.690571] Call Trace:
[  124.693389]  [<c012bd90>] worker_thread+0x207/0x225
[  124.698592]  [<c012fe68>] kthread+0xb7/0xbc
[  124.703070]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  124.708545] xfslogd/3     S 00000003     0   152     15           153   151 (L-TLB)
[  124.716871] f7e67f34 f7e67f24 00000004 00000003 00000000 00000001 00000100 00000080 
[  124.725097]        00000100 00000080 00000080 c05566ac c05566ac c05566a0 c2217ec0 c2217ec0 
[  124.734229]        00000000 00000202 c2227560 00000003 00000144 2e44ce9a 00000000 c2337a70 
[  124.743359] Call Trace:
[  124.746182]  [<c012bd90>] worker_thread+0x207/0x225
[  124.751388]  [<c012fe68>] kthread+0xb7/0xbc
[  124.755867]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  124.761337] xfsdatad/0    S 00000002     0   153     15           154   152 (L-TLB)
[  124.769565] f7d6df34 f7d6df20 00000004 00000002 00000000 00000001 00000100 00000000 
[  124.777695]        00000100 00000080 00000080 c05566a0 c2330a70 c05566ac c220fec0 c2330a70 
[  124.786724]        00000000 c220fec0 c220f560 00000000 000000dc 2e451b1e 00000000 c2330a70 
[  124.795750] Call Trace:
[  124.798534]  [<c012bd90>] worker_thread+0x207/0x225
[  124.803676]  [<c012fe68>] kthread+0xb7/0xbc
[  124.808105]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  124.813516] xfsdatad/1    S 00000001     0   154     15           155   153 (L-TLB)
[  124.821743] f7e69f34 f7e69f20 00000004 00000001 00000000 00000001 00000100 00000000 
[  124.829874]        00000100 00000080 00000080 c05566ac c2330a70 c05566a0 c2217ec0 c2330a70 
[  124.838911]        00000000 c2217ec0 c2217560 00000001 000000db 2e458238 00000000 c2330a70 
[  124.847944] Call Trace:
[  124.850730]  [<c012bd90>] worker_thread+0x207/0x225
[  124.855872]  [<c012fe68>] kthread+0xb7/0xbc
[  124.860304]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  124.865714] xfsdatad/2    S 00000004     0   155     15           156   154 (L-TLB)
[  124.873947] f7d6ef34 f7d6ef24 00000004 00000004 00000000 00000001 00000100 00000080 
[  124.882079]        00000100 00000080 00000080 c05566a0 c05566a0 c05566ac c220fec0 c220fec0 
[  124.891116]        00000000 00000202 c221f560 00000002 0000013b 2e45e865 00000000 c2330030 
[  124.900149] Call Trace:
[  124.902936]  [<c012bd90>] worker_thread+0x207/0x225
[  124.908080]  [<c012fe68>] kthread+0xb7/0xbc
[  124.912507]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  124.917916] xfsdatad/3    S 00000003     0   156     15           157   155 (L-TLB)
[  124.926149] f7e6af34 f7e6af24 00000004 00000003 00000000 00000001 00000100 00000080 
[  124.934282]        00000100 00000080 00000080 c05566ac c05566ac c05566a0 c2217ec0 c2217ec0 
[  124.943312]        00000000 00000202 c2227560 00000003 00000153 2e464063 00000000 c2337a70 
[  124.952345] Call Trace:
[  124.955130]  [<c012bd90>] worker_thread+0x207/0x225
[  124.960272]  [<c012fe68>] kthread+0xb7/0xbc
[  124.964700]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  124.970110] xfsbufd       S 00000001     0   157     15           742   156 (L-TLB)
[  124.978341] f7d70f44 f7d70f34 00000004 00000001 00000100 00000080 00000080 00000000 
[  124.986475]        f78d0030 c2330550 26d0b06e 00000016 00000202 c2217fe0 f7d70f58 00000202 
[  124.995507]        c2217fe0 f7d70f58 c2217560 00000001 000000a0 fc42aa7a 0000001c c2330550 
[  125.004539] Call Trace:
[  125.007325]  [<c04081f1>] schedule_timeout+0x54/0xa5
[  125.012561]  [<c040825c>] schedule_timeout_interruptible+0x1a/0x1c
[  125.019048]  [<c0272378>] xfsbufd+0x4e/0x1b0
[  125.023566]  [<c012fe68>] kthread+0xb7/0xbc
[  125.027995]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  125.033406] kseriod       S 00000002     0   742     15          1668   157 (L-TLB)
[  125.041635] f7e6bf5c f7e6bf4c 00000004 00000002 c02efb29 f8879da4 f7e6bf10 c02efab5 
[  125.049768]        f8879d88 c04a8738 f8879da4 c04a63e0 f7e6bf34 c02ef172 f8879d88 c04a8738 
[  125.058807]        00000282 00000202 c220f560 00000000 0000042c 6243e470 00000001 c048db40 
[  125.067839] Call Trace:
[  125.070624]  [<c02e4386>] serio_thread+0xea/0xec
[  125.075502]  [<c012fe68>] kthread+0xb7/0xbc
[  125.079930]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  125.085340] kjournald     S 00000001     0   804      1          1128   144 (L-TLB)
[  125.093571] f7fc1f58 f7fc1f48 00000004 00000001 f699004c 00000000 00000000 f7fbf800 
[  125.101702]        00000000 f7897b00 f71f6b30 000003b3 00000202 c2227fe0 f7fc1fb8 f7fc1f2c 
[  125.110733]        f7fae26c f7fae264 c2217560 00000001 00000319 951388b9 00000015 c2330550 
[  125.119770] Call Trace:
[  125.122555]  [<c01b0ecf>] kjournald+0x1d6/0x223
[  125.127342]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  125.132752] kjournald     S 00000001     0  1128      1          1129   804 (L-TLB)
[  125.140982] f7a79f58 f7a79f48 00000004 00000001 00000012 00000036 00000044 00000202 
[  125.149114]        5b3e363c 20202020 31382e35 33333337 f700205d 00000003 00000000 00000000 
[  125.158144]        f7b7986c f7b79864 c2217560 00000001 000001d4 5b3099b5 00000001 c2330550 
[  125.167178] Call Trace:
[  125.169962]  [<c01b0ecf>] kjournald+0x1d6/0x223
[  125.174746]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  125.180159] kjournald     S 00000001     0  1129      1          1130  1128 (L-TLB)
[  125.188396] f7a83f58 f7a83f48 00000004 00000001 f69d802c 00000000 00000000 f7160800 
[  125.196529]        00000000 f715b580 f7a9b518 00000291 00000202 c2217fe0 f7a83fb8 f7a83f2c 
[  125.205560]        f7b91e6c f7b91e64 c2217560 00000001 00000232 95aa661b 00000015 c2330550 
[  125.214596] Call Trace:
[  125.217381]  [<c01b0ecf>] kjournald+0x1d6/0x223
[  125.222167]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  125.227579] kjournald     D 00000001     0  1130      1          2762  1129 (L-TLB)
[  125.235812] f7a51c30 f7a51c20 00000004 00000001 c015aca8 c17c0c14 00000207 c015bc30 
[  125.243942]        f7a51c10 c2330550 f7f5d170 f7f5d170 00000202 c2217fe0 f7a51c44 00000202 
[  125.252974]        c2217fe0 f7a51c44 c2217560 00000001 000000e0 266d3497 0000001d c2330550 
[  125.262005] Call Trace:
[  125.264790]  [<c04081f1>] schedule_timeout+0x54/0xa5
[  125.270023]  [<c040817e>] io_schedule_timeout+0x29/0x33
[  125.275522]  [<c0287ce8>] blk_congestion_wait+0x70/0x85
[  125.281023]  [<c0141365>] throttle_vm_writeout+0x63/0x6d
[  125.286615]  [<c014712d>] shrink_zone+0xe0/0xfa
[  125.291403]  [<c01471b4>] shrink_caches+0x6d/0x6f
[  125.296366]  [<c0147286>] try_to_free_pages+0xd0/0x1b5
[  125.301780]  [<c013fa4b>] __alloc_pages+0x135/0x2e8
[  125.306925]  [<c013b51b>] find_or_create_page+0x83/0x97
[  125.312424]  [<c015b757>] grow_dev_page+0x38/0x15d
[  125.317481]  [<c015b934>] __getblk_slow+0xb8/0x124
[  125.322534]  [<c015bcfd>] __getblk+0x57/0x59
[  125.327052]  [<c01b1950>] journal_get_descriptor_buffer+0x4d/0xa4
[  125.333452]  [<c01ae62d>] journal_commit_transaction+0xdb0/0x127f
[  125.339846]  [<c01b0dba>] kjournald+0xc1/0x223
[  125.344542]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  125.349953] khubd         S 00000001     0  1668     15          2771   742 (L-TLB)
[  125.358186] f70bff60 f70bff4c 00000004 00000001 000003e8 00000007 f72cae80 f7a36d00 
[  125.366319]        f7238a70 f7f6e0b4 c0294056 f70bff1c f7aaea70 00000007 0000005d f7aaea70 
[  125.376164]        00000000 c2217ec0 c2217560 00000001 00001239 989deb94 00000002 f7aaea70 
[  125.385197] Call Trace:
[  125.387982]  [<f8b01331>] hub_thread+0xe3/0xe5 [usbcore]
[  125.393590]  [<c012fe68>] kthread+0xb7/0xbc
[  125.398018]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  125.403435] portmap       S 00000001     0  2762      1          2775  1130 (NOTLB)
[  125.411667] f78ebf00 f78ebef0 00000004 00000001 00000202 f7b8900c 00000202 f78ebeb4 
[  125.419798]        c0130113 f7b89000 f7b12980 00000202 f7b89028 00000202 f78ebed0 c0130113 
[  125.428828]        f7b89000 f7eb8880 c2217560 00000001 00000ad7 543e693b 00000006 c2330550 
[  125.437862] Call Trace:
[  125.440649]  [<c0408240>] schedule_timeout+0xa3/0xa5
[  125.445882]  [<c016ce68>] do_poll+0x9a/0xb9
[  125.450309]  [<c016cfdd>] sys_poll+0x156/0x221
[  125.455007]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  125.460327] rpciod/0      S 00000002     0  2771     15          2772  1668 (L-TLB)
[  125.468560] f7102f34 f7102f24 00000004 00000002 f7e45100 f7102ee8 c013e4eb f7e45100 
[  125.476696]        f7e31900 f7e45104 f7e45104 f7e45180 f7102ef8 c01c6bf5 f7e45100 f7dd72c0 
[  125.485730]        f715b730 f715b728 c220f560 00000000 000006ba 6b487038 00000014 c048db40 
[  125.494761] Call Trace:
[  125.497545]  [<c012bd90>] worker_thread+0x207/0x225
[  125.502695]  [<c012fe68>] kthread+0xb7/0xbc
[  125.507122]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  125.512533] rpciod/1      S 00000004     0  2772     15          2773  2771 (L-TLB)
[  125.520771] f7a67f34 c2361030 c22175a8 00000004 00000000 00000001 00000100 00000000 
[  125.528903]        00000100 00000000 8a3a855d 00000004 c2361030 00000000 c220fec0 c2361030 
[  125.537935]        00000000 c2217ec0 c2217560 00000001 00000195 8a3d45dc 00000004 c2361030 
[  125.546965] Call Trace:
[  125.549751]  [<c012bd90>] worker_thread+0x207/0x225
[  125.554893]  [<c012fe68>] kthread+0xb7/0xbc
[  125.559325]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  125.564740] rpciod/2      D 00000004     0  2773     15          2774  2772 (L-TLB)
[  125.572975] f7a77ab4 f7a77aa4 00000004 00000004 f72f4120 f72f4118 00000001 00000006 
[  125.581111]        f7a77a84 c2330030 f72f4118 00000003 00000202 c221ffe0 f7a77ac8 00000202 
[  125.590147]        c221ffe0 f7a77ac8 c221f560 00000002 00000108 9e7577bc 0000001c c2330030 
[  125.599187] Call Trace:
[  125.601974]  [<c04081f1>] schedule_timeout+0x54/0xa5
[  125.607212]  [<c040817e>] io_schedule_timeout+0x29/0x33
[  125.612780]  [<c0287ce8>] blk_congestion_wait+0x70/0x85
[  125.618347]  [<c0141365>] throttle_vm_writeout+0x63/0x6d
[  125.624004]  [<c014712d>] shrink_zone+0xe0/0xfa
[  125.628839]  [<c01471b4>] shrink_caches+0x6d/0x6f
[  125.633810]  [<c0147286>] try_to_free_pages+0xd0/0x1b5
[  125.639224]  [<c013fa4b>] __alloc_pages+0x135/0x2e8
[  125.644367]  [<c03b70d1>] tcp_sendmsg+0xaa0/0xb78
[  125.649336]  [<c03d428a>] inet_sendmsg+0x48/0x53
[  125.654213]  [<c038833a>] sock_sendmsg+0xb8/0xd3
[  125.659090]  [<c0388397>] kernel_sendmsg+0x42/0x4f
[  125.664145]  [<c038b824>] sock_no_sendpage+0x5e/0x77
[  125.669381]  [<c03ee3cb>] xs_tcp_send_request+0x2af/0x375
[  125.675063]  [<c03ed0e1>] xprt_transmit+0x4f/0x20b
[  125.680116]  [<c03eb8f7>] call_transmit+0x68/0xc8
[  125.685081]  [<c03f0625>] __rpc_execute+0x1cb/0x221
[  125.690227]  [<c03f06be>] rpc_async_schedule+0x11/0x15
[  125.695642]  [<c012bd1d>] worker_thread+0x194/0x225
[  125.700788]  [<c012fe68>] kthread+0xb7/0xbc
[  125.705271]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  125.710748] rpciod/3      S 00000003     0  2774     15                2773 (L-TLB)
[  125.719071] f7aa7f34 f7aa7f20 00000004 00000003 f68d3580 f7aa7ee8 c013e4eb 00000000 
[  125.727297]        f7e31900 f68d3584 f68d3584 f68d3600 f722c030 c01c6bf5 f68d3580 f722c030 
[  125.736429]        00000000 c2227ec0 c2227560 00000003 0000126c 248effa8 00000016 f722c030 
[  125.745561] Call Trace:
[  125.748380]  [<c012bd90>] worker_thread+0x207/0x225
[  125.753588]  [<c012fe68>] kthread+0xb7/0xbc
[  125.758069]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  125.763516] lockd         S 00000002     0  2775      1          2842  2762 (L-TLB)
[  125.771840] f7a78f14 f7a78f04 00000004 00000002 f72f4200 00000001 c2217560 f7a78ecc 
[  125.780024]        c01103a3 00000002 000000fc f789d030 c22175a8 f7a78edc c0115796 00000202 
[  125.789116]        f7b56aec 00000202 c220f560 00000000 00000914 8a48a8d1 00000004 c048db40 
[  125.798209] Call Trace:
[  125.801013]  [<c0408240>] schedule_timeout+0xa3/0xa5
[  125.806283]  [<c03f46d7>] svc_recv+0x387/0x4b9
[  125.811012]  [<c02045e6>] lockd+0x102/0x241
[  125.815472]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  125.820920] syslogd       D 00000002     0  2842      1          2845  2775 (NOTLB)
[  125.829245] f71b7cdc f71b7ccc 00000004 00000002 f71b7cb8 c02880f3 f7f464f4 f78e2c5c 
[  125.837433]        00000003 c048db40 00000000 c21bbde0 00000202 c220ffe0 f71b7cf0 00000202 
[  125.846528]        c220ffe0 f71b7cf0 c220f560 00000000 00000159 4916abe9 0000001d c048db40 
[  125.855618] Call Trace:
[  125.858421]  [<c04081f1>] schedule_timeout+0x54/0xa5
[  125.863691]  [<c040817e>] io_schedule_timeout+0x29/0x33
[  125.869230]  [<c0287ce8>] blk_congestion_wait+0x70/0x85
[  125.874769]  [<c0141365>] throttle_vm_writeout+0x63/0x6d
[  125.880414]  [<c014712d>] shrink_zone+0xe0/0xfa
[  125.885230]  [<c01471b4>] shrink_caches+0x6d/0x6f
[  125.890228]  [<c0147286>] try_to_free_pages+0xd0/0x1b5
[  125.895640]  [<c013fa4b>] __alloc_pages+0x135/0x2e8
[  125.900786]  [<c013fc2a>] __get_free_pages+0x2c/0x4f
[  125.906022]  [<c016c541>] __pollwait+0x3a/0xae
[  125.910718]  [<c038fa46>] datagram_poll+0x2b/0xcd
[  125.915683]  [<c0388b2a>] sock_poll+0x23/0x2a
[  125.920290]  [<c016c8b1>] do_select+0x217/0x26e
[  125.925076]  [<c016cba9>] sys_select+0x26a/0x3f7
[  125.929951]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  125.935276] klogd         S 00000001     0  2845      1          2860  2842 (NOTLB)
[  125.943501] f7a17d4c f7a17d3c 00000004 00000001 c220f5a8 f7a17cf4 c0115796 00000000 
[  125.951636]        00000096 c2330550 c0116055 c048db40 c220f560 00000000 c2381880 f6c04000 
[  125.960664]        00000a00 000000b7 c2217560 00000001 0000089f a21e21d1 0000001c c2330550 
[  125.969690] Call Trace:
[  125.972473]  [<c0408240>] schedule_timeout+0xa3/0xa5
[  125.977707]  [<f88b00ee>] unix_wait_for_peer+0xb3/0xb7 [unix]
[  125.983750]  [<f88b0abb>] unix_dgram_sendmsg+0x266/0x502 [unix]
[  125.989966]  [<c03886da>] sock_aio_write+0xea/0x106
[  125.995108]  [<c01592e7>] do_sync_write+0xa7/0xe7
[  126.000075]  [<c0159495>] vfs_write+0x16e/0x17d
[  126.004863]  [<c0159564>] sys_write+0x4b/0x75
[  126.009470]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  126.014790] ypbind        S 00000002     0  2860      1          2861  2845 (NOTLB)
[  126.023018] f7fc9f00 f7fc9ef0 00000004 00000002 00000202 f7ee600c 00000202 f7fc9eb4 
[  126.031149]        c0130113 f7ee6000 f7ebf280 00000202 f7ee6028 00000202 f7fc9ed0 c0130113 
[  126.040175]        f7ee6000 f716de80 c220f560 00000000 00001305 632220d6 00000005 c048db40 
[  126.049205] Call Trace:
[  126.051988]  [<c0408240>] schedule_timeout+0xa3/0xa5
[  126.057221]  [<c016ce68>] do_poll+0x9a/0xb9
[  126.061651]  [<c016cfdd>] sys_poll+0x156/0x221
[  126.066348]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  126.071668] ypbind        S 00000002     0  2861      1          2862  2860 (NOTLB)
[  126.079896] f7893eac f7893e9c 00000004 00000002 f7121a70 c220f560 00000001 c2330a70 
[  126.088023]        f7fc9e78 f7893ed4 c040732f c220c060 f7238710 f7121a70 f7238550 00000000 
[  126.097156]        f71a4080 f7ac1a70 c220f560 00000000 000025b0 2aa68eb6 00000005 c048db40 
[  126.106291] Call Trace:
[  126.109109]  [<c0408240>] schedule_timeout+0xa3/0xa5
[  126.114401]  [<c040825c>] schedule_timeout_interruptible+0x1a/0x1c
[  126.120883]  [<c0128559>] sys_rt_sigtimedwait+0x1f1/0x2e9
[  126.126568]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  126.131889] ypbind        D 00000002     0  2862      1          3022  2861 (NOTLB)
[  126.140116] f7247cf8 f7247ce8 00000004 00000002 f7b56680 f7b56680 f7247eb8 00000000 
[  126.148242]        00000000 c048db40 c2328e80 00000040 00000202 c220ffe0 f7247d0c 00000202 
[  126.157271]        c220ffe0 f7247d0c c220f560 00000000 0000016a 5e4a6eff 0000001d c048db40 
[  126.166301] Call Trace:
[  126.169088]  [<c04081f1>] schedule_timeout+0x54/0xa5
[  126.174319]  [<c040817e>] io_schedule_timeout+0x29/0x33
[  126.179819]  [<c0287ce8>] blk_congestion_wait+0x70/0x85
[  126.185319]  [<c0141365>] throttle_vm_writeout+0x63/0x6d
[  126.190914]  [<c014712d>] shrink_zone+0xe0/0xfa
[  126.196500]  [<c01471b4>] shrink_caches+0x6d/0x6f
[  126.201464]  [<c0147286>] try_to_free_pages+0xd0/0x1b5
[  126.206877]  [<c013fa4b>] __alloc_pages+0x135/0x2e8
[  126.212020]  [<c013fc2a>] __get_free_pages+0x2c/0x4f
[  126.217255]  [<c016c541>] __pollwait+0x3a/0xae
[  126.221984]  [<c038fa46>] datagram_poll+0x2b/0xcd
[  126.226952]  [<c03ce0ae>] udp_poll+0x22/0xfc
[  126.231471]  [<c0388b2a>] sock_poll+0x23/0x2a
[  126.236079]  [<c016cdca>] do_pollfd+0x94/0x98
[  126.240685]  [<c016ce29>] do_poll+0x5b/0xb9
[  126.245114]  [<c016cfdd>] sys_poll+0x156/0x221
[  126.249809]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  126.255135] automount     D 00000002     0  3022      1          3030  2862 (NOTLB)
[  126.263363] f739ad28 f739ad18 00000004 00000002 f739acf4 c014b7c2 f7ff7280 f78c9f94 
[  126.271492]        b7edb000 c048db40 f7a96b7c 00000001 00000202 c220ffe0 f739ad3c 00000202 
[  126.280528]        c220ffe0 f739ad3c c220f560 00000000 0000016f 65dac2db 0000001d c048db40 
[  126.289618] Call Trace:
[  126.292421]  [<c04081f1>] schedule_timeout+0x54/0xa5
[  126.297686]  [<c040817e>] io_schedule_timeout+0x29/0x33
[  126.303224]  [<c0287ce8>] blk_congestion_wait+0x70/0x85
[  126.308759]  [<c0141365>] throttle_vm_writeout+0x63/0x6d
[  126.314385]  [<c014712d>] shrink_zone+0xe0/0xfa
[  126.319204]  [<c01471b4>] shrink_caches+0x6d/0x6f
[  126.324212]  [<c0147286>] try_to_free_pages+0xd0/0x1b5
[  126.329659]  [<c013fa4b>] __alloc_pages+0x135/0x2e8
[  126.334836]  [<c013fc2a>] __get_free_pages+0x2c/0x4f
[  126.340104]  [<c016c541>] __pollwait+0x3a/0xae
[  126.344829]  [<c0165ff6>] pipe_poll+0x36/0xad
[  126.349466]  [<c016cdca>] do_pollfd+0x94/0x98
[  126.354097]  [<c016ce29>] do_poll+0x5b/0xb9
[  126.358531]  [<c016cfdd>] sys_poll+0x156/0x221
[  126.363230]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  126.368555] dbus-daemon-1 S 00000002     0  3030      1          3063  3022 (NOTLB)
[  126.376882] f7159f00 f7159ef0 00000004 00000002 000200d0 00000000 c0493880 00000044 
[  126.385019]        f7159ee0 c17b4488 00000010 c0493880 00000000 00000202 f6c92028 00000202 
[  126.394056]        f7159ed8 c0130113 c220f560 00000000 0000c1ab 86785d3f 00000005 c048db40 
[  126.403091] Call Trace:
[  126.405878]  [<c0408240>] schedule_timeout+0xa3/0xa5
[  126.411176]  [<c016ce68>] do_poll+0x9a/0xb9
[  126.415660]  [<c016cfdd>] sys_poll+0x156/0x221
[  126.420413]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  126.425800] exim4         S 00000001     0  3063      1          3068  3030 (NOTLB)
[  126.434124] f78baea0 f78bae90 00000004 00000001 c17c4cc4 c0493880 000000d0 f7892030 
[  126.442340]        f78bae80 c013f968 000200d0 00000000 c0493880 00000044 005cef3f c17c4cc4 
[  126.451427]        00000010 c0493880 c2217560 00000001 00001800 a5c81594 00000005 c2330550 
[  126.460515] Call Trace:
[  126.463317]  [<c0408240>] schedule_timeout+0xa3/0xa5
[  126.468584]  [<c016c7ec>] do_select+0x152/0x26e
[  126.473401]  [<c016cba9>] sys_select+0x26a/0x3f7
[  126.478307]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  126.483663] irqbalance    D 00000002     0  3068      1          3072  3063 (NOTLB)
[  126.491988] f73e0d14 f73e0d04 00000004 00000002 00000000 f73e0cc8 c01722a6 c2313200 
[  126.500170]        f00000dd c048db40 f7b10a00 c2313200 00000202 c220ffe0 f73e0d28 00000202 
[  126.509286]        c220ffe0 f73e0d28 c220f560 00000000 00000153 6f6232af 0000001d c048db40 
[  126.518319] Call Trace:
[  126.521102]  [<c04081f1>] schedule_timeout+0x54/0xa5
[  126.526333]  [<c040817e>] io_schedule_timeout+0x29/0x33
[  126.531832]  [<c0287ce8>] blk_congestion_wait+0x70/0x85
[  126.537330]  [<c0141365>] throttle_vm_writeout+0x63/0x6d
[  126.542919]  [<c014712d>] shrink_zone+0xe0/0xfa
[  126.547701]  [<c01471b4>] shrink_caches+0x6d/0x6f
[  126.552693]  [<c0147286>] try_to_free_pages+0xd0/0x1b5
[  126.558170]  [<c013fa4b>] __alloc_pages+0x135/0x2e8
[  126.563372]  [<c0142d15>] kmem_getpages+0x34/0x9d
[  126.568395]  [<c0143b71>] cache_grow+0xaf/0x16c
[  126.573238]  [<c0143d92>] cache_alloc_refill+0x164/0x207
[  126.578893]  [<c014401b>] kmem_cache_alloc+0x4e/0x50
[  126.584150]  [<c01668ec>] getname+0x24/0xb3
[  126.588581]  [<c015878d>] do_sys_open+0x1a/0xfa
[  126.593371]  [<c015888c>] sys_open+0x1f/0x23
[  126.597892]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  126.603223] nrpe          S 00000002     0  3072      1          3077  3068 (NOTLB)
[  126.611548] f7a62de8 f7a62dd8 00000004 00000002 f7fae200 c019da80 00000032 f7a62ea4 
[  126.619730]        f7a62df8 f7a62e84 c038833a f7a62df8 f7a24480 f7a62ea4 00000032 00000000 
[  126.628756]        f7a62dc8 c019daf0 c220f560 00000000 0000b007 af408ff0 00000005 c048db40 
[  126.637782] Call Trace:
[  126.640565]  [<c0408240>] schedule_timeout+0xa3/0xa5
[  126.645797]  [<c03b50ce>] inet_csk_wait_for_connect+0xd7/0xdc
[  126.651908]  [<c03b5131>] inet_csk_accept+0x5e/0x154
[  126.657204]  [<c03d4120>] inet_accept+0x30/0xb2
[  126.662018]  [<c0389379>] sys_accept+0xb3/0x173
[  126.666812]  [<c0389eed>] sys_socketcall+0xc1/0x234
[  126.671955]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  126.677277] snmpd         S 00000002     0  3077      1          3085  3072 (NOTLB)
[  126.685507] f70c5ea0 f70c5e90 00000004 00000002 c0493880 c0492f80 000200d0 00000002 
[  126.693639]        00000044 c17aeba8 c0493880 000000d0 00000202 f6a1a060 00000202 f70c5e74 
[  126.702740]        c0130113 f6a1a000 c220f560 00000000 00017d9c c9f7903b 00000005 c048db40 
[  126.711876] Call Trace:
[  126.714693]  [<c0408240>] schedule_timeout+0xa3/0xa5
[  126.719989]  [<c016c7ec>] do_select+0x152/0x26e
[  126.724836]  [<c016cba9>] sys_select+0x26a/0x3f7
[  126.729717]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  126.735037] sshd          S 00000002     0  3085      1          3096  3077 (NOTLB)
[  126.743261] f7b76ea0 f7b76e90 00000004 00000002 c17aedc4 c0493880 000000d0 f7ba0030 
[  126.751387]        f7b76e80 c013f968 000200d0 00000000 c0493880 00000044 f7b76ed4 c17aedc4 
[  126.760411]        00000010 c0493880 c220f560 00000000 00008e0f d7a313a7 00000005 c048db40 
[  126.769438] Call Trace:
[  126.772224]  [<c0408240>] schedule_timeout+0xa3/0xa5
[  126.777458]  [<c016c7ec>] do_select+0x152/0x26e
[  126.782241]  [<c016cba9>] sys_select+0x26a/0x3f7
[  126.787114]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  126.792434] xfs           S 00000002     0  3096      1          3100  3085 (NOTLB)
[  126.800687] f7a54ea0 f7a54e90 00000004 00000002 00000002 00000044 c17b3bd0 c0493880 
[  126.808848]        000000d0 f6a42a70 f7a54e88 c013f968 000200d0 00000000 c0493880 00000202 
[  126.817877]        c220ffe0 f7a54eb4 c220f560 00000000 000ff865 fbda9637 00000005 c048db40 
[  126.826904] Call Trace:
[  126.829688]  [<c04081f1>] schedule_timeout+0x54/0xa5
[  126.834919]  [<c016c7ec>] do_select+0x152/0x26e
[  126.839705]  [<c016cba9>] sys_select+0x26a/0x3f7
[  126.844580]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  126.849967] xfs-xtt       S 00000002     0  3100      1          3237  3096 (NOTLB)
[  126.858329] f739bea0 f739be90 00000004 00000002 00000002 00000044 c17b3ea0 c0493880 
[  126.866555]        000000d0 f6a42030 f739be88 c013f968 000200d0 00000000 c0493880 00000202 
[  126.875640]        c220ffe0 f739beb4 c220f560 00000000 001f709f 010b1144 00000006 c048db40 
[  126.884665] Call Trace:
[  126.887447]  [<c04081f1>] schedule_timeout+0x54/0xa5
[  126.892677]  [<c016c7ec>] do_select+0x152/0x26e
[  126.897460]  [<c016cba9>] sys_select+0x26a/0x3f7
[  126.902396]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  126.907783] S20xprint     S 00000002     0  3237      1  3238    3291  3100 (NOTLB)
[  126.916109] f78a6f08 f78a6ef4 00000004 00000002 00000099 00000001 f78a6ee0 f7b83780 
[  126.924332]        f7a75a70 0000002f c013f968 000200d2 f6a42550 00000002 00000000 f6a42550 
[  126.933471]        00000000 c220fec0 c220f560 00000000 00020dc6 122c8a50 00000006 f6a42550 
[  126.942606] Call Trace:
[  126.945423]  [<c011fa57>] do_wait+0x18a/0x376
[  126.950084]  [<c011fd03>] sys_wait4+0x3e/0x42
[  126.954746]  [<c011fd2e>] sys_waitpid+0x27/0x2b
[  126.959590]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  126.964975] S20xprint     S 00000001     0  3238   3237  3239    3241       (NOTLB)
[  126.973303] f7a8ef08 f7a8eef8 00000004 00000001 00000044 c2170bf4 c04938a8 000200d2 
[  126.981487]        f7a75a70 f7a8eeec c013f968 000200d2 00000000 c04938a8 00000044 f716d0cc 
[  126.990581]        c2170bf4 00000010 c2217560 00000001 0001eef6 125fdd55 00000006 c2330550 
[  126.999715] Call Trace:
[  127.002533]  [<c011fa57>] do_wait+0x18a/0x376
[  127.007201]  [<c011fd03>] sys_wait4+0x3e/0x42
[  127.011868]  [<c011fd2e>] sys_waitpid+0x27/0x2b
[  127.017526]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  127.022915] Xprt          S 00000002     0  3239   3238                     (NOTLB)
[  127.031242] f78cfea0 f78cfe90 00000004 00000002 00000002 00000044 c17b0b28 c0493880 
[  127.039466]        000000d0 f7ac1550 f78cfe88 c013f968 000200d0 00000000 c0493880 00000044 
[  127.048601]        00000257 c17b0b28 c220f560 00000000 000176ca 3308517b 00000006 c048db40 
[  127.057732] Call Trace:
[  127.060551]  [<c0408240>] schedule_timeout+0xa3/0xa5
[  127.065846]  [<c016c7ec>] do_select+0x152/0x26e
[  127.070690]  [<c016cba9>] sys_select+0x26a/0x3f7
[  127.075625]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  127.081011] S20xprint     S 00000002     0  3241   3237                3238 (NOTLB)
[  127.089337] f7a8fe9c f7a8fe8c 00000004 00000002 00000080 00000080 c05566a0 c05566a0 
[  127.097561]        c05566ac c220fec0 c220fec0 00000000 c0491680 00000082 00000011 f71e5144 
[  127.106696]        f7a8ff3c f7a8fe6c c220f560 00000000 0001b550 2c77edf0 00000006 c048db40 
[  127.115830] Call Trace:
[  127.118648]  [<c01656cb>] pipe_wait+0x6f/0x8e
[  127.123312]  [<c0165906>] pipe_readv+0x1c8/0x2f3
[  127.128246]  [<c0165a64>] pipe_read+0x33/0x37
[  127.132908]  [<c0159169>] vfs_read+0xa6/0x17d
[  127.137571]  [<c01594ef>] sys_read+0x4b/0x75
[  127.142143]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  127.147529] rpc.statd     S 00000002     0  3291      1          3294  3237 (NOTLB)
[  127.155857] f7399ea0 f7399e90 00000004 00000002 00000000 c0493880 00000044 f7399e74 
[  127.164084]        c17b0eac 00000010 c0493880 00000000 00000202 f6b13028 00000202 f7399e74 
[  127.173219]        c0130113 f6b13000 c220f560 00000000 00002198 544517ff 00000006 c048db40 
[  127.182355] Call Trace:
[  127.185172]  [<c0408240>] schedule_timeout+0xa3/0xa5
[  127.190468]  [<c016c7ec>] do_select+0x152/0x26e
[  127.195312]  [<c016cba9>] sys_select+0x26a/0x3f7
[  127.200247]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  127.205634] ntpd          D 00000002     0  3294      1          3307  3291 (NOTLB)
[  127.213959] f7350cb8 f7350ca8 00000004 00000002 00000001 00000282 00000000 00000003 
[  127.222186]        f7fedd38 c048db40 f7ae25cc f7350ca4 00000202 c220ffe0 f7350ccc 00000202 
[  127.231322]        c220ffe0 f7350ccc c220f560 00000000 00000180 9da8b971 0000001d c048db40 
[  127.240456] Call Trace:
[  127.243274]  [<c04081f1>] schedule_timeout+0x54/0xa5
[  127.248569]  [<c040817e>] io_schedule_timeout+0x29/0x33
[  127.254139]  [<c0287ce8>] blk_congestion_wait+0x70/0x85
[  127.259707]  [<c0141365>] throttle_vm_writeout+0x63/0x6d
[  127.265367]  [<c014712d>] shrink_zone+0xe0/0xfa
[  127.270217]  [<c01471b4>] shrink_caches+0x6d/0x6f
[  127.275240]  [<c0147286>] try_to_free_pages+0xd0/0x1b5
[  127.280718]  [<c013fa4b>] __alloc_pages+0x135/0x2e8
[  127.285920]  [<c013fc2a>] __get_free_pages+0x2c/0x4f
[  127.291216]  [<c016c541>] __pollwait+0x3a/0xae
[  127.295968]  [<c038fa46>] datagram_poll+0x2b/0xcd
[  127.300994]  [<c03ce0ae>] udp_poll+0x22/0xfc
[  127.305566]  [<c0388b2a>] sock_poll+0x23/0x2a
[  127.310228]  [<c016c8b1>] do_select+0x217/0x26e
[  127.315072]  [<c016cba9>] sys_select+0x26a/0x3f7
[  127.320006]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  127.325391] atd           S 00000001     0  3307      1          3310  3294 (NOTLB)
[  127.333720] f7261f38 f7261f28 00000004 00000001 f7fd9194 f7eb3180 c21f7264 c21f7264 
[  127.341946]        c17b135c f7261f1c c014b57b c21f7264 00000028 f7261f0c 00000000 00000202 
[  127.351081]        c2217fe0 f7261f4c c2217560 00000001 0000406f 65db04ff 00000006 c2330550 
[  127.360216] Call Trace:
[  127.363033]  [<c04081f1>] schedule_timeout+0x54/0xa5
[  127.368330]  [<c040825c>] schedule_timeout_interruptible+0x1a/0x1c
[  127.374892]  [<c012564b>] sys_nanosleep+0xc8/0x147
[  127.380008]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  127.385396] cron          S 00000002     0  3310      1          3316  3307 (NOTLB)
[  127.393722] f70e7f38 f70e7f28 00000004 00000002 00000000 f70e7f50 c016325c bfe2c0dc 
[  127.401954]        f70e7ee4 00000060 00000301 00000000 00000000 00008081 000081a4 00000202 
[  127.411093]        c220ffe0 f70e7f4c c220f560 00000000 00001742 03aa0958 00000013 c048db40 
[  127.420231] Call Trace:
[  127.423048]  [<c04081f1>] schedule_timeout+0x54/0xa5
[  127.428345]  [<c040825c>] schedule_timeout_interruptible+0x1a/0x1c
[  127.434907]  [<c012564b>] sys_nanosleep+0xc8/0x147
[  127.440022]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  127.445410] bash          S 00000001     0  3316      1  3388    3317  3310 (NOTLB)
[  127.453738] f78cef08 f78ceef8 00000004 00000001 00000044 c2167024 c04938a8 000200d2 
[  127.461966]        f7d6f030 f78ceeec c013f968 000200d2 00000000 c04938a8 00000044 0000000e 
[  127.471103]        c2167024 00000010 c2217560 00000001 0000258e 6b2eda41 00000014 c2330550 
[  127.480238] Call Trace:
[  127.483060]  [<c011fa57>] do_wait+0x18a/0x376
[  127.487720]  [<c011fd03>] sys_wait4+0x3e/0x42
[  127.492380]  [<c011fd2e>] sys_waitpid+0x27/0x2b
[  127.497222]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  127.502608] bash          S 00000003     0  3317      1  3387    3318  3316 (NOTLB)
[  127.510934] f78bff08 f78bfef8 00000004 00000003 00000044 c216544c c04938a8 000200d2 
[  127.519161]        f7f82a70 f78bfeec c013f968 000200d2 00000000 c04938a8 00000044 0000000e 
[  127.528296]        c216544c 00000010 c2227560 00000003 000029d1 f05376cd 00000012 c2337a70 
[  127.537431] Call Trace:
[  127.540248]  [<c011fa57>] do_wait+0x18a/0x376
[  127.544911]  [<c011fd03>] sys_wait4+0x3e/0x42
[  127.549573]  [<c011fd2e>] sys_waitpid+0x27/0x2b
[  127.554420]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  127.559802] getty         S 00000003     0  3318      1          3319  3317 (NOTLB)
[  127.568127] f7a85e6c f7a85e5c 00000004 00000003 c0167f20 c2316580 f718a000 f7f36bc8 
[  127.576350]        00000202 f70ce400 00000202 f7a85e3c c011c6dd 00000001 00000000 00000000 
[  127.585489]        f70ce400 ffffffff c2227560 00000003 00001206 a87ddbf8 00000006 c2337a70 
[  127.594625] Call Trace:
[  127.597442]  [<c0408240>] schedule_timeout+0xa3/0xa5
[  127.602740]  [<c02d6c9d>] read_chan+0x57c/0x64b
[  127.607585]  [<c02d180b>] tty_read+0xc9/0xcd
[  127.612160]  [<c0159169>] vfs_read+0xa6/0x17d
[  127.616824]  [<c01594ef>] sys_read+0x4b/0x75
[  127.621394]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  127.626778] getty         S 00000003     0  3319      1          3320  3318 (NOTLB)
[  127.635102] f7ac7e6c f7ac7e5c 00000004 00000003 c0167f20 c2316580 f718a000 f7f36bc8 
[  127.643328]        00000202 f70fca00 00000202 f7ac7e3c c011c6dd 00000001 00000000 00000000 
[  127.652471]        f70fca00 ffffffff c2227560 00000003 0000120b ab4a6a85 00000006 c2337a70 
[  127.661612] Call Trace:
[  127.664428]  [<c0408240>] schedule_timeout+0xa3/0xa5
[  127.669724]  [<c02d6c9d>] read_chan+0x57c/0x64b
[  127.674569]  [<c02d180b>] tty_read+0xc9/0xcd
[  127.679142]  [<c0159169>] vfs_read+0xa6/0x17d
[  127.683806]  [<c01594ef>] sys_read+0x4b/0x75
[  127.688378]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  127.693762] getty         S 00000006     0  3320      1          3321  3319 (NOTLB)
[  127.702092] f7a19e6c f7a7b550 c22275a8 00000006 c0167f20 c2316580 f718a000 00000000 
[  127.710319]        c2330030 f73a5e00 ab49c81c 00000006 f7a7b550 00000003 00000000 f7a7b550 
[  127.719457]        00000000 c2227ec0 c2227560 00000003 000011f4 ab49c81c 00000006 f7a7b550 
[  127.728595] Call Trace:
[  127.731420]  [<c0408240>] schedule_timeout+0xa3/0xa5
[  127.736717]  [<c02d6c9d>] read_chan+0x57c/0x64b
[  127.741561]  [<c02d180b>] tty_read+0xc9/0xcd
[  127.746134]  [<c0159169>] vfs_read+0xa6/0x17d
[  127.750797]  [<c01594ef>] sys_read+0x4b/0x75
[  127.755370]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  127.760755] getty         S 00000003     0  3321      1                3320 (NOTLB)
[  127.769079] f73dee6c f73dee5c 00000004 00000003 c0167f20 c2316580 f718a000 f7f36bc8 
[  127.777305]        00000202 f70d9600 00000202 f73dee3c c011c6dd 00000001 00000000 00000000 
[  127.786441]        f70d9600 ffffffff c2227560 00000003 0000122e abd302b7 00000006 c2337a70 
[  127.795580] Call Trace:
[  127.798400]  [<c0408240>] schedule_timeout+0xa3/0xa5
[  127.803696]  [<c02d6c9d>] read_chan+0x57c/0x64b
[  127.808539]  [<c02d180b>] tty_read+0xc9/0xcd
[  127.813113]  [<c0159169>] vfs_read+0xa6/0x17d
[  127.817776]  [<c01594ef>] sys_read+0x4b/0x75
[  127.822348]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  127.827734] slabtop       D 00000001     0  3387   3317                     (NOTLB)
[  127.836858] f6986cc8 f6986cb8 00000004 00000001 00000001 00000292 f7b15824 c2313200 
[  127.845082]        c2313200 c2330550 c018b6f3 f6986c80 00000202 c2217fe0 f6986cdc 00000202 
[  127.854217]        c2217fe0 f6986cdc c2217560 00000001 0000016b c4ba71c9 0000001d c2330550 
[  127.863350] Call Trace:
[  127.866169]  [<c04081f1>] schedule_timeout+0x54/0xa5
[  127.871464]  [<c040817e>] io_schedule_timeout+0x29/0x33
[  127.877031]  [<c0287ce8>] blk_congestion_wait+0x70/0x85
[  127.882571]  [<c0141365>] throttle_vm_writeout+0x63/0x6d
[  127.888160]  [<c014712d>] shrink_zone+0xe0/0xfa
[  127.892945]  [<c01471b4>] shrink_caches+0x6d/0x6f
[  127.897908]  [<c0147286>] try_to_free_pages+0xd0/0x1b5
[  127.903322]  [<c013fa4b>] __alloc_pages+0x135/0x2e8
[  127.908462]  [<c013fc2a>] __get_free_pages+0x2c/0x4f
[  127.913696]  [<c016c541>] __pollwait+0x3a/0xae
[  127.918391]  [<c02d6fba>] normal_poll+0x2f/0x155
[  127.923264]  [<c02d2fb2>] tty_poll+0x7a/0x7f
[  127.927781]  [<c016c8b1>] do_select+0x217/0x26e
[  127.932567]  [<c016cba9>] sys_select+0x26a/0x3f7
[  127.937439]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  127.942763] writetest     D 00000003     0  3388   3316                     (NOTLB)
[  127.951089] f6ba29e8 f6ba29d8 00000004 00000003 b3f79c28 f6ba2990 00000090 f78b3300 
[  127.959312]        f7b916e0 c2337a70 f6ba2ab4 f72eae00 00000202 c2227fe0 f6ba29fc 00000202 
[  127.968446]        c2227fe0 f6ba29fc c2227560 00000003 0000028e c74a15d2 0000001d c2337a70 
[  127.977585] Call Trace:
[  127.980403]  [<c04081f1>] schedule_timeout+0x54/0xa5
[  127.985700]  [<c040817e>] io_schedule_timeout+0x29/0x33
[  127.991267]  [<c0287ce8>] blk_congestion_wait+0x70/0x85
[  127.996838]  [<c0141365>] throttle_vm_writeout+0x63/0x6d
[  128.002496]  [<c014712d>] shrink_zone+0xe0/0xfa
[  128.007341]  [<c01471b4>] shrink_caches+0x6d/0x6f
[  128.012368]  [<c0147286>] try_to_free_pages+0xd0/0x1b5
[  128.017847]  [<c013fa4b>] __alloc_pages+0x135/0x2e8
[  128.023054]  [<c0142d15>] kmem_getpages+0x34/0x9d
[  128.028075]  [<c0143b71>] cache_grow+0xaf/0x16c
[  128.032919]  [<c0143d92>] cache_alloc_refill+0x164/0x207
[  128.038584]  [<c014401b>] kmem_cache_alloc+0x4e/0x50
[  128.043887]  [<c01c4ea9>] nfs_create_request+0x3d/0xcc
[  128.049364]  [<c01c9225>] nfs_update_request+0x172/0x1d9
[  128.055023]  [<c01c8734>] nfs_writepage_async+0x36/0x99
[  128.060589]  [<c01c8960>] nfs_writepage+0x1a8/0x1df
[  128.065796]  [<c017dd7b>] mpage_writepages+0x232/0x399
[  128.071270]  [<c01c89c5>] nfs_writepages+0x2e/0x11e
[  128.076475]  [<c01416da>] do_writepages+0x26/0x44
[  128.081502]  [<c013abd3>] __filemap_fdatawrite_range+0x97/0x99
[  128.087704]  [<c013ac0e>] filemap_fdatawrite+0x39/0x3d
[  128.093180]  [<c01c234a>] nfs_revalidate_mapping+0xd5/0x108
[  128.099110]  [<c01bfdbb>] nfs_file_write+0x76/0x115
[  128.104316]  [<c01592e7>] do_sync_write+0xa7/0xe7
[  128.109338]  [<c01593cd>] vfs_write+0xa6/0x17d
[  128.114091]  [<c015968c>] sys_pwrite64+0x7d/0x81
[  128.119025]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  241.896500] SysRq : Show Memory
[  241.899856] Mem-info:
[  241.902261] DMA per-cpu:
[  241.904981] cpu 0 hot: low 0, high 0, batch 1 used:0
[  241.910190] cpu 0 cold: low 0, high 0, batch 1 used:0
[  241.915487] cpu 1 hot: low 0, high 0, batch 1 used:0
[  241.920696] cpu 1 cold: low 0, high 0, batch 1 used:0
[  241.925995] cpu 2 hot: low 0, high 0, batch 1 used:0
[  241.931204] cpu 2 cold: low 0, high 0, batch 1 used:0
[  241.936505] cpu 3 hot: low 0, high 0, batch 1 used:0
[  241.941711] cpu 3 cold: low 0, high 0, batch 1 used:0
[  241.947009] DMA32 per-cpu: empty
[  241.950451] Normal per-cpu:
[  241.953442] cpu 0 hot: low 0, high 186, batch 31 used:111
[  241.959102] cpu 0 cold: low 0, high 62, batch 15 used:61
[  241.964670] cpu 1 hot: low 0, high 186, batch 31 used:73
[  241.970237] cpu 1 cold: low 0, high 62, batch 15 used:0
[  241.975715] cpu 2 hot: low 0, high 186, batch 31 used:171
[  241.981375] cpu 2 cold: low 0, high 62, batch 15 used:14
[  241.986940] cpu 3 hot: low 0, high 186, batch 31 used:30
[  241.992508] cpu 3 cold: low 0, high 62, batch 15 used:47
[  241.998078] HighMem per-cpu:
[  242.001158] cpu 0 hot: low 0, high 186, batch 31 used:38
[  242.006726] cpu 0 cold: low 0, high 62, batch 15 used:55
[  242.012290] cpu 1 hot: low 0, high 186, batch 31 used:20
[  242.017844] cpu 1 cold: low 0, high 62, batch 15 used:14
[  242.023415] cpu 2 hot: low 0, high 186, batch 31 used:174
[  242.029074] cpu 2 cold: low 0, high 62, batch 15 used:4
[  242.034554] cpu 3 hot: low 0, high 186, batch 31 used:17
[  242.040125] cpu 3 cold: low 0, high 62, batch 15 used:46
[  242.045700] Free pages:        9540kB (2928kB HighMem)
[  242.051089] Active:462754 inactive:34324 dirty:115243 writeback:288948 unstable:0 free:2385 slab:15137 mapped:9203 pagetables:100
[  242.063268] DMA free:3576kB min:68kB low:84kB high:100kB active:2708kB inactive:384kB present:16384kB pages_scanned:0 all_unreclaimable? no
[  242.076355] lowmem_reserve[]: 0 0 880 2031
[  242.080878] DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
[  242.092696] lowmem_reserve[]: 0 0 880 2031
[  242.097224] Normal free:3036kB min:3756kB low:4692kB high:5632kB active:739412kB inactive:74548kB present:901120kB pages_scanned:96 all_unreclaimable? no
[  242.111429] lowmem_reserve[]: 0 0 0 9211
[  242.115718] HighMem free:2928kB min:512kB low:1740kB high:2968kB active:1108896kB inactive:62364kB present:1179092kB pages_scanned:1055 all_unreclaimable? no
[  242.130242] lowmem_reserve[]: 0 0 0 0
[  242.134261] DMA: 2*4kB 0*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 3576kB
[  242.145096] DMA32: empty
[  242.147801] Normal: 1*4kB 1*8kB 15*16kB 1*32kB 1*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 3036kB
[  242.158998] HighMem: 210*4kB 49*8kB 30*16kB 14*32kB 4*64kB 2*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2928kB
[  242.170651] Swap cache: add 0, delete 0, find 0/0, race 0+0
[  242.176496] Free swap  = 4200956kB
[  242.180057] Total swap = 4200956kB
[  242.183636] Free swap:       4200956kB
[  242.196081] 524149 pages of RAM
[  242.199394] 294773 pages of HIGHMEM
[  242.203064] 6122 reserved pages
[  242.206376] 345487 pages shared
[  242.209683] 0 pages swap cached
[  242.212992] 115243 pages dirty
[  242.216210] 288948 pages writeback
[  242.219788] 9203 pages mapped
[  242.222918] 15137 pages slab
[  242.225956] 100 pages pagetables

Re: nfs unhappiness with memory pressure

[Trond Myklebust]

On Mon, 2005-12-05 at 10:01 -0800, Kenny Simpson wrote:
> Tested with rc5 - same results.  It was suggested that I run slabtop when the system freezed, so
> here is that info: (again, by hand, I'm getting another machine to use either netconsole or a
> serial cable).
> 
>  Active / Total Objects (% used)    : 478764 / 485383 (98.6%)
>  Active / Total Slabs (% used)      : 14618 / 14635 (99.9%)
>  Active / Total Caches (% used)     : 79 / 138 (57.2%)
>  Active / Total Size (% used)       : 56663.79K / 57566.41K (98.4%)
>  Minimum / Average / Maximum Object : 0.01K / 0.12K / 128.00K
> 
>   OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME                   
> 403088 403088 100%    0.06K   6832       59     27328K nfs_page
>  30380  30380 100%    0.50K   4340        7     17360K nfs_write_data
>  15134  15134 100%    0.27K   1081       14      4324K radix_tree_node
> ...
> 
> 
> The other thing is that the stack trace showsd slabtop as being halted in throttle_vm_writeout
> while allocating memory, and the writetest was halted waiting to allocate memory.

Can somebody VM-savvy please explain how on earth they expect something
like throttle_vm_writeout() to make progress? Shouldn't that thing at
the very least be kicking pdflush every time it loops?

Cheers,
  Trond

> I'll get more detailed stack traces once I get the second machine set up.
> 
> -Kenny
> 
> 
> 
> 		
> __________________________________ 
> Start your day with Yahoo! - Make it your home page! 
> http://www.yahoo.com/r/hs
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: nfs unhappiness with memory pressure

[Kenny Simpson]

[-- Attachment #1: Type: text/plain, Size: 327 bytes --]

Tested with rc5 + the ALL patch from http://linux-nfs.org/Linux-2.6.x/2.6.15-rc5/
 - same results.

I'm attaching the sysrq output from that run (no slabtop this time, just the test).

-Kenny


		
__________________________________________ 
Yahoo! DSL – Something to write home about. 
Just $16.99/mo. or less. 
dsl.yahoo.com 

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 3165733884-sysrq.log --]
[-- Type: text/x-log; name="sysrq.log", Size: 58018 bytes --]

[  113.760934] SysRq : Show Regs
[  113.764100] 
[  113.765684] Pid: 0, comm:              swapper
[  113.770347] EIP: 0060:[<c0100d4f>] CPU: 0
[  113.774562] EIP is at default_idle+0x36/0x56
[  113.779041]  EFLAGS: 00000246    Not tainted  (2.6.15-rc5+nfs+k2)
[  113.785428] EAX: 00000000 EBX: c0503000 ECX: c220e2e0 EDX: c0503000
[  113.791991] ESI: c053c380 EDI: c053c300 EBP: c0503fac DS: 007b ES: 007b
[  113.798964] CR0: 8005003b CR2: b7f10000 CR3: 37396000 CR4: 000006d0
[  118.286611] SysRq : Show State
[  118.289876] 
[  118.289878]                                                sibling
[  118.297851]   task             PC      pid father child younger older
[  118.304515] init          D 00000001     0     1      0     2               (NOTLB)
[  118.312833] c2331ce8 c2331cd8 00000004 00000001 c019af87 c2331c9c c015c8ad f7b16614 
[  118.321014]        00000000 c2330550 0001ba80 00000000 00000202 c2217fe0 c2331cfc 00000202 
[  118.330108]        c2217fe0 c2331cfc c2217560 00000001 0000024d 8cbc3672 0000001b c2330550 
[  118.339197] Call Trace:
[  118.342000]  [<c04081f1>] schedule_timeout+0x54/0xa5
[  118.347270]  [<c040817e>] io_schedule_timeout+0x29/0x33
[  118.352806]  [<c0287ce8>] blk_congestion_wait+0x70/0x85
[  118.358346]  [<c0141365>] throttle_vm_writeout+0x63/0x6d
[  118.364007]  [<c014712d>] shrink_zone+0xe0/0xfa
[  118.368851]  [<c01471b4>] shrink_caches+0x6d/0x6f
[  118.373846]  [<c0147286>] try_to_free_pages+0xd0/0x1b5
[  118.379293]  [<c013fa4b>] __alloc_pages+0x135/0x2e8
[  118.384470]  [<c013fc2a>] __get_free_pages+0x2c/0x4f
[  118.389735]  [<c016c541>] __pollwait+0x3a/0xae
[  118.394445]  [<c0165ff6>] pipe_poll+0x36/0xad
[  118.399057]  [<c016c8b1>] do_select+0x217/0x26e
[  118.403846]  [<c016cba9>] sys_select+0x26a/0x3f7
[  118.408764]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  118.414148] migration/0   S 00000002     0     2      1             3       (L-TLB)
[  118.422468] c2339f94 c2339f84 00000004 00000002 00000082 00000046 c053c98c c2339f50 
[  118.430687]        c0115555 c2337a70 00000003 c2227560 c2339f60 c01103a3 00000008 000000fc 
[  118.439817]        f69a3030 f72a6f34 c220f560 00000000 00001117 51095e93 0000000e c048db40 
[  118.448946] Call Trace:
[  118.451763]  [<c0118d0c>] migration_thread+0x81/0x10d
[  118.457121]  [<c012fe68>] kthread+0xb7/0xbc
[  118.461579]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  118.467026] ksoftirqd/0   S 00000002     0     3      1             4     2 (L-TLB)
[  118.475349] c233af98 c233af88 00000004 00000002 00000524 000e11a4 00000000 c233ba70 
[  118.483534]        c2337030 c2357a70 00000001 c2331f5c c2330a70 ffffffff c2337030 c233af6c 
[  118.492617]        00000000 00000000 c220f560 00000000 0000017a ee4ad1de 00000003 c048db40 
[  118.501707] Call Trace:
[  118.504514]  [<c0121786>] ksoftirqd+0xd6/0xf5
[  118.509151]  [<c012fe68>] kthread+0xb7/0xbc
[  118.513633]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  118.519108] migration/1   S 00000001     0     4      1             5     3 (L-TLB)
[  118.527437] c233cf94 c233cf80 00000004 00000001 00000082 f7a2fa70 c220fa20 00000000 
[  118.535664]        c048db40 c2337a70 00000003 c2227560 f78ada70 0000000b 00000008 f78ada70 
[  118.544800]        00000000 c2217ec0 c2217560 00000001 0000141d 86935902 0000000b f78ada70 
[  118.553937] Call Trace:
[  118.556755]  [<c0118d0c>] migration_thread+0x81/0x10d
[  118.562143]  [<c012fe68>] kthread+0xb7/0xbc
[  118.566624]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  118.572100] ksoftirqd/1   S 00000001     0     5      1             6     4 (L-TLB)
[  118.580426] c2345f98 c2345f88 00000004 00000001 000000cf 0017cd16 00000000 c233b550 
[  118.588650]        c233b550 c2361a70 00000001 c2331f28 c2345000 ffffffff c233b550 c2345f6c 
[  118.597792]        c04079e4 c22175a8 c2217560 00000001 00000168 416f839e 00000003 c2330550 
[  118.606928] Call Trace:
[  118.609744]  [<c0121786>] ksoftirqd+0xd6/0xf5
[  118.614406]  [<c012fe68>] kthread+0xb7/0xbc
[  118.618890]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  118.624364] migration/2   S 00000004     0     6      1             7     5 (L-TLB)
[  118.632689] c2346f94 c2346f80 00000004 00000004 00000082 c2330550 c22175a8 00000000 
[  118.640914]        c0115555 f71c3550 f7a85500 7f3c75cc f70c4a70 00000006 c2346f70 f70c4a70 
[  118.650052]        00000000 c221fec0 c221f560 00000002 00002dce 7f3c9134 00000006 f70c4a70 
[  118.659188] Call Trace:
[  118.662005]  [<c0118d0c>] migration_thread+0x81/0x10d
[  118.667393]  [<c012fe68>] kthread+0xb7/0xbc
[  118.671875]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  118.677351] ksoftirqd/2   S 00000002     0     7      1             8     6 (L-TLB)
[  118.685675] c234df98 c234ca70 c221f560 00000002 0000016e 00273899 00000000 c234ca70 
[  118.693896]        c234ca70 c234cb9c 00000001 c2331f28 c234d000 ffffffff c234ca70 c234df6c 
[  118.703032]        c04079e4 c221f5a8 c221f560 00000002 00000044 00273afd 00000000 c2330030 
[  118.712166] Call Trace:
[  118.714981]  [<c0121786>] ksoftirqd+0xd6/0xf5
[  118.719644]  [<c012fe68>] kthread+0xb7/0xbc
[  118.724126]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  118.729603] migration/3   S 00000003     0     8      1             9     7 (L-TLB)
[  118.737929] c234ef94 c234ef84 00000004 00000003 00000082 00000046 c053c98c c234ef50 
[  118.746155]        c0115555 c2330030 00000002 c221f560 c234ef60 c01103a3 00000004 000000fc 
[  118.755290]        f7f6da70 f7fdaf34 c2227560 00000003 0000189d 7f7b2f10 00000006 c2337a70 
[  118.764420] Call Trace:
[  118.767238]  [<c0118d0c>] migration_thread+0x81/0x10d
[  118.772623]  [<c012fe68>] kthread+0xb7/0xbc
[  118.777104]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  118.782581] ksoftirqd/3   S 00000003     0     9      1            10     8 (L-TLB)
[  118.790910] c2353f98 c234c030 c2227560 00000003 000000c7 00365820 00000000 c234c030 
[  118.799118]        c234c030 c234c15c 00000001 c2331f28 c2353000 ffffffff c234c030 c2353f6c 
[  118.808206]        c04079e4 c22275a8 c2227560 00000003 0000002b 003659aa 00000000 c2337a70 
[  118.817290] Call Trace:
[  118.820092]  [<c0121786>] ksoftirqd+0xd6/0xf5
[  118.824731]  [<c012fe68>] kthread+0xb7/0xbc
[  118.829189]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  118.834635] events/0      R running     0    10      1            11     9 (L-TLB)
[  118.842866] events/1      S 00000001     0    11      1            12    10 (L-TLB)
[  118.851194] c236af34 c236af24 00000004 00000001 00000202 00000001 c2354540 000007d0 
[  118.859417]        c236aef4 c012bb5c c22191b8 fffd4636 c2300790 c2300780 00000001 c236aefc 
[  118.868552]        c2348930 c2348928 c2217560 00000001 0000073e 65123bf2 0000001b c2330550 
[  118.877693] Call Trace:
[  118.880516]  [<c012bd90>] worker_thread+0x207/0x225
[  118.885721]  [<c012fe68>] kthread+0xb7/0xbc
[  118.890203]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  118.895681] events/2      S 00000004     0    12      1            13    11 (L-TLB)
[  118.904004] c2359f34 c2359f24 00000004 00000004 00000202 00000001 c2354540 000007d0 
[  118.912230]        c2359ef4 c012bb5c c22211b8 fffd4639 00000000 c2300780 c22211a4 c2359efc 
[  118.921365]        c23488b0 c23488a8 c221f560 00000002 000007bc 65400e9b 0000001b c2330030 
[  118.930504] Call Trace:
[  118.933322]  [<c012bd90>] worker_thread+0x207/0x225
[  118.938527]  [<c012fe68>] kthread+0xb7/0xbc
[  118.943009]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  118.948483] events/3      S 00000003     0    13      1            14    12 (L-TLB)
[  118.957618] c236cf34 c236cf24 00000004 00000003 00000202 00000001 c2354540 000007d0 
[  118.965798]        c236cef4 c012bb5c c22291b8 fffd4641 00000000 c2300780 c22291a4 c236cefc 
[  118.974889]        c2348830 c2348828 c2227560 00000003 000005e8 65b9eb0f 0000001b c2337a70 
[  118.983975] Call Trace:
[  118.986779]  [<c012bd90>] worker_thread+0x207/0x225
[  118.991957]  [<c012fe68>] kthread+0xb7/0xbc
[  118.996425]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  119.001902] khelper       S 00000003     0    14      1            15    13 (L-TLB)
[  119.010229] c235af34 c235af20 00000004 00000003 00000000 c235aef4 c0117b5d f738a480 
[  119.018453]        00000003 00000000 00000000 f78f5d58 f7f78030 f78f5d50 00000202 f7f78030 
[  119.027583]        00000000 c2227ec0 c2227560 00000003 00000366 7ea3b727 00000006 f7f78030 
[  119.036714] Call Trace:
[  119.039532]  [<c012bd90>] worker_thread+0x207/0x225
[  119.044736]  [<c012fe68>] kthread+0xb7/0xbc
[  119.049216]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  119.054685] kthread       S 00000002     0    15      1    20     144    14 (L-TLB)
[  119.063004] c2389f34 c2389f24 00000004 00000002 c2389ef0 c0117b5d f7118e88 00000003 
[  119.071199]        00000000 00000000 f7118f28 f7118f1c f7118f20 00000202 c2389f18 c0117cbb 
[  119.080286]        c2380b30 c2380b28 c220f560 00000000 00000199 a04f2a43 00000005 c048db40 
[  119.089371] Call Trace:
[  119.092172]  [<c012bd90>] worker_thread+0x207/0x225
[  119.097349]  [<c012fe68>] kthread+0xb7/0xbc
[  119.101833]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  119.107311] kblockd/0     S 00000002     0    20     15            21       (L-TLB)
[  119.115635] c238bf34 c238bf20 00000004 00000002 f6b2f75c 00c4d18d 00000000 00000000 
[  119.123846]        c2330550 f6b2f75c c238bee8 c0408b7a f78ada70 0000001e c0579414 f78ada70 
[  119.132934]        00000000 c220fec0 c220f560 00000000 00000c67 6c8d9dc5 0000000a f78ada70 
[  119.142040] Call Trace:
[  119.144857]  [<c012bd90>] worker_thread+0x207/0x225
[  119.150062]  [<c012fe68>] kthread+0xb7/0xbc
[  119.154544]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  119.160012] kblockd/1     S 00000001     0    21     15            22    20 (L-TLB)
[  119.168333] f7d58f34 f7d58f24 00000004 00000001 f7d58ed0 c0408b7a f7d58ef8 c02fc8e6 
[  119.176564]        c0579414 f7ae757c f7d58ee8 00000000 c23eef00 f7f5e4f4 c23eef00 f7f5e59c 
[  119.185700]        c23eef30 c23eef28 c2217560 00000001 00000fee 4ec7aa5e 00000010 c2330550 
[  119.194835] Call Trace:
[  119.197653]  [<c012bd90>] worker_thread+0x207/0x225
[  119.202857]  [<c012fe68>] kthread+0xb7/0xbc
[  119.207339]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  119.212859] kblockd/2     S 00000004     0    22     15            23    21 (L-TLB)
[  119.221188] c238ff34 c238ff20 00000004 00000004 f7a3639c 00d8d14d 00000000 00000000 
[  119.229322]        c2337a70 f7a3639c c238fee8 c0408b7a f710ca70 00000011 c0579414 f710ca70 
[  119.238350]        00000000 c221fec0 c221f560 00000002 00000e51 740a975b 00000006 f710ca70 
[  119.247384] Call Trace:
[  119.250167]  [<c012bd90>] worker_thread+0x207/0x225
[  119.255309]  [<c012fe68>] kthread+0xb7/0xbc
[  119.259738]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  119.265147] kblockd/3     S 00000003     0    23     15            24    22 (L-TLB)
[  119.273384] f7d59f34 f7d59f20 00000004 00000003 f71c089c 00c6abb5 00000000 00000000 
[  119.281565]        c2330030 f71c089c f7d59ee8 c0408b7a f7f6d550 00000023 c0579414 f7f6d550 
[  119.290653]        00000000 c2227ec0 c2227560 00000003 00000cf4 6adbe5b8 00000006 f7f6d550 
[  119.299736] Call Trace:
[  119.302538]  [<c012bd90>] worker_thread+0x207/0x225
[  119.307714]  [<c012fe68>] kthread+0xb7/0xbc
[  119.312197]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  119.317672] kacpid        S 00000002     0    24     15           142    23 (L-TLB)
[  119.325995] f7d78f34 f7d78f24 00000004 00000002 00000000 00000001 00000100 00000000 
[  119.334223]        00000080 00000080 00000080 c05566ac c05566ac c05566a0 c2217ec0 c2217ec0 
[  119.343324]        00000000 00000202 c220f560 00000000 000001b0 027f6e23 00000000 c048db40 
[  119.352412] Call Trace:
[  119.355213]  [<c012bd90>] worker_thread+0x207/0x225
[  119.360390]  [<c012fe68>] kthread+0xb7/0xbc
[  119.364853]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  119.370298] pdflush       S 00000002     0   142     15           143    24 (L-TLB)
[  119.378620] f7d5df68 f7d5df58 00000004 00000002 c0124c50 c0493a5c fffd4362 f7d5df68 
[  119.386801]        c014152c c0493a5c fffd4362 fffcbaaa 00000000 00000000 f7d5df18 00000400 
[  119.395887]        00000000 00000000 c220f560 00000000 00000278 874823b0 0000001a c048db40 
[  119.404974] Call Trace:
[  119.407777]  [<c0141dd8>] __pdflush+0x84/0x199
[  119.412506]  [<c0141f2d>] pdflush+0x40/0x47
[  119.416990]  [<c012fe68>] kthread+0xb7/0xbc
[  119.421475]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  119.426921] pdflush       D 00000002     0   143     15           145   142 (L-TLB)
[  119.435248] f7da0cb4 f7da0ca4 00000004 00000002 00000202 f6a81600 f7b16600 f7e36c40 
[  119.443392]        f697d01c c21645ac c21645ac 00000000 00000000 f6990d44 00000009 00000296 
[  119.452416]        00000000 f6990d44 c220f560 00000000 000006f6 b611579f 00000010 c048db40 
[  119.461441] Call Trace:
[  119.464225]  [<c040814b>] io_schedule+0x26/0x30
[  119.469009]  [<c013ab27>] sync_page+0x39/0x4e
[  119.473619]  [<c0408439>] __wait_on_bit_lock+0x58/0x67
[  119.479027]  [<c013b343>] __lock_page+0x84/0x8c
[  119.483812]  [<c017de94>] mpage_writepages+0x34b/0x399
[  119.489224]  [<c01c89c5>] nfs_writepages+0x2e/0x11e
[  119.494366]  [<c01416da>] do_writepages+0x26/0x44
[  119.499331]  [<c017c052>] __sync_single_inode+0x60/0x1f1
[  119.504919]  [<c017c275>] __writeback_single_inode+0x92/0x195
[  119.510957]  [<c017c535>] sync_sb_inodes+0x1bd/0x2c2
[  119.516189]  [<c017c714>] writeback_inodes+0xda/0xe6
[  119.521482]  [<c01413d8>] background_writeout+0x69/0xa2
[  119.527018]  [<c0141e15>] __pdflush+0xc1/0x199
[  119.531746]  [<c0141f2d>] pdflush+0x40/0x47
[  119.536204]  [<c012fe68>] kthread+0xb7/0xbc
[  119.540660]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  119.546104] kswapd0       D 00000002     0   144      1           804    15 (L-TLB)
[  119.554429] f7d6ac94 f7d6ac84 00000004 00000002 00011200 00000000 f72178d0 00000000 
[  119.562655]        00000003 00011200 f7e41580 0000001b 00011200 f7d6ac7c c0143d92 f7e3bb00 
[  119.571786]        00011200 00000000 c220f560 00000000 000037a5 fa8e7173 00000010 c048db40 
[  119.580874] Call Trace:
[  119.583663]  [<c040814b>] io_schedule+0x26/0x30
[  119.588452]  [<c013e464>] mempool_alloc+0xcf/0xd1
[  119.593478]  [<c01c9a4f>] nfs_flush_one+0x54/0x17f
[  119.598592]  [<c01c9bcf>] nfs_flush_list+0x55/0xa3
[  119.603705]  [<c01ca516>] nfs_flush_inode+0x82/0xad
[  119.608910]  [<c01c8992>] nfs_writepage+0x1da/0x1df
[  119.614118]  [<c01464f6>] pageout+0xb5/0x137
[  119.618688]  [<c0146762>] shrink_list+0x1ea/0x40c
[  119.623716]  [<c0146b20>] shrink_cache+0xf8/0x281
[  119.628739]  [<c0147118>] shrink_zone+0xcb/0xfa
[  119.633583]  [<c01475f1>] balance_pgdat+0x286/0x37e
[  119.638789]  [<c01477ce>] kswapd+0xe5/0x10f
[  119.643269]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  119.648744] aio/0         S 00000002     0   145     15           146   143 (L-TLB)
[  119.657070] f7de5f34 f7de5f20 00000004 00000002 00000000 00000001 00000100 00000000 
[  119.665275]        00000080 00000080 00000080 c05566ac f7d69550 c05566a0 c2217ec0 f7d69550 
[  119.674361]        00000000 c220fec0 c220f560 00000000 0000013a 2ca82088 00000000 f7d69550 
[  119.683454] Call Trace:
[  119.686268]  [<c012bd90>] worker_thread+0x207/0x225
[  119.691443]  [<c012fe68>] kthread+0xb7/0xbc
[  119.695901]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  119.701347] aio/1         S 00000000     0   146     15           147   145 (L-TLB)
[  119.709675] f7e2ef34 c2361030 c22175a8 00000000 00000000 00000001 00000100 00000000 
[  119.717857]        00000080 00000000 2ca651d4 00000000 c2361030 00000000 c220fec0 c2361030 
[  119.726992]        00000000 c2217ec0 c2217560 00000001 0000015d 2ca85df2 00000000 c2361030 
[  119.736129] Call Trace:
[  119.738947]  [<c012bd90>] worker_thread+0x207/0x225
[  119.744155]  [<c012fe68>] kthread+0xb7/0xbc
[  119.748615]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  119.754064] aio/2         S 00000004     0   147     15           148   146 (L-TLB)
[  119.763205] f7de6f34 f7de6f24 00000004 00000004 00000000 00000001 00000100 00000000 
[  119.771387]        00000080 00000080 00000080 c05566ac c05566ac c05566a0 c2217ec0 c2217ec0 
[  119.780471]        00000000 00000202 c221f560 00000002 00000197 2ca8baf9 00000000 c2330030 
[  119.789558] Call Trace:
[  119.792362]  [<c012bd90>] worker_thread+0x207/0x225
[  119.797539]  [<c012fe68>] kthread+0xb7/0xbc
[  119.801997]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  119.807456] aio/3         S 00000003     0   148     15           149   147 (L-TLB)
[  119.815784] f7e2ff34 f7e2ff24 00000004 00000003 00000000 00000001 00000100 00000080 
[  119.824011]        00000080 00000000 00000000 c05566a0 c05566a0 00000000 c220fec0 c220fec0 
[  119.833140]        00000000 00000202 c2227560 00000003 000001b1 2ca91759 00000000 c2337a70 
[  119.842274] Call Trace:
[  119.845089]  [<c012bd90>] worker_thread+0x207/0x225
[  119.850269]  [<c012fe68>] kthread+0xb7/0xbc
[  119.854726]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  119.860172] xfslogd/0     S 00000000     0   149     15           150   148 (L-TLB)
[  119.868500] f7de8f34 c2361030 c220fa20 00000000 00000000 00000001 00000100 00000000 
[  119.876682]        00000100 00000080 2e4b9798 00000000 c2361030 c05566a0 c2217ec0 c2361030 
[  119.885793]        00000000 c220fec0 c220f560 00000000 00000178 2e4c3ca3 00000000 c2361030 
[  119.894880] Call Trace:
[  119.897691]  [<c012bd90>] worker_thread+0x207/0x225
[  119.902852]  [<c012fe68>] kthread+0xb7/0xbc
[  119.907306]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  119.912751] xfslogd/1     S 00000001     0   150     15           151   149 (L-TLB)
[  119.921081] f7e69f34 f7e69f20 00000004 00000001 00000000 00000001 00000100 00000000 
[  119.929264]        00000100 00000080 00000080 c05566a0 f7de7550 c05566ac c220fec0 f7de7550 
[  119.938334]        00000000 c2217ec0 c2217560 00000001 00000101 2e4cc2be 00000000 f7de7550 
[  119.947360] Call Trace:
[  119.950178]  [<c012bd90>] worker_thread+0x207/0x225
[  119.955347]  [<c012fe68>] kthread+0xb7/0xbc
[  119.959779]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  119.965193] xfslogd/2     S 00000004     0   151     15           152   150 (L-TLB)
[  119.973515] f7de9f34 f7de9f24 00000004 00000004 00000000 00000001 00000100 00000080 
[  119.981740]        00000100 00000080 00000080 c05566ac c05566ac c05566a0 c2217ec0 c2217ec0 
[  119.990868]        00000000 00000202 c221f560 00000002 00000157 2e4d058a 00000000 c2330030 
[  119.999998] Call Trace:
[  120.002814]  [<c012bd90>] worker_thread+0x207/0x225
[  120.008021]  [<c012fe68>] kthread+0xb7/0xbc
[  120.012504]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  120.017980] xfslogd/3     S 00000003     0   152     15           153   151 (L-TLB)
[  120.026309] f7e6af34 f7e6af24 00000004 00000003 00000000 00000001 00000100 00000080 
[  120.034534]        00000100 00000080 00000080 c05566a0 c05566a0 c05566ac c220fec0 c220fec0 
[  120.043675]        00000000 00000202 c2227560 00000003 00000159 2e4d5ea8 00000000 c2337a70 
[  120.052807] Call Trace:
[  120.055622]  [<c012bd90>] worker_thread+0x207/0x225
[  120.060823]  [<c012fe68>] kthread+0xb7/0xbc
[  120.065302]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  120.070778] xfsdatad/0    S 00000000     0   153     15           154   152 (L-TLB)
[  120.079100] f7debf34 c2361030 c220fa20 00000000 00000000 00000001 00000100 00000000 
[  120.087322]        00000100 00000080 2e4d19a1 00000000 c2361030 c05566a0 c2217ec0 c2361030 
[  120.096451]        00000000 c220fec0 c220f560 00000000 00000138 2e4dbd90 00000000 c2361030 
[  120.105585] Call Trace:
[  120.108402]  [<c012bd90>] worker_thread+0x207/0x225
[  120.113604]  [<c012fe68>] kthread+0xb7/0xbc
[  120.118088]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  120.123534] xfsdatad/1    S 00000001     0   154     15           155   153 (L-TLB)
[  120.131861] f7e6cf34 f7e6cf20 00000004 00000001 00000000 00000001 00000100 00000000 
[  120.140066]        00000100 00000080 00000080 c05566a0 f7deca70 c05566ac c220fec0 f7deca70 
[  120.149151]        00000000 c2217ec0 c2217560 00000001 000000f1 2e4e3213 00000000 f7deca70 
[  120.158238] Call Trace:
[  120.161040]  [<c012bd90>] worker_thread+0x207/0x225
[  120.166219]  [<c012fe68>] kthread+0xb7/0xbc
[  120.170678]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  120.176124] xfsdatad/2    S 00000004     0   155     15           156   154 (L-TLB)
[  120.184450] f7dedf34 f7dedf24 00000004 00000004 00000000 00000001 00000100 00000080 
[  120.192635]        00000100 00000080 00000080 c05566ac c05566ac c05566a0 c2217ec0 c2217ec0 
[  120.201725]        00000000 00000202 c221f560 00000002 00000139 2e4e7238 00000000 c2330030 
[  120.210810] Call Trace:
[  120.213612]  [<c012bd90>] worker_thread+0x207/0x225
[  120.218769]  [<c012fe68>] kthread+0xb7/0xbc
[  120.223200]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  120.228617] xfsdatad/3    S 00000003     0   156     15           157   155 (L-TLB)
[  120.236943] f7e6ef34 f7e6ef24 00000004 00000003 00000000 00000001 00000100 00000080 
[  120.245171]        00000100 00000080 00000080 c05566a0 c05566a0 c05566ac c220fec0 c220fec0 
[  120.254311]        00000000 00000202 c2227560 00000003 00000133 2e4ec9a0 00000000 c2337a70 
[  120.263451] Call Trace:
[  120.266263]  [<c012bd90>] worker_thread+0x207/0x225
[  120.271467]  [<c012fe68>] kthread+0xb7/0xbc
[  120.275950]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  120.281426] xfsbufd       S 00000001     0   157     15           742   156 (L-TLB)
[  120.289751] f7deef44 f7deef34 00000004 00000001 00000080 00000000 00000000 00000000 
[  120.297976]        f70c4a70 c2330550 bc32a897 00000010 00000202 c2217fe0 f7deef58 00000202 
[  120.307109]        c2217fe0 f7deef58 c2217560 00000001 000000ab e6f6a233 0000001b c2330550 
[  120.316244] Call Trace:
[  120.319062]  [<c04081f1>] schedule_timeout+0x54/0xa5
[  120.324358]  [<c040825c>] schedule_timeout_interruptible+0x1a/0x1c
[  120.330918]  [<c0272378>] xfsbufd+0x4e/0x1b0
[  120.335491]  [<c012fe68>] kthread+0xb7/0xbc
[  120.339948]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  120.345395] kseriod       S 00000002     0   742     15          1668   157 (L-TLB)
[  120.353720] f7f0ef5c f7f0ef4c 00000004 00000002 c02efb29 f8879da4 f7f0ef10 c02efab5 
[  120.361950]        f8879d88 c04a8738 f8879da4 c04a63e0 f7f0ef34 c02ef172 f8879d88 c04a8738 
[  120.371090]        00000282 00000202 c220f560 00000000 00000617 5a39f23c 00000001 c048db40 
[  120.380227] Call Trace:
[  120.383042]  [<c02e4386>] serio_thread+0xea/0xec
[  120.387981]  [<c012fe68>] kthread+0xb7/0xbc
[  120.392454]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  120.397869] kjournald     D 00000004     0   804      1          1128   144 (L-TLB)
[  120.406196] f7fd2bec f7fd2bdc 00000004 00000004 c015aca8 c17d490c 000001de 00000000 
[  120.414336]        f7fd2bcc c2330030 f7f9eb30 f7f9eb30 00000202 c221ffe0 f7fd2c00 00000202 
[  120.423377]        c221ffe0 f7fd2c00 c221f560 00000002 000000f3 04aeb268 0000001c c2330030 
[  120.432414] Call Trace:
[  120.435203]  [<c04081f1>] schedule_timeout+0x54/0xa5
[  120.440440]  [<c040817e>] io_schedule_timeout+0x29/0x33
[  120.445952]  [<c0287ce8>] blk_congestion_wait+0x70/0x85
[  120.451457]  [<c0141365>] throttle_vm_writeout+0x63/0x6d
[  120.457053]  [<c014712d>] shrink_zone+0xe0/0xfa
[  120.461841]  [<c01471b4>] shrink_caches+0x6d/0x6f
[  120.466808]  [<c0147286>] try_to_free_pages+0xd0/0x1b5
[  120.472225]  [<c013fa4b>] __alloc_pages+0x135/0x2e8
[  120.477372]  [<c013b51b>] find_or_create_page+0x83/0x97
[  120.482876]  [<c015b757>] grow_dev_page+0x38/0x15d
[  120.487935]  [<c015b934>] __getblk_slow+0xb8/0x124
[  120.492993]  [<c015bcfd>] __getblk+0x57/0x59
[  120.497513]  [<c01b1950>] journal_get_descriptor_buffer+0x4d/0xa4
[  120.503912]  [<c01ad79f>] journal_write_commit_record+0x38/0x116
[  120.510222]  [<c01ade1f>] journal_commit_transaction+0x5a2/0x127f
[  120.516629]  [<c01b0dba>] kjournald+0xc1/0x223
[  120.521330]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  120.526745] kjournald     S 00000001     0  1128      1          1129   804 (L-TLB)
[  120.535077] f7abff58 f7abff48 00000004 00000001 00000012 00000036 00000044 00000202 
[  120.543226]        5b3e363c 20202020 37362e35 37383934 f700205d 00000003 00000000 00000000 
[  120.552306]        f7b8ee6c f7b8ee64 c2217560 00000001 000001db 52ac0ded 00000001 c2330550 
[  120.561387] Call Trace:
[  120.565000]  [<c01b0ecf>] kjournald+0x1d6/0x223
[  120.569818]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  120.575264] kjournald     S 00000001     0  1129      1          1130  1128 (L-TLB)
[  120.583590] f7188f58 f7188f48 00000004 00000001 f7acd02c 00000000 00000000 f7186800 
[  120.591774]        00000000 f73f3300 f699847c 0000027f 00000202 c2217fe0 f7188fb8 f7188f2c 
[  120.600861]        f7b1646c f7b16464 c2217560 00000001 000002a8 7bc98d57 0000000f c2330550 
[  120.609992] Call Trace:
[  120.612812]  [<c01b0ecf>] kjournald+0x1d6/0x223
[  120.617657]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  120.623135] kjournald     D 00000001     0  1130      1          2763  1129 (L-TLB)
[  120.631461] f717ebec f717ebdc 00000004 00000001 c015aca8 c17d4a2c 00000207 f7f9e6ec 
[  120.639685]        f717ebcc c2330550 f7f9eea4 f7f9eea4 00000202 c2217fe0 f717ec00 00000202 
[  120.648821]        c2217fe0 f717ec00 c2217560 00000001 00000118 168c9d61 0000001c c2330550 
[  120.657955] Call Trace:
[  120.660774]  [<c04081f1>] schedule_timeout+0x54/0xa5
[  120.666070]  [<c040817e>] io_schedule_timeout+0x29/0x33
[  120.671638]  [<c0287ce8>] blk_congestion_wait+0x70/0x85
[  120.677207]  [<c0141365>] throttle_vm_writeout+0x63/0x6d
[  120.682866]  [<c014712d>] shrink_zone+0xe0/0xfa
[  120.687711]  [<c01471b4>] shrink_caches+0x6d/0x6f
[  120.692736]  [<c0147286>] try_to_free_pages+0xd0/0x1b5
[  120.698212]  [<c013fa4b>] __alloc_pages+0x135/0x2e8
[  120.703418]  [<c013b51b>] find_or_create_page+0x83/0x97
[  120.708984]  [<c015b757>] grow_dev_page+0x38/0x15d
[  120.714100]  [<c015b934>] __getblk_slow+0xb8/0x124
[  120.719216]  [<c015bcfd>] __getblk+0x57/0x59
[  120.723788]  [<c01b1950>] journal_get_descriptor_buffer+0x4d/0xa4
[  120.730261]  [<c01ad79f>] journal_write_commit_record+0x38/0x116
[  120.736643]  [<c01ade1f>] journal_commit_transaction+0x5a2/0x127f
[  120.743115]  [<c01b0dba>] kjournald+0xc1/0x223
[  120.747870]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  120.753341] khubd         S 00000001     0  1668     15          2771   742 (L-TLB)
[  120.761662] f7a72f60 f7a72f4c 00000004 00000001 000003e8 00000007 f7169180 f738a880 
[  120.769883]        f78d9550 f7369ab4 c0294056 f7a72f1c f7b38a70 00000007 00000003 f7b38a70 
[  120.779013]        00000000 c2217ec0 c2217560 00000001 00000f95 87e73598 00000002 f7b38a70 
[  120.788116] Call Trace:
[  120.790919]  [<f8b01331>] hub_thread+0xe3/0xe5 [usbcore]
[  120.796564]  [<c012fe68>] kthread+0xb7/0xbc
[  120.801022]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  120.806468] portmap       S 00000002     0  2763      1          2775  1130 (NOTLB)
[  120.814792] f7a1bf00 f7a1bef0 00000004 00000002 00000202 f7f8b00c 00000202 f7a1beb4 
[  120.822973]        c0130113 f7f8b000 f7eb0a80 00000202 f7f8b028 00000202 f7a1bed0 c0130113 
[  120.832059]        f7f8b000 f70ee780 c220f560 00000000 00001794 563b7335 00000006 c048db40 
[  120.841194] Call Trace:
[  120.844009]  [<c0408240>] schedule_timeout+0xa3/0xa5
[  120.849305]  [<c016ce68>] do_poll+0x9a/0xb9
[  120.853783]  [<c016cfdd>] sys_poll+0x156/0x221
[  120.858534]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  120.863917] rpciod/0      S 00000002     0  2771     15          2772  1668 (L-TLB)
[  120.872237] f7a51f34 f7a51f24 00000004 00000002 f7e48100 f7a51ee8 c013e4eb f7e48100 
[  120.880424]        f7e3bb80 f7e48104 f7e48104 f7e48180 f7a51ef8 c01c6bf5 f7e48100 f7e36940 
[  120.889512]        f7338eb0 f7338ea8 c220f560 00000000 00000697 5119c8a1 0000000e c048db40 
[  120.898600] Call Trace:
[  120.901402]  [<c012bd90>] worker_thread+0x207/0x225
[  120.906580]  [<c012fe68>] kthread+0xb7/0xbc
[  120.911036]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  120.916483] rpciod/1      S 00000004     0  2772     15          2773  2771 (L-TLB)
[  120.924807] f7a31f34 c2361030 c2217a20 00000004 00000000 00000001 00000100 00000000 
[  120.933019]        00000100 00000000 7acd8eb6 00000004 c2361030 00000000 c220fec0 c2361030 
[  120.942107]        00000000 c2217ec0 c2217560 00000001 00000185 7ad0bcbc 00000004 c2361030 
[  120.951200] Call Trace:
[  120.954000]  [<c012bd90>] worker_thread+0x207/0x225
[  120.959178]  [<c012fe68>] kthread+0xb7/0xbc
[  120.963634]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  120.969080] rpciod/2      D 00000004     0  2773     15          2774  2772 (L-TLB)
[  120.977405] f7b25ab4 f7b25aa4 00000004 00000004 f7249ea0 f7249e98 00000001 00000006 
[  120.985592]        f7b25a84 c2330030 f7249e98 00000003 00000202 c221ffe0 f7b25ac8 00000202 
[  120.994675]        c221ffe0 f7b25ac8 c221f560 00000002 00000103 2b55b05e 0000001c c2330030 
[  121.003762] Call Trace:
[  121.006564]  [<c04081f1>] schedule_timeout+0x54/0xa5
[  121.011833]  [<c040817e>] io_schedule_timeout+0x29/0x33
[  121.017370]  [<c0287ce8>] blk_congestion_wait+0x70/0x85
[  121.022906]  [<c0141365>] throttle_vm_writeout+0x63/0x6d
[  121.028535]  [<c014712d>] shrink_zone+0xe0/0xfa
[  121.033355]  [<c01471b4>] shrink_caches+0x6d/0x6f
[  121.038354]  [<c0147286>] try_to_free_pages+0xd0/0x1b5
[  121.043799]  [<c013fa4b>] __alloc_pages+0x135/0x2e8
[  121.048976]  [<c03b70d1>] tcp_sendmsg+0xaa0/0xb78
[  121.053981]  [<c03d428a>] inet_sendmsg+0x48/0x53
[  121.058887]  [<c038833a>] sock_sendmsg+0xb8/0xd3
[  121.063795]  [<c0388397>] kernel_sendmsg+0x42/0x4f
[  121.068882]  [<c038b824>] sock_no_sendpage+0x5e/0x77
[  121.074148]  [<c03ee3cb>] xs_tcp_send_request+0x2af/0x375
[  121.079866]  [<c03ed0e1>] xprt_transmit+0x4f/0x20b
[  121.084957]  [<c03eb8f7>] call_transmit+0x68/0xc8
[  121.089956]  [<c03f0625>] __rpc_execute+0x1cb/0x221
[  121.095103]  [<c03f06be>] rpc_async_schedule+0x11/0x15
[  121.100512]  [<c012bd1d>] worker_thread+0x194/0x225
[  121.105654]  [<c012fe68>] kthread+0xb7/0xbc
[  121.110082]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  121.115490] rpciod/3      S 00000003     0  2774     15                2773 (L-TLB)
[  121.123764] f7a43f34 f7a43f20 00000004 00000003 f69a5c80 f7a43ee8 c013e4eb 00000000 
[  121.131954]        c2330030 f69a5c84 f69a5c84 f69a5d00 f69a3030 0000000c f69a5c80 f69a3030 
[  121.141045]        00000000 c2227ec0 c2227560 00000003 000010bc 112b45c7 00000010 f69a3030 
[  121.150131] Call Trace:
[  121.152948]  [<c012bd90>] worker_thread+0x207/0x225
[  121.158158]  [<c012fe68>] kthread+0xb7/0xbc
[  121.162641]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  121.168087] lockd         S 00000002     0  2775      1          2842  2763 (L-TLB)
[  121.176413] f7230f14 f7230f04 00000004 00000002 f7249180 00000001 c2217560 f7230ecc 
[  121.184637]        c01103a3 00000002 000000fc f78f6030 c2217a20 f7230edc c0115796 00000202 
[  121.193775]        f73eaaec 00000202 c220f560 00000000 0000089a 7adc3744 00000004 c048db40 
[  121.202888] Call Trace:
[  121.205690]  [<c0408240>] schedule_timeout+0xa3/0xa5
[  121.210959]  [<c03f46d7>] svc_recv+0x387/0x4b9
[  121.215714]  [<c02045e6>] lockd+0x102/0x241
[  121.220200]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  121.225677] syslogd       D 00000002     0  2842      1          2845  2775 (NOTLB)
[  121.234001] f7ab7cdc f7ab7ccc 00000004 00000002 f7ab7cb8 c02880f3 f7f5e4f4 f6b2f89c 
[  121.242151]        00000003 c048db40 c21e0000 c0493400 00000202 c220ffe0 f7ab7cf0 00000202 
[  121.251176]        c220ffe0 f7ab7cf0 c220f560 00000000 0000015b 862f2374 0000001b c048db40 
[  121.260200] Call Trace:
[  121.262983]  [<c04081f1>] schedule_timeout+0x54/0xa5
[  121.268212]  [<c040817e>] io_schedule_timeout+0x29/0x33
[  121.273713]  [<c0287ce8>] blk_congestion_wait+0x70/0x85
[  121.279214]  [<c0141365>] throttle_vm_writeout+0x63/0x6d
[  121.284803]  [<c014712d>] shrink_zone+0xe0/0xfa
[  121.289602]  [<c01471b4>] shrink_caches+0x6d/0x6f
[  121.294569]  [<c0147286>] try_to_free_pages+0xd0/0x1b5
[  121.300011]  [<c013fa4b>] __alloc_pages+0x135/0x2e8
[  121.305187]  [<c013fc2a>] __get_free_pages+0x2c/0x4f
[  121.310462]  [<c016c541>] __pollwait+0x3a/0xae
[  121.315190]  [<c038fa46>] datagram_poll+0x2b/0xcd
[  121.320198]  [<c0388b2a>] sock_poll+0x23/0x2a
[  121.324862]  [<c016c8b1>] do_select+0x217/0x26e
[  121.329704]  [<c016cba9>] sys_select+0x26a/0x3f7
[  121.334639]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  121.340024] klogd         S 00000001     0  2845      1          2860  2842 (NOTLB)
[  121.348350] f7a29d4c f7a29d3c 00000004 00000001 f7a29d00 c0142d54 00000014 00000001 
[  121.356532]        f7b63000 c2330550 c2383480 c23834a8 f7a29d30 c0143c07 c2381880 f7b63000 
[  121.365620]        00000001 000000d0 c2217560 00000001 000010f2 8aa0dcf9 0000001b c2330550 
[  121.374709] Call Trace:
[  121.377510]  [<c0408240>] schedule_timeout+0xa3/0xa5
[  121.382808]  [<f88b00ee>] unix_wait_for_peer+0xb3/0xb7 [unix]
[  121.388931]  [<f88b0abb>] unix_dgram_sendmsg+0x266/0x502 [unix]
[  121.395188]  [<c03886da>] sock_aio_write+0xea/0x106
[  121.401176]  [<c01592e7>] do_sync_write+0xa7/0xe7
[  121.406172]  [<c0159495>] vfs_write+0x16e/0x17d
[  121.410988]  [<c0159564>] sys_write+0x4b/0x75
[  121.415624]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  121.420981] ypbind        S 00000002     0  2860      1          2861  2845 (NOTLB)
[  121.429305] f70baf00 f70baef0 00000004 00000002 00000202 f7b2600c 00000202 f70baeb4 
[  121.437486]        c0130113 f7b26000 f70ea480 00000202 f7b26028 00000202 f70baed0 c0130113 
[  121.446609]        f7b26000 f7ef4180 c220f560 00000000 0000121d 94ce6a4c 00000005 c048db40 
[  121.455744] Call Trace:
[  121.458561]  [<c0408240>] schedule_timeout+0xa3/0xa5
[  121.463832]  [<c016ce68>] do_poll+0x9a/0xb9
[  121.468299]  [<c016cfdd>] sys_poll+0x156/0x221
[  121.473027]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  121.478384] ypbind        S 00000001     0  2861      1          2862  2860 (NOTLB)
[  121.486700] f7a88eac f7a88e9c 00000004 00000001 00000069 00000000 01cd9f60 c220f560 
[  121.494884]        c2217ec0 00000001 00000001 00000001 00000096 00000000 f70baedc 00000000 
[  121.503971]        f7a88e84 c0117b1a c2217560 00000001 0001050f 5c5940ce 00000005 c2330550 
[  121.513062] Call Trace:
[  121.515865]  [<c0408240>] schedule_timeout+0xa3/0xa5
[  121.521134]  [<c040825c>] schedule_timeout_interruptible+0x1a/0x1c
[  121.527662]  [<c0128559>] sys_rt_sigtimedwait+0x1f1/0x2e9
[  121.533381]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  121.538737] ypbind        D 00000002     0  2862      1          3022  2861 (NOTLB)
[  121.547057] f7acecf8 f7acece4 00000004 00000002 f73ea680 f73ea680 f7aceeb8 00000000 
[  121.555241]        c2330550 c048db40 f6cc7e80 00000040 f69b7a70 00000011 f7aced0c f69b7a70 
[  121.564327]        00000000 c220fec0 c220f560 00000000 0000016e 8559b23d 0000001b f69b7a70 
[  121.573436] Call Trace:
[  121.576250]  [<c04081f1>] schedule_timeout+0x54/0xa5
[  121.581544]  [<c040817e>] io_schedule_timeout+0x29/0x33
[  121.587110]  [<c0287ce8>] blk_congestion_wait+0x70/0x85
[  121.592681]  [<c0141365>] throttle_vm_writeout+0x63/0x6d
[  121.598339]  [<c014712d>] shrink_zone+0xe0/0xfa
[  121.603182]  [<c01471b4>] shrink_caches+0x6d/0x6f
[  121.608206]  [<c0147286>] try_to_free_pages+0xd0/0x1b5
[  121.613683]  [<c013fa4b>] __alloc_pages+0x135/0x2e8
[  121.618886]  [<c013fc2a>] __get_free_pages+0x2c/0x4f
[  121.624182]  [<c016c541>] __pollwait+0x3a/0xae
[  121.628934]  [<c038fa46>] datagram_poll+0x2b/0xcd
[  121.633959]  [<c03ce0ae>] udp_poll+0x22/0xfc
[  121.638523]  [<c0388b2a>] sock_poll+0x23/0x2a
[  121.643129]  [<c016cdca>] do_pollfd+0x94/0x98
[  121.647736]  [<c016ce29>] do_poll+0x5b/0xb9
[  121.652162]  [<c016cfdd>] sys_poll+0x156/0x221
[  121.656861]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  121.662185] automount     S 00000002     0  3022      1          3030  2862 (NOTLB)
[  121.670459] f7b1cf00 f7b1cef0 00000004 00000002 f78d9a70 f7b1ced4 c013f968 000200d0 
[  121.678647]        00000000 c0493880 00000044 f7251988 c17d5350 00000010 c0493880 00000000 
[  121.687733]        00000202 f7b34028 c220f560 00000000 00000f14 6c744686 0000000a c048db40 
[  121.696817] Call Trace:
[  121.699620]  [<c0408240>] schedule_timeout+0xa3/0xa5
[  121.704888]  [<c016ce68>] do_poll+0x9a/0xb9
[  121.709344]  [<c016cfdd>] sys_poll+0x156/0x221
[  121.714098]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  121.719483] dbus-daemon-1 S 00000002     0  3030      1          3063  3022 (NOTLB)
[  121.727806] f7a33f00 f7a33ef0 00000004 00000002 000200d0 00000000 c0493880 00000044 
[  121.736035]        f7a33ee0 c17c4f28 00000010 c0493880 00000000 00000202 f73fa028 00000202 
[  121.745174]        f7a33ed8 c0130113 c220f560 00000000 0000a1ec b930b7bb 00000005 c048db40 
[  121.754310] Call Trace:
[  121.757128]  [<c0408240>] schedule_timeout+0xa3/0xa5
[  121.762393]  [<c016ce68>] do_poll+0x9a/0xb9
[  121.766850]  [<c016cfdd>] sys_poll+0x156/0x221
[  121.771576]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  121.776933] exim4         S 00000001     0  3063      1          3068  3030 (NOTLB)
[  121.785263] f7b73ea0 f7b73e90 00000004 00000001 c17ade04 c0493880 000000d0 f69a3550 
[  121.793446]        f7b73e80 c013f968 000200d0 00000000 c0493880 00000044 005cbc10 c17ade04 
[  121.802532]        00000010 c0493880 c2217560 00000001 000018e7 d8fb1c8d 00000005 c2330550 
[  121.811615] Call Trace:
[  121.814420]  [<c0408240>] schedule_timeout+0xa3/0xa5
[  121.819688]  [<c016c7ec>] do_select+0x152/0x26e
[  121.824504]  [<c016cba9>] sys_select+0x26a/0x3f7
[  121.829412]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  121.834768] irqbalance    D 00000002     0  3068      1          3072  3063 (NOTLB)
[  121.843093] f7aa9d14 f7aa9d00 00000004 00000002 00000000 f7aa9cc8 c01722a6 00000000 
[  121.851312]        c2330550 f7bac550 f72c1180 c2313200 f78f6550 00000003 f7aa9d28 f78f6550 
[  121.860445]        00000000 c220fec0 c220f560 00000000 00000144 8559bc5f 0000001b f78f6550 
[  121.869579] Call Trace:
[  121.872396]  [<c04081f1>] schedule_timeout+0x54/0xa5
[  121.877695]  [<c040817e>] io_schedule_timeout+0x29/0x33
[  121.883268]  [<c0287ce8>] blk_congestion_wait+0x70/0x85
[  121.888832]  [<c0141365>] throttle_vm_writeout+0x63/0x6d
[  121.894490]  [<c014712d>] shrink_zone+0xe0/0xfa
[  121.899333]  [<c01471b4>] shrink_caches+0x6d/0x6f
[  121.904340]  [<c0147286>] try_to_free_pages+0xd0/0x1b5
[  121.909781]  [<c013fa4b>] __alloc_pages+0x135/0x2e8
[  121.914959]  [<c0142d15>] kmem_getpages+0x34/0x9d
[  121.919956]  [<c0143b71>] cache_grow+0xaf/0x16c
[  121.924777]  [<c0143d92>] cache_alloc_refill+0x164/0x207
[  121.930406]  [<c014401b>] kmem_cache_alloc+0x4e/0x50
[  121.935673]  [<c01668ec>] getname+0x24/0xb3
[  121.940133]  [<c015878d>] do_sys_open+0x1a/0xfa
[  121.944950]  [<c015888c>] sys_open+0x1f/0x23
[  121.949495]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  121.954852] nrpe          S 00000002     0  3072      1          3077  3068 (NOTLB)
[  121.963176] f7a6cde8 f7a6cdd8 00000004 00000002 f7fcee00 c019da80 00000032 f7a6cea4 
[  121.971393]        f7a6cdf8 f7a6ce84 c038833a f7a6cdf8 c2387880 f7a6cea4 00000032 00000000 
[  121.980480]        f7a6cdc8 c019daf0 c220f560 00000000 0000b73d e27c0cf9 00000005 c048db40 
[  121.989566] Call Trace:
[  121.992369]  [<c0408240>] schedule_timeout+0xa3/0xa5
[  121.997637]  [<c03b50ce>] inet_csk_wait_for_connect+0xd7/0xdc
[  122.003713]  [<c03b5131>] inet_csk_accept+0x5e/0x154
[  122.008981]  [<c03d4120>] inet_accept+0x30/0xb2
[  122.013799]  [<c0389379>] sys_accept+0xb3/0x173
[  122.018617]  [<c0389eed>] sys_socketcall+0xc1/0x234
[  122.023795]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  122.029152] snmpd         S 00000002     0  3077      1          3085  3072 (NOTLB)
[  122.037476] f70dcea0 f70dce90 00000004 00000002 c0493880 c0492f80 000200d0 00000002 
[  122.045658]        00000044 c17aea88 c0493880 000000d0 00000202 f6a12060 00000202 f70dce74 
[  122.054743]        c0130113 f6a12000 c220f560 00000000 0001b1a4 fcad0c4a 00000005 c048db40 
[  122.063829] Call Trace:
[  122.066632]  [<c0408240>] schedule_timeout+0xa3/0xa5
[  122.071930]  [<c016c7ec>] do_select+0x152/0x26e
[  122.076773]  [<c016cba9>] sys_select+0x26a/0x3f7
[  122.081707]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  122.087091] sshd          S 00000002     0  3085      1          3096  3077 (NOTLB)
[  122.095419] f73e0ea0 f73e0e90 00000004 00000002 c17aebf0 c0493880 000000d0 f710c550 
[  122.103616]        f73e0e80 c013f968 000200d0 00000000 c0493880 00000044 f73e0ed4 c17aebf0 
[  122.112645]        00000010 c0493880 c220f560 00000000 00008cc7 ffe83822 00000005 c048db40 
[  122.121670] Call Trace:
[  122.124454]  [<c0408240>] schedule_timeout+0xa3/0xa5
[  122.129685]  [<c016c7ec>] do_select+0x152/0x26e
[  122.134469]  [<c016cba9>] sys_select+0x26a/0x3f7
[  122.139344]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  122.144665] xfs           S 00000002     0  3096      1          3100  3085 (NOTLB)
[  122.152888] f71a2ea0 f71a2e90 00000004 00000002 00000002 00000044 c17af9dc c0493880 
[  122.161014]        000000d0 f7f6d030 f71a2e88 c013f968 000200d0 00000000 c0493880 00000202 
[  122.170040]        c220ffe0 f71a2eb4 c220f560 00000000 000feae2 2c919ee0 00000006 c048db40 
[  122.179066] Call Trace:
[  122.181850]  [<c04081f1>] schedule_timeout+0x54/0xa5
[  122.187082]  [<c016c7ec>] do_select+0x152/0x26e
[  122.191913]  [<c016cba9>] sys_select+0x26a/0x3f7
[  122.196794]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  122.202149] xfs-xtt       S 00000004     0  3100      1          3237  3096 (NOTLB)
[  122.210473] f7b74ea0 f7b74e90 00000004 00000004 00000002 00000044 c17af850 c0493880 
[  122.218659]        000000d0 f7baca70 f7b74e88 c013f968 000200d0 00000000 c0493880 00000202 
[  122.228556]        c221ffe0 f7b74eb4 c221f560 00000002 00193300 2e4020b0 00000006 c2330030 
[  122.237645] Call Trace:
[  122.240449]  [<c04081f1>] schedule_timeout+0x54/0xa5
[  122.245716]  [<c016c7ec>] do_select+0x152/0x26e
[  122.250557]  [<c016cba9>] sys_select+0x26a/0x3f7
[  122.255490]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  122.260871] S20xprint     S 00000002     0  3237      1  3238    3250  3100 (NOTLB)
[  122.269196] f7a91f08 f7a91ef4 00000004 00000002 00000099 00000001 f7a91ee0 f7a79180 
[  122.277420]        f7f6d550 0000002f c013f968 000200d2 f7a2f030 0000004b 00000000 f7a2f030 
[  122.286555]        00000000 c220fec0 c220f560 00000000 00013e60 3f693dd0 00000006 f7a2f030 
[  122.295688] Call Trace:
[  122.298505]  [<c011fa57>] do_wait+0x18a/0x376
[  122.303166]  [<c011fd03>] sys_wait4+0x3e/0x42
[  122.307826]  [<c011fd2e>] sys_waitpid+0x27/0x2b
[  122.312673]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  122.318054] S20xprint     S 00000004     0  3238   3237  3239    3240       (NOTLB)
[  122.326378] f72b3f08 f72b3ef8 00000004 00000004 00000044 c2173474 c04938a8 000200d2 
[  122.334561]        f717d030 f72b3eec c013f968 000200d2 00000000 c04938a8 00000044 f70e6ccc 
[  122.343650]        c2173474 c2174c38 c221f560 00000002 000225ec 3f98e492 00000006 c2330030 
[  122.352781] Call Trace:
[  122.355601]  [<c011fa57>] do_wait+0x18a/0x376
[  122.360261]  [<c011fd03>] sys_wait4+0x3e/0x42
[  122.364924]  [<c011fd2e>] sys_waitpid+0x27/0x2b
[  122.369763]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  122.375150] Xprt          S 00000001     0  3239   3238                     (NOTLB)
[  122.383474] f7118ea0 f7118e8c 00000004 00000001 00000002 00000044 c17b1764 f7b9f880 
[  122.391699]        f6b34550 f7f6d550 f7118e88 c013f968 000200d0 00000000 00000087 f6cc0550 
[  122.400804]        00000001 c2217ec0 c2217560 00000001 0002235b 7e6b4944 00000006 c2330550 
[  122.409836] Call Trace:
[  122.412621]  [<c0408240>] schedule_timeout+0xa3/0xa5
[  122.417854]  [<c016c7ec>] do_select+0x152/0x26e
[  122.422641]  [<c016cba9>] sys_select+0x26a/0x3f7
[  122.427520]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  122.432840] S20xprint     S 00000001     0  3240   3237                3238 (NOTLB)
[  122.441065] f720fe9c f720fe8c 00000004 00000001 00000000 00000000 c05566ac c05566ac 
[  122.449290]        00000000 c2217ec0 c2217ec0 00000000 c0491680 00000082 00000011 f71e9a44 
[  122.458425]        f720ff3c f720fe6c c2217560 00000001 00013577 700f9522 00000006 c2330550 
[  122.467555] Call Trace:
[  122.470376]  [<c01656cb>] pipe_wait+0x6f/0x8e
[  122.475038]  [<c0165906>] pipe_readv+0x1c8/0x2f3
[  122.479970]  [<c0165a64>] pipe_read+0x33/0x37
[  122.484633]  [<c0159169>] vfs_read+0xa6/0x17d
[  122.489293]  [<c01594ef>] sys_read+0x4b/0x75
[  122.493866]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  122.499250] rpc.statd     S 00000002     0  3250      1          3289  3237 (NOTLB)
[  122.507573] f7210ea0 f7210e90 00000004 00000002 00000000 c0493880 00000044 f7210e74 
[  122.515724]        c17b3fc0 00000010 c0493880 00000000 00000202 f6c70028 00000202 f7210e74 
[  122.524753]        c0130113 f6c70000 c220f560 00000000 00001f32 564270a1 00000006 c048db40 
[  122.533780] Call Trace:
[  122.536564]  [<c0408240>] schedule_timeout+0xa3/0xa5
[  122.541802]  [<c016c7ec>] do_select+0x152/0x26e
[  122.546592]  [<c016cba9>] sys_select+0x26a/0x3f7
[  122.551471]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  122.556797] ntpd          D 00000002     0  3289      1          3305  3250 (NOTLB)
[  122.565119] f7a98cb8 f7a98ca8 00000004 00000002 00000001 00000282 00000000 00000003 
[  122.573300]        f6a81eb8 c048db40 f7b274f0 f7a98ca4 00000202 c220ffe0 f7a98ccc 00000202 
[  122.582390]        c220ffe0 f7a98ccc c220f560 00000000 00000166 870495e8 0000001b c048db40 
[  122.591475] Call Trace:
[  122.594278]  [<c04081f1>] schedule_timeout+0x54/0xa5
[  122.599545]  [<c040817e>] io_schedule_timeout+0x29/0x33
[  122.605082]  [<c0287ce8>] blk_congestion_wait+0x70/0x85
[  122.610619]  [<c0141365>] throttle_vm_writeout+0x63/0x6d
[  122.616246]  [<c014712d>] shrink_zone+0xe0/0xfa
[  122.621064]  [<c01471b4>] shrink_caches+0x6d/0x6f
[  122.626060]  [<c0147286>] try_to_free_pages+0xd0/0x1b5
[  122.631505]  [<c013fa4b>] __alloc_pages+0x135/0x2e8
[  122.636681]  [<c013fc2a>] __get_free_pages+0x2c/0x4f
[  122.641946]  [<c016c541>] __pollwait+0x3a/0xae
[  122.646686]  [<c038fa46>] datagram_poll+0x2b/0xcd
[  122.651651]  [<c03ce0ae>] udp_poll+0x22/0xfc
[  122.656169]  [<c0388b2a>] sock_poll+0x23/0x2a
[  122.660778]  [<c016c8b1>] do_select+0x217/0x26e
[  122.665560]  [<c016cba9>] sys_select+0x26a/0x3f7
[  122.670439]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  122.675762] atd           S 00000004     0  3305      1          3308  3289 (NOTLB)
[  122.683988] f7219f38 f7219f28 00000004 00000004 f7889194 f7eb6180 c21f7264 c21f7264 
[  122.692151]        c17bd8cc f7219f1c c014b57b c21f7264 00000028 f7219f0c 00000000 00000202 
[  122.701179]        c221ffe0 f7219f4c c221f560 00000002 00003745 7451c85d 00000006 c2330030 
[  122.710204] Call Trace:
[  122.712989]  [<c04081f1>] schedule_timeout+0x54/0xa5
[  122.718222]  [<c040825c>] schedule_timeout_interruptible+0x1a/0x1c
[  122.724703]  [<c012564b>] sys_nanosleep+0xc8/0x147
[  122.729756]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  122.735076] cron          D 00000002     0  3308      1          3314  3305 (NOTLB)
[  122.743302] f7b7bc64 f7b7bc54 00000004 00000002 f7b7bc14 f7216a94 00070026 00000000 
[  122.751530]        f7b16600 f69b7a70 00000000 f7b7bc3c 00000202 c220ffe0 f7b7bc78 00000202 
[  122.760664]        c220ffe0 f7b7bc78 c220f560 00000000 00000136 8559c610 0000001b c048db40 
[  122.769800] Call Trace:
[  122.772615]  [<c04081f1>] schedule_timeout+0x54/0xa5
[  122.777875]  [<c040817e>] io_schedule_timeout+0x29/0x33
[  122.783374]  [<c0287ce8>] blk_congestion_wait+0x70/0x85
[  122.788875]  [<c0141365>] throttle_vm_writeout+0x63/0x6d
[  122.794462]  [<c014712d>] shrink_zone+0xe0/0xfa
[  122.799247]  [<c01471b4>] shrink_caches+0x6d/0x6f
[  122.804210]  [<c0147286>] try_to_free_pages+0xd0/0x1b5
[  122.809621]  [<c013fa4b>] __alloc_pages+0x135/0x2e8
[  122.814764]  [<c0142d15>] kmem_getpages+0x34/0x9d
[  122.819728]  [<c0143b71>] cache_grow+0xaf/0x16c
[  122.824512]  [<c0143d92>] cache_alloc_refill+0x164/0x207
[  122.830102]  [<c014401b>] kmem_cache_alloc+0x4e/0x50
[  122.835332]  [<c01668ec>] getname+0x24/0xb3
[  122.839761]  [<c0168593>] __user_walk+0x1c/0x55
[  122.844545]  [<c0162c22>] vfs_stat+0x1e/0x55
[  122.849062]  [<c016328b>] sys_stat64+0x1b/0x37
[  122.853756]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  122.859075] bash          S 00000002     0  3314      1  3384    3315  3308 (NOTLB)
[  122.867298] f70fff08 f70ffef8 00000004 00000002 00000044 c21687c4 c04938a8 000200d2 
[  122.875429]        f78ada70 f70ffeec c013f968 000200d2 00000000 c04938a8 00000044 0000000e 
[  122.884508]        c21687c4 00000010 c220f560 00000000 000020f3 51053239 0000000e c048db40 
[  122.893596] Call Trace:
[  122.896398]  [<c011fa57>] do_wait+0x18a/0x376
[  122.901035]  [<c011fd03>] sys_wait4+0x3e/0x42
[  122.905674]  [<c011fd2e>] sys_waitpid+0x27/0x2b
[  122.910490]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  122.915846] getty         S 00000001     0  3315      1          3316  3314 (NOTLB)
[  122.924166] f7115e6c f7115e5c 00000004 00000001 c0167f20 c2316480 f73be000 f7ef2358 
[  122.932351]        00000202 f7b8ea00 00000202 f7115e3c c011c6dd 00000001 00000000 00000000 
[  122.941436]        f7b8ea00 ffffffff c2217560 00000001 000012d8 ba6a1921 00000006 c2330550 
[  122.950525] Call Trace:
[  122.953325]  [<c0408240>] schedule_timeout+0xa3/0xa5
[  122.958596]  [<c02d6c9d>] read_chan+0x57c/0x64b
[  122.963442]  [<c02d180b>] tty_read+0xc9/0xcd
[  122.968010]  [<c0159169>] vfs_read+0xa6/0x17d
[  122.972658]  [<c01594ef>] sys_read+0x4b/0x75
[  122.977189]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  122.982515] getty         S 00000001     0  3316      1          3317  3315 (NOTLB)
[  122.990746] f78f5e6c f78f5e5c 00000004 00000001 c0167f20 c2316480 f73be000 f7ef2358 
[  122.998878]        00000202 f7aaa000 00000202 f78f5e3c c011c6dd 00000001 00000000 00000000 
[  123.007909]        f7aaa000 ffffffff c2217560 00000001 00001235 ba79575b 00000006 c2330550 
[  123.016937] Call Trace:
[  123.019722]  [<c0408240>] schedule_timeout+0xa3/0xa5
[  123.024956]  [<c02d6c9d>] read_chan+0x57c/0x64b
[  123.029801]  [<c02d180b>] tty_read+0xc9/0xcd
[  123.034373]  [<c0159169>] vfs_read+0xa6/0x17d
[  123.039847]  [<c01594ef>] sys_read+0x4b/0x75
[  123.044419]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  123.049802] getty         S 00000006     0  3317      1          3318  3316 (NOTLB)
[  123.058127] f715ee6c f78f6030 c2227a20 00000006 c0167f20 c2316480 f7334000 00000000 
[  123.066345]        c2330030 f7b2d200 b9b43335 00000006 f78f6030 00000000 00000000 f78f6030 
[  123.075472]        00000000 c2227ec0 c2227560 00000003 0000120b b9b43335 00000006 f78f6030 
[  123.084604] Call Trace:
[  123.087421]  [<c0408240>] schedule_timeout+0xa3/0xa5
[  123.092715]  [<c02d6c9d>] read_chan+0x57c/0x64b
[  123.097557]  [<c02d180b>] tty_read+0xc9/0xcd
[  123.102127]  [<c0159169>] vfs_read+0xa6/0x17d
[  123.106786]  [<c01594ef>] sys_read+0x4b/0x75
[  123.111353]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  123.116737] getty         S 00000006     0  3318      1          3319  3317 (NOTLB)
[  123.125063] f7354e6c f7e6d030 c2227a20 00000006 c0167f20 c2316480 f7334000 00000000 
[  123.133253]        c2330030 f7a0f000 b9b390d1 00000006 f7e6d030 00000003 00000000 f7e6d030 
[  123.142284]        00000000 c2227ec0 c2227560 00000003 00001d80 b9b390d1 00000006 f7e6d030 
[  123.151312] Call Trace:
[  123.154097]  [<c0408240>] schedule_timeout+0xa3/0xa5
[  123.159332]  [<c02d6c9d>] read_chan+0x57c/0x64b
[  123.164150]  [<c02d180b>] tty_read+0xc9/0xcd
[  123.168699]  [<c0159169>] vfs_read+0xa6/0x17d
[  123.173341]  [<c01594ef>] sys_read+0x4b/0x75
[  123.177890]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  123.183249] getty         S 00000003     0  3319      1                3318 (NOTLB)
[  123.191577] f7b6ce6c f7b6ce5c 00000004 00000003 c0167f20 c2316480 f7334000 f7ef2358 
[  123.199799]        00000202 f732ae00 00000202 f7b6ce3c c011c6dd 00000001 00000000 00000000 
[  123.208851]        f732ae00 ffffffff c2227560 00000003 00001231 b9b4d6ee 00000006 c2337a70 
[  123.217893] Call Trace:
[  123.220681]  [<c0408240>] schedule_timeout+0xa3/0xa5
[  123.225976]  [<c02d6c9d>] read_chan+0x57c/0x64b
[  123.230815]  [<c02d180b>] tty_read+0xc9/0xcd
[  123.235386]  [<c0159169>] vfs_read+0xa6/0x17d
[  123.240046]  [<c01594ef>] sys_read+0x4b/0x75
[  123.244592]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  123.249947] writetest     D 00000003     0  3384   3314                     (NOTLB)
[  123.258278] f72a69e8 f72a69d8 00000004 00000003 00000000 f72a69b4 c0117800 00000003 
[  123.266501]        c2227560 c2337a70 bf71bf17 00000010 00000202 c2227fe0 f72a69fc 00000202 
[  123.275586]        c2227fe0 f72a69fc c2227560 00000003 0000020f af03b035 0000001c c2337a70 
[  123.284677] Call Trace:
[  123.287478]  [<c04081f1>] schedule_timeout+0x54/0xa5
[  123.292742]  [<c040817e>] io_schedule_timeout+0x29/0x33
[  123.298304]  [<c0287ce8>] blk_congestion_wait+0x70/0x85
[  123.303843]  [<c0141365>] throttle_vm_writeout+0x63/0x6d
[  123.309470]  [<c014712d>] shrink_zone+0xe0/0xfa
[  123.314287]  [<c01471b4>] shrink_caches+0x6d/0x6f
[  123.319284]  [<c0147286>] try_to_free_pages+0xd0/0x1b5
[  123.324727]  [<c013fa4b>] __alloc_pages+0x135/0x2e8
[  123.329903]  [<c0142d15>] kmem_getpages+0x34/0x9d
[  123.334900]  [<c0143b71>] cache_grow+0xaf/0x16c
[  123.339717]  [<c0143d92>] cache_alloc_refill+0x164/0x207
[  123.345344]  [<c014401b>] kmem_cache_alloc+0x4e/0x50
[  123.350609]  [<c01c4ea9>] nfs_create_request+0x3d/0xcc
[  123.356054]  [<c01c9225>] nfs_update_request+0x172/0x1d9
[  123.361679]  [<c01c8734>] nfs_writepage_async+0x36/0x99
[  123.367216]  [<c01c8960>] nfs_writepage+0x1a8/0x1df
[  123.372393]  [<c017dd7b>] mpage_writepages+0x232/0x399
[  123.377840]  [<c01c89c5>] nfs_writepages+0x2e/0x11e
[  123.382990]  [<c01416da>] do_writepages+0x26/0x44
[  123.387962]  [<c013abd3>] __filemap_fdatawrite_range+0x97/0x99
[  123.394094]  [<c013ac0e>] filemap_fdatawrite+0x39/0x3d
[  123.399511]  [<c01c234a>] nfs_revalidate_mapping+0xd5/0x108
[  123.405380]  [<c01bfdbb>] nfs_file_write+0x76/0x115
[  123.410527]  [<c01592e7>] do_sync_write+0xa7/0xe7
[  123.415497]  [<c01593cd>] vfs_write+0xa6/0x17d
[  123.420197]  [<c015968c>] sys_pwrite64+0x7d/0x81
[  123.425075]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  124.929031] SysRq : Show Memory
[  124.932384] Mem-info:
[  124.934788] DMA per-cpu:
[  124.937512] cpu 0 hot: low 0, high 0, batch 1 used:0
[  124.942717] cpu 0 cold: low 0, high 0, batch 1 used:0
[  124.948015] cpu 1 hot: low 0, high 0, batch 1 used:0
[  124.953222] cpu 1 cold: low 0, high 0, batch 1 used:0
[  124.958520] cpu 2 hot: low 0, high 0, batch 1 used:0
[  124.963727] cpu 2 cold: low 0, high 0, batch 1 used:0
[  124.969023] cpu 3 hot: low 0, high 0, batch 1 used:0
[  124.974205] cpu 3 cold: low 0, high 0, batch 1 used:0
[  124.979472] DMA32 per-cpu: empty
[  124.982895] Normal per-cpu:
[  124.985870] cpu 0 hot: low 0, high 186, batch 31 used:51
[  124.991410] cpu 0 cold: low 0, high 62, batch 15 used:12
[  124.996950] cpu 1 hot: low 0, high 186, batch 31 used:91
[  125.002490] cpu 1 cold: low 0, high 62, batch 15 used:57
[  125.008030] cpu 2 hot: low 0, high 186, batch 31 used:178
[  125.013660] cpu 2 cold: low 0, high 62, batch 15 used:12
[  125.019199] cpu 3 hot: low 0, high 186, batch 31 used:29
[  125.024738] cpu 3 cold: low 0, high 62, batch 15 used:18
[  125.030304] HighMem per-cpu:
[  125.033368] cpu 0 hot: low 0, high 186, batch 31 used:77
[  125.038909] cpu 0 cold: low 0, high 62, batch 15 used:50
[  125.044451] cpu 1 hot: low 0, high 186, batch 31 used:58
[  125.049993] cpu 1 cold: low 0, high 62, batch 15 used:54
[  125.055496] cpu 2 hot: low 0, high 186, batch 31 used:178
[  125.061090] cpu 2 cold: low 0, high 62, batch 15 used:7
[  125.066503] cpu 3 hot: low 0, high 186, batch 31 used:4
[  125.071920] cpu 3 cold: low 0, high 62, batch 15 used:46
[  125.077423] Free pages:        9140kB (2944kB HighMem)
[  125.082750] Active:454665 inactive:42267 dirty:58557 writeback:244276 unstable:0 free:2285 slab:15340 mapped:8992 pagetables:96
[  125.094606] DMA free:3572kB min:68kB low:84kB high:100kB active:2700kB inactive:408kB present:16384kB pages_scanned:0 all_unreclaimable? no
[  125.107535] lowmem_reserve[]: 0 0 880 2031
[  125.112005] DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
[  125.123683] lowmem_reserve[]: 0 0 880 2031
[  125.128151] Normal free:2624kB min:3756kB low:4692kB high:5632kB active:699080kB inactive:114720kB present:901120kB pages_scanned:291 all_unreclaimable? no
[  125.142622] lowmem_reserve[]: 0 0 0 9211
[  125.146944] HighMem free:2944kB min:512kB low:1740kB high:2968kB active:1116880kB inactive:53940kB present:1179092kB pages_scanned:701 all_unreclaimable? no
[  125.161484] lowmem_reserve[]: 0 0 0 0
[  125.165529] DMA: 1*4kB 0*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 3572kB
[  125.176368] DMA32: empty
[  125.179072] Normal: 0*4kB 0*8kB 0*16kB 0*32kB 1*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2624kB
[  125.190175] HighMem: 196*4kB 34*8kB 24*16kB 13*32kB 11*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2944kB
[  125.201909] Swap cache: add 0, delete 0, find 0/0, race 0+0
[  125.207717] Free swap  = 4200956kB
[  125.211277] Total swap = 4200956kB
[  125.214837] Free swap:       4200956kB
[  125.227299] 524149 pages of RAM
[  125.230612] 294773 pages of HIGHMEM
[  125.234282] 6122 reserved pages
[  125.237589] 397419 pages shared
[  125.240895] 0 pages swap cached
[  125.244205] 58557 pages dirty
[  125.247333] 244276 pages writeback
[  125.250912] 8992 pages mapped
[  125.254038] 15340 pages slab
[  125.257075] 96 pages pagetables

Re: nfs unhappiness with memory pressure

[Trond Myklebust]

[-- Attachment #1: Type: text/plain, Size: 1446 bytes --]

On Mon, 2005-12-05 at 15:13 -0500, Trond Myklebust wrote:
> On Mon, 2005-12-05 at 10:01 -0800, Kenny Simpson wrote:
> > Tested with rc5 - same results.  It was suggested that I run slabtop when the system freezed, so
> > here is that info: (again, by hand, I'm getting another machine to use either netconsole or a
> > serial cable).
> > 
> >  Active / Total Objects (% used)    : 478764 / 485383 (98.6%)
> >  Active / Total Slabs (% used)      : 14618 / 14635 (99.9%)
> >  Active / Total Caches (% used)     : 79 / 138 (57.2%)
> >  Active / Total Size (% used)       : 56663.79K / 57566.41K (98.4%)
> >  Minimum / Average / Maximum Object : 0.01K / 0.12K / 128.00K
> > 
> >   OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME                   
> > 403088 403088 100%    0.06K   6832       59     27328K nfs_page
> >  30380  30380 100%    0.50K   4340        7     17360K nfs_write_data
> >  15134  15134 100%    0.27K   1081       14      4324K radix_tree_node
> > ...
> > 
> > 
> > The other thing is that the stack trace showsd slabtop as being halted in throttle_vm_writeout
> > while allocating memory, and the writetest was halted waiting to allocate memory.
> 
> Can somebody VM-savvy please explain how on earth they expect something
> like throttle_vm_writeout() to make progress? Shouldn't that thing at
> the very least be kicking pdflush every time it loops?

Can you try something like this patch, Kenny?

Cheers,
  Trond



[-- Attachment #2: linux-2.6.15-throttle_kicks_pdflush.dif --]
[-- Type: text/plain, Size: 822 bytes --]

VM: Ensure that throttle_vm_writeout() can make progress

 Once a process is in the loop inside throttle_vm_writeout(), it has
 no guarantee that it will ever get out, since there is nothing that
 will kickstart the flushing of unstable writes.

 Signed-off-by: Trond Myklebust <Trond.Myklebust@netapp.com>
---

 mm/page-writeback.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index 5240e42..9a66dee 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -306,6 +306,8 @@ void throttle_vm_writeout(void)
 
                 if (wbs.nr_unstable + wbs.nr_writeback <= dirty_thresh)
                         break;
+		if (wbs.nr_unstable != 0)
+			wakeup_pdflush(wbs.nr_unstable);
                 blk_congestion_wait(WRITE, HZ/10);
         }
 }

Re: nfs unhappiness with memory pressure

[Nick Piggin]

Trond Myklebust wrote:
> On Mon, 2005-12-05 at 15:13 -0500, Trond Myklebust wrote:
> 

>>Can somebody VM-savvy please explain how on earth they expect something
>>like throttle_vm_writeout() to make progress? Shouldn't that thing at
>>the very least be kicking pdflush every time it loops?
> 
> 
> Can you try something like this patch, Kenny?
> 

The VM doesn't expect to have to rely on pdflush to write out pages
for it. ->writepage should be enough. Adding wakeup_pdflush here
actually could do the wrong thing for non-NFS filesystems if it
starts more writeback.

Nick

Send instant messages to your online friends http://au.messenger.yahoo.com 

Re: nfs unhappiness with memory pressure

[Kenny Simpson]

[-- Attachment #1: Type: text/plain, Size: 564 bytes --]

--- Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> > Can somebody VM-savvy please explain how on earth they expect something
> > like throttle_vm_writeout() to make progress? Shouldn't that thing at
> > the very least be kicking pdflush every time it loops?
> 
> Can you try something like this patch, Kenny?
> 
> Cheers,
>   Trond
> 
> 
> > VM: Ensure that throttle_vm_writeout() can make progress

No change. :(

-Kenny


		
__________________________________________ 
Yahoo! DSL – Something to write home about. 
Just $16.99/mo. or less. 
dsl.yahoo.com 

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 3165733884-sysrq.log --]
[-- Type: text/x-log; name="sysrq.log", Size: 57053 bytes --]

[  120.491185] SysRq : Show Regs
[  120.494350] 
[  120.495933] Pid: 0, comm:              swapper
[  120.500590] EIP: 0060:[<c0100d4f>] CPU: 3
[  120.504798] EIP is at default_idle+0x36/0x56
[  120.509273]  EFLAGS: 00000246    Not tainted  (2.6.15-rc5+nfs+k2)
[  120.515656] EAX: 00000000 EBX: c2338000 ECX: c22262e0 EDX: c2338000
[  120.522219] ESI: c053c380 EDI: c053c300 EBP: c2338f84 DS: 007b ES: 007b
[  120.529197] CR0: 8005003b CR2: b7d3f000 CR3: 36b4d000 CR4: 000006d0
[  122.665018] SysRq : Show State
[  122.668281] 
[  122.668282]                                                sibling
[  122.676341]   task             PC      pid father child younger older
[  122.683084] init          D 00000001     0     1      0     2               (NOTLB)
[  122.691406] c2331ce8 c2331cd8 00000004 00000001 c019afa7 c2331c9c c015c8cd f71c0a14 
[  122.699584]        00000000 c2330550 00022f80 00000000 00000202 c2217fe0 c2331cfc 00000202 
[  122.708671]        c2217fe0 c2331cfc c2217560 00000001 00000133 8d20aa2d 0000001c c2330550 
[  122.717755] Call Trace:
[  122.720557]  [<c04086e1>] schedule_timeout+0x54/0xa5
[  122.725825]  [<c040866e>] io_schedule_timeout+0x29/0x33
[  122.731357]  [<c02880c4>] blk_congestion_wait+0x70/0x85
[  122.736895]  [<c014136b>] throttle_vm_writeout+0x69/0x7d
[  122.742524]  [<c014714d>] shrink_zone+0xe0/0xfa
[  122.747339]  [<c01471d4>] shrink_caches+0x6d/0x6f
[  122.752333]  [<c01472a6>] try_to_free_pages+0xd0/0x1b5
[  122.757777]  [<c013fa4b>] __alloc_pages+0x135/0x2e8
[  122.762951]  [<c013fc2a>] __get_free_pages+0x2c/0x4f
[  122.768215]  [<c016c561>] __pollwait+0x3a/0xae
[  122.772942]  [<c0166016>] pipe_poll+0x36/0xad
[  122.777579]  [<c016c8d1>] do_select+0x217/0x26e
[  122.782395]  [<c016cbc9>] sys_select+0x26a/0x3f7
[  122.787301]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  122.792658] migration/0   S 00000002     0     2      1             3       (L-TLB)
[  122.800978] c2339f94 c2339f84 00000004 00000002 c0116b9c c2337a70 c2227a20 00000001 
[  122.809158]        c220f5ac 00000001 00000073 c220f958 c220f958 c2227a20 c220f5a8 00000000 
[  122.818241]        c2227560 00000003 c220f560 00000000 00003394 026b476a 0000000e c048db40 
[  122.827324] Call Trace:
[  122.830125]  [<c0118d0c>] migration_thread+0x81/0x10d
[  122.835478]  [<c012fe68>] kthread+0xb7/0xbc
[  122.839934]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  122.845377] ksoftirqd/0   S 00000002     0     3      1             4     2 (L-TLB)
[  122.853697] c233af98 c233af88 00000004 00000002 000004d3 000e12b9 00000000 c233ba70 
[  122.861880]        c2337030 c2357a70 00000001 c2331f5c c2330a70 ffffffff c2337030 c233af6c 
[  122.870964]        00000000 00000000 c220f560 00000000 00000129 0678d7c6 00000004 c048db40 
[  122.880048] Call Trace:
[  122.882849]  [<c0121786>] ksoftirqd+0xd6/0xf5
[  122.887484]  [<c012fe68>] kthread+0xb7/0xbc
[  122.891939]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  122.897384] migration/1   S 00000001     0     4      1             5     3 (L-TLB)
[  122.905700] c233cf94 c233cf80 00000004 00000001 00000082 00000001 c233cf44 00000000 
[  122.913876]        c048db40 c2337a70 00000003 c2227560 f78b7a70 00000005 00000008 f78b7a70 
[  122.922961]        00000000 c2217ec0 c2217560 00000001 00001493 42f120b6 00000009 f78b7a70 
[  122.932040] Call Trace:
[  122.934842]  [<c0118d0c>] migration_thread+0x81/0x10d
[  122.940194]  [<c012fe68>] kthread+0xb7/0xbc
[  122.944648]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  122.950091] ksoftirqd/1   S 00000001     0     5      1             6     4 (L-TLB)
[  122.958412] c2345f98 c2345f88 00000004 00000001 000000bd 00180d6e 00000000 c233b550 
[  122.966589]        c233b550 c2330a70 00000001 c2331f28 c2345000 ffffffff c233b550 c2345f6c 
[  122.975671]        c0407ed4 c22175a8 c2217560 00000001 000001ab 2fca1753 00000000 c2330550 
[  122.984759] Call Trace:
[  122.987575]  [<c0121786>] ksoftirqd+0xd6/0xf5
[  122.992235]  [<c012fe68>] kthread+0xb7/0xbc
[  122.996717]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  123.002192] migration/2   S 00000004     0     6      1             7     5 (L-TLB)
[  123.010514] c2346f94 c2346f80 00000004 00000004 00000082 c2330550 c22175a8 f78a8700 
[  123.018737]        f7b7a550 c2330550 00000001 c2217560 f7a54030 000000b9 00000002 f7a54030 
[  123.027869]        00000000 c221fec0 c221f560 00000002 00001664 25534bef 00000006 f7a54030 
[  123.036999] Call Trace:
[  123.039816]  [<c0118d0c>] migration_thread+0x81/0x10d
[  123.045199]  [<c012fe68>] kthread+0xb7/0xbc
[  123.049680]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  123.055157] ksoftirqd/2   S 00000002     0     7      1             8     6 (L-TLB)
[  123.063480] c234df98 c234ca70 c221f560 00000002 00000164 00275ccf 00000000 c234ca70 
[  123.071701]        c234ca70 c234cb9c 00000001 c2331f28 c234d000 ffffffff c234ca70 c234df6c 
[  123.080829]        c0407ed4 c221f5a8 c221f560 00000002 00000045 00275f43 00000000 c2330030 
[  123.089958] Call Trace:
[  123.092774]  [<c0121786>] ksoftirqd+0xd6/0xf5
[  123.097437]  [<c012fe68>] kthread+0xb7/0xbc
[  123.101916]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  123.107386] migration/3   S 00000003     0     8      1             9     7 (L-TLB)
[  123.115706] c234ef94 c234ef80 00000004 00000003 00000082 f7120550 c221fa20 00000000 
[  123.123926]        c2330030 c2330550 00000001 c2217560 f7ae1a70 00000012 00000002 f7ae1a70 
[  123.133057]        00000000 c2227ec0 c2227560 00000003 00000eff 529edf4c 00000008 f7ae1a70 
[  123.142186] Call Trace:
[  123.145001]  [<c0118d0c>] migration_thread+0x81/0x10d
[  123.150384]  [<c012fe68>] kthread+0xb7/0xbc
[  123.154864]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  123.160337] ksoftirqd/3   S 00000003     0     9      1            10     8 (L-TLB)
[  123.168657] c2353f98 c2353f88 00000004 00000003 000000c0 003698fe 00000000 c234c030 
[  123.176873]        c234c030 c2361550 00000001 c2331f28 c2353000 ffffffff c234c030 c2353f6c 
[  123.185990]        c0407ed4 c22275a8 c2227560 00000003 0000014c 4497dad4 00000003 c2337a70 
[  123.195074] Call Trace:
[  123.197865]  [<c0121786>] ksoftirqd+0xd6/0xf5
[  123.202473]  [<c012fe68>] kthread+0xb7/0xbc
[  123.206901]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  123.212312] events/0      S 00000002     0    10      1            11     9 (L-TLB)
[  123.220639] c2358f34 c2358f24 00000004 00000002 c220ffe0 00000202 c2358ef0 c0124bab 
[  123.228773]        c220ffe0 f7a38398 00000202 00000002 f7a38380 f7a38000 c2358f00 c0124c50 
[  123.237801]        c23489b0 c23489a8 c220f560 00000000 00003db9 af434da0 0000001c c048db40 
[  123.246831] Call Trace:
[  123.249615]  [<c012bd90>] worker_thread+0x207/0x225
[  123.254760]  [<c012fe68>] kthread+0xb7/0xbc
[  123.259189]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  123.264603] events/1      S 00000001     0    11      1            12    10 (L-TLB)
[  123.272924] c236af34 c236af24 00000004 00000001 00000202 00000001 c2354540 000007d0 
[  123.281057]        c236aef4 c012bb5c c22191b8 fffd55d5 00000000 c2300780 c22191a4 c236aefc 
[  123.290162]        c2348930 c2348928 c2217560 00000001 000005d9 533d40e5 0000001c c2330550 
[  123.299292] Call Trace:
[  123.302109]  [<c012bd90>] worker_thread+0x207/0x225
[  123.307312]  [<c012fe68>] kthread+0xb7/0xbc
[  123.311790]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  123.317262] events/2      S 00000004     0    12      1            13    11 (L-TLB)
[  123.325582] c2359f34 c2359f24 00000004 00000004 00000202 00000001 c2354540 000007d0 
[  123.334617]        c2359ef4 c012bb5c c22211b8 fffd55d7 00000000 c2300780 c22211a4 c2359efc 
[  123.343746]        c23488b0 c23488a8 c221f560 00000002 000008a0 535bc77c 0000001c c2330030 
[  123.352873] Call Trace:
[  123.355689]  [<c012bd90>] worker_thread+0x207/0x225
[  123.360891]  [<c012fe68>] kthread+0xb7/0xbc
[  123.365370]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  123.370843] events/3      R running     0    13      1            14    12 (L-TLB)
[  123.379071] khelper       S 00000004     0    14      1            15    13 (L-TLB)
[  123.387394] c235af34 c235af20 00000004 00000004 00000000 00000000 00000000 f7361d80 
[  123.395620]        00000000 f7a29d2c 00000000 f7a29d58 f7b97a70 f7a29d50 00000202 f7b97a70 
[  123.404753]        00000000 c221fec0 c221f560 00000002 0000017b 249842c5 00000006 f7b97a70 
[  123.413889] Call Trace:
[  123.416705]  [<c012bd90>] worker_thread+0x207/0x225
[  123.421905]  [<c012fe68>] kthread+0xb7/0xbc
[  123.426387]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  123.431861] kthread       S 00000002     0    15      1    20     144    14 (L-TLB)
[  123.440182] c2389f34 c2389f24 00000004 00000002 c2389ef0 c0117b5d f7ad9e88 00000003 
[  123.448403]        00000000 00000000 f7ad9f28 f7ad9f1c f7ad9f20 00000202 c2389f18 c0117cbb 
[  123.457535]        c2380b30 c2380b28 c220f560 00000000 000001b0 2318e05f 00000005 c048db40 
[  123.466668] Call Trace:
[  123.469483]  [<c012bd90>] worker_thread+0x207/0x225
[  123.474687]  [<c012fe68>] kthread+0xb7/0xbc
[  123.479166]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  123.484639] kblockd/0     S 00000002     0    20     15            21       (L-TLB)
[  123.492959] c238bf34 c238bf20 00000004 00000002 f7fd0c5c 00c4d18d 00000000 00000000 
[  123.501180]        c2330550 f7fd0c5c c238bee8 c040906a f78b7a70 0000001f c0579414 f78b7a70 
[  123.510310]        00000000 c220fec0 c220f560 00000000 00000c12 5303b35e 00000008 f78b7a70 
[  123.519436] Call Trace:
[  123.522252]  [<c012bd90>] worker_thread+0x207/0x225
[  123.527454]  [<c012fe68>] kthread+0xb7/0xbc
[  123.531933]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  123.537405] kblockd/1     S 00000001     0    21     15            22    20 (L-TLB)
[  123.545725] f7d58f34 f7d58f20 00000004 00000001 f7fd09dc 0014012f 00000000 f7a5d180 
[  123.553910]        f6c37550 f7fd09dc f7d58ee8 c040906a f6b0ba70 c02fccc2 00000003 f6b0ba70 
[  123.562994]        00000000 c2217ec0 c2217560 00000001 000014ab 21553fb6 00000006 f6b0ba70 
[  123.572076] Call Trace:
[  123.574921]  [<c012bd90>] worker_thread+0x207/0x225
[  123.580092]  [<c012fe68>] kthread+0xb7/0xbc
[  123.584546]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  123.589990] kblockd/2     S 00000004     0    22     15            23    21 (L-TLB)
[  123.598306] c238ff34 c238ff20 00000004 00000004 c238fed0 c040906a c238fef8 f7a43280 
[  123.606487]        f7120550 f78c707c c238fee8 00000000 f7206550 00000172 c23eee80 f7206550 
[  123.615583]        00000000 c221fec0 c221f560 00000002 00000ed0 628d2b7d 00000002 f7206550 
[  123.624714] Call Trace:
[  123.627528]  [<c012bd90>] worker_thread+0x207/0x225
[  123.632731]  [<c012fe68>] kthread+0xb7/0xbc
[  123.637209]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  123.642680] kblockd/3     S 00000003     0    23     15            24    22 (L-TLB)
[  123.651002] f7d59f34 f7d59f24 00000004 00000003 f7a3b25c 00a06705 00000000 00000202 
[  123.659214]        00000001 f7a3b25c f7d59ee8 c040906a f7d59f10 c02fccc2 c0579414 f7a3b25c 
[  123.668294]        c23eee30 c23eee28 c2227560 00000003 00000bf5 55d51770 00000008 c2337a70 
[  123.677361] Call Trace:
[  123.680149]  [<c012bd90>] worker_thread+0x207/0x225
[  123.685315]  [<c012fe68>] kthread+0xb7/0xbc
[  123.689775]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  123.695221] kacpid        S 00000002     0    24     15           142    23 (L-TLB)
[  123.703544] f7d78f34 f7d78f24 00000004 00000002 00000000 00000001 00000100 00000000 
[  123.711728]        00000080 00000080 00000080 c05566ac c05566ac c05566a0 c2217ec0 c2217ec0 
[  123.720818]        00000000 00000202 c220f560 00000000 00000185 027fd001 00000000 c048db40 
[  123.729909] Call Trace:
[  123.732714]  [<c012bd90>] worker_thread+0x207/0x225
[  123.737889]  [<c012fe68>] kthread+0xb7/0xbc
[  123.742345]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  123.747790] pdflush       S 00000004     0   142     15           143    24 (L-TLB)
[  123.756107] f7dadf68 f7dadf58 00000004 00000004 c0124c50 c0493a5c fffd56de f7dadf68 
[  123.764288]        c014153c c0493a5c fffd56de fffcce26 00000000 00000000 f7dadf18 00000400 
[  123.773380]        00000000 00000000 c221f560 00000002 00000181 b059cc3e 0000001b c2330030 
[  123.782465] Call Trace:
[  123.785271]  [<c0141df8>] __pdflush+0x84/0x199
[  123.789995]  [<c0141f4d>] pdflush+0x40/0x47
[  123.794451]  [<c012fe68>] kthread+0xb7/0xbc
[  123.798910]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  123.804354] pdflush       D 00000003     0   143     15           145   142 (L-TLB)
[  123.812676] f7d5dcb0 f7d5dca0 00000004 00000003 0000001b 00000000 f7d5dc80 c0117800 
[  123.820864]        00000003 c2227560 00000086 03d8f695 0000000e c23ef030 f7d5dc6c f7d5dc6c 
[  123.829954]        c23ef030 00000003 c2227560 00000003 002e3da9 0579218a 0000000e c2337a70 
[  123.839041] Call Trace:
[  123.841844]  [<c040863b>] io_schedule+0x26/0x30
[  123.846662]  [<c013ab27>] sync_page+0x39/0x4e
[  123.851302]  [<c0408929>] __wait_on_bit_lock+0x58/0x67
[  123.856746]  [<c013b343>] __lock_page+0x84/0x8c
[  123.861561]  [<c017deb4>] mpage_writepages+0x34b/0x399
[  123.867008]  [<c01c88f1>] nfs_writepages+0x2e/0x11e
[  123.872185]  [<c01416f2>] do_writepages+0x2e/0x53
[  123.877182]  [<c017c072>] __sync_single_inode+0x60/0x1f1
[  123.882808]  [<c017c295>] __writeback_single_inode+0x92/0x195
[  123.888898]  [<c017c555>] sync_sb_inodes+0x1bd/0x2c2
[  123.894190]  [<c017c734>] writeback_inodes+0xda/0xe6
[  123.899482]  [<c01413e8>] background_writeout+0x69/0xa2
[  123.905048]  [<c0141e35>] __pdflush+0xc1/0x199
[  123.909801]  [<c0141f4d>] pdflush+0x40/0x47
[  123.914257]  [<c012fe68>] kthread+0xb7/0xbc
[  123.918713]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  123.924156] kswapd0       D 00000001     0   144      1           804    15 (L-TLB)
[  123.932476] f7de6c94 f7de6c84 00000004 00000001 00011200 00000000 c0409378 00000000 
[  123.940649]        00000003 00011200 f7e3f580 0000001b 00011200 f7de6c7c c0143db2 f7e31880 
[  123.949731]        00011200 00000000 c2217560 00000001 001e5571 05ebddf0 0000000e c2330550 
[  123.958804] Call Trace:
[  123.961604]  [<c040863b>] io_schedule+0x26/0x30
[  123.966416]  [<c013e464>] mempool_alloc+0xcf/0xd1
[  123.971406]  [<c01c996e>] nfs_flush_one+0x54/0x17f
[  123.976488]  [<c01c9aee>] nfs_flush_list+0x55/0xa3
[  123.981569]  [<c01ca420>] nfs_flush_inode+0x82/0xad
[  123.986741]  [<c01c88be>] nfs_writepage+0x1d7/0x1dc
[  123.991913]  [<c0146516>] pageout+0xb5/0x137
[  123.996468]  [<c0146782>] shrink_list+0x1ea/0x40c
[  124.001487]  [<c0146b40>] shrink_cache+0xf8/0x281
[  124.006508]  [<c0147138>] shrink_zone+0xcb/0xfa
[  124.011348]  [<c0147611>] balance_pgdat+0x286/0x37e
[  124.016551]  [<c01477ee>] kswapd+0xe5/0x10f
[  124.021030]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  124.026506] aio/0         S 00000002     0   145     15           146   143 (L-TLB)
[  124.034822] f7d69f34 f7d69f20 00000004 00000002 00000000 00000001 00000100 00000000 
[  124.043043]        00000080 00000080 00000080 c05566a0 c2330a70 c05566ac c220fec0 c2330a70 
[  124.052168]        00000000 c220fec0 c220f560 00000000 00000124 2ca5cbf6 00000000 c2330a70 
[  124.061297] Call Trace:
[  124.064112]  [<c012bd90>] worker_thread+0x207/0x225
[  124.069312]  [<c012fe68>] kthread+0xb7/0xbc
[  124.073792]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  124.079263] aio/1         S 00000001     0   146     15           147   145 (L-TLB)
[  124.087580] f7e2cf34 f7e2cf20 00000004 00000001 00000000 00000001 00000100 00000000 
[  124.095759]        00000080 00000000 00000000 c05566ac c2330a70 00000000 c2217ec0 c2330a70 
[  124.104839]        00000000 c2217ec0 c2217560 00000001 000000f3 2ca62a97 00000000 c2330a70 
[  124.113923] Call Trace:
[  124.116724]  [<c012bd90>] worker_thread+0x207/0x225
[  124.121900]  [<c012fe68>] kthread+0xb7/0xbc
[  124.126357]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  124.131801] aio/2         S 00000004     0   147     15           148   146 (L-TLB)
[  124.140928] f7d6af34 f7d6af24 00000004 00000004 00000000 00000001 00000100 00000000 
[  124.149058]        00000080 00000080 00000080 c05566a0 c05566a0 c05566ac c220fec0 c220fec0 
[  124.158084]        00000000 00000202 c221f560 00000002 000001ab 2ca695a2 00000000 c2330030 
[  124.167110] Call Trace:
[  124.169895]  [<c012bd90>] worker_thread+0x207/0x225
[  124.175036]  [<c012fe68>] kthread+0xb7/0xbc
[  124.179464]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  124.184875] aio/3         S 00000003     0   148     15           149   147 (L-TLB)
[  124.193106] f7e2ef34 f7e2ef24 00000004 00000003 00000000 00000001 00000100 00000080 
[  124.201234]        00000080 00000000 00000000 c05566ac c05566ac 00000000 c2217ec0 c2217ec0 
[  124.210259]        00000000 00000202 c2227560 00000003 0000017e 2ca6ee55 00000000 c2337a70 
[  124.219287] Call Trace:
[  124.222070]  [<c012bd90>] worker_thread+0x207/0x225
[  124.227211]  [<c012fe68>] kthread+0xb7/0xbc
[  124.231638]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  124.237047] xfslogd/0     S 00000002     0   149     15           150   148 (L-TLB)
[  124.245274] f7d6bf34 f7d6bf20 00000004 00000002 00000000 00000001 00000100 00000000 
[  124.253404]        00000100 00000080 00000080 c05566a0 c2330a70 c05566ac c220fec0 c2330a70 
[  124.262495]        00000000 c220fec0 c220f560 00000000 000000d8 2e49df27 00000000 c2330a70 
[  124.271614] Call Trace:
[  124.274430]  [<c012bd90>] worker_thread+0x207/0x225
[  124.279633]  [<c012fe68>] kthread+0xb7/0xbc
[  124.284115]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  124.289587] xfslogd/1     S 00000001     0   150     15           151   149 (L-TLB)
[  124.297908] f7e64f34 f7e64f20 00000004 00000001 00000000 00000001 00000100 00000000 
[  124.306130]        00000100 00000080 00000080 c05566ac c2330a70 c05566a0 c2217ec0 c2330a70 
[  124.315261]        00000000 c2217ec0 c2217560 00000001 000000e1 2e4a40dc 00000000 c2330a70 
[  124.324393] Call Trace:
[  124.327208]  [<c012bd90>] worker_thread+0x207/0x225
[  124.332409]  [<c012fe68>] kthread+0xb7/0xbc
[  124.336888]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  124.342359] xfslogd/2     S 00000004     0   151     15           152   150 (L-TLB)
[  124.350683] f7d6ef34 f7d6ef24 00000004 00000004 00000000 00000001 00000100 00000080 
[  124.358907]        00000100 00000080 00000080 c05566a0 c05566a0 c05566ac c220fec0 c220fec0 
[  124.368037]        00000000 00000202 c221f560 00000002 00000135 2e4ab2b7 00000000 c2330030 
[  124.377169] Call Trace:
[  124.379984]  [<c012bd90>] worker_thread+0x207/0x225
[  124.385181]  [<c012fe68>] kthread+0xb7/0xbc
[  124.389660]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  124.395129] xfslogd/3     S 00000003     0   152     15           153   151 (L-TLB)
[  124.403451] f7e66f34 f7e66f24 00000004 00000003 00000000 00000001 00000100 00000080 
[  124.411667]        00000100 00000080 00000080 c05566ac c05566ac c05566a0 c2217ec0 c2217ec0 
[  124.420786]        00000000 00000202 c2227560 00000003 00000146 2e4b118c 00000000 c2337a70 
[  124.429910] Call Trace:
[  124.432724]  [<c012bd90>] worker_thread+0x207/0x225
[  124.437925]  [<c012fe68>] kthread+0xb7/0xbc
[  124.442404]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  124.447874] xfsdatad/0    S 00000002     0   153     15           154   152 (L-TLB)
[  124.456194] f7d6ff34 f7d6ff20 00000004 00000002 00000000 00000001 00000100 00000000 
[  124.464408]        00000100 00000080 00000080 c05566a0 c2330a70 c05566ac c220fec0 c2330a70 
[  124.473532]        00000000 c220fec0 c220f560 00000000 000000e0 2e4b6256 00000000 c2330a70 
[  124.482656] Call Trace:
[  124.485469]  [<c012bd90>] worker_thread+0x207/0x225
[  124.490669]  [<c012fe68>] kthread+0xb7/0xbc
[  124.495146]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  124.500616] xfsdatad/1    S 00000001     0   154     15           155   153 (L-TLB)
[  124.508932] f7e68f34 f7e68f20 00000004 00000001 00000000 00000001 00000100 00000000 
[  124.517149]        00000100 00000080 00000080 c05566ac c2330a70 c05566a0 c2217ec0 c2330a70 
[  124.526275]        00000000 c2217ec0 c2217560 00000001 000000d9 2e4bc69b 00000000 c2330a70 
[  124.535401] Call Trace:
[  124.538216]  [<c012bd90>] worker_thread+0x207/0x225
[  124.543415]  [<c012fe68>] kthread+0xb7/0xbc
[  124.547891]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  124.553360] xfsdatad/2    S 00000004     0   155     15           156   154 (L-TLB)
[  124.561676] f7d70f34 f7d70f24 00000004 00000004 00000000 00000001 00000100 00000080 
[  124.569891]        00000100 00000080 00000080 c05566a0 c05566a0 c05566ac c220fec0 c220fec0 
[  124.579021]        00000000 00000202 c221f560 00000002 0000013a 2e4c2d9c 00000000 c2330030 
[  124.588154] Call Trace:
[  124.590967]  [<c012bd90>] worker_thread+0x207/0x225
[  124.596166]  [<c012fe68>] kthread+0xb7/0xbc
[  124.600641]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  124.606110] xfsdatad/3    S 00000003     0   156     15           157   155 (L-TLB)
[  124.614405] f7e69f34 f7e69f24 00000004 00000003 00000000 00000001 00000100 00000080 
[  124.622529]        00000100 00000080 00000080 c05566ac c05566ac c05566a0 c2217ec0 c2217ec0 
[  124.631552]        00000000 00000202 c2227560 00000003 00000147 2e4c83ef 00000000 c2337a70 
[  124.640581] Call Trace:
[  124.643364]  [<c012bd90>] worker_thread+0x207/0x225
[  124.648503]  [<c012fe68>] kthread+0xb7/0xbc
[  124.652929]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  124.658337] xfsbufd       S 00000001     0   157     15           742   156 (L-TLB)
[  124.666557] f7d72f44 f7d72f34 00000004 00000001 00000100 00000080 00000080 00000000 
[  124.674682]        f7206030 c2330550 0216915a 0000000e 00000202 c2217fe0 f7d72f58 00000202 
[  124.683704]        c2217fe0 f7d72f58 c2217560 00000001 000000b1 e5ed67a9 0000001c c2330550 
[  124.692731] Call Trace:
[  124.695515]  [<c04086e1>] schedule_timeout+0x54/0xa5
[  124.700742]  [<c040874c>] schedule_timeout_interruptible+0x1a/0x1c
[  124.707221]  [<c0272754>] xfsbufd+0x4e/0x1b0
[  124.711734]  [<c012fe68>] kthread+0xb7/0xbc
[  124.716165]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  124.721572] kseriod       S 00000002     0   742     15          1668   157 (L-TLB)
[  124.729796] f7f10f5c f7f10f4c 00000004 00000002 c02eff05 f8879da4 f7f10f10 c02efe91 
[  124.737924]        f8879d88 c04a8738 f8879da4 c04a63e0 f7f10f34 c02ef54e f8879d88 c04a8738 
[  124.746953]        00000286 00000202 c220f560 00000000 0000063d 6358b857 00000001 c048db40 
[  124.755975] Call Trace:
[  124.758757]  [<c02e4762>] serio_thread+0xea/0xec
[  124.763633]  [<c012fe68>] kthread+0xb7/0xbc
[  124.768057]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  124.773464] kjournald     S 00000001     0   804      1          1128   144 (L-TLB)
[  124.781686] f7fb9f58 f7fb9f48 00000004 00000001 f6b4504c 00000000 00000000 f7fb7800 
[  124.789808]        00000000 f726b200 f6a48e70 0000037f 00000202 c2227fe0 f7fb9fb8 f7fb9f2c 
[  124.798829]        f7fadc6c f7fadc64 c2217560 00000001 0000030b d4604302 0000000c c2330550 
[  124.807852] Call Trace:
[  124.810639]  [<c01b0eef>] kjournald+0x1d6/0x223
[  124.815425]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  124.820832] kjournald     S 00000004     0  1128      1          1129   804 (L-TLB)
[  124.829059] f7ac0f58 f7ac0f48 00000004 00000004 00000012 00000036 00000044 00000202 
[  124.837188]        5b3e363c 20202020 32382e35 36333038 f700205d 00000003 00000000 00000000 
[  124.846216]        f716606c f7166064 c221f560 00000002 0000033a 5bd83b5f 00000001 c2330030 
[  124.855239] Call Trace:
[  124.858020]  [<c01b0eef>] kjournald+0x1d6/0x223
[  124.862800]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  124.868209] kjournald     S 00000001     0  1129      1          1130  1128 (L-TLB)
[  124.876431] f7b3ef58 f7b3ef48 00000004 00000001 f722a02c 00000000 00000000 f71c4800 
[  124.884552]        00000000 f7344900 f788d6ec 0000028a 00000202 c2217fe0 f7b3efb8 f7b3ef2c 
[  124.893577]        f71c0c6c f71c0c64 c2217560 00000001 0000021e d4d9438e 0000000c c2330550 
[  124.902607] Call Trace:
[  124.905420]  [<c01b0eef>] kjournald+0x1d6/0x223
[  124.910260]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  124.915730] kjournald     D 00000001     0  1130      1          2761  1129 (L-TLB)
[  124.924863] f7fdfbec f7fdfbdc 00000004 00000001 f6a4520c c015a4f0 c21ff6d0 f7fdfbe0 
[  124.933083]        00000202 c2330550 c0492f80 00000006 00000202 c2217fe0 f7fdfc00 00000202 
[  124.942212]        c2217fe0 f7fdfc00 c2217560 00000001 000000fc 15ede76c 0000001d c2330550 
[  124.951339] Call Trace:
[  124.954154]  [<c04086e1>] schedule_timeout+0x54/0xa5
[  124.959446]  [<c040866e>] io_schedule_timeout+0x29/0x33
[  124.965010]  [<c02880c4>] blk_congestion_wait+0x70/0x85
[  124.970574]  [<c014136b>] throttle_vm_writeout+0x69/0x7d
[  124.976228]  [<c014714d>] shrink_zone+0xe0/0xfa
[  124.981068]  [<c01471d4>] shrink_caches+0x6d/0x6f
[  124.986089]  [<c01472a6>] try_to_free_pages+0xd0/0x1b5
[  124.991566]  [<c013fa4b>] __alloc_pages+0x135/0x2e8
[  124.996767]  [<c013b51b>] find_or_create_page+0x83/0x97
[  125.002330]  [<c015b777>] grow_dev_page+0x38/0x15d
[  125.007444]  [<c015b954>] __getblk_slow+0xb8/0x124
[  125.012557]  [<c015bd1d>] __getblk+0x57/0x59
[  125.017131]  [<c01b1970>] journal_get_descriptor_buffer+0x4d/0xa4
[  125.023599]  [<c01ad7bf>] journal_write_commit_record+0x38/0x116
[  125.029974]  [<c01ade3f>] journal_commit_transaction+0x5a2/0x127f
[  125.036446]  [<c01b0dda>] kjournald+0xc1/0x223
[  125.041194]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  125.046668] khubd         S 00000001     0  1668     15          2770   742 (L-TLB)
[  125.054989] f72e0f60 f72e0f4c 00000004 00000001 000003e8 00000007 f72cab80 f7a5dd80 
[  125.063213]        f7206030 f73580b4 c0294432 f72e0f1c f7206550 00000007 00000010 f7206550 
[  125.072347]        00000000 c2217ec0 c2217560 00000001 00000e03 91e8c7a3 00000002 f7206550 
[  125.081480] Call Trace:
[  125.084295]  [<f8b01331>] hub_thread+0xe3/0xe5 [usbcore]
[  125.089968]  [<c012fe68>] kthread+0xb7/0xbc
[  125.094446]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  125.099919] portmap       S 00000001     0  2761      1          2774  1130 (NOTLB)
[  125.108238] f7a2ff00 f7a2fef0 00000004 00000001 00000202 f72ea00c 00000202 f7a2feb4 
[  125.116463]        c0130113 f72ea000 f7b9a680 00000202 f72ea028 00000202 f7a2fed0 c0130113 
[  125.125593]        f72ea000 f72f4480 c2217560 00000001 00000ba3 0a410b84 00000006 c2330550 
[  125.134727] Call Trace:
[  125.137546]  [<c0408730>] schedule_timeout+0xa3/0xa5
[  125.142842]  [<c016ce88>] do_poll+0x9a/0xb9
[  125.147321]  [<c016cffd>] sys_poll+0x156/0x221
[  125.152073]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  125.157455] rpciod/0      S 00000002     0  2770     15          2771  1668 (L-TLB)
[  125.165779] f6d24f34 f6d24f24 00000004 00000002 f7e45104 c04100d4 f7e45100 f6d24ee4 
[  125.174000]        c01c6b66 f7e45100 f7e3eec0 f6d24f00 c03f0ee4 f7e45100 f6d24f00 00000000 
[  125.183128]        f73448b0 f73448a8 c220f560 00000000 000006ec aa8c16d7 0000000b c048db40 
[  125.192261] Call Trace:
[  125.195078]  [<c012bd90>] worker_thread+0x207/0x225
[  125.200281]  [<c012fe68>] kthread+0xb7/0xbc
[  125.204761]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  125.210233] rpciod/1      S 00000004     0  2771     15          2772  2770 (L-TLB)
[  125.218555] f7a12f34 c2361030 c2217a20 00000004 00000000 00000001 00000100 00000000 
[  125.226781]        00000100 00000000 42e170b2 00000004 c2361030 00000000 c220fec0 c2361030 
[  125.235914]        00000000 c2217ec0 c2217560 00000001 00000191 42e4db8d 00000004 c2361030 
[  125.245045] Call Trace:
[  125.247861]  [<c012bd90>] worker_thread+0x207/0x225
[  125.253065]  [<c012fe68>] kthread+0xb7/0xbc
[  125.257546]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  125.263017] rpciod/2      D 00000004     0  2772     15          2773  2771 (L-TLB)
[  125.271338] f70ccaa8 f70cca98 00000004 00000004 ffffffce f70cca74 c0117800 00000002 
[  125.279560]        c221f560 c2330030 05f561fa 0000000e 00000202 c221ffe0 f70ccabc 00000202 
[  125.288692]        c221ffe0 f70ccabc c221f560 00000002 00000143 2b8c50f7 0000001d c2330030 
[  125.297821] Call Trace:
[  125.300636]  [<c04086e1>] schedule_timeout+0x54/0xa5
[  125.305931]  [<c040866e>] io_schedule_timeout+0x29/0x33
[  125.311495]  [<c02880c4>] blk_congestion_wait+0x70/0x85
[  125.317058]  [<c014136b>] throttle_vm_writeout+0x69/0x7d
[  125.322720]  [<c014714d>] shrink_zone+0xe0/0xfa
[  125.327560]  [<c01471d4>] shrink_caches+0x6d/0x6f
[  125.332581]  [<c01472a6>] try_to_free_pages+0xd0/0x1b5
[  125.338056]  [<c013fa4b>] __alloc_pages+0x135/0x2e8
[  125.343258]  [<c03b74ad>] tcp_sendmsg+0xaa0/0xb78
[  125.348281]  [<c03d4666>] inet_sendmsg+0x48/0x53
[  125.353212]  [<c0388716>] sock_sendmsg+0xb8/0xd3
[  125.358147]  [<c0388773>] kernel_sendmsg+0x42/0x4f
[  125.363259]  [<c038bc00>] sock_no_sendpage+0x5e/0x77
[  125.368556]  [<c03ee7af>] xs_tcp_send_request+0x2af/0x375
[  125.374306]  [<c03ed4c5>] xprt_transmit+0x4f/0x20b
[  125.379414]  [<c03ebce1>] call_transmit+0x68/0xc8
[  125.384435]  [<c03f0962>] __rpc_execute+0x5c/0x1ea
[  125.389548]  [<c03f0b27>] rpc_async_schedule+0x11/0x15
[  125.395019]  [<c012bd1d>] worker_thread+0x194/0x225
[  125.400221]  [<c012fe68>] kthread+0xb7/0xbc
[  125.404699]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  125.410173] rpciod/3      S 00000003     0  2773     15                2772 (L-TLB)
[  125.418494] f7a85f34 f7a85f20 00000004 00000003 f6cc0784 c04100d4 f6cc0780 00000000 
[  125.426691]        c2330030 f6cc0780 f7e3eec0 f7a85f00 f7b97550 00000039 f7a85f00 f7b97550 
[  125.435777]        00000000 c2227ec0 c2227560 00000003 00000efa 68208a87 0000000d f7b97550 
[  125.444862] Call Trace:
[  125.447662]  [<c012bd90>] worker_thread+0x207/0x225
[  125.452836]  [<c012fe68>] kthread+0xb7/0xbc
[  125.457292]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  125.462739] lockd         S 00000002     0  2774      1          2841  2761 (L-TLB)
[  125.471060] f7236f14 f7236f04 00000004 00000002 f720e380 00000001 c2217560 f7236ecc 
[  125.479240]        c01103a3 00000002 000000fc f7b7a030 c2217a20 f7236edc c0115796 00000202 
[  125.488319]        f73a0aec 00000202 c220f560 00000000 0000090e 42f05f34 00000004 c048db40 
[  125.497408] Call Trace:
[  125.500211]  [<c0408730>] schedule_timeout+0xa3/0xa5
[  125.505475]  [<c03f4bcb>] svc_recv+0x387/0x4b9
[  125.510199]  [<c02049d1>] lockd+0x101/0x239
[  125.514657]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  125.520103] syslogd       D 00000002     0  2841      1          2844  2774 (NOTLB)
[  125.528421] f7368cdc f7368ccc 00000004 00000002 c02884cf f7f634f4 f7a3b25c 00000003 
[  125.536601]        00000000 c048db40 c21b1e20 c0493400 00000202 c220ffe0 f7368cf0 00000202 
[  125.545684]        c220ffe0 f7368cf0 c220f560 00000000 00000150 397c1343 0000001d c048db40 
[  125.554766] Call Trace:
[  125.557566]  [<c04086e1>] schedule_timeout+0x54/0xa5
[  125.562831]  [<c040866e>] io_schedule_timeout+0x29/0x33
[  125.568363]  [<c02880c4>] blk_congestion_wait+0x70/0x85
[  125.573898]  [<c014136b>] throttle_vm_writeout+0x69/0x7d
[  125.579522]  [<c014714d>] shrink_zone+0xe0/0xfa
[  125.584336]  [<c01471d4>] shrink_caches+0x6d/0x6f
[  125.589330]  [<c01472a6>] try_to_free_pages+0xd0/0x1b5
[  125.594772]  [<c013fa4b>] __alloc_pages+0x135/0x2e8
[  125.599946]  [<c013fc2a>] __get_free_pages+0x2c/0x4f
[  125.605212]  [<c016c561>] __pollwait+0x3a/0xae
[  125.609937]  [<c038fe22>] datagram_poll+0x2b/0xcd
[  125.614936]  [<c0388f06>] sock_poll+0x23/0x2a
[  125.619574]  [<c016c8d1>] do_select+0x217/0x26e
[  125.624389]  [<c016cbc9>] sys_select+0x26a/0x3f7
[  125.629294]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  125.634649] klogd         S 00000001     0  2844      1          2859  2841 (NOTLB)
[  125.642976] f7304d4c f7304d3c 00000004 00000001 c220fa20 f7304cf4 c0115796 00000000 
[  125.651159]        00000096 c2330550 c0116055 c048db40 c220f560 00000000 c2381880 f7380000 
[  125.660245]        00000a00 000000b7 c2217560 00000001 00000847 8f99b9ea 0000001c c2330550 
[  125.669324] Call Trace:
[  125.672125]  [<c0408730>] schedule_timeout+0xa3/0xa5
[  125.677408]  [<f88b00ee>] unix_wait_for_peer+0xb3/0xb7 [unix]
[  125.683525]  [<f88b0abb>] unix_dgram_sendmsg+0x266/0x502 [unix]
[  125.689785]  [<c0388ab6>] sock_aio_write+0xea/0x106
[  125.694960]  [<c0159307>] do_sync_write+0xa7/0xe7
[  125.699968]  [<c01594b5>] vfs_write+0x16e/0x17d
[  125.704808]  [<c0159584>] sys_write+0x4b/0x75
[  125.709468]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  125.714852] ypbind        S 00000002     0  2859      1          2860  2844 (NOTLB)
[  125.723173] f7e6af00 f7e6aef0 00000004 00000002 00000202 f69a200c 00000202 f7e6aeb4 
[  125.731401]        c0130113 f69a2000 f70df880 00000202 f69a2028 00000202 f7e6aed0 c0130113 
[  125.740539]        f69a2000 f7ecc280 c220f560 00000000 0000130c 17ad0134 00000005 c048db40 
[  125.749670] Call Trace:
[  125.752492]  [<c0408730>] schedule_timeout+0xa3/0xa5
[  125.757784]  [<c016ce88>] do_poll+0x9a/0xb9
[  125.763079]  [<c016cffd>] sys_poll+0x156/0x221
[  125.767831]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  125.773214] ypbind        S 00000002     0  2860      1          2861  2859 (NOTLB)
[  125.781531] f723beac f723be9c 00000004 00000002 c0493400 f723be5c f723bed4 c220f560 
[  125.789754]        00000000 00000001 00000001 00000000 00000046 00000000 f7e6aedc 00000000 
[  125.798835]        f723be84 c0117b1a c220f560 00000000 0001069f df266306 00000004 c048db40 
[  125.807916] Call Trace:
[  125.810719]  [<c0408730>] schedule_timeout+0xa3/0xa5
[  125.815981]  [<c040874c>] schedule_timeout_interruptible+0x1a/0x1c
[  125.822503]  [<c0128559>] sys_rt_sigtimedwait+0x1f1/0x2e9
[  125.828216]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  125.833570] ypbind        D 00000002     0  2861      1          3021  2860 (NOTLB)
[  125.841890] f71dfcf8 f71dfce8 00000004 00000002 f73eb100 f73eb100 f71dfeb8 00000000 
[  125.850076]        00000000 c048db40 f7a60e80 00000040 00000202 c220ffe0 f71dfd0c 00000202 
[  125.859210]        c220ffe0 f71dfd0c c220f560 00000000 00000174 4de9a1b2 0000001d c048db40 
[  125.868340] Call Trace:
[  125.871157]  [<c04086e1>] schedule_timeout+0x54/0xa5
[  125.876450]  [<c040866e>] io_schedule_timeout+0x29/0x33
[  125.882014]  [<c02880c4>] blk_congestion_wait+0x70/0x85
[  125.887579]  [<c014136b>] throttle_vm_writeout+0x69/0x7d
[  125.893239]  [<c014714d>] shrink_zone+0xe0/0xfa
[  125.898084]  [<c01471d4>] shrink_caches+0x6d/0x6f
[  125.903105]  [<c01472a6>] try_to_free_pages+0xd0/0x1b5
[  125.908578]  [<c013fa4b>] __alloc_pages+0x135/0x2e8
[  125.913782]  [<c013fc2a>] __get_free_pages+0x2c/0x4f
[  125.919074]  [<c016c561>] __pollwait+0x3a/0xae
[  125.923822]  [<c038fe22>] datagram_poll+0x2b/0xcd
[  125.928842]  [<c03ce48a>] udp_poll+0x22/0xfc
[  125.933413]  [<c0388f06>] sock_poll+0x23/0x2a
[  125.938073]  [<c016cdea>] do_pollfd+0x94/0x98
[  125.942730]  [<c016ce49>] do_poll+0x5b/0xb9
[  125.947211]  [<c016cffd>] sys_poll+0x156/0x221
[  125.951961]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  125.957343] automount     D 00000002     0  3021      1          3029  2861 (NOTLB)
[  125.965660] f7150d28 f7150d18 00000004 00000002 f7150cf4 c014b7e2 f7a9f300 f72d28b4 
[  125.973880]        b7f86000 c048db40 f78edb7c 00000001 00000202 c220ffe0 f7150d3c 00000202 
[  125.983009]        c220ffe0 f7150d3c c220f560 00000000 00000162 550f363f 0000001d c048db40 
[  125.992137] Call Trace:
[  125.994953]  [<c04086e1>] schedule_timeout+0x54/0xa5
[  126.000246]  [<c040866e>] io_schedule_timeout+0x29/0x33
[  126.005810]  [<c02880c4>] blk_congestion_wait+0x70/0x85
[  126.011374]  [<c014136b>] throttle_vm_writeout+0x69/0x7d
[  126.017028]  [<c014714d>] shrink_zone+0xe0/0xfa
[  126.021870]  [<c01471d4>] shrink_caches+0x6d/0x6f
[  126.026896]  [<c01472a6>] try_to_free_pages+0xd0/0x1b5
[  126.032371]  [<c013fa4b>] __alloc_pages+0x135/0x2e8
[  126.037573]  [<c013fc2a>] __get_free_pages+0x2c/0x4f
[  126.042864]  [<c016c561>] __pollwait+0x3a/0xae
[  126.047614]  [<c0166016>] pipe_poll+0x36/0xad
[  126.052274]  [<c016cdea>] do_pollfd+0x94/0x98
[  126.056934]  [<c016ce49>] do_poll+0x5b/0xb9
[  126.061413]  [<c016cffd>] sys_poll+0x156/0x221
[  126.066163]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  126.071548] dbus-daemon-1 S 00000002     0  3029      1          3062  3021 (NOTLB)
[  126.079870] f70c6f00 f70c6ef0 00000004 00000002 000200d0 00000000 c0493880 00000044 
[  126.088097]        f70c6ee0 c17b4fa4 00000010 c0493880 00000000 00000202 f6ce1028 00000202 
[  126.097222]        f70c6ed8 c0130113 c220f560 00000000 0000c283 40012bcd 00000005 c048db40 
[  126.106351] Call Trace:
[  126.109154]  [<c0408730>] schedule_timeout+0xa3/0xa5
[  126.114414]  [<c016ce88>] do_poll+0x9a/0xb9
[  126.118871]  [<c016cffd>] sys_poll+0x156/0x221
[  126.123598]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  126.128954] exim4         S 00000001     0  3062      1          3067  3029 (NOTLB)
[  126.137276] f7af6ea0 f7af6e90 00000004 00000001 c17ae530 c0493880 000000d0 f72e7030 
[  126.145457]        f7af6e80 c013f968 000200d0 00000000 c0493880 00000044 00000286 c17ae530 
[  126.154552]        00000010 c0493880 c2217560 00000001 00001913 69393fd9 00000005 c2330550 
[  126.163678] Call Trace:
[  126.166492]  [<c0408730>] schedule_timeout+0xa3/0xa5
[  126.171783]  [<c016c80c>] do_select+0x152/0x26e
[  126.176620]  [<c016cbc9>] sys_select+0x26a/0x3f7
[  126.181546]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  126.186926] irqbalance    D 00000002     0  3067      1          3071  3062 (NOTLB)
[  126.195242] f723ad14 f723ad04 00000004 00000002 00000000 f723acc8 c01722c6 c2313200 
[  126.203461]        f00000dd c048db40 f720eb00 c2313200 00000202 c220ffe0 f723ad28 00000202 
[  126.212588]        c220ffe0 f723ad28 c220f560 00000000 00000153 5e4a6d73 0000001d c048db40 
[  126.221717] Call Trace:
[  126.224531]  [<c04086e1>] schedule_timeout+0x54/0xa5
[  126.229819]  [<c040866e>] io_schedule_timeout+0x29/0x33
[  126.235379]  [<c02880c4>] blk_congestion_wait+0x70/0x85
[  126.240940]  [<c014136b>] throttle_vm_writeout+0x69/0x7d
[  126.246591]  [<c014714d>] shrink_zone+0xe0/0xfa
[  126.251428]  [<c01471d4>] shrink_caches+0x6d/0x6f
[  126.256448]  [<c01472a6>] try_to_free_pages+0xd0/0x1b5
[  126.261918]  [<c013fa4b>] __alloc_pages+0x135/0x2e8
[  126.267118]  [<c0142d35>] kmem_getpages+0x34/0x9d
[  126.272136]  [<c0143b91>] cache_grow+0xaf/0x16c
[  126.276977]  [<c0143db2>] cache_alloc_refill+0x164/0x207
[  126.282627]  [<c014403b>] kmem_cache_alloc+0x4e/0x50
[  126.287917]  [<c016690c>] getname+0x24/0xb3
[  126.292392]  [<c01587ad>] do_sys_open+0x1a/0xfa
[  126.297228]  [<c01588ac>] sys_open+0x1f/0x23
[  126.301794]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  126.307172] nrpe          S 00000002     0  3071      1          3076  3067 (NOTLB)
[  126.315492] f71e7de8 f71e7dd8 00000004 00000002 f7fadc00 c019daa0 00000032 f71e7ea4 
[  126.323709]        f71e7df8 f71e7e84 c0388716 f71e7df8 f7382d00 f71e7ea4 00000032 00000000 
[  126.332835]        f71e7dc8 c019db10 c220f560 00000000 0000a806 68cca1c6 00000005 c048db40 
[  126.341958] Call Trace:
[  126.344773]  [<c0408730>] schedule_timeout+0xa3/0xa5
[  126.350069]  [<c03b54aa>] inet_csk_wait_for_connect+0xd7/0xdc
[  126.356174]  [<c03b550d>] inet_csk_accept+0x5e/0x154
[  126.361467]  [<c03d44fc>] inet_accept+0x30/0xb2
[  126.366309]  [<c0389755>] sys_accept+0xb3/0x173
[  126.371148]  [<c038a2c9>] sys_socketcall+0xc1/0x234
[  126.376352]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  126.381735] snmpd         S 00000002     0  3076      1          3084  3071 (NOTLB)
[  126.390056] f72d7ea0 f72d7e90 00000004 00000002 c0493880 c0492f80 000200d0 00000002 
[  126.398276]        00000044 c17af16c c0493880 000000d0 00000202 f6a43060 00000202 f72d7e74 
[  126.407405]        c0130113 f6a43000 c220f560 00000000 0001d0fd 83f7c635 00000005 c048db40 
[  126.416536] Call Trace:
[  126.419352]  [<c0408730>] schedule_timeout+0xa3/0xa5
[  126.424648]  [<c016c80c>] do_select+0x152/0x26e
[  126.429492]  [<c016cbc9>] sys_select+0x26a/0x3f7
[  126.434422]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  126.439808] sshd          S 00000002     0  3084      1          3095  3076 (NOTLB)
[  126.448127] f727eea0 f727ee90 00000004 00000002 c17d3a24 c0493880 000000d0 f78d7030 
[  126.456347]        f727ee80 c013f968 000200d0 00000000 c0493880 00000044 f727eed4 c17d3a24 
[  126.465478]        00000010 c0493880 c220f560 00000000 0000939a 87aed9de 00000005 c048db40 
[  126.474506] Call Trace:
[  126.477289]  [<c0408730>] schedule_timeout+0xa3/0xa5
[  126.482517]  [<c016c80c>] do_select+0x152/0x26e
[  126.487303]  [<c016cbc9>] sys_select+0x26a/0x3f7
[  126.492173]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  126.497496] xfs           S 00000002     0  3095      1          3099  3084 (NOTLB)
[  126.505720] f7aabea0 f7aabe90 00000004 00000002 00000002 00000044 c17afe38 c0493880 
[  126.513844]        000000d0 f7a82a70 f7aabe88 c013f968 000200d0 00000000 c0493880 00000202 
[  126.522867]        c220ffe0 f7aabeb4 c220f560 00000000 0013f3e3 b174fdf3 00000005 c048db40 
[  126.531889] Call Trace:
[  126.534673]  [<c04086e1>] schedule_timeout+0x54/0xa5
[  126.539903]  [<c016c80c>] do_select+0x152/0x26e
[  126.544684]  [<c016cbc9>] sys_select+0x26a/0x3f7
[  126.549554]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  126.554874] xfs-xtt       S 00000002     0  3099      1          3236  3095 (NOTLB)
[  126.563099] f7269ea0 f7269e90 00000004 00000002 00000002 00000044 c17afa48 c0493880 
[  126.571226]        000000d0 f78b7550 f7269e88 c013f968 000200d0 00000000 c0493880 00000202 
[  126.580249]        c220ffe0 f7269eb4 c220f560 00000000 001ce8c6 b6f5fbe5 00000005 c048db40 
[  126.589276] Call Trace:
[  126.592865]  [<c04086e1>] schedule_timeout+0x54/0xa5
[  126.598100]  [<c016c80c>] do_select+0x152/0x26e
[  126.602933]  [<c016cbc9>] sys_select+0x26a/0x3f7
[  126.607837]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  126.613159] S20xprint     S C220FEC0     0  3236      1  3237    3290  3099 (NOTLB)
[  126.621387] f7a83f08 00000000 c220f560 c220fec0 00000099 00000001 f7a83ee0 c013c381 
[  126.629514]        f7ffd800 0000002f c013f968 000200d2 f7fd7a70 00000002 00000000 f71133cc 
[  126.638539]        f7ffd800 f7113380 c220f560 00000000 00010daa c78685a6 00000005 f7fd7a70 
[  126.647566] Call Trace:
[  126.650382]  [<c011fa57>] do_wait+0x18a/0x376
[  126.655045]  [<c011fd03>] sys_wait4+0x3e/0x42
[  126.659704]  [<c011fd2e>] sys_waitpid+0x27/0x2b
[  126.664545]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  126.669927] S20xprint     S 00000002     0  3237   3236  3240    3238       (NOTLB)
[  126.678249] f7a94f08 f7a94ef8 00000004 00000002 00000044 c2174cc8 c04938a8 000200d2 
[  126.686465]        f7fd7a70 f7a94eec c013f968 000200d2 00000000 c04938a8 00000044 f71133cc 
[  126.695593]        c2174cc8 00000010 c220f560 00000000 0001d4c3 c7ce7231 00000005 c048db40 
[  126.704722] Call Trace:
[  126.707538]  [<c011fa57>] do_wait+0x18a/0x376
[  126.712179]  [<c011fd03>] sys_wait4+0x3e/0x42
[  126.716821]  [<c011fd2e>] sys_waitpid+0x27/0x2b
[  126.721661]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  126.727044] S20xprint     S 00000002     0  3238   3236                3237 (NOTLB)
[  126.735368] f7a61e9c f7a61e8c 00000004 00000002 00000000 00000000 c05566ac c05566ac 
[  126.743594]        00000000 c2217ec0 c2217ec0 00000286 c0491680 00000082 00000011 f7a2abf4 
[  126.752726]        f7a61f3c f7a61e6c c220f560 00000000 00016ca5 e2f5840d 00000005 c048db40 
[  126.761861] Call Trace:
[  126.764677]  [<c01656eb>] pipe_wait+0x6f/0x8e
[  126.769337]  [<c0165926>] pipe_readv+0x1c8/0x2f3
[  126.774267]  [<c0165a84>] pipe_read+0x33/0x37
[  126.778926]  [<c0159189>] vfs_read+0xa6/0x17d
[  126.783592]  [<c015950f>] sys_read+0x4b/0x75
[  126.788161]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  126.793548] Xprt          S 00000002     0  3240   3237                     (NOTLB)
[  126.801872] f7b95ea0 f7b95e90 00000004 00000002 00000002 00000044 c17b0834 c0493880 
[  126.810092]        000000d0 f7206a70 f7b95e88 c013f968 000200d0 00000000 c0493880 00000044 
[  126.819221]        00000257 c17b0834 c220f560 00000000 000176f9 e98a6f02 00000005 c048db40 
[  126.828345] Call Trace:
[  126.831160]  [<c0408730>] schedule_timeout+0xa3/0xa5
[  126.836453]  [<c016c80c>] do_select+0x152/0x26e
[  126.841289]  [<c016cbc9>] sys_select+0x26a/0x3f7
[  126.846216]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  126.851596] rpc.statd     S 00000002     0  3290      1          3293  3236 (NOTLB)
[  126.859917] f7a5bea0 f7a5be90 00000004 00000002 00000000 c0493880 00000044 f7a5be74 
[  126.868134]        c17b0ae0 00000010 c0493880 00000000 00000202 f6af8028 00000202 f7a5be74 
[  126.877262]        c0130113 f6af8000 c220f560 00000000 000020f3 0a47ac1f 00000006 c048db40 
[  126.886395] Call Trace:
[  126.889211]  [<c0408730>] schedule_timeout+0xa3/0xa5
[  126.894458]  [<c016c80c>] do_select+0x152/0x26e
[  126.899244]  [<c016cbc9>] sys_select+0x26a/0x3f7
[  126.904119]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  126.909440] ntpd          D 00000002     0  3293      1          3306  3290 (NOTLB)
[  126.917665] f714dcb8 f714dca8 00000004 00000002 00000001 00000282 00000000 00000003 
[  126.925790]        f73312b8 c048db40 f7b063c4 f714dca4 00000202 c220ffe0 f714dccc 00000202 
[  126.934817]        c220ffe0 f714dccc c220f560 00000000 00000168 8c633820 0000001d c048db40 
[  126.943843] Call Trace:
[  126.946627]  [<c04086e1>] schedule_timeout+0x54/0xa5
[  126.951860]  [<c040866e>] io_schedule_timeout+0x29/0x33
[  126.957358]  [<c02880c4>] blk_congestion_wait+0x70/0x85
[  126.962859]  [<c014136b>] throttle_vm_writeout+0x69/0x7d
[  126.968450]  [<c014714d>] shrink_zone+0xe0/0xfa
[  126.973234]  [<c01471d4>] shrink_caches+0x6d/0x6f
[  126.978197]  [<c01472a6>] try_to_free_pages+0xd0/0x1b5
[  126.983607]  [<c013fa4b>] __alloc_pages+0x135/0x2e8
[  126.988750]  [<c013fc2a>] __get_free_pages+0x2c/0x4f
[  126.993981]  [<c016c561>] __pollwait+0x3a/0xae
[  126.998678]  [<c038fe22>] datagram_poll+0x2b/0xcd
[  127.003642]  [<c03ce48a>] udp_poll+0x22/0xfc
[  127.008164]  [<c0388f06>] sock_poll+0x23/0x2a
[  127.012771]  [<c016c8d1>] do_select+0x217/0x26e
[  127.017560]  [<c016cbc9>] sys_select+0x26a/0x3f7
[  127.022433]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  127.027755] atd           S 00000001     0  3306      1          3309  3293 (NOTLB)
[  127.036075] f7366f38 f7366f28 00000004 00000001 f7ff1194 f7113180 c21f7264 c21f7264 
[  127.044293]        c17b0f24 f7366f1c c014b59b c21f7264 00000028 f7366f0c 00000000 00000202 
[  127.053419]        c2217fe0 f7366f4c c2217560 00000001 00003e97 1adef83c 00000006 c2330550 
[  127.062543] Call Trace:
[  127.065358]  [<c04086e1>] schedule_timeout+0x54/0xa5
[  127.070649]  [<c040874c>] schedule_timeout_interruptible+0x1a/0x1c
[  127.077202]  [<c012564b>] sys_nanosleep+0xc8/0x147
[  127.082313]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  127.087694] cron          S 00000002     0  3309      1          3315  3306 (NOTLB)
[  127.096017] f72e9f38 f72e9f28 00000004 00000002 00000000 f72e9f50 c016327c bf8c4a0c 
[  127.104149]        f72e9ee4 00000060 00000301 00000000 00000000 00008081 000081a4 00000202 
[  127.113216]        c220ffe0 f72e9f4c c220f560 00000000 00001af4 b1d9ad24 0000000f c048db40 
[  127.122301] Call Trace:
[  127.125101]  [<c04086e1>] schedule_timeout+0x54/0xa5
[  127.130370]  [<c040874c>] schedule_timeout_interruptible+0x1a/0x1c
[  127.136913]  [<c012564b>] sys_nanosleep+0xc8/0x147
[  127.142002]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  127.147358] bash          S 00000002     0  3315      1  3385    3316  3309 (NOTLB)
[  127.155678] f7bc2f08 f7bc2ef8 00000004 00000002 00000044 c215fdf4 c04938a8 000200d2 
[  127.163874]        f78b7a70 f7bc2eec c013f968 000200d2 00000000 c04938a8 00000044 0000000e 
[  127.172995]        c215fdf4 00000010 c220f560 00000000 00002499 aa74e8ec 0000000b c048db40 
[  127.182017] Call Trace:
[  127.184802]  [<c011fa57>] do_wait+0x18a/0x376
[  127.189407]  [<c011fd03>] sys_wait4+0x3e/0x42
[  127.194009]  [<c011fd2e>] sys_waitpid+0x27/0x2b
[  127.198791]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  127.204114] getty         S 00000003     0  3316      1          3317  3315 (NOTLB)
[  127.212334] f7a4ee6c f7a4ee5c 00000004 00000003 c0167f40 c2316580 f7b3d000 f7ec0b38 
[  127.220459]        00000202 f7b15e00 00000202 f7a4ee3c c011c6dd 00000001 00000000 00000000 
[  127.229487]        f7b15e00 ffffffff c2227560 00000003 0000245c 5da0f3bd 00000006 c2337a70 
[  127.238512] Call Trace:
[  127.241293]  [<c0408730>] schedule_timeout+0xa3/0xa5
[  127.246522]  [<c02d7079>] read_chan+0x57c/0x64b
[  127.251306]  [<c02d1be7>] tty_read+0xc9/0xcd
[  127.255825]  [<c0159189>] vfs_read+0xa6/0x17d
[  127.260436]  [<c015950f>] sys_read+0x4b/0x75
[  127.264949]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  127.270267] getty         S 00000006     0  3317      1          3318  3316 (NOTLB)
[  127.278487] f78ebe6c f6b0b550 c2227a20 00000006 c0167f40 c2316580 f7b3d000 00000000 
[  127.286611]        c2330030 f7180600 5dec8adc 00000006 f6b0b550 00000003 00000000 f6b0b550 
[  127.295636]        00000000 c2227ec0 c2227560 00000003 00001232 5dec8adc 00000006 f6b0b550 
[  127.304661] Call Trace:
[  127.307446]  [<c0408730>] schedule_timeout+0xa3/0xa5
[  127.312681]  [<c02d7079>] read_chan+0x57c/0x64b
[  127.317467]  [<c02d1be7>] tty_read+0xc9/0xcd
[  127.321982]  [<c0159189>] vfs_read+0xa6/0x17d
[  127.326590]  [<c015950f>] sys_read+0x4b/0x75
[  127.331110]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  127.336429] getty         S 00000003     0  3318      1          3319  3317 (NOTLB)
[  127.344660] f78e7e6c f78e7e5c 00000004 00000003 c0167f40 c2316580 f7b3d000 f7ec0b38 
[  127.352792]        00000202 f78ba400 00000202 f78e7e3c c011c6dd 00000001 00000000 00000000 
[  127.361817]        f78ba400 ffffffff c2227560 00000003 000011c4 5ded2ac3 00000006 c2337a70 
[  127.370842] Call Trace:
[  127.373625]  [<c0408730>] schedule_timeout+0xa3/0xa5
[  127.378851]  [<c02d7079>] read_chan+0x57c/0x64b
[  127.383633]  [<c02d1be7>] tty_read+0xc9/0xcd
[  127.388151]  [<c0159189>] vfs_read+0xa6/0x17d
[  127.392757]  [<c015950f>] sys_read+0x4b/0x75
[  127.397276]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  127.403408] getty         S 00000004     0  3319      1          3320  3318 (NOTLB)
[  127.411638] f7ad8e6c f7ad8e5c 00000004 00000004 c0167f40 c2316580 f69c8000 f7ec0b38 
[  127.419769]        00000202 f7b53000 00000202 f7ad8e3c c011c6dd 00000001 00000000 00000000 
[  127.428795]        f7b53000 ffffffff c221f560 00000002 0000154d 603f2ae7 00000006 c2330030 
[  127.437822] Call Trace:
[  127.440610]  [<c0408730>] schedule_timeout+0xa3/0xa5
[  127.445846]  [<c02d7079>] read_chan+0x57c/0x64b
[  127.450634]  [<c02d1be7>] tty_read+0xc9/0xcd
[  127.455149]  [<c0159189>] vfs_read+0xa6/0x17d
[  127.459755]  [<c015950f>] sys_read+0x4b/0x75
[  127.464276]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  127.469597] getty         S 00000004     0  3320      1                3319 (NOTLB)
[  127.477824] f7a29e6c f7a29e5c 00000004 00000004 c0167f40 c2316580 f69c8000 f7ec0b38 
[  127.485955]        00000202 f6c81000 00000202 f7a29e3c c011c6dd 00000001 00000000 00000000 
[  127.494985]        f6c81000 ffffffff c221f560 00000002 000012ad 609a8e0f 00000006 c2330030 
[  127.504084] Call Trace:
[  127.506899]  [<c0408730>] schedule_timeout+0xa3/0xa5
[  127.512186]  [<c02d7079>] read_chan+0x57c/0x64b
[  127.517026]  [<c02d1be7>] tty_read+0xc9/0xcd
[  127.521594]  [<c0159189>] vfs_read+0xa6/0x17d
[  127.526254]  [<c015950f>] sys_read+0x4b/0x75
[  127.530822]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  127.536206] writetest     D 00000003     0  3385   3315                     (NOTLB)
[  127.544531] f72c09e4 f72c09d4 00000004 00000003 c0111ca1 00000000 f72c099c c053c988 
[  127.552751]        f72c0a6c c2337a70 f73f41f0 f72c0a3c 00000202 c2227fe0 f72c09f8 00000202 
[  127.561884]        c2227fe0 f72c09f8 c2227560 00000003 00000192 8a911700 0000001c c2337a70 
[  127.571016] Call Trace:
[  127.573832]  [<c04086e1>] schedule_timeout+0x54/0xa5
[  127.579124]  [<c040866e>] io_schedule_timeout+0x29/0x33
[  127.584691]  [<c02880c4>] blk_congestion_wait+0x70/0x85
[  127.590254]  [<c014136b>] throttle_vm_writeout+0x69/0x7d
[  127.595909]  [<c014714d>] shrink_zone+0xe0/0xfa
[  127.600749]  [<c01471d4>] shrink_caches+0x6d/0x6f
[  127.605770]  [<c01472a6>] try_to_free_pages+0xd0/0x1b5
[  127.611243]  [<c013fa4b>] __alloc_pages+0x135/0x2e8
[  127.616450]  [<c0142d35>] kmem_getpages+0x34/0x9d
[  127.621473]  [<c0143b91>] cache_grow+0xaf/0x16c
[  127.626313]  [<c0143db2>] cache_alloc_refill+0x164/0x207
[  127.631968]  [<c014403b>] kmem_cache_alloc+0x4e/0x50
[  127.637257]  [<c01c4e05>] nfs_create_request+0x3d/0xcc
[  127.642735]  [<c01c9151>] nfs_update_request+0x172/0x1d9
[  127.648369]  [<c01c867e>] nfs_writepage_async+0x36/0x7e
[  127.653934]  [<c01c888f>] nfs_writepage+0x1a8/0x1dc
[  127.659134]  [<c017dd9b>] mpage_writepages+0x232/0x399
[  127.664604]  [<c01c88f1>] nfs_writepages+0x2e/0x11e
[  127.669804]  [<c01416f2>] do_writepages+0x2e/0x53
[  127.674823]  [<c013abd3>] __filemap_fdatawrite_range+0x97/0x99
[  127.681016]  [<c013ac0e>] filemap_fdatawrite+0x39/0x3d
[  127.686486]  [<c01c236a>] nfs_revalidate_mapping+0xd5/0x108
[  127.692411]  [<c01bfddb>] nfs_file_write+0x76/0x115
[  127.697610]  [<c0159307>] do_sync_write+0xa7/0xe7
[  127.702628]  [<c01593ed>] vfs_write+0xa6/0x17d
[  127.707377]  [<c01596ac>] sys_pwrite64+0x7d/0x81
[  127.712307]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  130.310296] SysRq : Show Memory
[  130.313648] Mem-info:
[  130.316052] DMA per-cpu:
[  130.318771] cpu 0 hot: low 0, high 0, batch 1 used:0
[  130.323977] cpu 0 cold: low 0, high 0, batch 1 used:0
[  130.329272] cpu 1 hot: low 0, high 0, batch 1 used:0
[  130.334479] cpu 1 cold: low 0, high 0, batch 1 used:0
[  130.339774] cpu 2 hot: low 0, high 0, batch 1 used:0
[  130.344980] cpu 2 cold: low 0, high 0, batch 1 used:0
[  130.350276] cpu 3 hot: low 0, high 0, batch 1 used:0
[  130.355480] cpu 3 cold: low 0, high 0, batch 1 used:0
[  130.360774] DMA32 per-cpu: empty
[  130.364213] Normal per-cpu:
[  130.367203] cpu 0 hot: low 0, high 186, batch 31 used:105
[  130.372861] cpu 0 cold: low 0, high 62, batch 15 used:49
[  130.378428] cpu 1 hot: low 0, high 186, batch 31 used:32
[  130.383966] cpu 1 cold: low 0, high 62, batch 15 used:6
[  130.389414] cpu 2 hot: low 0, high 186, batch 31 used:180
[  130.395041] cpu 2 cold: low 0, high 62, batch 15 used:14
[  130.400577] cpu 3 hot: low 0, high 186, batch 31 used:31
[  130.406148] cpu 3 cold: low 0, high 62, batch 15 used:9
[  130.411623] HighMem per-cpu:
[  130.414701] cpu 0 hot: low 0, high 186, batch 31 used:90
[  130.420267] cpu 0 cold: low 0, high 62, batch 15 used:52
[  130.425834] cpu 1 hot: low 0, high 186, batch 31 used:20
[  130.431399] cpu 1 cold: low 0, high 62, batch 15 used:12
[  130.436966] cpu 2 hot: low 0, high 186, batch 31 used:181
[  130.442623] cpu 2 cold: low 0, high 62, batch 15 used:8
[  130.448099] cpu 3 hot: low 0, high 186, batch 31 used:0
[  130.453576] cpu 3 cold: low 0, high 62, batch 15 used:14
[  130.459143] Free pages:        9528kB (3064kB HighMem)
[  130.464531] Active:459749 inactive:37222 dirty:56603 writeback:241915 unstable:0 free:2382 slab:15300 mapped:9000 pagetables:97
[  130.476527] DMA free:3576kB min:68kB low:84kB high:100kB active:2696kB inactive:404kB present:16384kB pages_scanned:0 all_unreclaimable? no
[  130.489607] lowmem_reserve[]: 0 0 880 2031
[  130.494126] DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
[  130.505935] lowmem_reserve[]: 0 0 880 2031
[  130.510455] Normal free:2888kB min:3756kB low:4692kB high:5632kB active:716876kB inactive:96820kB present:901120kB pages_scanned:250 all_unreclaimable? no
[  130.524885] lowmem_reserve[]: 0 0 0 9211
[  130.529223] HighMem free:3064kB min:512kB low:1740kB high:2968kB active:1119424kB inactive:51664kB present:1179092kB pages_scanned:0 all_unreclaimable? no
[  130.543656] lowmem_reserve[]: 0 0 0 0
[  130.547723] DMA: 2*4kB 0*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 3576kB
[  130.558613] DMA32: empty
[  130.561332] Normal: 0*4kB 1*8kB 6*16kB 1*32kB 1*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2888kB
[  130.572499] HighMem: 204*4kB 57*8kB 22*16kB 15*32kB 7*64kB 2*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3064kB
[  130.584205] Swap cache: add 0, delete 0, find 0/0, race 0+0
[  130.590041] Free swap  = 4200956kB
[  130.593620] Total swap = 4200956kB
[  130.597196] Free swap:       4200956kB
[  130.609675] 524149 pages of RAM
[  130.612986] 294773 pages of HIGHMEM
[  130.616655] 6122 reserved pages
[  130.619961] 399280 pages shared
[  130.623267] 0 pages swap cached
[  130.626575] 56603 pages dirty
[  130.629703] 241915 pages writeback
[  130.633280] 9000 pages mapped
[  130.636407] 15300 pages slab
[  130.639443] 97 pages pagetables

Re: nfs unhappiness with memory pressure

[Trond Myklebust]

On Tue, 2005-12-06 at 07:52 +1100, Nick Piggin wrote:

> The VM doesn't expect to have to rely on pdflush to write out pages
> for it. ->writepage should be enough. Adding wakeup_pdflush here
> actually could do the wrong thing for non-NFS filesystems if it
> starts more writeback.

nr_unstable is not going to be set for non-NFS filesystems. 'unstable'
is a caching state in which pages have been written out to the NFS
server, but the server has not yet flushed the data to disk.

Cheers,
  Trond

Re: nfs unhappiness with memory pressure

[Trond Myklebust]

On Mon, 2005-12-05 at 16:18 -0500, Trond Myklebust wrote:
> On Tue, 2005-12-06 at 07:52 +1100, Nick Piggin wrote:
> 
> > The VM doesn't expect to have to rely on pdflush to write out pages
> > for it. ->writepage should be enough. Adding wakeup_pdflush here
> > actually could do the wrong thing for non-NFS filesystems if it
> > starts more writeback.
> 
> nr_unstable is not going to be set for non-NFS filesystems. 'unstable'
> is a caching state in which pages have been written out to the NFS
> server, but the server has not yet flushed the data to disk.

...and most important of all: 'unstable' does _not_ mean that I/O is
active on those pages (unlike the apparent assumption in
vm_throttle_write.
That is why the choice is either to kick pdflush there, or to remove
nr_unstable from the accounting in that loop.

Cheers,
  Trond

Re: nfs unhappiness with memory pressure

[Kenny Simpson]

[-- Attachment #1: Type: text/plain, Size: 714 bytes --]

--- Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> nr_unstable is not going to be set for non-NFS filesystems. 'unstable'
> is a caching state in which pages have been written out to the NFS
> server, but the server has not yet flushed the data to disk.

The NetApp always seems to return stable writes (even when the request are not).

Either way, I ran the system under minimal user mode (i.e. init=/bin/bash), to try to cut down on
the superfluous processes (cron, ntp, etc..).

I ran the test again.. same general result.
The tesr program is unkillable - even with sysrq's.

-Kenny



		
__________________________________ 
Start your day with Yahoo! - Make it your home page! 
http://www.yahoo.com/r/hs

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 3165733884-sysrq.log --]
[-- Type: text/x-log; name="sysrq.log", Size: 88511 bytes --]

root@(none):/mnt/src# ./writetest -m /mnt/foo
pwrite to 200000  2097152
wrote 100352k 98M bytes in 1133800490.814289 seconds -> 0.000000M/sec
wrote 137216k 134M bytes in 1.000662 seconds -> 140.416228M/sec
wrote 143360k 140M bytes in 1.021878 seconds -> 143.657697M/sec
wrote 126976k 124M bytes in 0.993586 seconds -> 130.862778M/sec
wrote 141312k 138M bytes in 1.006713 seconds -> 143.738571M/sec
wrote 145408k 142M bytes in 0.980652 seconds -> 151.835505M/sec
wrote 112640k 110M bytes in 1.034777 seconds -> 111.466876M/sec
wrote 139264k 136M bytes in 0.967575 seconds -> 147.385304M/sec
wrote 106496k 104M bytes in 1.010874 seconds -> 107.878830M/sec
wrote 137216k 134M bytes in 0.990759 seconds -> 141.819740M/sec
wrote 124928k 122M bytes in 1.005555 seconds -> 127.219567M/sec
wrote 133120k 130M bytes in 1.016719 seconds -> 134.073308M/sec
wrote 129024k 126M bytes in 1.017694 seconds -> 129.823479M/sec
wrote 122880k 120M bytes in 0.963566 seconds -> 130.586924M/sec
wrote 102400k 100M bytes in 1.000683 seconds -> 104.786031M/sec
wrote 90112k 88M bytes in 1.083051 seconds -> 85.198839M/sec
[  744.911172] SysRq : HELP : loglevel0-8 reBoot tErm Full kIll saK showMem Nice powerOff showPc unRaw Sync showTasks Unmount 
[  748.595938] SysRq : Show Regs
[  748.599109] 
[  748.600699] Pid: 0, comm:              swapper
[  748.605363] EIP: 0060:[<c0100d4f>] CPU: 0
[  748.609578] EIP is at default_idle+0x36/0x56
[  748.614896]  EFLAGS: 00000246    Not tainted  (2.6.15-rc5+nfs+k2)
[  748.621274] EAX: 00000000 EBX: c0503000 ECX: c220e2e0 EDX: c0503000
[  748.627837] ESI: c053c380 EDI: c053c300 EBP: c0503fac DS: 007b ES: 007b
[  748.634804] CR0: 8005003b CR2: b7afd000 CR3: 00540000 CR4: 000006d0
[  751.044910] SysRq : Show State
[  751.048172] 
[  751.048173]                                                sibling
[  751.056232]   task             PC      pid father child younger older
[  751.062974] bash          S 00000001     0     1      0     2               (NOTLB)
[  751.071296] c2331f08 c2331ef8 00000004 00000001 00000044 c21fd780 c04938a8 000200d2 
[  751.079518]        c2330a70 c2331eec c013f968 000200d2 00000000 c04938a8 00000044 0000000e 
[  751.088650]        c21fd780 00000010 c2217560 00000001 0000391a 0183f473 000000a6 c2330550 
[  751.097780] Call Trace:
[  751.100597]  [<c011fa57>] do_wait+0x18a/0x376
[  751.105258]  [<c011fd03>] sys_wait4+0x3e/0x42
[  751.109918]  [<c011fd2e>] sys_waitpid+0x27/0x2b
[  751.114760]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  751.120145] migration/0   S 00000002     0     2      1             3       (L-TLB)
[  751.128365] c2339f94 c2339f84 00000004 00000002 c0116b9c c2337a70 c22275a8 00000001 
[  751.136510]        c220f5ac 00000001 00000073 c220f958 c220f958 c22275a8 c220f5a8 00000000 
[  751.145590]        c2227560 00000003 c220f560 00000000 000017c2 aa8db191 000000a9 c048db40 
[  751.154653] Call Trace:
[  751.157435]  [<c0118d0c>] migration_thread+0x81/0x10d
[  751.162753]  [<c012fe68>] kthread+0xb7/0xbc
[  751.167179]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  751.172584] ksoftirqd/0   S 00000000     0     3      1             4     2 (L-TLB)
[  751.180802] c233af98 c2337030 c220f560 00000000 000004db 000e078d 00000000 c233ba70 
[  751.188926]        c2337030 c233715c 00000001 c2331f5c c2330a70 ffffffff c2337030 c233af6c 
[  751.197945]        00000000 00000000 c220f560 00000000 000002b2 000ea053 00000000 c2330a70 
[  751.206965] Call Trace:
[  751.209747]  [<c0121786>] ksoftirqd+0xd6/0xf5
[  751.214350]  [<c012fe68>] kthread+0xb7/0xbc
[  751.218774]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  751.224179] migration/1   S 00000001     0     4      1             5     3 (L-TLB)
[  751.232400] c233cf94 c233cf80 00000004 00000001 00000082 00000001 c233cf44 00000000 
[  751.240522]        c048db40 c2337a70 00000003 c2227560 c2330a70 00000009 00000008 c2330a70 
[  751.249542]        00000000 c2217ec0 c2217560 00000001 00000eaf 3827d003 000000a1 c2330a70 
[  751.258563] Call Trace:
[  751.261344]  [<c0118d0c>] migration_thread+0x81/0x10d
[  751.266661]  [<c012fe68>] kthread+0xb7/0xbc
[  751.271085]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  751.276490] ksoftirqd/1   S 00000001     0     5      1             6     4 (L-TLB)
[  751.284710] c2345f98 c2345f88 00000004 00000001 000000c7 00180089 00000000 c233b550 
[  751.292832]        c233b550 c2330a70 00000001 c2331f28 c2345000 ffffffff c233b550 c2345f6c 
[  751.301852]        c0407ed4 c22175a8 c2217560 00000001 00000177 2fc9a4b5 00000000 c2330550 
[  751.310872] Call Trace:
[  751.313654]  [<c0121786>] ksoftirqd+0xd6/0xf5
[  751.318257]  [<c012fe68>] kthread+0xb7/0xbc
[  751.322680]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  751.328085] migration/2   S 00000004     0     6      1             7     5 (L-TLB)
[  751.336306] c2346f94 c2346f80 00000004 00000004 00000082 c2330550 c22175a8 00000000 
[  751.344427]        c2337a70 c2330550 00000001 c2217560 f7e79030 00000002 00000002 f7e79030 
[  751.353447]        00000000 c221fec0 c221f560 00000002 00000c37 37d42ab3 00000058 f7e79030 
[  751.362469] Call Trace:
[  751.365251]  [<c0118d0c>] migration_thread+0x81/0x10d
[  751.370568]  [<c012fe68>] kthread+0xb7/0xbc
[  751.374993]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  751.380399] ksoftirqd/2   S 00000002     0     7      1             8     6 (L-TLB)
[  751.388618] c234df98 c234ca70 c221f560 00000002 00000163 00275526 00000000 c234ca70 
[  751.396740]        c234ca70 c234cb9c 00000001 c2331f28 c234d000 ffffffff c234ca70 c234df6c 
[  751.405760]        c0407ed4 c221f5a8 c221f560 00000002 00000043 00275786 00000000 c2330030 
[  751.414781] Call Trace:
[  751.417562]  [<c0121786>] ksoftirqd+0xd6/0xf5
[  751.422165]  [<c012fe68>] kthread+0xb7/0xbc
[  751.426590]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  751.431995] migration/3   S 00000003     0     8      1             9     7 (L-TLB)
[  751.440214] c234ef94 c234ef80 00000004 00000003 00000082 00000001 c234ef44 00000000 
[  751.448336]        c2330030 c2330550 00000001 c2217560 f70f8030 00000011 00000002 f70f8030 
[  751.457359]        00000000 c2227ec0 c2227560 00000003 0000105f 353ed25e 00000058 f70f8030 
[  751.466380] Call Trace:
[  751.469162]  [<c0118d0c>] migration_thread+0x81/0x10d
[  751.474479]  [<c012fe68>] kthread+0xb7/0xbc
[  751.478903]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  751.484309] ksoftirqd/3   S 00000003     0     9      1            10     8 (L-TLB)
[  751.492530] c2353f98 c2353f88 00000004 00000003 000000c9 00367434 00000000 c234c030 
[  751.500654]        c234c030 c2361550 00000001 c2331f28 c2353000 ffffffff c234c030 c2353f6c 
[  751.509673]        c0407ed4 c22275a8 c2227560 00000003 00000182 364e7604 00000058 c2337a70 
[  751.518694] Call Trace:
[  751.521476]  [<c0121786>] ksoftirqd+0xd6/0xf5
[  751.526078]  [<c012fe68>] kthread+0xb7/0xbc
[  751.530502]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  751.535907] events/0      R running     0    10      1            11     9 (L-TLB)
[  751.544036] events/1      S 00000001     0    11      1            12    10 (L-TLB)
[  751.552256] c236af34 c236af24 00000004 00000001 00000202 00000001 c2354540 000007d0 
[  751.560377]        c236aef4 c012bb5c c22191b8 0006eafe c2300790 c2300780 00000001 c236aefc 
[  751.569398]        c2348930 c2348928 c2217560 00000001 00000af9 cec7134c 000000ae c2330550 
[  751.578417] Call Trace:
[  751.581199]  [<c012bd90>] worker_thread+0x207/0x225
[  751.586336]  [<c012fe68>] kthread+0xb7/0xbc
[  751.590761]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  751.596166] events/2      S 00000004     0    12      1            13    11 (L-TLB)
[  751.604385] c2359f34 c2359f24 00000004 00000004 00000202 00000001 c2354540 000007d0 
[  751.612506]        c2359ef4 c012bb5c c22211b8 0006eb01 00000000 c2300780 c22211a4 c2359efc 
[  751.621526]        c23488b0 c23488a8 c221f560 00000002 00000801 cef4a869 000000ae c2330030 
[  751.630546] Call Trace:
[  751.633328]  [<c012bd90>] worker_thread+0x207/0x225
[  751.638465]  [<c012fe68>] kthread+0xb7/0xbc
[  751.642889]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  751.648295] events/3      S 00000003     0    13      1            14    12 (L-TLB)
[  751.656514] c236cf34 c236cf24 00000004 00000003 00000202 00000001 c2354540 000007d0 
[  751.664638]        c236cef4 c012bb5c c22291b8 0006eb02 00000000 c2300780 c22291a4 c236cefc 
[  751.673658]        c2348830 c2348828 c2227560 00000003 00000529 cf03c93c 000000ae c2337a70 
[  751.682681] Call Trace:
[  751.685462]  [<c012bd90>] worker_thread+0x207/0x225
[  751.690602]  [<c012fe68>] kthread+0xb7/0xbc
[  751.695027]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  751.700432] khelper       S 00000004     0    14      1            15    13 (L-TLB)
[  751.708654] c235af34 c235af24 00000004 00000004 00000000 c235aef4 c0117b5d f7ba3c70 
[  751.716776]        00000003 00000000 00000000 f7ba3d00 f7ba3cf4 f7ba3cf8 00000202 c235af1c 
[  751.725796]        c23487b0 c23487a8 c221f560 00000002 000001db da3e6193 00000056 c2330030 
[  751.734818] Call Trace:
[  751.737599]  [<c012bd90>] worker_thread+0x207/0x225
[  751.742737]  [<c012fe68>] kthread+0xb7/0xbc
[  751.747160]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  751.752565] kthread       S 00000001     0    15      1    20     144    14 (L-TLB)
[  751.760786] c2389f34 c2389f24 00000004 00000001 c2389ef0 c0117b5d f78ebd80 00000003 
[  751.768908]        00000000 00000000 f78ebe20 f78ebe14 f78ebe18 00000202 c2389f18 c0117cbb 
[  751.777928]        c2380b30 c2380b28 c2217560 00000001 00000187 fa724d73 00000066 c2330550 
[  751.786947] Call Trace:
[  751.789730]  [<c012bd90>] worker_thread+0x207/0x225
[  751.794868]  [<c012fe68>] kthread+0xb7/0xbc
[  751.799292]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  751.804697] kblockd/0     S 00000002     0    20     15            21       (L-TLB)
[  751.813752] c238bf34 c238bf20 00000004 00000002 c238bed0 c040906a c238bef8 00000000 
[  751.821916]        c0579414 f7f7aedc c238bee8 00000000 f7a95a70 f7f734f4 c23eef80 f7a95a70 
[  751.830997]        00000000 c220fec0 c220f560 00000000 000017d6 aee7a4ca 000000a9 f7a95a70 
[  751.840077] Call Trace:
[  751.842877]  [<c012bd90>] worker_thread+0x207/0x225
[  751.848051]  [<c012fe68>] kthread+0xb7/0xbc
[  751.852506]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  751.857949] kblockd/1     S 00000001     0    21     15            22    20 (L-TLB)
[  751.866271] f7d58f34 f7d58f24 00000004 00000001 f7d58ed0 c040906a f7d58ef8 c02fccc2 
[  751.874450]        c0579414 f7f7ac5c f7d58ee8 00000000 c23eef00 f7f734f4 c23eef00 f7f7359c 
[  751.883480]        c23eef30 c23eef28 c2217560 00000001 00000c15 9a287d8b 00000091 c2330550 
[  751.892508] Call Trace:
[  751.895292]  [<c012bd90>] worker_thread+0x207/0x225
[  751.900436]  [<c012fe68>] kthread+0xb7/0xbc
[  751.904863]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  751.910273] kblockd/2     S 00000004     0    22     15            23    21 (L-TLB)
[  751.918595] c238ff34 c238ff24 00000004 00000004 c238fed0 c040906a c238fef8 c02fccc2 
[  751.926724]        c0579414 d6cb69dc 00000080 00000000 c23eee80 f7f734f4 c23eee80 f7f7359c 
[  751.935754]        c23eeeb0 c23eeea8 c221f560 00000002 000011d4 afbc9d2d 000000a9 c2330030 
[  751.944784] Call Trace:
[  751.947568]  [<c012bd90>] worker_thread+0x207/0x225
[  751.952711]  [<c012fe68>] kthread+0xb7/0xbc
[  751.957139]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  751.962550] kblockd/3     S 00000003     0    23     15            24    22 (L-TLB)
[  751.970871] f7d59f34 f7d59f24 00000004 00000003 f7f7a2fc 000b884f 00000000 00000202 
[  751.979001]        00000001 f7f7a2fc f7d59ee8 c040906a f7d59f10 c02fccc2 c0579414 f7f7a2fc 
[  751.988039]        c23eee30 c23eee28 c2227560 00000003 00000d8c 29f94785 0000003a c2337a70 
[  751.997120] Call Trace:
[  751.999921]  [<c012bd90>] worker_thread+0x207/0x225
[  752.005093]  [<c012fe68>] kthread+0xb7/0xbc
[  752.009548]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  752.014990] kacpid        S 00000002     0    24     15           142    23 (L-TLB)
[  752.023312] f7d78f34 f7d78f24 00000004 00000002 00000000 00000001 00000100 00000000 
[  752.031490]        00000080 00000080 00000080 c05566ac c05566ac c05566a0 c2217ec0 c2217ec0 
[  752.040571]        00000000 00000202 c220f560 00000000 00000189 027f7e52 00000000 c048db40 
[  752.049651] Call Trace:
[  752.052452]  [<c012bd90>] worker_thread+0x207/0x225
[  752.057624]  [<c012fe68>] kthread+0xb7/0xbc
[  752.062078]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  752.067491] pdflush       S 00000002     0   142     15           143    24 (L-TLB)
[  752.075814] f7d5df68 f7d5df58 00000004 00000002 c0124c50 c0493a5c 0006f49d f7d5df68 
[  752.083979]        c014153c c0493a5c 0006f49d 00066be5 00000000 00000000 f7d5df18 00000400 
[  752.093099]        00000000 00000000 c220f560 00000000 000001d3 aecc404f 000000ae c048db40 
[  752.102231] Call Trace:
[  752.105048]  [<c0141df8>] __pdflush+0x84/0x199
[  752.109800]  [<c0141f4d>] pdflush+0x40/0x47
[  752.114279]  [<c012fe68>] kthread+0xb7/0xbc
[  752.118759]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  752.124233] pdflush       D 00000001     0   143     15           145   142 (L-TLB)
[  752.132554] f7d8aa48 f7d8aa38 00000004 00000001 f7d8aa34 00000202 f7d8a9f8 c01302cf 
[  752.140776]        f7b26080 c2330550 f7d8aa28 f7d8aa54 00000202 c2217fe0 f7d8aa5c 00000202 
[  752.149907]        c2217fe0 f7d8aa5c c2217560 00000001 000000ed dd6ce28f 000000ae c2330550 
[  752.159038] Call Trace:
[  752.161855]  [<c04086e1>] schedule_timeout+0x54/0xa5
[  752.167148]  [<c040866e>] io_schedule_timeout+0x29/0x33
[  752.172712]  [<c02880c4>] blk_congestion_wait+0x70/0x85
[  752.178280]  [<c014136b>] throttle_vm_writeout+0x69/0x7d
[  752.183936]  [<c014714d>] shrink_zone+0xe0/0xfa
[  752.188777]  [<c01471d4>] shrink_caches+0x6d/0x6f
[  752.193746]  [<c01472a6>] try_to_free_pages+0xd0/0x1b5
[  752.199157]  [<c013fa4b>] __alloc_pages+0x135/0x2e8
[  752.204300]  [<c0142d35>] kmem_getpages+0x34/0x9d
[  752.209264]  [<c0143b91>] cache_grow+0xaf/0x16c
[  752.214050]  [<c0143db2>] cache_alloc_refill+0x164/0x207
[  752.219639]  [<c014403b>] kmem_cache_alloc+0x4e/0x50
[  752.224871]  [<c01c4e05>] nfs_create_request+0x3d/0xcc
[  752.230283]  [<c01c9151>] nfs_update_request+0x172/0x1d9
[  752.235874]  [<c01c867e>] nfs_writepage_async+0x36/0x7e
[  752.241376]  [<c01c888f>] nfs_writepage+0x1a8/0x1dc
[  752.246518]  [<c017dd9b>] mpage_writepages+0x232/0x399
[  752.251930]  [<c01c88f1>] nfs_writepages+0x2e/0x11e
[  752.257073]  [<c01416f2>] do_writepages+0x2e/0x53
[  752.262067]  [<c017c072>] __sync_single_inode+0x60/0x1f1
[  752.267689]  [<c017c295>] __writeback_single_inode+0x92/0x195
[  752.273761]  [<c017c555>] sync_sb_inodes+0x1bd/0x2c2
[  752.279025]  [<c017c734>] writeback_inodes+0xda/0xe6
[  752.284287]  [<c01413e8>] background_writeout+0x69/0xa2
[  752.289819]  [<c0141e35>] __pdflush+0xc1/0x199
[  752.294545]  [<c0141f4d>] pdflush+0x40/0x47
[  752.298998]  [<c012fe68>] kthread+0xb7/0xbc
[  752.303452]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  752.308895] kswapd0       D 00000003     0   144      1           804    15 (L-TLB)
[  752.317218] f7dd8e14 f7dd8e04 00000004 00000003 f7dd8dac 00000004 00000001 c21ebd68 
[  752.325395]        c21ebd8c c2337a70 c21ebdd4 c21ebc00 00000202 c2227fe0 f7dd8e28 00000202 
[  752.334477]        c2227fe0 f7dd8e28 c2227560 00000003 0000013a d9cae466 000000ae c2337a70 
[  752.343561] Call Trace:
[  752.346348]  [<c04086e1>] schedule_timeout+0x54/0xa5
[  752.351581]  [<c040866e>] io_schedule_timeout+0x29/0x33
[  752.357081]  [<c02880c4>] blk_congestion_wait+0x70/0x85
[  752.362581]  [<c014136b>] throttle_vm_writeout+0x69/0x7d
[  752.368171]  [<c014714d>] shrink_zone+0xe0/0xfa
[  752.372956]  [<c0147611>] balance_pgdat+0x286/0x37e
[  752.378102]  [<c01477ee>] kswapd+0xe5/0x10f
[  752.382530]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  752.387941] aio/0         S 00000002     0   145     15           146   143 (L-TLB)
[  752.396263] f7e34f34 f7e34f20 00000004 00000002 00000000 00000001 00000100 00000000 
[  752.404393]        00000080 00000080 00000080 c05566ac f7dd7550 c05566a0 c2217ec0 f7dd7550 
[  752.413421]        00000000 c220fec0 c220f560 00000000 0000013f 2ca67fde 00000000 f7dd7550 
[  752.422449] Call Trace:
[  752.425233]  [<c012bd90>] worker_thread+0x207/0x225
[  752.430375]  [<c012fe68>] kthread+0xb7/0xbc
[  752.434802]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  752.440212] aio/1         S 00000000     0   146     15           147   145 (L-TLB)
[  752.448533] f7e1df34 c2361030 c22175a8 00000000 00000000 00000001 00000100 00000000 
[  752.456662]        00000080 00000000 2ca50f66 00000000 c2361030 00000000 c220fec0 c2361030 
[  752.465692]        00000000 c2217ec0 c2217560 00000001 0000016c 2ca6bdf9 00000000 c2361030 
[  752.474725] Call Trace:
[  752.477525]  [<c012bd90>] worker_thread+0x207/0x225
[  752.482699]  [<c012fe68>] kthread+0xb7/0xbc
[  752.487154]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  752.492596] aio/2         S 00000004     0   147     15           148   146 (L-TLB)
[  752.500917] f7e35f34 f7e35f24 00000004 00000004 00000000 00000001 00000100 00000000 
[  752.509093]        00000080 00000080 00000080 c05566ac c05566ac c05566a0 c2217ec0 c2217ec0 
[  752.518173]        00000000 00000202 c221f560 00000002 000001a5 2ca71c86 00000000 c2330030 
[  752.527257] Call Trace:
[  752.530057]  [<c012bd90>] worker_thread+0x207/0x225
[  752.535230]  [<c012fe68>] kthread+0xb7/0xbc
[  752.539685]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  752.545128] aio/3         S 00000003     0   148     15           149   147 (L-TLB)
[  752.553449] f7e1ef34 f7e1ef24 00000004 00000003 00000000 00000001 00000100 00000080 
[  752.561668]        00000080 00000000 00000000 c05566a0 c05566a0 00000000 c220fec0 c220fec0 
[  752.570747]        00000000 00000202 c2227560 00000003 00000191 2ca776c8 00000000 c2337a70 
[  752.579829] Call Trace:
[  752.582629]  [<c012bd90>] worker_thread+0x207/0x225
[  752.587803]  [<c012fe68>] kthread+0xb7/0xbc
[  752.592258]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  752.597733] xfslogd/0     S 00000000     0   149     15           150   148 (L-TLB)
[  752.606054] f7e37f34 c2361030 c220f5a8 00000000 00000000 00000001 00000100 00000000 
[  752.614275]        00000100 00000080 2e49f705 00000000 c2361030 c05566a0 c2217ec0 c2361030 
[  752.623405]        00000000 c220fec0 c220f560 00000000 0000016d 2e4aa29a 00000000 c2361030 
[  752.633358] Call Trace:
[  752.636175]  [<c012bd90>] worker_thread+0x207/0x225
[  752.641378]  [<c012fe68>] kthread+0xb7/0xbc
[  752.645857]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  752.651331] xfslogd/1     S 00000001     0   150     15           151   149 (L-TLB)
[  752.659652] f7e75f34 f7e75f20 00000004 00000001 00000000 00000001 00000100 00000000 
[  752.667875]        00000100 00000080 00000080 c05566a0 f7e36550 c05566ac c220fec0 f7e36550 
[  752.677006]        00000000 c2217ec0 c2217560 00000001 000000f4 2e4b28d6 00000000 f7e36550 
[  752.686136] Call Trace:
[  752.688953]  [<c012bd90>] worker_thread+0x207/0x225
[  752.694155]  [<c012fe68>] kthread+0xb7/0xbc
[  752.698634]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  752.704107] xfslogd/2     S 00000004     0   151     15           152   150 (L-TLB)
[  752.712427] f7e38f34 f7e38f24 00000004 00000004 00000000 00000001 00000100 00000080 
[  752.720649]        00000100 00000080 00000080 c05566ac c05566ac c05566a0 c2217ec0 c2217ec0 
[  752.729781]        00000000 00000202 c221f560 00000002 0000015d 2e4b69d2 00000000 c2330030 
[  752.738911] Call Trace:
[  752.741727]  [<c012bd90>] worker_thread+0x207/0x225
[  752.746928]  [<c012fe68>] kthread+0xb7/0xbc
[  752.751408]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  752.756882] xfslogd/3     S 00000003     0   152     15           153   151 (L-TLB)
[  752.765203] f7e76f34 f7e76f24 00000004 00000003 00000000 00000001 00000100 00000080 
[  752.773426]        00000100 00000080 00000080 c05566a0 c05566a0 c05566ac c220fec0 c220fec0 
[  752.782557]        00000000 00000202 c2227560 00000003 00000173 2e4bc489 00000000 c2337a70 
[  752.791687] Call Trace:
[  752.794503]  [<c012bd90>] worker_thread+0x207/0x225
[  752.799706]  [<c012fe68>] kthread+0xb7/0xbc
[  752.804185]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  752.809659] xfsdatad/0    S 00000000     0   153     15           154   152 (L-TLB)
[  752.817979] f7e3af34 c2361030 c220f5a8 00000000 00000000 00000001 00000100 00000000 
[  752.826200]        00000100 00000080 2e4b784f 00000000 c2361030 c05566a0 c2217ec0 c2361030 
[  752.835272]        00000000 c220fec0 c220f560 00000000 00000131 2e4c227a 00000000 c2361030 
[  752.844354] Call Trace:
[  752.847155]  [<c012bd90>] worker_thread+0x207/0x225
[  752.852329]  [<c012fe68>] kthread+0xb7/0xbc
[  752.856783]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  752.862226] xfsdatad/1    S 00000001     0   154     15           155   153 (L-TLB)
[  752.870547] f7e78f34 f7e78f20 00000004 00000001 00000000 00000001 00000100 00000000 
[  752.878723]        00000100 00000080 00000080 c05566a0 f7e3ba70 c05566ac c220fec0 f7e3ba70 
[  752.887803]        00000000 c2217ec0 c2217560 00000001 000000f6 2e4c9b2f 00000000 f7e3ba70 
[  752.896885] Call Trace:
[  752.899686]  [<c012bd90>] worker_thread+0x207/0x225
[  752.904859]  [<c012fe68>] kthread+0xb7/0xbc
[  752.909313]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  752.914755] xfsdatad/2    S 00000004     0   155     15           156   154 (L-TLB)
[  752.923077] f7e3cf34 f7e3cf24 00000004 00000004 00000000 00000001 00000100 00000080 
[  752.931254]        00000100 00000080 00000080 c05566ac c05566ac c05566a0 c2217ec0 c2217ec0 
[  752.940335]        00000000 00000202 c221f560 00000002 00000136 2e4cd917 00000000 c2330030 
[  752.949449] Call Trace:
[  752.952265]  [<c012bd90>] worker_thread+0x207/0x225
[  752.957468]  [<c012fe68>] kthread+0xb7/0xbc
[  752.961947]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  752.967421] xfsdatad/3    S 00000003     0   156     15           157   155 (L-TLB)
[  752.975743] f7e7af34 f7e7af24 00000004 00000003 00000000 00000001 00000100 00000080 
[  752.983965]        00000100 00000080 00000080 c05566a0 c05566a0 c05566ac c220fec0 c220fec0 
[  752.993096]        00000000 00000202 c2227560 00000003 0000013e 2e4d313a 00000000 c2337a70 
[  753.002227] Call Trace:
[  753.005043]  [<c012bd90>] worker_thread+0x207/0x225
[  753.010247]  [<c012fe68>] kthread+0xb7/0xbc
[  753.014726]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  753.020200] xfsbufd       S 00000004     0   157     15           742   156 (L-TLB)
[  753.028521] f7e3df44 f7e3df34 00000004 00000004 00000100 00000080 00000080 00000000 
[  753.036743]        c2337a70 c2330030 af846c0b 000000a9 00000202 c221ffe0 f7e3df58 00000202 
[  753.045874]        c221ffe0 f7e3df58 c221f560 00000002 00000096 abf07eb3 000000ae c2330030 
[  753.055005] Call Trace:
[  753.057822]  [<c04086e1>] schedule_timeout+0x54/0xa5
[  753.063115]  [<c040874c>] schedule_timeout_interruptible+0x1a/0x1c
[  753.069673]  [<c0272754>] xfsbufd+0x4e/0x1b0
[  753.074245]  [<c012fe68>] kthread+0xb7/0xbc
[  753.078723]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  753.084196] kseriod       S 00000002     0   742     15          1514   157 (L-TLB)
[  753.092517] f7f16f5c f7f16f4c 00000004 00000002 c02eff05 c04b8f24 f7f16f10 c02efe91 
[  753.100730]        c04b8f08 c04a8738 c04b8f24 c04a63e0 f7f16f34 c02ef54e c04b8f08 c04a8738 
[  753.109812]        00000282 00000202 c220f560 00000000 000a8380 b437cbfc 00000000 c048db40 
[  753.118894] Call Trace:
[  753.121695]  [<c02e4762>] serio_thread+0xea/0xec
[  753.126600]  [<c012fe68>] kthread+0xb7/0xbc
[  753.131055]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  753.136498] kjournald     S 00000002     0   804      1          1519   144 (L-TLB)
[  753.144821] f7eaff58 f7eaff48 00000004 00000002 f726204c 00000000 00000000 f7fc6800 
[  753.152997]        00000000 f7171400 f7afcfa8 000003d0 00000202 c2217fe0 f7eaffb8 f7eaff2c 
[  753.162079]        f7fb086c f7fb0864 c220f560 00000000 000004a8 8a6284da 000000a6 c048db40 
[  753.171159] Call Trace:
[  753.173961]  [<c01b0eef>] kjournald+0x1d6/0x223
[  753.178777]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  753.184220] rpciod/0      D 00000002     0  1514     15          1515   742 (L-TLB)
[  753.192540] f7b96aa8 f7b96a98 00000004 00000002 f7ff8f18 00000001 00000003 00000b9d 
[  753.200669]        324272cc c048db40 02000000 f7e4f46c 00000202 c220ffe0 f7b96abc 00000202 
[  753.209698]        c220ffe0 f7b96abc c220f560 00000000 0000012d dd6cd092 000000ae c048db40 
[  753.218726] Call Trace:
[  753.221510]  [<c04086e1>] schedule_timeout+0x54/0xa5
[  753.226743]  [<c040866e>] io_schedule_timeout+0x29/0x33
[  753.232242]  [<c02880c4>] blk_congestion_wait+0x70/0x85
[  753.237743]  [<c014136b>] throttle_vm_writeout+0x69/0x7d
[  753.243332]  [<c014714d>] shrink_zone+0xe0/0xfa
[  753.248116]  [<c01471d4>] shrink_caches+0x6d/0x6f
[  753.253080]  [<c01472a6>] try_to_free_pages+0xd0/0x1b5
[  753.258492]  [<c013fa4b>] __alloc_pages+0x135/0x2e8
[  753.263635]  [<c03b74ad>] tcp_sendmsg+0xaa0/0xb78
[  753.268602]  [<c03d4666>] inet_sendmsg+0x48/0x53
[  753.273476]  [<c0388716>] sock_sendmsg+0xb8/0xd3
[  753.278353]  [<c0388773>] kernel_sendmsg+0x42/0x4f
[  753.283406]  [<c038bc00>] sock_no_sendpage+0x5e/0x77
[  753.288638]  [<c03ee7af>] xs_tcp_send_request+0x2af/0x375
[  753.294320]  [<c03ed4c5>] xprt_transmit+0x4f/0x20b
[  753.299374]  [<c03ebce1>] call_transmit+0x68/0xc8
[  753.304340]  [<c03f0962>] __rpc_execute+0x5c/0x1ea
[  753.309423]  [<c03f0b27>] rpc_async_schedule+0x11/0x15
[  753.314844]  [<c012bd1d>] worker_thread+0x194/0x225
[  753.319987]  [<c012fe68>] kthread+0xb7/0xbc
[  753.324415]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  753.329826] rpciod/1      S 00000001     0  1515     15          1516  1514 (L-TLB)
[  753.338147] f78dbf34 f78dbf20 00000004 00000001 00000000 c03ed2f0 f7b13fc0 00000000 
[  753.346277]        f7b13e00 c3a2e504 c3a2e504 00000000 f7d89a70 c03ebce1 c3a2e504 f7d89a70 
[  753.355308]        00000000 c2217ec0 c2217560 00000001 00002aff 0db3409e 000000aa f7d89a70 
[  753.364338] Call Trace:
[  753.367122]  [<c012bd90>] worker_thread+0x207/0x225
[  753.372265]  [<c012fe68>] kthread+0xb7/0xbc
[  753.376695]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  753.382106] rpciod/2      S 00000004     0  1516     15          1517  1515 (L-TLB)
[  753.390429] f7ba6f34 f7ba6f24 00000004 00000004 00000000 00000000 00000003 00000000 
[  753.398570]        c4866584 00000000 f7ba6eec c0296593 00000000 f7ba6f00 c04071af f7b13e00 
[  753.407652]        f71717b0 f71717a8 c221f560 00000002 000021eb af7442db 000000a9 c2330030 
[  753.416734] Call Trace:
[  753.419535]  [<c012bd90>] worker_thread+0x207/0x225
[  753.424709]  [<c012fe68>] kthread+0xb7/0xbc
[  753.429970]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  753.435413] rpciod/3      S 00000003     0  1517     15                1516 (L-TLB)
[  753.443733] f7154f34 f7154f20 00000004 00000003 00000000 00000000 00000003 00000000 
[  753.451909]        c2330030 00000000 f7154eec c0296593 f717b030 00000040 c04071af f717b030 
[  753.460991]        00000000 c2227ec0 c2227560 00000003 00003481 af9995ef 000000a9 f717b030 
[  753.470072] Call Trace:
[  753.472873]  [<c012bd90>] worker_thread+0x207/0x225
[  753.478046]  [<c012fe68>] kthread+0xb7/0xbc
[  753.482501]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  753.487976] portmap       S 00000002     0  1519      1          1541   804 (NOTLB)
[  753.496297] f717cf00 f717ceec 00000004 00000002 00000202 f7b8100c 00000202 00000000 
[  753.504473]        c2330550 f7b81000 f7e87080 00000202 f7e79030 00000003 f717ced0 f7e79030 
[  753.513568]        00000000 c220fec0 c220f560 00000000 0000c0e9 607461be 00000089 f7e79030 
[  753.522700] Call Trace:
[  753.525517]  [<c0408730>] schedule_timeout+0xa3/0xa5
[  753.530808]  [<c016ce88>] do_poll+0x9a/0xb9
[  753.535289]  [<c016cffd>] sys_poll+0x156/0x221
[  753.540039]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  753.545422] kjournald     S 00000002     0  1541      1          1544  1519 (L-TLB)
[  753.553744] f71d3f58 f71d3f48 00000004 00000002 f724d02c 00000000 00000000 f71d1800 
[  753.561965]        00000000 f7171e00 f7afcd6c 0000020c 00000202 c220ffe0 f71d3fb8 f71d3f2c 
[  753.571070]        f7fb066c f7fb0664 c220f560 00000000 00000585 2cebc346 000000a7 c048db40 
[  753.580151] Call Trace:
[  753.582953]  [<c01b0eef>] kjournald+0x1d6/0x223
[  753.587766]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  753.593210] writetest     D 00000003     0  1544      1                1541 (NOTLB)
[  753.601532] f7adce10 f7adce00 00000004 00000003 f7adcdd4 f7ecfd80 f7adcde8 c01c7bcf 
[  753.609709]        f71b25c0 00000008 c01c7a21 f7adcdcc 00000082 f7b13600 f7adcdd4 f71b25c0 
[  753.618790]        f7adcdd4 f7adcdd4 c2227560 00000003 00000630 c562b75c 000000a9 c2337a70 
[  753.627872] Call Trace:
[  753.630672]  [<c040863b>] io_schedule+0x26/0x30
[  753.635488]  [<c013ab27>] sync_page+0x39/0x4e
[  753.640123]  [<c0408929>] __wait_on_bit_lock+0x58/0x67
[  753.645561]  [<c013b343>] __lock_page+0x84/0x8c
[  753.650402]  [<c013c3c9>] filemap_nopage+0x1e4/0x347
[  753.655695]  [<c014b4b4>] do_no_page+0x70/0x25b
[  753.660536]  [<c014b812>] __handle_mm_fault+0xef/0x22a
[  753.666011]  [<c0409a84>] do_page_fault+0x179/0x5e6
[  753.671214]  [<c01039cf>] error_code+0x4f/0x54

[  807.227512] SysRq : Show State
[  807.230773] 
[  807.230774]                                                sibling
[  807.238834]   task             PC      pid father child younger older
[  807.245576] bash          S 00000001     0     1      0     2               (NOTLB)
[  807.253896] c2331f08 c2331ef8 00000004 00000001 00000044 c21fd780 c04938a8 000200d2 
[  807.262113]        c2330a70 c2331eec c013f968 000200d2 00000000 c04938a8 00000044 0000000e 
[  807.271238]        c21fd780 00000010 c2217560 00000001 0000391a 0183f473 000000a6 c2330550 
[  807.280363] Call Trace:
[  807.283179]  [<c011fa57>] do_wait+0x18a/0x376
[  807.287838]  [<c011fd03>] sys_wait4+0x3e/0x42
[  807.292495]  [<c011fd2e>] sys_waitpid+0x27/0x2b
[  807.297333]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  807.302714] migration/0   S 00000002     0     2      1             3       (L-TLB)
[  807.311031] c2339f94 c2339f84 00000004 00000002 c0116b9c c2337a70 c22275a8 00000001 
[  807.319190]        c220f5ac 00000001 00000073 c220f958 c220f958 c22275a8 c220f5a8 00000000 
[  807.328211]        c2227560 00000003 c220f560 00000000 000017c2 aa8db191 000000a9 c048db40 
[  807.337230] Call Trace:
[  807.340011]  [<c0118d0c>] migration_thread+0x81/0x10d
[  807.345330]  [<c012fe68>] kthread+0xb7/0xbc
[  807.349757]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  807.355163] ksoftirqd/0   S 00000000     0     3      1             4     2 (L-TLB)
[  807.363383] c233af98 c2337030 c220f560 00000000 000004db 000e078d 00000000 c233ba70 
[  807.371505]        c2337030 c233715c 00000001 c2331f5c c2330a70 ffffffff c2337030 c233af6c 
[  807.380526]        00000000 00000000 c220f560 00000000 000002b2 000ea053 00000000 c2330a70 
[  807.389545] Call Trace:
[  807.392327]  [<c0121786>] ksoftirqd+0xd6/0xf5
[  807.396929]  [<c012fe68>] kthread+0xb7/0xbc
[  807.401353]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  807.406758] migration/1   S 00000001     0     4      1             5     3 (L-TLB)
[  807.414979] c233cf94 c233cf80 00000004 00000001 00000082 00000001 c233cf44 00000000 
[  807.423103]        c048db40 c2337a70 00000003 c2227560 c2330a70 00000009 00000008 c2330a70 
[  807.432126]        00000000 c2217ec0 c2217560 00000001 00000eaf 3827d003 000000a1 c2330a70 
[  807.441147] Call Trace:
[  807.443931]  [<c0118d0c>] migration_thread+0x81/0x10d
[  807.449250]  [<c012fe68>] kthread+0xb7/0xbc
[  807.453675]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  807.459083] ksoftirqd/1   S 00000001     0     5      1             6     4 (L-TLB)
[  807.467305] c2345f98 c2345f88 00000004 00000001 000000c7 00180089 00000000 c233b550 
[  807.475432]        c233b550 c2330a70 00000001 c2331f28 c2345000 ffffffff c233b550 c2345f6c 
[  807.484477]        c0407ed4 c22175a8 c2217560 00000001 00000177 2fc9a4b5 00000000 c2330550 
[  807.493558] Call Trace:
[  807.496359]  [<c0121786>] ksoftirqd+0xd6/0xf5
[  807.500994]  [<c012fe68>] kthread+0xb7/0xbc
[  807.505448]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  807.510890] migration/2   S 00000004     0     6      1             7     5 (L-TLB)
[  807.519212] c2346f94 c2346f80 00000004 00000004 00000082 c2330550 c22175a8 00000000 
[  807.527390]        c2337a70 c2330550 00000001 c2217560 f7e79030 00000002 00000002 f7e79030 
[  807.536470]        00000000 c221fec0 c221f560 00000002 00000c37 37d42ab3 00000058 f7e79030 
[  807.545552] Call Trace:
[  807.548352]  [<c0118d0c>] migration_thread+0x81/0x10d
[  807.553706]  [<c012fe68>] kthread+0xb7/0xbc
[  807.558161]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  807.563602] ksoftirqd/2   S 00000002     0     7      1             8     6 (L-TLB)
[  807.571923] c234df98 c234ca70 c221f560 00000002 00000163 00275526 00000000 c234ca70 
[  807.580100]        c234ca70 c234cb9c 00000001 c2331f28 c234d000 ffffffff c234ca70 c234df6c 
[  807.589183]        c0407ed4 c221f5a8 c221f560 00000002 00000043 00275786 00000000 c2330030 
[  807.598264] Call Trace:
[  807.601065]  [<c0121786>] ksoftirqd+0xd6/0xf5
[  807.605700]  [<c012fe68>] kthread+0xb7/0xbc
[  807.610155]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  807.615598] migration/3   S 00000003     0     8      1             9     7 (L-TLB)
[  807.623918] c234ef94 c234ef80 00000004 00000003 00000082 00000001 c234ef44 00000000 
[  807.632095]        c2330030 c2330550 00000001 c2217560 f70f8030 00000011 00000002 f70f8030 
[  807.641174]        00000000 c2227ec0 c2227560 00000003 0000105f 353ed25e 00000058 f70f8030 
[  807.650256] Call Trace:
[  807.653057]  [<c0118d0c>] migration_thread+0x81/0x10d
[  807.658410]  [<c012fe68>] kthread+0xb7/0xbc
[  807.662866]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  807.668308] ksoftirqd/3   S 00000003     0     9      1            10     8 (L-TLB)
[  807.676629] c2353f98 c2353f88 00000004 00000003 000000c9 00367434 00000000 c234c030 
[  807.684806]        c234c030 c2361550 00000001 c2331f28 c2353000 ffffffff c234c030 c2353f6c 
[  807.693887]        c0407ed4 c22275a8 c2227560 00000003 00000182 364e7604 00000058 c2337a70 
[  807.702968] Call Trace:
[  807.705768]  [<c0121786>] ksoftirqd+0xd6/0xf5
[  807.710402]  [<c012fe68>] kthread+0xb7/0xbc
[  807.714856]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  807.720299] events/0      R running     0    10      1            11     9 (L-TLB)
[  807.728530] events/1      S 00000001     0    11      1            12    10 (L-TLB)
[  807.736850] c236af34 c236af24 00000004 00000001 00000202 00000001 c2354540 000007d0 
[  807.745072]        c236aef4 c012bb5c c22191b8 0007bdee c2300290 c2300780 00000004 c236aefc 
[  807.755044]        c2348930 c2348928 c2217560 00000001 00000918 bf47472a 000000bb c2330550 
[  807.764175] Call Trace:
[  807.766991]  [<c012bd90>] worker_thread+0x207/0x225
[  807.772194]  [<c012fe68>] kthread+0xb7/0xbc
[  807.776673]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  807.782116] events/2      S 00000004     0    12      1            13    11 (L-TLB)
[  807.790437] c2359f34 c2359f24 00000004 00000004 00000202 00000001 c2354540 000007d0 
[  807.798612]        c2359ef4 c012bb5c c22211b8 0007bdf1 f7e6f1d0 c2300780 00000002 c2359efc 
[  807.807692]        c23488b0 c23488a8 c221f560 00000002 00000805 bf74eeb2 000000bb c2330030 
[  807.816810] Call Trace:
[  807.819624]  [<c012bd90>] worker_thread+0x207/0x225
[  807.824824]  [<c012fe68>] kthread+0xb7/0xbc
[  807.829302]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  807.834772] events/3      S 00000003     0    13      1            14    12 (L-TLB)
[  807.843088] c236cf34 c236cf24 00000004 00000003 00000202 00000001 c2354540 000007d0 
[  807.851306]        c236cef4 c012bb5c c22291b8 0007bdf2 00000000 c2300780 c22291a4 c236cefc 
[  807.860430]        c2348830 c2348828 c2227560 00000003 00000521 bf840f13 000000bb c2337a70 
[  807.869556] Call Trace:
[  807.872371]  [<c012bd90>] worker_thread+0x207/0x225
[  807.877571]  [<c012fe68>] kthread+0xb7/0xbc
[  807.882047]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  807.887517] khelper       S 00000004     0    14      1            15    13 (L-TLB)
[  807.895835] c235af34 c235af24 00000004 00000004 00000000 c235aef4 c0117b5d f7ba3c70 
[  807.904053]        00000003 00000000 00000000 f7ba3d00 f7ba3cf4 f7ba3cf8 00000202 c235af1c 
[  807.913140]        c23487b0 c23487a8 c221f560 00000002 000001db da3e6193 00000056 c2330030 
[  807.922272] Call Trace:
[  807.925088]  [<c012bd90>] worker_thread+0x207/0x225
[  807.930291]  [<c012fe68>] kthread+0xb7/0xbc
[  807.934771]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  807.940245] kthread       S 00000001     0    15      1    20     144    14 (L-TLB)
[  807.948567] c2389f34 c2389f24 00000004 00000001 c2389ef0 c0117b5d f78ebd80 00000003 
[  807.956789]        00000000 00000000 f78ebe20 f78ebe14 f78ebe18 00000202 c2389f18 c0117cbb 
[  807.965920]        c2380b30 c2380b28 c2217560 00000001 00000187 fa724d73 00000066 c2330550 
[  807.975051] Call Trace:
[  807.977867]  [<c012bd90>] worker_thread+0x207/0x225
[  807.983070]  [<c012fe68>] kthread+0xb7/0xbc
[  807.987549]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  807.993022] kblockd/0     S 00000002     0    20     15            21       (L-TLB)
[  808.001343] c238bf34 c238bf20 00000004 00000002 c238bed0 c040906a c238bef8 00000000 
[  808.009566]        c0579414 f7f7aedc c238bee8 00000000 f7a95a70 f7f734f4 c23eef80 f7a95a70 
[  808.018595]        00000000 c220fec0 c220f560 00000000 000017d6 aee7a4ca 000000a9 f7a95a70 
[  808.027623] Call Trace:
[  808.030408]  [<c012bd90>] worker_thread+0x207/0x225
[  808.035551]  [<c012fe68>] kthread+0xb7/0xbc
[  808.039980]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  808.045389] kblockd/1     S 00000001     0    21     15            22    20 (L-TLB)
[  808.053710] f7d58f34 f7d58f24 00000004 00000001 f7d58ed0 c040906a f7d58ef8 c02fccc2 
[  808.061840]        c0579414 f7f7ac5c f7d58ee8 00000000 c23eef00 f7f734f4 c23eef00 f7f7359c 
[  808.070869]        c23eef30 c23eef28 c2217560 00000001 00000c15 9a287d8b 00000091 c2330550 
[  808.079898] Call Trace:
[  808.082683]  [<c012bd90>] worker_thread+0x207/0x225
[  808.087825]  [<c012fe68>] kthread+0xb7/0xbc
[  808.092253]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  808.097664] kblockd/2     S 00000004     0    22     15            23    21 (L-TLB)
[  808.105985] c238ff34 c238ff24 00000004 00000004 c238fed0 c040906a c238fef8 c02fccc2 
[  808.114151]        c0579414 d6cb69dc 00000080 00000000 c23eee80 f7f734f4 c23eee80 f7f7359c 
[  808.123182]        c23eeeb0 c23eeea8 c221f560 00000002 000011d4 afbc9d2d 000000a9 c2330030 
[  808.132211] Call Trace:
[  808.134995]  [<c012bd90>] worker_thread+0x207/0x225
[  808.140138]  [<c012fe68>] kthread+0xb7/0xbc
[  808.144567]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  808.149977] kblockd/3     S 00000003     0    23     15            24    22 (L-TLB)
[  808.158298] f7d59f34 f7d59f24 00000004 00000003 f7f7a2fc 000b884f 00000000 00000202 
[  808.166427]        00000001 f7f7a2fc f7d59ee8 c040906a f7d59f10 c02fccc2 c0579414 f7f7a2fc 
[  808.175456]        c23eee30 c23eee28 c2227560 00000003 00000d8c 29f94785 0000003a c2337a70 
[  808.184484] Call Trace:
[  808.187267]  [<c012bd90>] worker_thread+0x207/0x225
[  808.192409]  [<c012fe68>] kthread+0xb7/0xbc
[  808.196837]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  808.202313] kacpid        S 00000002     0    24     15           142    23 (L-TLB)
[  808.210635] f7d78f34 f7d78f24 00000004 00000002 00000000 00000001 00000100 00000000 
[  808.218858]        00000080 00000080 00000080 c05566ac c05566ac c05566a0 c2217ec0 c2217ec0 
[  808.227991]        00000000 00000202 c220f560 00000000 00000189 027f7e52 00000000 c048db40 
[  808.237125] Call Trace:
[  808.239942]  [<c012bd90>] worker_thread+0x207/0x225
[  808.245144]  [<c012fe68>] kthread+0xb7/0xbc
[  808.249624]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  808.255097] pdflush       S 00000002     0   142     15           143    24 (L-TLB)
[  808.263418] f7d5df68 f7d5df58 00000004 00000002 c0124c50 c0493a5c 0007cb75 f7d5df68 
[  808.271640]        c014153c c0493a5c 0007cb75 000742bd 00000000 00000000 f7d5df18 00000400 
[  808.280772]        00000000 00000000 c220f560 00000000 000001c7 dad93105 000000bb c048db40 
[  808.289903] Call Trace:
[  808.292719]  [<c0141df8>] __pdflush+0x84/0x199
[  808.297471]  [<c0141f4d>] pdflush+0x40/0x47
[  808.301950]  [<c012fe68>] kthread+0xb7/0xbc
[  808.306430]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  808.311904] pdflush       D 00000001     0   143     15           145   142 (L-TLB)
[  808.320226] f7d8aa48 f7d8aa38 00000004 00000001 f7d8aa34 00000202 f7d8a9f8 c01302cf 
[  808.328447]        f7b26080 c2330550 f7d8aa28 f7d8aa54 00000202 c2217fe0 f7d8aa5c 00000202 
[  808.337578]        c2217fe0 f7d8aa5c c2217560 00000001 00000115 ed121eb1 000000bb c2330550 
[  808.346710] Call Trace:
[  808.349526]  [<c04086e1>] schedule_timeout+0x54/0xa5
[  808.354818]  [<c040866e>] io_schedule_timeout+0x29/0x33
[  808.360383]  [<c02880c4>] blk_congestion_wait+0x70/0x85
[  808.365948]  [<c014136b>] throttle_vm_writeout+0x69/0x7d
[  808.371603]  [<c014714d>] shrink_zone+0xe0/0xfa
[  808.376444]  [<c01471d4>] shrink_caches+0x6d/0x6f
[  808.381465]  [<c01472a6>] try_to_free_pages+0xd0/0x1b5
[  808.386939]  [<c013fa4b>] __alloc_pages+0x135/0x2e8
[  808.392142]  [<c0142d35>] kmem_getpages+0x34/0x9d
[  808.397166]  [<c0143b91>] cache_grow+0xaf/0x16c
[  808.402007]  [<c0143db2>] cache_alloc_refill+0x164/0x207
[  808.407662]  [<c014403b>] kmem_cache_alloc+0x4e/0x50
[  808.412955]  [<c01c4e05>] nfs_create_request+0x3d/0xcc
[  808.418430]  [<c01c9151>] nfs_update_request+0x172/0x1d9
[  808.424086]  [<c01c867e>] nfs_writepage_async+0x36/0x7e
[  808.429649]  [<c01c888f>] nfs_writepage+0x1a8/0x1dc
[  808.434851]  [<c017dd9b>] mpage_writepages+0x232/0x399
[  808.440326]  [<c01c88f1>] nfs_writepages+0x2e/0x11e
[  808.445529]  [<c01416f2>] do_writepages+0x2e/0x53
[  808.450552]  [<c017c072>] __sync_single_inode+0x60/0x1f1
[  808.456207]  [<c017c295>] __writeback_single_inode+0x92/0x195
[  808.462314]  [<c017c555>] sync_sb_inodes+0x1bd/0x2c2
[  808.467607]  [<c017c734>] writeback_inodes+0xda/0xe6
[  808.472900]  [<c01413e8>] background_writeout+0x69/0xa2
[  808.478465]  [<c0141e35>] __pdflush+0xc1/0x199
[  808.483216]  [<c0141f4d>] pdflush+0x40/0x47
[  808.487696]  [<c012fe68>] kthread+0xb7/0xbc
[  808.492176]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  808.497650] kswapd0       D 00000003     0   144      1           804    15 (L-TLB)
[  808.505971] f7dd8e14 f7dd8e04 00000004 00000003 f7dd8dac 00000004 00000001 c21ebd68 
[  808.514101]        c21ebd8c c2337a70 c21ebdd4 c21ebc00 00000202 c2227fe0 f7dd8e28 00000202 
[  808.523130]        c2227fe0 f7dd8e28 c2227560 00000003 00000132 ed120e86 000000bb c2337a70 
[  808.532160] Call Trace:
[  808.534945]  [<c04086e1>] schedule_timeout+0x54/0xa5
[  808.540177]  [<c040866e>] io_schedule_timeout+0x29/0x33
[  808.545677]  [<c02880c4>] blk_congestion_wait+0x70/0x85
[  808.551177]  [<c014136b>] throttle_vm_writeout+0x69/0x7d
[  808.556766]  [<c014714d>] shrink_zone+0xe0/0xfa
[  808.562395]  [<c0147611>] balance_pgdat+0x286/0x37e
[  808.567538]  [<c01477ee>] kswapd+0xe5/0x10f
[  808.571965]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  808.577375] aio/0         S 00000002     0   145     15           146   143 (L-TLB)
[  808.585696] f7e34f34 f7e34f20 00000004 00000002 00000000 00000001 00000100 00000000 
[  808.593825]        00000080 00000080 00000080 c05566ac f7dd7550 c05566a0 c2217ec0 f7dd7550 
[  808.602889]        00000000 c220fec0 c220f560 00000000 0000013f 2ca67fde 00000000 f7dd7550 
[  808.611968] Call Trace:
[  808.614784]  [<c012bd90>] worker_thread+0x207/0x225
[  808.619988]  [<c012fe68>] kthread+0xb7/0xbc
[  808.624467]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  808.629941] aio/1         S 00000000     0   146     15           147   145 (L-TLB)
[  808.638263] f7e1df34 c2361030 c22175a8 00000000 00000000 00000001 00000100 00000000 
[  808.646485]        00000080 00000000 2ca50f66 00000000 c2361030 00000000 c220fec0 c2361030 
[  808.655616]        00000000 c2217ec0 c2217560 00000001 0000016c 2ca6bdf9 00000000 c2361030 
[  808.664722] Call Trace:
[  808.667537]  [<c012bd90>] worker_thread+0x207/0x225
[  808.672736]  [<c012fe68>] kthread+0xb7/0xbc
[  808.677212]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  808.682683] aio/2         S 00000004     0   147     15           148   146 (L-TLB)
[  808.691000] f7e35f34 f7e35f24 00000004 00000004 00000000 00000001 00000100 00000000 
[  808.699216]        00000080 00000080 00000080 c05566ac c05566ac c05566a0 c2217ec0 c2217ec0 
[  808.708341]        00000000 00000202 c221f560 00000002 000001a5 2ca71c86 00000000 c2330030 
[  808.717466] Call Trace:
[  808.720281]  [<c012bd90>] worker_thread+0x207/0x225
[  808.725482]  [<c012fe68>] kthread+0xb7/0xbc
[  808.729959]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  808.735429] aio/3         S 00000003     0   148     15           149   147 (L-TLB)
[  808.743746] f7e1ef34 f7e1ef24 00000004 00000003 00000000 00000001 00000100 00000080 
[  808.751877]        00000080 00000000 00000000 c05566a0 c05566a0 00000000 c220fec0 c220fec0 
[  808.760959]        00000000 00000202 c2227560 00000003 00000191 2ca776c8 00000000 c2337a70 
[  808.770040] Call Trace:
[  808.772841]  [<c012bd90>] worker_thread+0x207/0x225
[  808.778016]  [<c012fe68>] kthread+0xb7/0xbc
[  808.782470]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  808.787912] xfslogd/0     S 00000000     0   149     15           150   148 (L-TLB)
[  808.796234] f7e37f34 c2361030 c220f5a8 00000000 00000000 00000001 00000100 00000000 
[  808.804411]        00000100 00000080 2e49f705 00000000 c2361030 c05566a0 c2217ec0 c2361030 
[  808.813492]        00000000 c220fec0 c220f560 00000000 0000016d 2e4aa29a 00000000 c2361030 
[  808.822579] Call Trace:
[  808.825396]  [<c012bd90>] worker_thread+0x207/0x225
[  808.830600]  [<c012fe68>] kthread+0xb7/0xbc
[  808.835080]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  808.840554] xfslogd/1     S 00000001     0   150     15           151   149 (L-TLB)
[  808.848875] f7e75f34 f7e75f20 00000004 00000001 00000000 00000001 00000100 00000000 
[  808.857097]        00000100 00000080 00000080 c05566a0 f7e36550 c05566ac c220fec0 f7e36550 
[  808.866228]        00000000 c2217ec0 c2217560 00000001 000000f4 2e4b28d6 00000000 f7e36550 
[  808.875357] Call Trace:
[  808.878173]  [<c012bd90>] worker_thread+0x207/0x225
[  808.883376]  [<c012fe68>] kthread+0xb7/0xbc
[  808.887856]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  808.893329] xfslogd/2     S 00000004     0   151     15           152   150 (L-TLB)
[  808.901650] f7e38f34 f7e38f24 00000004 00000004 00000000 00000001 00000100 00000080 
[  808.909872]        00000100 00000080 00000080 c05566ac c05566ac c05566a0 c2217ec0 c2217ec0 
[  808.919003]        00000000 00000202 c221f560 00000002 0000015d 2e4b69d2 00000000 c2330030 
[  808.928134] Call Trace:
[  808.930950]  [<c012bd90>] worker_thread+0x207/0x225
[  808.936153]  [<c012fe68>] kthread+0xb7/0xbc
[  808.940633]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  808.946106] xfslogd/3     S 00000003     0   152     15           153   151 (L-TLB)
[  808.954428] f7e76f34 f7e76f24 00000004 00000003 00000000 00000001 00000100 00000080 
[  808.962649]        00000100 00000080 00000080 c05566a0 c05566a0 c05566ac c220fec0 c220fec0 
[  808.971782]        00000000 00000202 c2227560 00000003 00000173 2e4bc489 00000000 c2337a70 
[  808.980913] Call Trace:
[  808.983729]  [<c012bd90>] worker_thread+0x207/0x225
[  808.988932]  [<c012fe68>] kthread+0xb7/0xbc
[  808.993411]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  808.998885] xfsdatad/0    S 00000000     0   153     15           154   152 (L-TLB)
[  809.007206] f7e3af34 c2361030 c220f5a8 00000000 00000000 00000001 00000100 00000000 
[  809.015428]        00000100 00000080 2e4b784f 00000000 c2361030 c05566a0 c2217ec0 c2361030 
[  809.024560]        00000000 c220fec0 c220f560 00000000 00000131 2e4c227a 00000000 c2361030 
[  809.033653] Call Trace:
[  809.036453]  [<c012bd90>] worker_thread+0x207/0x225
[  809.041626]  [<c012fe68>] kthread+0xb7/0xbc
[  809.046080]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  809.051507] xfsdatad/1    S 00000001     0   154     15           155   153 (L-TLB)
[  809.059829] f7e78f34 f7e78f20 00000004 00000001 00000000 00000001 00000100 00000000 
[  809.068030]        00000100 00000080 00000080 c05566a0 f7e3ba70 c05566ac c220fec0 f7e3ba70 
[  809.077112]        00000000 c2217ec0 c2217560 00000001 000000f6 2e4c9b2f 00000000 f7e3ba70 
[  809.086193] Call Trace:
[  809.088993]  [<c012bd90>] worker_thread+0x207/0x225
[  809.094167]  [<c012fe68>] kthread+0xb7/0xbc
[  809.098621]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  809.104064] xfsdatad/2    S 00000004     0   155     15           156   154 (L-TLB)
[  809.112385] f7e3cf34 f7e3cf24 00000004 00000004 00000000 00000001 00000100 00000080 
[  809.120562]        00000100 00000080 00000080 c05566ac c05566ac c05566a0 c2217ec0 c2217ec0 
[  809.129644]        00000000 00000202 c221f560 00000002 00000136 2e4cd917 00000000 c2330030 
[  809.138725] Call Trace:
[  809.141526]  [<c012bd90>] worker_thread+0x207/0x225
[  809.146699]  [<c012fe68>] kthread+0xb7/0xbc
[  809.151153]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  809.156595] xfsdatad/3    S 00000003     0   156     15           157   155 (L-TLB)
[  809.164917] f7e7af34 f7e7af24 00000004 00000003 00000000 00000001 00000100 00000080 
[  809.173139]        00000100 00000080 00000080 c05566a0 c05566a0 c05566ac c220fec0 c220fec0 
[  809.182270]        00000000 00000202 c2227560 00000003 0000013e 2e4d313a 00000000 c2337a70 
[  809.191401] Call Trace:
[  809.194218]  [<c012bd90>] worker_thread+0x207/0x225
[  809.199419]  [<c012fe68>] kthread+0xb7/0xbc
[  809.203899]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  809.209372] xfsbufd       S 00000004     0   157     15           742   156 (L-TLB)
[  809.217693] f7e3df44 f7e3df34 00000004 00000004 00000100 00000080 00000080 00000000 
[  809.225916]        c2337a70 c2330030 af846c0b 000000a9 00000202 c221ffe0 f7e3df58 00000202 
[  809.235047]        c221ffe0 f7e3df58 c221f560 00000002 000000a9 cf3c7dbc 000000bb c2330030 
[  809.244178] Call Trace:
[  809.246994]  [<c04086e1>] schedule_timeout+0x54/0xa5
[  809.252287]  [<c040874c>] schedule_timeout_interruptible+0x1a/0x1c
[  809.258845]  [<c0272754>] xfsbufd+0x4e/0x1b0
[  809.263416]  [<c012fe68>] kthread+0xb7/0xbc
[  809.267896]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  809.273371] kseriod       S 00000002     0   742     15          1514   157 (L-TLB)
[  809.281691] f7f16f5c f7f16f4c 00000004 00000002 c02eff05 c04b8f24 f7f16f10 c02efe91 
[  809.289868]        c04b8f08 c04a8738 c04b8f24 c04a63e0 f7f16f34 c02ef54e c04b8f08 c04a8738 
[  809.298950]        00000282 00000202 c220f560 00000000 000a8380 b437cbfc 00000000 c048db40 
[  809.308030] Call Trace:
[  809.310830]  [<c02e4762>] serio_thread+0xea/0xec
[  809.315735]  [<c012fe68>] kthread+0xb7/0xbc
[  809.320189]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  809.325631] kjournald     S 00000002     0   804      1          1519   144 (L-TLB)
[  809.333953] f7eaff58 f7eaff48 00000004 00000002 f726204c 00000000 00000000 f7fc6800 
[  809.342129]        00000000 f7171400 f7afcfa8 000003d0 00000202 c2217fe0 f7eaffb8 f7eaff2c 
[  809.351209]        f7fb086c f7fb0864 c220f560 00000000 000004a8 8a6284da 000000a6 c048db40 
[  809.361134] Call Trace:
[  809.363931]  [<c01b0eef>] kjournald+0x1d6/0x223
[  809.368746]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  809.374189] rpciod/0      D 00000002     0  1514     15          1515   742 (L-TLB)
[  809.382510] f7b96aa8 f7b96a98 00000004 00000002 f7ff8f18 00000001 00000003 00000b9d 
[  809.390687]        324272cc c048db40 02000000 f7e4f46c 00000202 c220ffe0 f7b96abc 00000202 
[  809.399767]        c220ffe0 f7b96abc c220f560 00000000 00000137 ed120e43 000000bb c048db40 
[  809.408849] Call Trace:
[  809.411649]  [<c04086e1>] schedule_timeout+0x54/0xa5
[  809.416913]  [<c040866e>] io_schedule_timeout+0x29/0x33
[  809.422446]  [<c02880c4>] blk_congestion_wait+0x70/0x85
[  809.427979]  [<c014136b>] throttle_vm_writeout+0x69/0x7d
[  809.433602]  [<c014714d>] shrink_zone+0xe0/0xfa
[  809.438415]  [<c01471d4>] shrink_caches+0x6d/0x6f
[  809.443408]  [<c01472a6>] try_to_free_pages+0xd0/0x1b5
[  809.448851]  [<c013fa4b>] __alloc_pages+0x135/0x2e8
[  809.454024]  [<c03b74ad>] tcp_sendmsg+0xaa0/0xb78
[  809.459020]  [<c03d4666>] inet_sendmsg+0x48/0x53
[  809.463924]  [<c0388716>] sock_sendmsg+0xb8/0xd3
[  809.468829]  [<c0388773>] kernel_sendmsg+0x42/0x4f
[  809.473913]  [<c038bc00>] sock_no_sendpage+0x5e/0x77
[  809.479177]  [<c03ee7af>] xs_tcp_send_request+0x2af/0x375
[  809.484891]  [<c03ed4c5>] xprt_transmit+0x4f/0x20b
[  809.489975]  [<c03ebce1>] call_transmit+0x68/0xc8
[  809.494969]  [<c03f0962>] __rpc_execute+0x5c/0x1ea
[  809.500053]  [<c03f0b27>] rpc_async_schedule+0x11/0x15
[  809.505495]  [<c012bd1d>] worker_thread+0x194/0x225
[  809.510669]  [<c012fe68>] kthread+0xb7/0xbc
[  809.515123]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  809.520567] rpciod/1      S 00000001     0  1515     15          1516  1514 (L-TLB)
[  809.528890] f78dbf34 f78dbf20 00000004 00000001 00000000 c03ed2f0 f7b13fc0 00000000 
[  809.537066]        f7b13e00 c3a2e504 c3a2e504 00000000 f7d89a70 c03ebce1 c3a2e504 f7d89a70 
[  809.546148]        00000000 c2217ec0 c2217560 00000001 00002aff 0db3409e 000000aa f7d89a70 
[  809.555229] Call Trace:
[  809.558031]  [<c012bd90>] worker_thread+0x207/0x225
[  809.563205]  [<c012fe68>] kthread+0xb7/0xbc
[  809.567660]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  809.573103] rpciod/2      S 00000004     0  1516     15          1517  1515 (L-TLB)
[  809.581423] f7ba6f34 f7ba6f24 00000004 00000004 00000000 00000000 00000003 00000000 
[  809.589555]        c4866584 00000000 f7ba6eec c0296593 00000000 f7ba6f00 c04071af f7b13e00 
[  809.598583]        f71717b0 f71717a8 c221f560 00000002 000021eb af7442db 000000a9 c2330030 
[  809.607612] Call Trace:
[  809.610396]  [<c012bd90>] worker_thread+0x207/0x225
[  809.615539]  [<c012fe68>] kthread+0xb7/0xbc
[  809.619967]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  809.625377] rpciod/3      S 00000003     0  1517     15                1516 (L-TLB)
[  809.633698] f7154f34 f7154f20 00000004 00000003 00000000 00000000 00000003 00000000 
[  809.641827]        c2330030 00000000 f7154eec c0296593 f717b030 00000040 c04071af f717b030 
[  809.650858]        00000000 c2227ec0 c2227560 00000003 00003481 af9995ef 000000a9 f717b030 
[  809.659885] Call Trace:
[  809.662669]  [<c012bd90>] worker_thread+0x207/0x225
[  809.667811]  [<c012fe68>] kthread+0xb7/0xbc
[  809.672240]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  809.677650] portmap       S 00000002     0  1519      1          1541   804 (NOTLB)
[  809.685971] f717cf00 f717ceec 00000004 00000002 00000202 f7b8100c 00000202 00000000 
[  809.694100]        c2330550 f7b81000 f7e87080 00000202 f7e79030 00000003 f717ced0 f7e79030 
[  809.703128]        00000000 c220fec0 c220f560 00000000 0000c0e9 607461be 00000089 f7e79030 
[  809.712156] Call Trace:
[  809.714941]  [<c0408730>] schedule_timeout+0xa3/0xa5
[  809.720173]  [<c016ce88>] do_poll+0x9a/0xb9
[  809.724601]  [<c016cffd>] sys_poll+0x156/0x221
[  809.729298]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  809.734618] kjournald     S 00000002     0  1541      1          1544  1519 (L-TLB)
[  809.742939] f71d3f58 f71d3f48 00000004 00000002 f724d02c 00000000 00000000 f71d1800 
[  809.751070]        00000000 f7171e00 f7afcd6c 0000020c 00000202 c220ffe0 f71d3fb8 f71d3f2c 
[  809.760116]        f7fb066c f7fb0664 c220f560 00000000 00000585 2cebc346 000000a7 c048db40 
[  809.769198] Call Trace:
[  809.771998]  [<c01b0eef>] kjournald+0x1d6/0x223
[  809.776813]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  809.782256] writetest     D 00000003     0  1544      1                1541 (NOTLB)
[  809.790577] f7adce10 f7adce00 00000004 00000003 f7adcdd4 f7ecfd80 f7adcde8 c01c7bcf 
[  809.798754]        f71b25c0 00000008 c01c7a21 f7adcdcc 00000082 f7b13600 f7adcdd4 f71b25c0 
[  809.807838]        f7adcdd4 f7adcdd4 c2227560 00000003 00000630 c562b75c 000000a9 c2337a70 
[  809.816918] Call Trace:
[  809.819719]  [<c040863b>] io_schedule+0x26/0x30
[  809.824532]  [<c013ab27>] sync_page+0x39/0x4e
[  809.829168]  [<c0408929>] __wait_on_bit_lock+0x58/0x67
[  809.834611]  [<c013b343>] __lock_page+0x84/0x8c
[  809.839425]  [<c013c3c9>] filemap_nopage+0x1e4/0x347
[  809.844688]  [<c014b4b4>] do_no_page+0x70/0x25b
[  809.849503]  [<c014b812>] __handle_mm_fault+0xef/0x22a
[  809.854945]  [<c0409a84>] do_page_fault+0x179/0x5e6
[  809.860119]  [<c01039cf>] error_code+0x4f/0x54

[  816.203560] SysRq : Show Memory
[  816.206914] Mem-info:
[  816.209316] DMA per-cpu:
[  816.212033] cpu 0 hot: low 0, high 0, batch 1 used:0
[  816.217238] cpu 0 cold: low 0, high 0, batch 1 used:0
[  816.222532] cpu 1 hot: low 0, high 0, batch 1 used:0
[  816.227737] cpu 1 cold: low 0, high 0, batch 1 used:0
[  816.233033] cpu 2 hot: low 0, high 0, batch 1 used:0
[  816.238237] cpu 2 cold: low 0, high 0, batch 1 used:0
[  816.243532] cpu 3 hot: low 0, high 0, batch 1 used:0
[  816.248736] cpu 3 cold: low 0, high 0, batch 1 used:0
[  816.254031] DMA32 per-cpu: empty
[  816.257470] Normal per-cpu:
[  816.260458] cpu 0 hot: low 0, high 186, batch 31 used:172
[  816.266115] cpu 0 cold: low 0, high 62, batch 15 used:7
[  816.271592] cpu 1 hot: low 0, high 186, batch 31 used:104
[  816.277249] cpu 1 cold: low 0, high 62, batch 15 used:29
[  816.282815] cpu 2 hot: low 0, high 186, batch 31 used:10
[  816.288379] cpu 2 cold: low 0, high 62, batch 15 used:14
[  816.293945] cpu 3 hot: low 0, high 186, batch 31 used:18
[  816.299509] cpu 3 cold: low 0, high 62, batch 15 used:57
[  816.305072] HighMem per-cpu:
[  816.308150] cpu 0 hot: low 0, high 186, batch 31 used:118
[  816.313805] cpu 0 cold: low 0, high 62, batch 15 used:12
[  816.319370] cpu 1 hot: low 0, high 186, batch 31 used:26
[  816.324933] cpu 1 cold: low 0, high 62, batch 15 used:61
[  816.330498] cpu 2 hot: low 0, high 186, batch 31 used:92
[  816.336062] cpu 2 cold: low 0, high 62, batch 15 used:14
[  816.341626] cpu 3 hot: low 0, high 186, batch 31 used:160
[  816.347282] cpu 3 cold: low 0, high 62, batch 15 used:48
[  816.352848] Free pages:       10336kB (3008kB HighMem)
[  816.358232] Active:462053 inactive:37639 dirty:164757 writeback:302602 unstable:0 free:2584 slab:13246 mapped:1247 pagetables:10
[  816.370311] DMA free:3588kB min:68kB low:84kB high:100kB active:2696kB inactive:512kB present:16384kB pages_scanned:33 all_unreclaimable? no
[  816.383472] lowmem_reserve[]: 0 0 880 2031
[  816.387989] DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
[  816.399742] lowmem_reserve[]: 0 0 880 2031
[  816.404212] Normal free:3740kB min:3756kB low:4692kB high:5632kB active:747500kB inactive:74408kB present:901120kB pages_scanned:96 all_unreclaimable? no
[  816.418389] lowmem_reserve[]: 0 0 0 9211
[  816.422680] HighMem free:3008kB min:512kB low:1740kB high:2968kB active:1098016kB inactive:75636kB present:1179092kB pages_scanned:0 all_unreclaimable? no
[  816.436944] lowmem_reserve[]: 0 0 0 0
[  816.440965] DMA: 1*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3588kB
[  816.451733] DMA32: empty
[  816.454419] Normal: 1*4kB 1*8kB 1*16kB 0*32kB 0*64kB 1*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3740kB
[  816.465455] HighMem: 60*4kB 40*8kB 17*16kB 8*32kB 4*64kB 3*128kB 1*256kB 2*512kB 0*1024kB 0*2048kB 0*4096kB = 3008kB
[  816.476852] Swap cache: add 0, delete 0, find 0/0, race 0+0
[  816.482623] Free swap  = 0kB
[  816.485624] Total swap = 0kB
[  816.488624] Free swap:            0kB
[  816.500924] 524149 pages of RAM
[  816.504196] 294773 pages of HIGHMEM
[  816.507822] 6073 reserved pages
[  816.511091] 304404 pages shared
[  816.514360] 0 pages swap cached
[  816.517630] 164757 pages dirty
[  816.520809] 302602 pages writeback
[  816.524347] 1247 pages mapped
[  816.527437] 13246 pages slab
[  816.530438] 10 pages pagetables
[  865.060642] SysRq : HELP : loglevel0-8 reBoot tErm Full kIll saK showMem Nice powerOff showPc unRaw Sync showTasks Unmount 
[  871.180739] SysRq : Kill All Tasks


[  891.618438] SysRq : Terminate All Tasks


[  913.929600] SysRq : Show Regs
[  913.932773] 
[  913.934362] Pid: 0, comm:              swapper
[  913.939025] EIP: 0060:[<c0100d4f>] CPU: 0
[  913.943240] EIP is at default_idle+0x36/0x56
[  913.947724]  EFLAGS: 00000246    Not tainted  (2.6.15-rc5+nfs+k2)
[  913.954105] EAX: 00000000 EBX: c0503000 ECX: c220e2e0 EDX: c0503000
[  913.960666] ESI: c053c380 EDI: c053c300 EBP: c0503fac DS: 007b ES: 007b
[  913.967634] CR0: 8005003b CR2: b7afd000 CR3: 00540000 CR4: 000006d0
[  915.322037] SysRq : Show State
[  915.325299] 
[  915.325300]                                                sibling
[  915.333359]   task             PC      pid father child younger older
[  915.340102] bash          S 00000001     0     1      0     2               (NOTLB)
[  915.348424] c2331f08 c2331ef8 00000004 00000001 00000092 f717ba70 f717ba70 f717ba70 
[  915.356645]        c2331ed0 c011d794 f717ba70 000200d2 00000000 00000000 f717ba70 000005ef 
[  915.365776]        00000000 c2331f08 c2217560 00000001 0000083d d6a485f6 000000ca c2330550 
[  915.374907] Call Trace:
[  915.377722]  [<c011fa57>] do_wait+0x18a/0x376
[  915.382382]  [<c011fd03>] sys_wait4+0x3e/0x42
[  915.387040]  [<c011fd2e>] sys_waitpid+0x27/0x2b
[  915.391878]  [<c0102dd7>] sysenter_past_esp+0x54/0x75
[  915.397259] migration/0   S 00000002     0     2      1             3       (L-TLB)
[  915.405576] c2339f94 c2339f84 00000004 00000002 c0116b9c c2337a70 c22275a8 00000001 
[  915.413793]        c220f5ac 00000001 00000073 c220f958 c220f958 c22275a8 c220f5a8 00000000 
[  915.422918]        c2227560 00000003 c220f560 00000000 000017c2 aa8db191 000000a9 c048db40 
[  915.432045] Call Trace:
[  915.434859]  [<c0118d0c>] migration_thread+0x81/0x10d
[  915.440242]  [<c012fe68>] kthread+0xb7/0xbc
[  915.444720]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  915.450189] ksoftirqd/0   S 00000000     0     3      1             4     2 (L-TLB)
[  915.458511] c233af98 c2337030 c220f560 00000000 000004db 000e078d 00000000 c233ba70 
[  915.466641]        c2337030 c233715c 00000001 c2331f5c c2330a70 ffffffff c2337030 c233af6c 
[  915.475671]        00000000 00000000 c220f560 00000000 000002b2 000ea053 00000000 c2330a70 
[  915.484700] Call Trace:
[  915.487485]  [<c0121786>] ksoftirqd+0xd6/0xf5
[  915.492093]  [<c012fe68>] kthread+0xb7/0xbc
[  915.496522]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  915.501997] migration/1   S 00000001     0     4      1             5     3 (L-TLB)
[  915.510319] c233cf94 c233cf80 00000004 00000001 00000082 00000001 c233cf44 00000000 
[  915.518541]        c048db40 c2337a70 00000003 c2227560 c2330a70 00000009 00000008 c2330a70 
[  915.527674]        00000000 c2217ec0 c2217560 00000001 00000eaf 3827d003 000000a1 c2330a70 
[  915.536805] Call Trace:
[  915.539621]  [<c0118d0c>] migration_thread+0x81/0x10d
[  915.545007]  [<c012fe68>] kthread+0xb7/0xbc
[  915.549486]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  915.554961] ksoftirqd/1   S 00000001     0     5      1             6     4 (L-TLB)
[  915.563281] c2345f98 c2345f88 00000004 00000001 000000c7 00180089 00000000 c233b550 
[  915.571503]        c233b550 c2330a70 00000001 c2331f28 c2345000 ffffffff c233b550 c2345f6c 
[  915.580635]        c0407ed4 c22175a8 c2217560 00000001 00000177 2fc9a4b5 00000000 c2330550 
[  915.589767] Call Trace:
[  915.592577]  [<c0121786>] ksoftirqd+0xd6/0xf5
[  915.597185]  [<c012fe68>] kthread+0xb7/0xbc
[  915.601612]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  915.607023] migration/2   S 00000004     0     6      1             7     5 (L-TLB)
[  915.615343] c2346f94 c2346f80 00000004 00000004 00000082 c2330550 c22175a8 00000000 
[  915.623473]        c2337a70 c2330550 00000001 c2217560 f7e79030 00000002 00000002 f7e79030 
[  915.632501]        00000000 c221fec0 c221f560 00000002 00000c37 37d42ab3 00000058 f7e79030 
[  915.641529] Call Trace:
[  915.644314]  [<c0118d0c>] migration_thread+0x81/0x10d
[  915.649638]  [<c012fe68>] kthread+0xb7/0xbc
[  915.654066]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  915.659476] ksoftirqd/2   S 00000002     0     7      1             8     6 (L-TLB)
[  915.667798] c234df98 c234ca70 c221f560 00000002 00000163 00275526 00000000 c234ca70 
[  915.675975]        c234ca70 c234cb9c 00000001 c2331f28 c234d000 ffffffff c234ca70 c234df6c 
[  915.685055]        c0407ed4 c221f5a8 c221f560 00000002 00000043 00275786 00000000 c2330030 
[  915.694136] Call Trace:
[  915.696936]  [<c0121786>] ksoftirqd+0xd6/0xf5
[  915.701569]  [<c012fe68>] kthread+0xb7/0xbc
[  915.706023]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  915.711465] migration/3   S 00000003     0     8      1             9     7 (L-TLB)
[  915.719786] c234ef94 c234ef80 00000004 00000003 00000082 00000001 c234ef44 00000000 
[  915.727961]        c2330030 c2330550 00000001 c2217560 f70f8030 00000011 00000002 f70f8030 
[  915.737042]        00000000 c2227ec0 c2227560 00000003 0000105f 353ed25e 00000058 f70f8030 
[  915.746123] Call Trace:
[  915.748923]  [<c0118d0c>] migration_thread+0x81/0x10d
[  915.754277]  [<c012fe68>] kthread+0xb7/0xbc
[  915.758731]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  915.764173] ksoftirqd/3   S 00000003     0     9      1            10     8 (L-TLB)
[  915.772495] c2353f98 c2353f88 00000004 00000003 000000c9 00367434 00000000 c234c030 
[  915.780670]        c234c030 c2361550 00000001 c2331f28 c2353000 ffffffff c234c030 c2353f6c 
[  915.789752]        c0407ed4 c22275a8 c2227560 00000003 00000182 364e7604 00000058 c2337a70 
[  915.798831] Call Trace:
[  915.801645]  [<c0121786>] ksoftirqd+0xd6/0xf5
[  915.806303]  [<c012fe68>] kthread+0xb7/0xbc
[  915.810781]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  915.816251] events/0      R running     0    10      1            11     9 (L-TLB)
[  915.824477] events/1      S 00000001     0    11      1            12    10 (L-TLB)
[  915.832792] c236af34 c236af24 00000004 00000001 00000202 00000001 c2354540 000007d0 
[  915.841010]        c236aef4 c012bb5c c22191b8 00095bff c23518d0 c2300780 00000002 c236aefc 
[  915.850135]        c2348930 c2348928 c2217560 00000001 00000923 c8d49673 000000d4 c2330550 
[  915.859262] Call Trace:
[  915.862076]  [<c012bd90>] worker_thread+0x207/0x225
[  915.867276]  [<c012fe68>] kthread+0xb7/0xbc
[  915.871754]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  915.877224] events/2      S 00000004     0    12      1            13    11 (L-TLB)
[  915.885541] c2359f34 c2359f24 00000004 00000004 00000202 00000001 c2354540 000007d0 
[  915.893759]        c2359ef4 c012bb5c c22211b8 00095c02 00000000 c2300780 c22211a4 c2359efc 
[  915.902885]        c23488b0 c23488a8 c221f560 00000002 000007dd c902406c 000000d4 c2330030 
[  915.912011] Call Trace:
[  915.914826]  [<c012bd90>] worker_thread+0x207/0x225
[  915.920025]  [<c012fe68>] kthread+0xb7/0xbc
[  915.924501]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  915.929971] events/3      S 00000003     0    13      1            14    12 (L-TLB)
[  915.938287] c236cf34 c236cf24 00000004 00000003 00000202 00000001 c2354540 000007d0 
[  915.946504]        c236cef4 c012bb5c c22291b8 00095c03 00000000 c2300780 c22291a4 c236cefc 
[  915.955631]        c2348830 c2348828 c2227560 00000003 00000533 c9115d4f 000000d4 c2337a70 
[  915.964755] Call Trace:
[  915.967570]  [<c012bd90>] worker_thread+0x207/0x225
[  915.972769]  [<c012fe68>] kthread+0xb7/0xbc
[  915.977246]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  915.982717] khelper       S 00000004     0    14      1            15    13 (L-TLB)
[  915.991016] c235af34 c235af24 00000004 00000004 00000000 c235aef4 c0117b5d f7ba3c70 
[  915.999141]        00000003 00000000 00000000 f7ba3d00 f7ba3cf4 f7ba3cf8 00000202 c235af1c 
[  916.008162]        c23487b0 c23487a8 c221f560 00000002 000001db da3e6193 00000056 c2330030 
[  916.017185] Call Trace:
[  916.019968]  [<c012bd90>] worker_thread+0x207/0x225
[  916.025107]  [<c012fe68>] kthread+0xb7/0xbc
[  916.029532]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  916.034939] kthread       S 00000001     0    15      1    20     144    14 (L-TLB)
[  916.043161] c2389f34 c2389f24 00000004 00000001 c2389ef0 c0117b5d f78ebd80 00000003 
[  916.051285]        00000000 00000000 f78ebe20 f78ebe14 f78ebe18 00000202 c2389f18 c0117cbb 
[  916.060309]        c2380b30 c2380b28 c2217560 00000001 00000187 fa724d73 00000066 c2330550 
[  916.070150] Call Trace:
[  916.072933]  [<c012bd90>] worker_thread+0x207/0x225
[  916.078072]  [<c012fe68>] kthread+0xb7/0xbc
[  916.082498]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  916.087905] kblockd/0     S 00000002     0    20     15            21       (L-TLB)
[  916.096128] c238bf34 c238bf20 00000004 00000002 c238bed0 c040906a c238bef8 00000000 
[  916.104252]        c0579414 f7f7aedc c238bee8 00000000 f7a95a70 f7f734f4 c23eef80 f7a95a70 
[  916.113276]        00000000 c220fec0 c220f560 00000000 000017d6 aee7a4ca 000000a9 f7a95a70 
[  916.122300] Call Trace:
[  916.125083]  [<c012bd90>] worker_thread+0x207/0x225
[  916.130224]  [<c012fe68>] kthread+0xb7/0xbc
[  916.134650]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  916.140057] kblockd/1     S 00000001     0    21     15            22    20 (L-TLB)
[  916.148280] f7d58f34 f7d58f24 00000004 00000001 f7d58ed0 c040906a f7d58ef8 c02fccc2 
[  916.156407]        c0579414 f7f7ac5c f7d58ee8 00000000 c23eef00 f7f734f4 c23eef00 f7f7359c 
[  916.165430]        c23eef30 c23eef28 c2217560 00000001 00000c15 9a287d8b 00000091 c2330550 
[  916.174455] Call Trace:
[  916.177269]  [<c012bd90>] worker_thread+0x207/0x225
[  916.182471]  [<c012fe68>] kthread+0xb7/0xbc
[  916.186947]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  916.192417] kblockd/2     S 00000004     0    22     15            23    21 (L-TLB)
[  916.200733] c238ff34 c238ff24 00000004 00000004 c238fed0 c040906a c238fef8 c02fccc2 
[  916.208950]        c0579414 d6cb69dc 00000080 00000000 c23eee80 f7f734f4 c23eee80 f7f7359c 
[  916.218076]        c23eeeb0 c23eeea8 c221f560 00000002 000011d4 afbc9d2d 000000a9 c2330030 
[  916.227201] Call Trace:
[  916.230015]  [<c012bd90>] worker_thread+0x207/0x225
[  916.235215]  [<c012fe68>] kthread+0xb7/0xbc
[  916.239692]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  916.245162] kblockd/3     S 00000003     0    23     15            24    22 (L-TLB)
[  916.253452] f7d59f34 f7d59f24 00000004 00000003 f7f7a2fc 000b884f 00000000 00000202 
[  916.261573]        00000001 f7f7a2fc f7d59ee8 c040906a f7d59f10 c02fccc2 c0579414 f7f7a2fc 
[  916.270594]        c23eee30 c23eee28 c2227560 00000003 00000d8c 29f94785 0000003a c2337a70 
[  916.279616] Call Trace:
[  916.282398]  [<c012bd90>] worker_thread+0x207/0x225
[  916.287539]  [<c012fe68>] kthread+0xb7/0xbc
[  916.291964]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  916.297372] kacpid        S 00000002     0    24     15           142    23 (L-TLB)
[  916.305592] f7d78f34 f7d78f24 00000004 00000002 00000000 00000001 00000100 00000000 
[  916.313712]        00000080 00000080 00000080 c05566ac c05566ac c05566a0 c2217ec0 c2217ec0 
[  916.322730]        00000000 00000202 c220f560 00000000 00000189 027f7e52 00000000 c048db40 
[  916.331748] Call Trace:
[  916.334530]  [<c012bd90>] worker_thread+0x207/0x225
[  916.339668]  [<c012fe68>] kthread+0xb7/0xbc
[  916.344091]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  916.349496] pdflush       S 00000002     0   142     15           143    24 (L-TLB)
[  916.357714] f7d5df68 f7d5df58 00000004 00000002 c0124c50 c0493a5c 0009659d f7d5df68 
[  916.365835]        c014153c c0493a5c 0009659d 0008dce5 00000000 00000000 f7d5df18 00000400 
[  916.374856]        00000000 00000000 c220f560 00000000 000001ac a8ca9576 000000d4 c048db40 
[  916.383874] Call Trace:
[  916.386656]  [<c0141df8>] __pdflush+0x84/0x199
[  916.391348]  [<c0141f4d>] pdflush+0x40/0x47
[  916.395771]  [<c012fe68>] kthread+0xb7/0xbc
[  916.400195]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  916.405599] pdflush       D 00000001     0   143     15           145   142 (L-TLB)
[  916.413816] f7d8aa48 f7d8aa38 00000004 00000001 f7d8aa34 00000202 f7d8a9f8 c01302cf 
[  916.421936]        f7b26080 c2330550 f7d8aa28 f7d8aa54 00000202 c2217fe0 f7d8aa5c 00000202 
[  916.430954]        c2217fe0 f7d8aa5c c2217560 00000001 0000011b 1a87fa49 000000d5 c2330550 
[  916.439974] Call Trace:
[  916.442755]  [<c04086e1>] schedule_timeout+0x54/0xa5
[  916.447982]  [<c040866e>] io_schedule_timeout+0x29/0x33
[  916.453476]  [<c02880c4>] blk_congestion_wait+0x70/0x85
[  916.458972]  [<c014136b>] throttle_vm_writeout+0x69/0x7d
[  916.464555]  [<c014714d>] shrink_zone+0xe0/0xfa
[  916.469336]  [<c01471d4>] shrink_caches+0x6d/0x6f
[  916.474295]  [<c01472a6>] try_to_free_pages+0xd0/0x1b5
[  916.479699]  [<c013fa4b>] __alloc_pages+0x135/0x2e8
[  916.484836]  [<c0142d35>] kmem_getpages+0x34/0x9d
[  916.489796]  [<c0143b91>] cache_grow+0xaf/0x16c
[  916.494578]  [<c0143db2>] cache_alloc_refill+0x164/0x207
[  916.500163]  [<c014403b>] kmem_cache_alloc+0x4e/0x50
[  916.505389]  [<c01c4e05>] nfs_create_request+0x3d/0xcc
[  916.510794]  [<c01c9151>] nfs_update_request+0x172/0x1d9
[  916.516377]  [<c01c867e>] nfs_writepage_async+0x36/0x7e
[  916.521871]  [<c01c888f>] nfs_writepage+0x1a8/0x1dc
[  916.527007]  [<c017dd9b>] mpage_writepages+0x232/0x399
[  916.532413]  [<c01c88f1>] nfs_writepages+0x2e/0x11e
[  916.537551]  [<c01416f2>] do_writepages+0x2e/0x53
[  916.542510]  [<c017c072>] __sync_single_inode+0x60/0x1f1
[  916.548094]  [<c017c295>] __writeback_single_inode+0x92/0x195
[  916.554124]  [<c017c555>] sync_sb_inodes+0x1bd/0x2c2
[  916.559350]  [<c017c734>] writeback_inodes+0xda/0xe6
[  916.564578]  [<c01413e8>] background_writeout+0x69/0xa2
[  916.570072]  [<c0141e35>] __pdflush+0xc1/0x199
[  916.574763]  [<c0141f4d>] pdflush+0x40/0x47
[  916.579186]  [<c012fe68>] kthread+0xb7/0xbc
[  916.583609]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  916.589014] kswapd0       D 00000003     0   144      1           804    15 (L-TLB)
[  916.597234] f7dd8e14 f7dd8e04 00000004 00000003 f7dd8dac 00000004 00000001 c21ebd68 
[  916.605355]        c21ebd8c c2337a70 c21ebdd4 c21ebc00 00000202 c2227fe0 f7dd8e28 00000202 
[  916.614374]        c2227fe0 f7dd8e28 c2227560 00000003 00000136 1a87e819 000000d5 c2337a70 
[  916.623394] Call Trace:
[  916.626176]  [<c04086e1>] schedule_timeout+0x54/0xa5
[  916.631403]  [<c040866e>] io_schedule_timeout+0x29/0x33
[  916.636900]  [<c02880c4>] blk_congestion_wait+0x70/0x85
[  916.642395]  [<c014136b>] throttle_vm_writeout+0x69/0x7d
[  916.647978]  [<c014714d>] shrink_zone+0xe0/0xfa
[  916.652759]  [<c0147611>] balance_pgdat+0x286/0x37e
[  916.657896]  [<c01477ee>] kswapd+0xe5/0x10f
[  916.662320]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  916.667724] aio/0         S 00000002     0   145     15           146   143 (L-TLB)
[  916.675944] f7e34f34 f7e34f20 00000004 00000002 00000000 00000001 00000100 00000000 
[  916.684064]        00000080 00000080 00000080 c05566ac f7dd7550 c05566a0 c2217ec0 f7dd7550 
[  916.693084]        00000000 c220fec0 c220f560 00000000 0000013f 2ca67fde 00000000 f7dd7550 
[  916.702102] Call Trace:
[  916.704884]  [<c012bd90>] worker_thread+0x207/0x225
[  916.710021]  [<c012fe68>] kthread+0xb7/0xbc
[  916.714444]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  916.719850] aio/1         S 00000000     0   146     15           147   145 (L-TLB)
[  916.728070] f7e1df34 c2361030 c22175a8 00000000 00000000 00000001 00000100 00000000 
[  916.736192]        00000080 00000000 2ca50f66 00000000 c2361030 00000000 c220fec0 c2361030 
[  916.745215]        00000000 c2217ec0 c2217560 00000001 0000016c 2ca6bdf9 00000000 c2361030 
[  916.754233] Call Trace:
[  916.757014]  [<c012bd90>] worker_thread+0x207/0x225
[  916.762151]  [<c012fe68>] kthread+0xb7/0xbc
[  916.766574]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  916.771979] aio/2         S 00000004     0   147     15           148   146 (L-TLB)
[  916.780198] f7e35f34 f7e35f24 00000004 00000004 00000000 00000001 00000100 00000000 
[  916.788317]        00000080 00000080 00000080 c05566ac c05566ac c05566a0 c2217ec0 c2217ec0 
[  916.797334]        00000000 00000202 c221f560 00000002 000001a5 2ca71c86 00000000 c2330030 
[  916.806353] Call Trace:
[  916.809134]  [<c012bd90>] worker_thread+0x207/0x225
[  916.814271]  [<c012fe68>] kthread+0xb7/0xbc
[  916.818695]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  916.824099] aio/3         S 00000003     0   148     15           149   147 (L-TLB)
[  916.832317] f7e1ef34 f7e1ef24 00000004 00000003 00000000 00000001 00000100 00000080 
[  916.840438]        00000080 00000000 00000000 c05566a0 c05566a0 00000000 c220fec0 c220fec0 
[  916.849457]        00000000 00000202 c2227560 00000003 00000191 2ca776c8 00000000 c2337a70 
[  916.858475] Call Trace:
[  916.861256]  [<c012bd90>] worker_thread+0x207/0x225
[  916.866393]  [<c012fe68>] kthread+0xb7/0xbc
[  916.871618]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  916.877023] xfslogd/0     S 00000000     0   149     15           150   148 (L-TLB)
[  916.885241] f7e37f34 c2361030 c220f5a8 00000000 00000000 00000001 00000100 00000000 
[  916.893361]        00000100 00000080 2e49f705 00000000 c2361030 c05566a0 c2217ec0 c2361030 
[  916.902383]        00000000 c220fec0 c220f560 00000000 0000016d 2e4aa29a 00000000 c2361030 
[  916.911401] Call Trace:
[  916.914183]  [<c012bd90>] worker_thread+0x207/0x225
[  916.919319]  [<c012fe68>] kthread+0xb7/0xbc
[  916.923742]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  916.929147] xfslogd/1     S 00000001     0   150     15           151   149 (L-TLB)
[  916.937365] f7e75f34 f7e75f20 00000004 00000001 00000000 00000001 00000100 00000000 
[  916.945486]        00000100 00000080 00000080 c05566a0 f7e36550 c05566ac c220fec0 f7e36550 
[  916.954505]        00000000 c2217ec0 c2217560 00000001 000000f4 2e4b28d6 00000000 f7e36550 
[  916.963525] Call Trace:
[  916.966306]  [<c012bd90>] worker_thread+0x207/0x225
[  916.971443]  [<c012fe68>] kthread+0xb7/0xbc
[  916.975867]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  916.981271] xfslogd/2     S 00000004     0   151     15           152   150 (L-TLB)
[  916.989489] f7e38f34 f7e38f24 00000004 00000004 00000000 00000001 00000100 00000080 
[  916.997609]        00000100 00000080 00000080 c05566ac c05566ac c05566a0 c2217ec0 c2217ec0 
[  917.006626]        00000000 00000202 c221f560 00000002 0000015d 2e4b69d2 00000000 c2330030 
[  917.015643] Call Trace:
[  917.018424]  [<c012bd90>] worker_thread+0x207/0x225
[  917.023563]  [<c012fe68>] kthread+0xb7/0xbc
[  917.027988]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  917.033393] xfslogd/3     S 00000003     0   152     15           153   151 (L-TLB)
[  917.041612] f7e76f34 f7e76f24 00000004 00000003 00000000 00000001 00000100 00000080 
[  917.049733]        00000100 00000080 00000080 c05566a0 c05566a0 c05566ac c220fec0 c220fec0 
[  917.058751]        00000000 00000202 c2227560 00000003 00000173 2e4bc489 00000000 c2337a70 
[  917.067770] Call Trace:
[  917.070552]  [<c012bd90>] worker_thread+0x207/0x225
[  917.075689]  [<c012fe68>] kthread+0xb7/0xbc
[  917.080112]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  917.085517] xfsdatad/0    S 00000000     0   153     15           154   152 (L-TLB)
[  917.093734] f7e3af34 c2361030 c220f5a8 00000000 00000000 00000001 00000100 00000000 
[  917.101897]        00000100 00000080 2e4b784f 00000000 c2361030 c05566a0 c2217ec0 c2361030 
[  917.110914]        00000000 c220fec0 c220f560 00000000 00000131 2e4c227a 00000000 c2361030 
[  917.119982] Call Trace:
[  917.122783]  [<c012bd90>] worker_thread+0x207/0x225
[  917.127957]  [<c012fe68>] kthread+0xb7/0xbc
[  917.132412]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  917.137855] xfsdatad/1    S 00000001     0   154     15           155   153 (L-TLB)
[  917.146175] f7e78f34 f7e78f20 00000004 00000001 00000000 00000001 00000100 00000000 
[  917.154352]        00000100 00000080 00000080 c05566a0 f7e3ba70 c05566ac c220fec0 f7e3ba70 
[  917.163434]        00000000 c2217ec0 c2217560 00000001 000000f6 2e4c9b2f 00000000 f7e3ba70 
[  917.172519] Call Trace:
[  917.175319]  [<c012bd90>] worker_thread+0x207/0x225
[  917.180493]  [<c012fe68>] kthread+0xb7/0xbc
[  917.184948]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  917.190391] xfsdatad/2    S 00000004     0   155     15           156   154 (L-TLB)
[  917.198714] f7e3cf34 f7e3cf24 00000004 00000004 00000000 00000001 00000100 00000080 
[  917.206892]        00000100 00000080 00000080 c05566ac c05566ac c05566a0 c2217ec0 c2217ec0 
[  917.215972]        00000000 00000202 c221f560 00000002 00000136 2e4cd917 00000000 c2330030 
[  917.225063] Call Trace:
[  917.227879]  [<c012bd90>] worker_thread+0x207/0x225
[  917.233081]  [<c012fe68>] kthread+0xb7/0xbc
[  917.237560]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  917.243034] xfsdatad/3    S 00000003     0   156     15           157   155 (L-TLB)
[  917.251357] f7e7af34 f7e7af24 00000004 00000003 00000000 00000001 00000100 00000080 
[  917.259486]        00000100 00000080 00000080 c05566a0 c05566a0 c05566ac c220fec0 c220fec0 
[  917.268514]        00000000 00000202 c2227560 00000003 0000013e 2e4d313a 00000000 c2337a70 
[  917.277566] Call Trace:
[  917.280365]  [<c012bd90>] worker_thread+0x207/0x225
[  917.285538]  [<c012fe68>] kthread+0xb7/0xbc
[  917.289991]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  917.295434] xfsbufd       S 00000004     0   157     15           742   156 (L-TLB)
[  917.303756] f7e3df44 f7e3df34 00000004 00000004 00000100 00000080 00000080 00000000 
[  917.311931]        c2337a70 c2330030 af846c0b 000000a9 00000202 c221ffe0 f7e3df58 00000202 
[  917.321011]        c221ffe0 f7e3df58 c221f560 00000002 0000009c fd694784 000000d4 c2330030 
[  917.330093] Call Trace:
[  917.332893]  [<c04086e1>] schedule_timeout+0x54/0xa5
[  917.338155]  [<c040874c>] schedule_timeout_interruptible+0x1a/0x1c
[  917.344677]  [<c0272754>] xfsbufd+0x4e/0x1b0
[  917.349221]  [<c012fe68>] kthread+0xb7/0xbc
[  917.353675]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  917.359118] kseriod       S 00000002     0   742     15          1514   157 (L-TLB)
[  917.367440] f7f16f5c f7f16f4c 00000004 00000002 c02eff05 c04b8f24 f7f16f10 c02efe91 
[  917.375616]        c04b8f08 c04a8738 c04b8f24 c04a63e0 f7f16f34 c02ef54e c04b8f08 c04a8738 
[  917.384696]        00000282 00000202 c220f560 00000000 000a8380 b437cbfc 00000000 c048db40 
[  917.393776] Call Trace:
[  917.396576]  [<c02e4762>] serio_thread+0xea/0xec
[  917.401481]  [<c012fe68>] kthread+0xb7/0xbc
[  917.405934]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  917.411375] kjournald     S 00000002     0   804      1          1541   144 (L-TLB)
[  917.419696] f7eaff58 f7eaff48 00000004 00000002 f726204c 00000000 00000000 f7fc6800 
[  917.427860]        00000000 f7171400 f7afcfa8 000003d0 00000202 c2217fe0 f7eaffb8 f7eaff2c 
[  917.436888]        f7fb086c f7fb0864 c220f560 00000000 000004a8 8a6284da 000000a6 c048db40 
[  917.445918] Call Trace:
[  917.448703]  [<c01b0eef>] kjournald+0x1d6/0x223
[  917.453490]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  917.458917] rpciod/0      D 00000002     0  1514     15          1515   742 (L-TLB)
[  917.467239] f7b96aa8 f7b96a98 00000004 00000002 f7ff8f18 00000001 00000003 00000b9d 
[  917.475462]        324272cc c048db40 02000000 f7e4f46c 00000202 c220ffe0 f7b96abc 00000202 
[  917.484593]        c220ffe0 f7b96abc c220f560 00000000 0000012c 1a87e85c 000000d5 c048db40 
[  917.493725] Call Trace:
[  917.496542]  [<c04086e1>] schedule_timeout+0x54/0xa5
[  917.501835]  [<c040866e>] io_schedule_timeout+0x29/0x33
[  917.507400]  [<c02880c4>] blk_congestion_wait+0x70/0x85
[  917.512965]  [<c014136b>] throttle_vm_writeout+0x69/0x7d
[  917.518619]  [<c014714d>] shrink_zone+0xe0/0xfa
[  917.523461]  [<c01471d4>] shrink_caches+0x6d/0x6f
[  917.528483]  [<c01472a6>] try_to_free_pages+0xd0/0x1b5
[  917.533956]  [<c013fa4b>] __alloc_pages+0x135/0x2e8
[  917.539159]  [<c03b74ad>] tcp_sendmsg+0xaa0/0xb78
[  917.544182]  [<c03d4666>] inet_sendmsg+0x48/0x53
[  917.549114]  [<c0388716>] sock_sendmsg+0xb8/0xd3
[  917.554047]  [<c0388773>] kernel_sendmsg+0x42/0x4f
[  917.559159]  [<c038bc00>] sock_no_sendpage+0x5e/0x77
[  917.564452]  [<c03ee7af>] xs_tcp_send_request+0x2af/0x375
[  917.570199]  [<c03ed4c5>] xprt_transmit+0x4f/0x20b
[  917.575312]  [<c03ebce1>] call_transmit+0x68/0xc8
[  917.580334]  [<c03f0962>] __rpc_execute+0x5c/0x1ea
[  917.585445]  [<c03f0b27>] rpc_async_schedule+0x11/0x15
[  917.590920]  [<c012bd1d>] worker_thread+0x194/0x225
[  917.596123]  [<c012fe68>] kthread+0xb7/0xbc
[  917.600602]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  917.606076] rpciod/1      S 00000001     0  1515     15          1516  1514 (L-TLB)
[  917.614398] f78dbf34 f78dbf20 00000004 00000001 00000000 c03ed2f0 f7b13fc0 00000000 
[  917.622620]        f7b13e00 c3a2e504 c3a2e504 00000000 f7d89a70 c03ebce1 c3a2e504 f7d89a70 
[  917.631752]        00000000 c2217ec0 c2217560 00000001 00002aff 0db3409e 000000aa f7d89a70 
[  917.640885] Call Trace:
[  917.643702]  [<c012bd90>] worker_thread+0x207/0x225
[  917.648904]  [<c012fe68>] kthread+0xb7/0xbc
[  917.653384]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  917.658837] rpciod/2      S 00000004     0  1516     15          1517  1515 (L-TLB)
[  917.667159] f7ba6f34 f7ba6f24 00000004 00000004 00000000 00000000 00000003 00000000 
[  917.675333]        c4866584 00000000 f7ba6eec c0296593 00000000 f7ba6f00 c04071af f7b13e00 
[  917.685250]        f71717b0 f71717a8 c221f560 00000002 000021eb af7442db 000000a9 c2330030 
[  917.694331] Call Trace:
[  917.697132]  [<c012bd90>] worker_thread+0x207/0x225
[  917.702304]  [<c012fe68>] kthread+0xb7/0xbc
[  917.706758]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  917.712177] rpciod/3      S 00000003     0  1517     15                1516 (L-TLB)
[  917.720499] f7154f34 f7154f20 00000004 00000003 00000000 00000000 00000003 00000000 
[  917.728627]        c2330030 00000000 f7154eec c0296593 f717b030 00000040 c04071af f717b030 
[  917.737686]        00000000 c2227ec0 c2227560 00000003 00003481 af9995ef 000000a9 f717b030 
[  917.746718] Call Trace:
[  917.749502]  [<c012bd90>] worker_thread+0x207/0x225
[  917.754644]  [<c012fe68>] kthread+0xb7/0xbc
[  917.759072]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  917.764482] kjournald     S 00000002     0  1541      1          1544   804 (L-TLB)
[  917.772805] f71d3f58 f71d3f48 00000004 00000002 f724d02c 00000000 00000000 f71d1800 
[  917.780935]        00000000 f7171e00 f7afcd6c 0000020c 00000202 c220ffe0 f71d3fb8 f71d3f2c 
[  917.789964]        f7fb066c f7fb0664 c220f560 00000000 00000585 2cebc346 000000a7 c048db40 
[  917.798993] Call Trace:
[  917.801778]  [<c01b0eef>] kjournald+0x1d6/0x223
[  917.806564]  [<c0101175>] kernel_thread_helper+0x5/0xb
[  917.811974] writetest     D 00000003     0  1544      1                1541 (NOTLB)
[  917.820295] f7adce10 f7adce00 00000004 00000003 f7adcdd4 f7ecfd80 f7adcde8 c01c7bcf 
[  917.828424]        f71b25c0 00000008 c01c7a21 f7adcdcc 00000082 f7b13600 f7adcdd4 f71b25c0 
[  917.837453]        f7adcdd4 f7adcdd4 c2227560 00000003 00000630 c562b75c 000000a9 c2337a70 
[  917.846483] Call Trace:
[  917.849268]  [<c040863b>] io_schedule+0x26/0x30
[  917.854053]  [<c013ab27>] sync_page+0x39/0x4e
[  917.858662]  [<c0408929>] __wait_on_bit_lock+0x58/0x67
[  917.864073]  [<c013b343>] __lock_page+0x84/0x8c
[  917.868858]  [<c013c3c9>] filemap_nopage+0x1e4/0x347
[  917.874089]  [<c014b4b4>] do_no_page+0x70/0x25b
[  917.878876]  [<c014b812>] __handle_mm_fault+0xef/0x22a
[  917.884286]  [<c0409a84>] do_page_fault+0x179/0x5e6
[  917.889429]  [<c01039cf>] error_code+0x4f/0x54
[  932.729798] SysRq : Emergency Sync
[  932.733470] Emergency Sync complete
[  936.457722] Restarting system.
[  936.460951] .

Re: nfs unhappiness with memory pressure

[Trond Myklebust]

[-- Attachment #1: Type: text/plain, Size: 1780 bytes --]

Gah... This is the problem:


[  125.263017] rpciod/2      D 00000004     0  2772     15          2773
2771 (L-TLB)
[  125.271338] f70ccaa8 f70cca98 00000004 00000004 ffffffce f70cca74
c0117800 00000002 
[  125.279560]        c221f560 c2330030 05f561fa 0000000e 00000202
c221ffe0 f70ccabc 00000202 
[  125.288692]        c221ffe0 f70ccabc c221f560 00000002 00000143
2b8c50f7 0000001d c2330030 
[  125.297821] Call Trace:
[  125.300636]  [<c04086e1>] schedule_timeout+0x54/0xa5
[  125.305931]  [<c040866e>] io_schedule_timeout+0x29/0x33
[  125.311495]  [<c02880c4>] blk_congestion_wait+0x70/0x85
[  125.317058]  [<c014136b>] throttle_vm_writeout+0x69/0x7d
[  125.322720]  [<c014714d>] shrink_zone+0xe0/0xfa
[  125.327560]  [<c01471d4>] shrink_caches+0x6d/0x6f
[  125.332581]  [<c01472a6>] try_to_free_pages+0xd0/0x1b5
[  125.338056]  [<c013fa4b>] __alloc_pages+0x135/0x2e8
[  125.343258]  [<c03b74ad>] tcp_sendmsg+0xaa0/0xb78
[  125.348281]  [<c03d4666>] inet_sendmsg+0x48/0x53
[  125.353212]  [<c0388716>] sock_sendmsg+0xb8/0xd3
[  125.358147]  [<c0388773>] kernel_sendmsg+0x42/0x4f
[  125.363259]  [<c038bc00>] sock_no_sendpage+0x5e/0x77
[  125.368556]  [<c03ee7af>] xs_tcp_send_request+0x2af/0x375
[  125.374306]  [<c03ed4c5>] xprt_transmit+0x4f/0x20b
....

If a process is blocking in an OOM situation in the socket layer, then
you are hosed.

Argh... Not sure entirely how to deal with that... We definitely don't
want the thing futzing around inside throttle_vm_writeout(), 'cos
writeout isn't going to happen while the socket blocks.

...OTOH, I'm not entirely sure we want to use GFP_ATOMIC and risk
bleeding the emergency pools dry: we also need those in order to receive
replies from the server.

Ah, well... Try GFP_ATOMIC first, and see if that helps.

Cheers,
 Trond


[-- Attachment #2: linux-2.6.15-fix_sock_allocation.dif --]
[-- Type: text/plain, Size: 1865 bytes --]

RPC: Do not block on skb allocation

 If we get something like the following,
 [  125.300636]  [<c04086e1>] schedule_timeout+0x54/0xa5
 [  125.305931]  [<c040866e>] io_schedule_timeout+0x29/0x33
 [  125.311495]  [<c02880c4>] blk_congestion_wait+0x70/0x85
 [  125.317058]  [<c014136b>] throttle_vm_writeout+0x69/0x7d
 [  125.322720]  [<c014714d>] shrink_zone+0xe0/0xfa
 [  125.327560]  [<c01471d4>] shrink_caches+0x6d/0x6f
 [  125.332581]  [<c01472a6>] try_to_free_pages+0xd0/0x1b5
 [  125.338056]  [<c013fa4b>] __alloc_pages+0x135/0x2e8
 [  125.343258]  [<c03b74ad>] tcp_sendmsg+0xaa0/0xb78
 [  125.348281]  [<c03d4666>] inet_sendmsg+0x48/0x53
 [  125.353212]  [<c0388716>] sock_sendmsg+0xb8/0xd3
 [  125.358147]  [<c0388773>] kernel_sendmsg+0x42/0x4f
 [  125.363259]  [<c038bc00>] sock_no_sendpage+0x5e/0x77
 [  125.368556]  [<c03ee7af>] xs_tcp_send_request+0x2af/0x375
 then the socket is blocked until memory is reclaimed, and no
 progress can ever be made.

 Try to access the emergency pools by using GFP_ATOMIC.

 Signed-off-by: Trond Myklebust <Trond.Myklebust@netapp.com>
---

 net/sunrpc/xprtsock.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c
index 0a51fd4..77e8800 100644
--- a/net/sunrpc/xprtsock.c
+++ b/net/sunrpc/xprtsock.c
@@ -990,6 +990,7 @@ static void xs_udp_connect_worker(void *
 		sk->sk_data_ready = xs_udp_data_ready;
 		sk->sk_write_space = xs_udp_write_space;
 		sk->sk_no_check = UDP_CSUM_NORCV;
+		sk->sk_allocation = GFP_ATOMIC;
 
 		xprt_set_connected(xprt);
 
@@ -1074,6 +1075,7 @@ static void xs_tcp_connect_worker(void *
 		sk->sk_data_ready = xs_tcp_data_ready;
 		sk->sk_state_change = xs_tcp_state_change;
 		sk->sk_write_space = xs_tcp_write_space;
+		sk->sk_allocation = GFP_ATOMIC;
 
 		/* socket options */
 		sk->sk_userlocks |= SOCK_BINDPORT_LOCK;

Re: nfs unhappiness with memory pressure

[Nick Piggin]

Trond Myklebust wrote:
> On Mon, 2005-12-05 at 16:18 -0500, Trond Myklebust wrote:
> 
>>On Tue, 2005-12-06 at 07:52 +1100, Nick Piggin wrote:
>>
>>
>>>The VM doesn't expect to have to rely on pdflush to write out pages
>>>for it. ->writepage should be enough. Adding wakeup_pdflush here
>>>actually could do the wrong thing for non-NFS filesystems if it
>>>starts more writeback.
>>
>>nr_unstable is not going to be set for non-NFS filesystems. 'unstable'
>>is a caching state in which pages have been written out to the NFS
>>server, but the server has not yet flushed the data to disk.
> 

But if you have NFS and non-NFS filesystems, wakeup_pdflush isn't
always going to do the right thing.

> 
> ...and most important of all: 'unstable' does _not_ mean that I/O is
> active on those pages (unlike the apparent assumption in
> vm_throttle_write.
> That is why the choice is either to kick pdflush there, or to remove
> nr_unstable from the accounting in that loop.
> 

Doesn't matter if IO is actually active or not, if you've allocated
memory for these unstable pages, then page reclaim can scan itself
to death, which is what seems to have happened here. And which is
what vm_throttle_write is supposed to help.

-- 
SUSE Labs, Novell Inc.

Send instant messages to your online friends http://au.messenger.yahoo.com 

Re: nfs unhappiness with memory pressure

[Trond Myklebust]

On Tue, 2005-12-06 at 10:40 +1100, Nick Piggin wrote:

> > ...and most important of all: 'unstable' does _not_ mean that I/O is
> > active on those pages (unlike the apparent assumption in
> > vm_throttle_write.
> > That is why the choice is either to kick pdflush there, or to remove
> > nr_unstable from the accounting in that loop.
> > 
> 
> Doesn't matter if IO is actually active or not, if you've allocated
> memory for these unstable pages, then page reclaim can scan itself
> to death, which is what seems to have happened here. And which is
> what vm_throttle_write is supposed to help.

Unless someone somehow triggers an NFS commit, then nr_unstable is not
ever going to decrease, and your process will end up looping forever. In
fact, those nr_writeback that refer to NFS pages, will end up being
added to nr_unstable (because they have been written to the server, but
not committed to disk).

Cheers,
  Trond

Re: nfs unhappiness with memory pressure

[Andrew Morton]

Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
>
> Argh... Not sure entirely how to deal with that... We definitely don't
>  want the thing futzing around inside throttle_vm_writeout(), 'cos
>  writeout isn't going to happen while the socket blocks.
> 

As far as the core VM is concerned, these pages are really "dirty", only it
happens to be a different flavour of dirtiness.  So perhaps we should
continue to mark these pages as dirty and let NFS internally take care
of which end of the wire they're dirty at.

Presumably calling writepage() a second time won't be very useful.  Or will
it?  Perhaps when NFS sees writepage against a PageDirty && PageUnstable
page it can recognise that as a hint to kick off a server-side write.

>  ...OTOH, I'm not entirely sure we want to use GFP_ATOMIC and risk
>  bleeding the emergency pools dry: we also need those in order to receive
>  replies from the server.

You can use (GFP_ATOMIC & !__GFP_HIGH) to avoid draining emergency pools.

Re: nfs unhappiness with memory pressure

[Andrew Morton]

Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
>
> Argh... Not sure entirely how to deal with that... We definitely don't
>  want the thing futzing around inside throttle_vm_writeout(), 'cos
>  writeout isn't going to happen while the socket blocks.
> 

As far as the core VM is concerned, these pages are really "dirty", only it
happens to be a different flavour of dirtiness.  So perhaps we should
continue to mark these pages as dirty and let NFS internally take care
of which end of the wire they're dirty at.

Presumably calling writepage() a second time won't be very useful.  Or will
it?  Perhaps when NFS sees writepage against a PageDirty && PageUnstable
page it can recognise that as a hint to kick off a server-side write.

>  ...OTOH, I'm not entirely sure we want to use GFP_ATOMIC and risk
>  bleeding the emergency pools dry: we also need those in order to receive
>  replies from the server.

You can use (GFP_ATOMIC & ~__GFP_HIGH) to avoid draining emergency pools.

Re: nfs unhappiness with memory pressure

[Trond Myklebust]

On Tue, 2005-12-06 at 14:36 +1100, Andrew Morton wrote:
> Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> >
> > Argh... Not sure entirely how to deal with that... We definitely don't
> >  want the thing futzing around inside throttle_vm_writeout(), 'cos
> >  writeout isn't going to happen while the socket blocks.
> > 
> 
> As far as the core VM is concerned, these pages are really "dirty", only it
> happens to be a different flavour of dirtiness.  So perhaps we should
> continue to mark these pages as dirty and let NFS internally take care
> of which end of the wire they're dirty at.
> 
> Presumably calling writepage() a second time won't be very useful.  Or will
> it?  Perhaps when NFS sees writepage against a PageDirty && PageUnstable
> page it can recognise that as a hint to kick off a server-side write.

Calling writepages() would actually be better. That will do the right
thing, and trigger a commit if there are unstable writes.

Cheers,
  Trond

Re: nfs unhappiness with memory pressure

[Nick Piggin]

Trond Myklebust wrote:
> On Tue, 2005-12-06 at 14:36 +1100, Andrew Morton wrote:
> 
>>Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
>>
>>>Argh... Not sure entirely how to deal with that... We definitely don't
>>> want the thing futzing around inside throttle_vm_writeout(), 'cos
>>> writeout isn't going to happen while the socket blocks.
>>>
>>
>>As far as the core VM is concerned, these pages are really "dirty", only it
>>happens to be a different flavour of dirtiness.  So perhaps we should
>>continue to mark these pages as dirty and let NFS internally take care
>>of which end of the wire they're dirty at.
>>
>>Presumably calling writepage() a second time won't be very useful.  Or will
>>it?  Perhaps when NFS sees writepage against a PageDirty && PageUnstable
>>page it can recognise that as a hint to kick off a server-side write.
> 
> 
> Calling writepages() would actually be better. That will do the right
> thing, and trigger a commit if there are unstable writes.
> 

writepage should as well, then it would have a better chance
of just doing the right thing.

-- 
SUSE Labs, Novell Inc.

Send instant messages to your online friends http://au.messenger.yahoo.com 

Re: nfs unhappiness with memory pressure

[Trond Myklebust]

On Tue, 2005-12-06 at 16:42 +1100, Nick Piggin wrote:

> writepage should as well, then it would have a better chance
> of just doing the right thing.

writepage triggers a stable write of the page (i.e. the page is written
directly to disk) if asked to reclaim it.

If the VM wants the unstable writes from the mapping to be committed, it
should call writepages.

Cheers,
  Trond

Re: nfs unhappiness with memory pressure

[Kenny Simpson]

--- Trond Myklebust <trond.myklebust@fys.uio.no> wrote:

> Gah... This is the problem:
...
> 
> Ah, well... Try GFP_ATOMIC first, and see if that helps.


That DOES IT!!!!

-Kenny



		
__________________________________________ 
Yahoo! DSL – Something to write home about. 
Just $16.99/mo. or less. 
dsl.yahoo.com 

Re: nfs unhappiness with memory pressure

[Kenny Simpson]

--- Andrew Morton <akpm@osdl.org> wrote:
> Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
> >  ...OTOH, I'm not entirely sure we want to use GFP_ATOMIC and risk
> >  bleeding the emergency pools dry: we also need those in order to receive
> >  replies from the server.
> 
> You can use (GFP_ATOMIC & ~__GFP_HIGH) to avoid draining emergency pools.
> 
> 

After beating on this for a while now, it all seems very happy.  The write out to nfs are a little
choppy, but make forward progress.

Any chance of this being in 2.6.15?

-Kenny



		
__________________________________________ 
Yahoo! DSL – Something to write home about. 
Just $16.99/mo. or less. 
dsl.yahoo.com 

Re: nfs unhappiness with memory pressure

[Albert Herranz]

Hi,

The patch linux-2.6.15-fix_sock_allocation.dif seems
to have helped with this issue.

With this patch applied I have been unable to
reproduce the system freezes I was experiencing in
latest 2.6.x kernels when using nfs-root on my
GameCube (24MB RAM).

Thanks,
Albert



	
	
		
______________________________________________ 
Renovamos el Correo Yahoo! 
Nuevos servicios, más seguridad 
http://correo.yahoo.es