pdflush stuck in D state with v2.6.24-rc1-192-gef49c32

pdflush stuck in D state with v2.6.24-rc1-192-gef49c32

[Florin Iucha]

[-- Attachment #1: Type: text/plain, Size: 3457 bytes --]

Hello,

For a week or two I started noticing that some time after I'm logged
in, my keyboard input becomes a bit staggering, there is a small delay
between the keypress and the actual character appearing in the
terminal.  This is on a AMD Athlon x2 4200+ with 2 GB RAM and just a
gnome-terminal open.  The machine is as idle as possible - monitored
via the system monitor applet.  I could not get any hard data on it,
until now.

After I logged off from GNOME, I switched to the text console and ran
top, with the option of showing one CPU stats line for each CPU.  Lo
and behold, one core is 100% idle, and the other one is 25% idle and
75% waiting.  Periodically, a pdflush process in 'D' state raises to
the top.  I did a 'echo t > /proc/sysrequest-trigger' and this is what
is says for the two pdflush processes:

   [ 3687.824424] pdflush       S ffff8100057ffef8     0   247      2
   [ 3687.824427]  ffff8100057ffed0 0000000000000046 ffff8100057ffe70 ffffffff8022a96c
   [ 3687.824431]  ffff8100057fc000 ffff810003040770 ffff8100057fc208 0000000000000297
   [ 3687.824434]  ffff8100057ffe90 ffff810002c1ba10 ffff8100057ffed0 ffffffff8022b9d2
   [ 3687.824438] Call Trace:
   [ 3687.824440]  [<ffffffff8022a96c>] enqueue_task_fair+0x21/0x34
   [ 3687.824444]  [<ffffffff8022b9d2>] set_user_nice+0x110/0x12c
   [ 3687.824448]  [<ffffffff80267165>] pdflush+0x0/0x1c3
   [ 3687.824451]  [<ffffffff80267234>] pdflush+0xcf/0x1c3
   [ 3687.824455]  [<ffffffff80245876>] kthread+0x49/0x77
   [ 3687.824458]  [<ffffffff8020c598>] child_rip+0xa/0x12
   [ 3687.824463]  [<ffffffff8024582d>] kthread+0x0/0x77
   [ 3687.824466]  [<ffffffff8020c58e>] child_rip+0x0/0x12
   [ 3687.824468] 
   [ 3687.824470] pdflush       D ffffffff805787c0     0   248      2
   [ 3687.824473]  ffff810006001d90 0000000000000046 0000000000000000 0000000000000286
   [ 3687.824476]  ffff8100057fc770 ffff810003062000 ffff8100057fc978 0000000106001da0
   [ 3687.824480]  0000000000000003 ffffffff8023b1b2 0000000000000000 0000000000000000
   [ 3687.824483] Call Trace:
   [ 3687.824488]  [<ffffffff8023b1b2>] __mod_timer+0xb8/0xca
   [ 3687.824492]  [<ffffffff8055c87a>] schedule_timeout+0x8d/0xb4
   [ 3687.824496]  [<ffffffff8023ad6c>] process_timeout+0x0/0xb
   [ 3687.824499]  [<ffffffff8055c79a>] io_schedule_timeout+0x28/0x33
   [ 3687.824503]  [<ffffffff8026bb24>] congestion_wait+0x6b/0x87
   [ 3687.824506]  [<ffffffff80245983>] autoremove_wake_function+0x0/0x38
   [ 3687.824510]  [<ffffffff8029e684>] writeback_inodes+0xcd/0xd5
   [ 3687.824514]  [<ffffffff80266dc4>] wb_kupdate+0xbb/0x10d
   [ 3687.824518]  [<ffffffff80267165>] pdflush+0x0/0x1c3
   [ 3687.824520]  [<ffffffff8026727d>] pdflush+0x118/0x1c3
   [ 3687.824523]  [<ffffffff80266d09>] wb_kupdate+0x0/0x10d
   [ 3687.824527]  [<ffffffff80245876>] kthread+0x49/0x77
   [ 3687.824530]  [<ffffffff8020c598>] child_rip+0xa/0x12
   [ 3687.824535]  [<ffffffff8024582d>] kthread+0x0/0x77
   [ 3687.824538]  [<ffffffff8020c58e>] child_rip+0x0/0x12
   [ 3687.824540] 

What could cause this?  I use NFS4 to automount the home directories
from a Solaris10 server, and this box found a few bugs in the NFS4
code (fixed in the 2.6.22 kernel).

I'll try running with 2.6.23 again for a few days, to see if I get the
pdflush stuck.  Any other ideas?

florin

-- 
Bruce Schneier expects the Spanish Inquisition.
      http://geekz.co.uk/schneierfacts/fact/163

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: pdflush stuck in D state with v2.6.24-rc1-192-gef49c32

[Trond Myklebust]

On Sun, 2007-10-28 at 10:24 -0500, Florin Iucha wrote:
> Hello,
> 
> For a week or two I started noticing that some time after I'm logged
> in, my keyboard input becomes a bit staggering, there is a small delay
> between the keypress and the actual character appearing in the
> terminal.  This is on a AMD Athlon x2 4200+ with 2 GB RAM and just a
> gnome-terminal open.  The machine is as idle as possible - monitored
> via the system monitor applet.  I could not get any hard data on it,
> until now.
> 
> After I logged off from GNOME, I switched to the text console and ran
> top, with the option of showing one CPU stats line for each CPU.  Lo
> and behold, one core is 100% idle, and the other one is 25% idle and
> 75% waiting.  Periodically, a pdflush process in 'D' state raises to
> the top.  I did a 'echo t > /proc/sysrequest-trigger' and this is what
> is says for the two pdflush processes:
> 
>    [ 3687.824424] pdflush       S ffff8100057ffef8     0   247      2
>    [ 3687.824427]  ffff8100057ffed0 0000000000000046 ffff8100057ffe70 ffffffff8022a96c
>    [ 3687.824431]  ffff8100057fc000 ffff810003040770 ffff8100057fc208 0000000000000297
>    [ 3687.824434]  ffff8100057ffe90 ffff810002c1ba10 ffff8100057ffed0 ffffffff8022b9d2
>    [ 3687.824438] Call Trace:
>    [ 3687.824440]  [<ffffffff8022a96c>] enqueue_task_fair+0x21/0x34
>    [ 3687.824444]  [<ffffffff8022b9d2>] set_user_nice+0x110/0x12c
>    [ 3687.824448]  [<ffffffff80267165>] pdflush+0x0/0x1c3
>    [ 3687.824451]  [<ffffffff80267234>] pdflush+0xcf/0x1c3
>    [ 3687.824455]  [<ffffffff80245876>] kthread+0x49/0x77
>    [ 3687.824458]  [<ffffffff8020c598>] child_rip+0xa/0x12
>    [ 3687.824463]  [<ffffffff8024582d>] kthread+0x0/0x77
>    [ 3687.824466]  [<ffffffff8020c58e>] child_rip+0x0/0x12
>    [ 3687.824468] 
>    [ 3687.824470] pdflush       D ffffffff805787c0     0   248      2
>    [ 3687.824473]  ffff810006001d90 0000000000000046 0000000000000000 0000000000000286
>    [ 3687.824476]  ffff8100057fc770 ffff810003062000 ffff8100057fc978 0000000106001da0
>    [ 3687.824480]  0000000000000003 ffffffff8023b1b2 0000000000000000 0000000000000000
>    [ 3687.824483] Call Trace:
>    [ 3687.824488]  [<ffffffff8023b1b2>] __mod_timer+0xb8/0xca
>    [ 3687.824492]  [<ffffffff8055c87a>] schedule_timeout+0x8d/0xb4
>    [ 3687.824496]  [<ffffffff8023ad6c>] process_timeout+0x0/0xb
>    [ 3687.824499]  [<ffffffff8055c79a>] io_schedule_timeout+0x28/0x33
>    [ 3687.824503]  [<ffffffff8026bb24>] congestion_wait+0x6b/0x87
>    [ 3687.824506]  [<ffffffff80245983>] autoremove_wake_function+0x0/0x38
>    [ 3687.824510]  [<ffffffff8029e684>] writeback_inodes+0xcd/0xd5
>    [ 3687.824514]  [<ffffffff80266dc4>] wb_kupdate+0xbb/0x10d
>    [ 3687.824518]  [<ffffffff80267165>] pdflush+0x0/0x1c3
>    [ 3687.824520]  [<ffffffff8026727d>] pdflush+0x118/0x1c3
>    [ 3687.824523]  [<ffffffff80266d09>] wb_kupdate+0x0/0x10d
>    [ 3687.824527]  [<ffffffff80245876>] kthread+0x49/0x77
>    [ 3687.824530]  [<ffffffff8020c598>] child_rip+0xa/0x12
>    [ 3687.824535]  [<ffffffff8024582d>] kthread+0x0/0x77
>    [ 3687.824538]  [<ffffffff8020c58e>] child_rip+0x0/0x12
>    [ 3687.824540] 
> 
> What could cause this?  I use NFS4 to automount the home directories
> from a Solaris10 server, and this box found a few bugs in the NFS4
> code (fixed in the 2.6.22 kernel).
> 
> I'll try running with 2.6.23 again for a few days, to see if I get the
> pdflush stuck.  Any other ideas?

One of them appears to be waiting for i/o congestion to clear up. If the
filesystem is NFS, then that means that some other thread is busy
writing data out to the server. You'll need to look at the rest of the
thread dump to figure out which thread is writing the data out, and
where it is getting stuck.

Trond

Re: pdflush stuck in D state with v2.6.24-rc1-192-gef49c32

[Florin Iucha]

[-- Attachment #1: Type: text/plain, Size: 905 bytes --]

On Mon, Oct 29, 2007 at 09:46:59AM -0400, Trond Myklebust wrote:
> > What could cause this?  I use NFS4 to automount the home directories
> > from a Solaris10 server, and this box found a few bugs in the NFS4
> > code (fixed in the 2.6.22 kernel).
> > 
> > I'll try running with 2.6.23 again for a few days, to see if I get the
> > pdflush stuck.  Any other ideas?
> 
> One of them appears to be waiting for i/o congestion to clear up. If the
> filesystem is NFS, then that means that some other thread is busy
> writing data out to the server. You'll need to look at the rest of the
> thread dump to figure out which thread is writing the data out, and
> where it is getting stuck.

Trond,

The full dmesg is at http://iucha.net/2.6.24-rc1/dmesg.stuck_pdflush.gz

Cheers,
florin

-- 
Bruce Schneier expects the Spanish Inquisition.
      http://geekz.co.uk/schneierfacts/fact/163

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: pdflush stuck in D state with v2.6.24-rc1-192-gef49c32

[Trond Myklebust]

On Mon, 2007-10-29 at 10:01 -0500, Florin Iucha wrote:
> On Mon, Oct 29, 2007 at 09:46:59AM -0400, Trond Myklebust wrote:
> > > What could cause this?  I use NFS4 to automount the home directories
> > > from a Solaris10 server, and this box found a few bugs in the NFS4
> > > code (fixed in the 2.6.22 kernel).
> > > 
> > > I'll try running with 2.6.23 again for a few days, to see if I get the
> > > pdflush stuck.  Any other ideas?
> > 
> > One of them appears to be waiting for i/o congestion to clear up. If the
> > filesystem is NFS, then that means that some other thread is busy
> > writing data out to the server. You'll need to look at the rest of the
> > thread dump to figure out which thread is writing the data out, and
> > where it is getting stuck.
> 
> Trond,
> 
> The full dmesg is at http://iucha.net/2.6.24-rc1/dmesg.stuck_pdflush.gz
> 
> Cheers,
> florin

I can't see any evidence of NFS traffic at all in those traces, but
there is a fuse process that is sleeping in :fuse:fuse_dev_read(). Could
that perhaps be relevant.

Trond

Re: pdflush stuck in D state with v2.6.24-rc1-192-gef49c32

[Florin Iucha]

[-- Attachment #1: Type: text/plain, Size: 1543 bytes --]

On Mon, Oct 29, 2007 at 02:43:32PM -0400, Trond Myklebust wrote:
> 
> On Mon, 2007-10-29 at 10:01 -0500, Florin Iucha wrote:
> > On Mon, Oct 29, 2007 at 09:46:59AM -0400, Trond Myklebust wrote:
> > > > What could cause this?  I use NFS4 to automount the home directories
> > > > from a Solaris10 server, and this box found a few bugs in the NFS4
> > > > code (fixed in the 2.6.22 kernel).
> > > > 
> > > > I'll try running with 2.6.23 again for a few days, to see if I get the
> > > > pdflush stuck.  Any other ideas?
> > > 
> > > One of them appears to be waiting for i/o congestion to clear up. If the
> > > filesystem is NFS, then that means that some other thread is busy
> > > writing data out to the server. You'll need to look at the rest of the
> > > thread dump to figure out which thread is writing the data out, and
> > > where it is getting stuck.
> > 
> > Trond,
> > 
> > The full dmesg is at http://iucha.net/2.6.24-rc1/dmesg.stuck_pdflush.gz
> 
> I can't see any evidence of NFS traffic at all in those traces, but
> there is a fuse process that is sleeping in :fuse:fuse_dev_read(). Could
> that perhaps be relevant.

That might be the overzealous Ubuntu trying to make the NTFS partition
available.  I will try to disable it and see if I can reproduce the
hang.  BTW: With 2.6.24-rc1+ it happens after a couple of hours.  With
2.6.23 it did not happen after 6 hours or so.

Cheers,
florin

-- 
Bruce Schneier expects the Spanish Inquisition.
      http://geekz.co.uk/schneierfacts/fact/163

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: pdflush stuck in D state with v2.6.24-rc1-192-gef49c32

[Fengguang Wu]

[-- Attachment #1: Type: text/plain, Size: 2060 bytes --]

On Sun, Oct 28, 2007 at 10:24:29AM -0500, Florin Iucha wrote:
[...]
>    [ 3687.824468] 
>    [ 3687.824470] pdflush       D ffffffff805787c0     0   248      2
>    [ 3687.824473]  ffff810006001d90 0000000000000046 0000000000000000 0000000000000286
>    [ 3687.824476]  ffff8100057fc770 ffff810003062000 ffff8100057fc978 0000000106001da0
>    [ 3687.824480]  0000000000000003 ffffffff8023b1b2 0000000000000000 0000000000000000
>    [ 3687.824483] Call Trace:
>    [ 3687.824488]  [<ffffffff8023b1b2>] __mod_timer+0xb8/0xca
>    [ 3687.824492]  [<ffffffff8055c87a>] schedule_timeout+0x8d/0xb4
>    [ 3687.824496]  [<ffffffff8023ad6c>] process_timeout+0x0/0xb
>    [ 3687.824499]  [<ffffffff8055c79a>] io_schedule_timeout+0x28/0x33
>    [ 3687.824503]  [<ffffffff8026bb24>] congestion_wait+0x6b/0x87
>    [ 3687.824506]  [<ffffffff80245983>] autoremove_wake_function+0x0/0x38
>    [ 3687.824510]  [<ffffffff8029e684>] writeback_inodes+0xcd/0xd5
>    [ 3687.824514]  [<ffffffff80266dc4>] wb_kupdate+0xbb/0x10d
>    [ 3687.824518]  [<ffffffff80267165>] pdflush+0x0/0x1c3
>    [ 3687.824520]  [<ffffffff8026727d>] pdflush+0x118/0x1c3
>    [ 3687.824523]  [<ffffffff80266d09>] wb_kupdate+0x0/0x10d
>    [ 3687.824527]  [<ffffffff80245876>] kthread+0x49/0x77
>    [ 3687.824530]  [<ffffffff8020c598>] child_rip+0xa/0x12
>    [ 3687.824535]  [<ffffffff8024582d>] kthread+0x0/0x77
>    [ 3687.824538]  [<ffffffff8020c58e>] child_rip+0x0/0x12
>    [ 3687.824540] 
> 
> What could cause this?  I use NFS4 to automount the home directories
> from a Solaris10 server, and this box found a few bugs in the NFS4
> code (fixed in the 2.6.22 kernel).
> 
> I'll try running with 2.6.23 again for a few days, to see if I get the
> pdflush stuck.  Any other ideas?

It could be triggered by the more aggressive writeback behavior - the
new code will keep on retrying as long as there are dirty inodes pending.

Florin, would you try the attached patches against 2.6.24-git?
They may generate big traffic of printk messages, but will help
debug the problem.

Thank you,
Fengguang

[-- Attachment #2: writeback-debug.patch --]
[-- Type: text/x-diff, Size: 1926 bytes --]

---
 mm/page-writeback.c |   23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

--- linux-2.6.23-rc8-mm2.orig/mm/page-writeback.c
+++ linux-2.6.23-rc8-mm2/mm/page-writeback.c
@@ -98,6 +98,26 @@ EXPORT_SYMBOL(laptop_mode);
 
 /* End of sysctl-exported parameters */
 
+#define writeback_debug_report(n, wbc) do {                               \
+	__writeback_debug_report(n, wbc, __FILE__, __LINE__, __FUNCTION__); \
+} while (0)
+
+void __writeback_debug_report(long n, struct writeback_control *wbc,
+		const char *file, int line, const char *func)
+{
+	printk("%s %d %s: %s(%d) %ld "
+			"global %lu %lu %lu "
+			"wc %c%c tw %ld sk %ld\n",
+			file, line, func,
+			current->comm, current->pid, n,
+			global_page_state(NR_FILE_DIRTY),
+			global_page_state(NR_WRITEBACK),
+			global_page_state(NR_UNSTABLE_NFS),
+			wbc->encountered_congestion ? 'C':'_',
+			wbc->more_io ? 'M':'_',
+			wbc->nr_to_write,
+			wbc->pages_skipped);
+}
 
 static void background_writeout(unsigned long _min_pages);
 
@@ -404,6 +424,7 @@ static void balance_dirty_pages(struct a
 			pages_written += write_chunk - wbc.nr_to_write;
 			get_dirty_limits(&background_thresh, &dirty_thresh,
 				       &bdi_thresh, bdi);
+			writeback_debug_report(pages_written, &wbc);
 		}
 
 		/*
@@ -568,6 +589,7 @@ static void background_writeout(unsigned
 		wbc.pages_skipped = 0;
 		writeback_inodes(&wbc);
 		min_pages -= MAX_WRITEBACK_PAGES - wbc.nr_to_write;
+		writeback_debug_report(min_pages, &wbc);
 		if (wbc.nr_to_write > 0 || wbc.pages_skipped > 0) {
 			/* Wrote less than expected */
 			if (wbc.encountered_congestion)
@@ -643,6 +665,7 @@ static void wb_kupdate(unsigned long arg
 		wbc.encountered_congestion = 0;
 		wbc.nr_to_write = MAX_WRITEBACK_PAGES;
 		writeback_inodes(&wbc);
+		writeback_debug_report(nr_to_write, &wbc);
 		if (wbc.nr_to_write > 0) {
 			if (wbc.encountered_congestion)
 				congestion_wait(WRITE, HZ/10);

[-- Attachment #3: requeue_io-debug.patch --]
[-- Type: text/x-diff, Size: 1140 bytes --]

Subject: track redirty_tail() calls

It helps a lot to know how redirty_tail() are called.

Cc: Ken Chen <kenchen@google.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Fengguang Wu <wfg@mail.ustc.edu.cn>
---
 fs/fs-writeback.c |   16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

--- linux-2.6.24-git17.orig/fs/fs-writeback.c
+++ linux-2.6.24-git17/fs/fs-writeback.c
@@ -164,12 +164,26 @@ static void redirty_tail(struct inode *i
 	list_move(&inode->i_list, &sb->s_dirty);
 }
 
+#define requeue_io(inode)						\
+	do {								\
+		__requeue_io(inode, __LINE__);				\
+	} while (0)
+
 /*
  * requeue inode for re-scanning after sb->s_io list is exhausted.
  */
-static void requeue_io(struct inode *inode)
+static void __requeue_io(struct inode *inode, int line)
 {
 	list_move(&inode->i_list, &inode->i_sb->s_more_io);
+
+	printk(KERN_DEBUG "requeue_io %d: inode %lu size %llu at %02x:%02x(%s)\n",
+			line,
+			inode->i_ino,
+			i_size_read(inode),
+			MAJOR(inode->i_sb->s_dev),
+			MINOR(inode->i_sb->s_dev),
+			inode->i_sb->s_id
+			);
 }
 
 static void inode_sync_complete(struct inode *inode)

Re: pdflush stuck in D state with v2.6.24-rc1-192-gef49c32

[Florin Iucha]

[-- Attachment #1: Type: text/plain, Size: 2552 bytes --]

On Tue, Oct 30, 2007 at 03:54:03PM +0800, Fengguang Wu wrote:
> On Sun, Oct 28, 2007 at 10:24:29AM -0500, Florin Iucha wrote:
> [...]
> >    [ 3687.824468] 
> >    [ 3687.824470] pdflush       D ffffffff805787c0     0   248      2
> >    [ 3687.824473]  ffff810006001d90 0000000000000046 0000000000000000 0000000000000286
> >    [ 3687.824476]  ffff8100057fc770 ffff810003062000 ffff8100057fc978 0000000106001da0
> >    [ 3687.824480]  0000000000000003 ffffffff8023b1b2 0000000000000000 0000000000000000
> >    [ 3687.824483] Call Trace:
> >    [ 3687.824488]  [<ffffffff8023b1b2>] __mod_timer+0xb8/0xca
> >    [ 3687.824492]  [<ffffffff8055c87a>] schedule_timeout+0x8d/0xb4
> >    [ 3687.824496]  [<ffffffff8023ad6c>] process_timeout+0x0/0xb
> >    [ 3687.824499]  [<ffffffff8055c79a>] io_schedule_timeout+0x28/0x33
> >    [ 3687.824503]  [<ffffffff8026bb24>] congestion_wait+0x6b/0x87
> >    [ 3687.824506]  [<ffffffff80245983>] autoremove_wake_function+0x0/0x38
> >    [ 3687.824510]  [<ffffffff8029e684>] writeback_inodes+0xcd/0xd5
> >    [ 3687.824514]  [<ffffffff80266dc4>] wb_kupdate+0xbb/0x10d
> >    [ 3687.824518]  [<ffffffff80267165>] pdflush+0x0/0x1c3
> >    [ 3687.824520]  [<ffffffff8026727d>] pdflush+0x118/0x1c3
> >    [ 3687.824523]  [<ffffffff80266d09>] wb_kupdate+0x0/0x10d
> >    [ 3687.824527]  [<ffffffff80245876>] kthread+0x49/0x77
> >    [ 3687.824530]  [<ffffffff8020c598>] child_rip+0xa/0x12
> >    [ 3687.824535]  [<ffffffff8024582d>] kthread+0x0/0x77
> >    [ 3687.824538]  [<ffffffff8020c58e>] child_rip+0x0/0x12
> >    [ 3687.824540] 
> > 
> > What could cause this?  I use NFS4 to automount the home directories
> > from a Solaris10 server, and this box found a few bugs in the NFS4
> > code (fixed in the 2.6.22 kernel).
> > 
> > I'll try running with 2.6.23 again for a few days, to see if I get the
> > pdflush stuck.  Any other ideas?
> 
> It could be triggered by the more aggressive writeback behavior - the
> new code will keep on retrying as long as there are dirty inodes pending.
> 
> Florin, would you try the attached patches against 2.6.24-git?
> They may generate big traffic of printk messages, but will help
> debug the problem.

I have updated to v2.6.24-rc1-334-g82798a1.  After using my computer
for two hours, I left the computer idle overnight.  This morning,
pdflushd is again consuming 25% of a CPU.  I will try Fengguang's
patches today.

florin

-- 
Bruce Schneier expects the Spanish Inquisition.
      http://geekz.co.uk/schneierfacts/fact/163

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: pdflush stuck in D state with v2.6.24-rc1-192-gef49c32

[Fengguang Wu]

On Tue, Oct 30, 2007 at 06:42:50AM -0500, Florin Iucha wrote:
> On Tue, Oct 30, 2007 at 03:54:03PM +0800, Fengguang Wu wrote:
> > It could be triggered by the more aggressive writeback behavior - the
> > new code will keep on retrying as long as there are dirty inodes pending.
> > 
> > Florin, would you try the attached patches against 2.6.24-git?
> > They may generate big traffic of printk messages, but will help
> > debug the problem.
> 
> I have updated to v2.6.24-rc1-334-g82798a1.  After using my computer
> for two hours, I left the computer idle overnight.  This morning,
> pdflushd is again consuming 25% of a CPU.  I will try Fengguang's
> patches today.

Thank you.

Be sure to stop klogd ;-)
A `dmesg` will be sufficient for debugging.

btw, I cannot reproduce it with simple NFSv3 tests.

Fengguang

Re: pdflush stuck in D state with v2.6.24-rc1-192-gef49c32

[Florin Iucha]

[-- Attachment #1: Type: text/plain, Size: 1194 bytes --]

On Tue, Oct 30, 2007 at 07:49:41PM +0800, Fengguang Wu wrote:
> On Tue, Oct 30, 2007 at 06:42:50AM -0500, Florin Iucha wrote:
> > On Tue, Oct 30, 2007 at 03:54:03PM +0800, Fengguang Wu wrote:
> > > It could be triggered by the more aggressive writeback behavior - the
> > > new code will keep on retrying as long as there are dirty inodes pending.
> > > 
> > > Florin, would you try the attached patches against 2.6.24-git?
> > > They may generate big traffic of printk messages, but will help
> > > debug the problem.
> > 
> > I have updated to v2.6.24-rc1-334-g82798a1.  After using my computer
> > for two hours, I left the computer idle overnight.  This morning,
> > pdflushd is again consuming 25% of a CPU.  I will try Fengguang's
> > patches today.
> 
> Thank you.
> 
> Be sure to stop klogd ;-)
> A `dmesg` will be sufficient for debugging.
> 
> btw, I cannot reproduce it with simple NFSv3 tests.

This is using nfsv4.  And being idle in a GNOME session with home
mounted from the server is hardly a stressful experience.  But what do
I know ;)

florin

-- 
Bruce Schneier expects the Spanish Inquisition.
      http://geekz.co.uk/schneierfacts/fact/163

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: pdflush stuck in D state with v2.6.24-rc1-192-gef49c32

[Florin Iucha]

[-- Attachment #1: Type: text/plain, Size: 1566 bytes --]

On Tue, Oct 30, 2007 at 07:49:41PM +0800, Fengguang Wu wrote:
> > > It could be triggered by the more aggressive writeback behavior - the
> > > new code will keep on retrying as long as there are dirty inodes pending.
> > > 
> > > Florin, would you try the attached patches against 2.6.24-git?
> > > They may generate big traffic of printk messages, but will help
> > > debug the problem.
> > 
> > I have updated to v2.6.24-rc1-334-g82798a1.  After using my computer
> > for two hours, I left the computer idle overnight.  This morning,
> > pdflushd is again consuming 25% of a CPU.  I will try Fengguang's
> > patches today.
> 
> Thank you.
> 
> Be sure to stop klogd ;-)
> A `dmesg` will be sufficient for debugging.

I have added the patches and started a linux kernel compilation, and
something really interesting happens.  I run the build with the
equivalent of "make -j3" and in a separate console I am watching the
build with 'top'.  The build consumes 98% of both CPUs.  If I stop the
output in the build console with "Ctrl-S", one core goes to idle,
while the other is in 50% waiting, then goes to 75% waiting.  When I
resume the build with "Ctrl-Q", the build starts to use both CPUs at
98-99%.  The NFS4 use was minimal, as I did not login with Gnome, but
just logged on the console.  Also, the CPU that is in 75% waiting
state changes occasionally.  'Top' shows pdflush in D state, using
5-6% of CPU.

Cheers,
florin

-- 
Bruce Schneier expects the Spanish Inquisition.
      http://geekz.co.uk/schneierfacts/fact/163

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: pdflush stuck in D state with v2.6.24-rc1-192-gef49c32

[Florin Iucha]

[-- Attachment #1: Type: text/plain, Size: 1050 bytes --]

On Tue, Oct 30, 2007 at 07:02:42PM -0500, Florin Iucha wrote:
> I have added the patches and started a linux kernel compilation, and
> something really interesting happens.  I run the build with the
> equivalent of "make -j3" and in a separate console I am watching the
> build with 'top'.  The build consumes 98% of both CPUs.  If I stop the
> output in the build console with "Ctrl-S", one core goes to idle,
> while the other is in 50% waiting, then goes to 75% waiting.  When I
> resume the build with "Ctrl-Q", the build starts to use both CPUs at
> 98-99%.  The NFS4 use was minimal, as I did not login with Gnome, but
> just logged on the console.  Also, the CPU that is in 75% waiting
> state changes occasionally.  'Top' shows pdflush in D state, using
> 5-6% of CPU.

I forgot the traces:

   http://iucha.net/2.6.24-rc1/fw.1.gz
   http://iucha.net/2.6.24-rc1/fw.2.gz
   http://iucha.net/2.6.24-rc1/fw.3.gz

florin

-- 
Bruce Schneier expects the Spanish Inquisition.
      http://geekz.co.uk/schneierfacts/fact/163

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: pdflush stuck in D state with v2.6.24-rc1-192-gef49c32

[Fengguang Wu]

On Tue, Oct 30, 2007 at 10:52:45PM -0500, Florin Iucha wrote:
> On Tue, Oct 30, 2007 at 07:02:42PM -0500, Florin Iucha wrote:
> > I have added the patches and started a linux kernel compilation, and
> > something really interesting happens.  I run the build with the
> > equivalent of "make -j3" and in a separate console I am watching the
> > build with 'top'.  The build consumes 98% of both CPUs.  If I stop the
> > output in the build console with "Ctrl-S", one core goes to idle,
> > while the other is in 50% waiting, then goes to 75% waiting.  When I
> > resume the build with "Ctrl-Q", the build starts to use both CPUs at
> > 98-99%.  The NFS4 use was minimal, as I did not login with Gnome, but
> > just logged on the console.  Also, the CPU that is in 75% waiting
> > state changes occasionally.  'Top' shows pdflush in D state, using
> > 5-6% of CPU.
> 
> I forgot the traces:
> 
>    http://iucha.net/2.6.24-rc1/fw.1.gz
>    http://iucha.net/2.6.24-rc1/fw.2.gz
>    http://iucha.net/2.6.24-rc1/fw.3.gz

Sorry for the delay - I've been fixing our server today.

[  263.685691] mm/page-writeback.c 655 wb_kupdate: pdflush(248) 24235 global 4593 0 0 wc _M tw 1024 sk 0
[  263.789648] requeue_io 301: inode 4031199 size 562 at 08:07(sda7)
[  263.789656] requeue_io 301: inode 4031231 size 329 at 08:07(sda7)
[  263.789660] requeue_io 301: inode 4031255 size 177 at 08:07(sda7)
[  263.789664] requeue_io 301: inode 4031268 size 94 at 08:07(sda7)
[  263.789667] requeue_io 301: inode 4031329 size 88 at 08:07(sda7)
[  263.789671] requeue_io 301: inode 4031351 size 74 at 08:07(sda7)
[  263.789674] requeue_io 301: inode 4031408 size 175 at 08:07(sda7)
[  263.789678] requeue_io 301: inode 4031413 size 129 at 08:07(sda7)
[  263.789681] requeue_io 301: inode 4031415 size 391 at 08:07(sda7)
[  263.789690] mm/page-writeback.c 655 wb_kupdate: pdflush(248) 24235 global 4593 0 0 wc _M tw 1024 sk 0
[  263.890184] requeue_io 301: inode 4031199 size 562 at 08:07(sda7)
[  263.890191] requeue_io 301: inode 4031231 size 329 at 08:07(sda7)
[  263.890195] requeue_io 301: inode 4031255 size 177 at 08:07(sda7)
[  263.890198] requeue_io 301: inode 4031268 size 94 at 08:07(sda7)
[  263.890202] requeue_io 301: inode 4031329 size 88 at 08:07(sda7)
[  263.890205] requeue_io 301: inode 4031351 size 74 at 08:07(sda7)
[  263.890208] requeue_io 301: inode 4031408 size 175 at 08:07(sda7)
[  263.890212] requeue_io 301: inode 4031413 size 129 at 08:07(sda7)
[  263.890215] requeue_io 301: inode 4031415 size 391 at 08:07(sda7)
[  263.890223] mm/page-writeback.c 655 wb_kupdate: pdflush(248) 24235 global 4593 0 0 wc _M tw 1024 sk 0

It's about sda7, not NFSv4.

Is it a Reiserfs? We have a fresh fix for it: http://lkml.org/lkml/2007/10/23/93

Thank you,
Fengguang

Re: pdflush stuck in D state with v2.6.24-rc1-192-gef49c32

[Florin Iucha]

[-- Attachment #1: Type: text/plain, Size: 3331 bytes --]

On Wed, Oct 31, 2007 at 02:53:25PM +0800, Fengguang Wu wrote:
> On Tue, Oct 30, 2007 at 10:52:45PM -0500, Florin Iucha wrote:
> > On Tue, Oct 30, 2007 at 07:02:42PM -0500, Florin Iucha wrote:
> > > I have added the patches and started a linux kernel compilation, and
> > > something really interesting happens.  I run the build with the
> > > equivalent of "make -j3" and in a separate console I am watching the
> > > build with 'top'.  The build consumes 98% of both CPUs.  If I stop the
> > > output in the build console with "Ctrl-S", one core goes to idle,
> > > while the other is in 50% waiting, then goes to 75% waiting.  When I
> > > resume the build with "Ctrl-Q", the build starts to use both CPUs at
> > > 98-99%.  The NFS4 use was minimal, as I did not login with Gnome, but
> > > just logged on the console.  Also, the CPU that is in 75% waiting
> > > state changes occasionally.  'Top' shows pdflush in D state, using
> > > 5-6% of CPU.
> > 
> > I forgot the traces:
> > 
> >    http://iucha.net/2.6.24-rc1/fw.1.gz
> >    http://iucha.net/2.6.24-rc1/fw.2.gz
> >    http://iucha.net/2.6.24-rc1/fw.3.gz
> 
> Sorry for the delay - I've been fixing our server today.
> 
> [  263.685691] mm/page-writeback.c 655 wb_kupdate: pdflush(248) 24235 global 4593 0 0 wc _M tw 1024 sk 0
> [  263.789648] requeue_io 301: inode 4031199 size 562 at 08:07(sda7)
> [  263.789656] requeue_io 301: inode 4031231 size 329 at 08:07(sda7)
> [  263.789660] requeue_io 301: inode 4031255 size 177 at 08:07(sda7)
> [  263.789664] requeue_io 301: inode 4031268 size 94 at 08:07(sda7)
> [  263.789667] requeue_io 301: inode 4031329 size 88 at 08:07(sda7)
> [  263.789671] requeue_io 301: inode 4031351 size 74 at 08:07(sda7)
> [  263.789674] requeue_io 301: inode 4031408 size 175 at 08:07(sda7)
> [  263.789678] requeue_io 301: inode 4031413 size 129 at 08:07(sda7)
> [  263.789681] requeue_io 301: inode 4031415 size 391 at 08:07(sda7)
> [  263.789690] mm/page-writeback.c 655 wb_kupdate: pdflush(248) 24235 global 4593 0 0 wc _M tw 1024 sk 0
> [  263.890184] requeue_io 301: inode 4031199 size 562 at 08:07(sda7)
> [  263.890191] requeue_io 301: inode 4031231 size 329 at 08:07(sda7)
> [  263.890195] requeue_io 301: inode 4031255 size 177 at 08:07(sda7)
> [  263.890198] requeue_io 301: inode 4031268 size 94 at 08:07(sda7)
> [  263.890202] requeue_io 301: inode 4031329 size 88 at 08:07(sda7)
> [  263.890205] requeue_io 301: inode 4031351 size 74 at 08:07(sda7)
> [  263.890208] requeue_io 301: inode 4031408 size 175 at 08:07(sda7)
> [  263.890212] requeue_io 301: inode 4031413 size 129 at 08:07(sda7)
> [  263.890215] requeue_io 301: inode 4031415 size 391 at 08:07(sda7)
> [  263.890223] mm/page-writeback.c 655 wb_kupdate: pdflush(248) 24235 global 4593 0 0 wc _M tw 1024 sk 0
> 
> It's about sda7, not NFSv4.
> 
> Is it a Reiserfs? We have a fresh fix for it: http://lkml.org/lkml/2007/10/23/93

Yes, it is a Reiserfs.  Incidentally it is the partition that holds
the kernel sources and build directory.  The message states that the
same bug exists in 2.6.23 but I do not see the same behavior in
2.6.23.  Anyway, I will apply the patch and see what I get.

Thanks,
florin

-- 
Bruce Schneier expects the Spanish Inquisition.
      http://geekz.co.uk/schneierfacts/fact/163

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: pdflush stuck in D state with v2.6.24-rc1-192-gef49c32

[Florin Iucha]

[-- Attachment #1: Type: text/plain, Size: 3677 bytes --]

On Wed, Oct 31, 2007 at 07:16:06AM -0500, Florin Iucha wrote:
> On Wed, Oct 31, 2007 at 02:53:25PM +0800, Fengguang Wu wrote:
> > On Tue, Oct 30, 2007 at 10:52:45PM -0500, Florin Iucha wrote:
> > > On Tue, Oct 30, 2007 at 07:02:42PM -0500, Florin Iucha wrote:
> > > > I have added the patches and started a linux kernel compilation, and
> > > > something really interesting happens.  I run the build with the
> > > > equivalent of "make -j3" and in a separate console I am watching the
> > > > build with 'top'.  The build consumes 98% of both CPUs.  If I stop the
> > > > output in the build console with "Ctrl-S", one core goes to idle,
> > > > while the other is in 50% waiting, then goes to 75% waiting.  When I
> > > > resume the build with "Ctrl-Q", the build starts to use both CPUs at
> > > > 98-99%.  The NFS4 use was minimal, as I did not login with Gnome, but
> > > > just logged on the console.  Also, the CPU that is in 75% waiting
> > > > state changes occasionally.  'Top' shows pdflush in D state, using
> > > > 5-6% of CPU.
> > > 
> > > I forgot the traces:
> > > 
> > >    http://iucha.net/2.6.24-rc1/fw.1.gz
> > >    http://iucha.net/2.6.24-rc1/fw.2.gz
> > >    http://iucha.net/2.6.24-rc1/fw.3.gz
> > 
> > Sorry for the delay - I've been fixing our server today.
> > 
> > [  263.685691] mm/page-writeback.c 655 wb_kupdate: pdflush(248) 24235 global 4593 0 0 wc _M tw 1024 sk 0
> > [  263.789648] requeue_io 301: inode 4031199 size 562 at 08:07(sda7)
> > [  263.789656] requeue_io 301: inode 4031231 size 329 at 08:07(sda7)
> > [  263.789660] requeue_io 301: inode 4031255 size 177 at 08:07(sda7)
> > [  263.789664] requeue_io 301: inode 4031268 size 94 at 08:07(sda7)
> > [  263.789667] requeue_io 301: inode 4031329 size 88 at 08:07(sda7)
> > [  263.789671] requeue_io 301: inode 4031351 size 74 at 08:07(sda7)
> > [  263.789674] requeue_io 301: inode 4031408 size 175 at 08:07(sda7)
> > [  263.789678] requeue_io 301: inode 4031413 size 129 at 08:07(sda7)
> > [  263.789681] requeue_io 301: inode 4031415 size 391 at 08:07(sda7)
> > [  263.789690] mm/page-writeback.c 655 wb_kupdate: pdflush(248) 24235 global 4593 0 0 wc _M tw 1024 sk 0
> > [  263.890184] requeue_io 301: inode 4031199 size 562 at 08:07(sda7)
> > [  263.890191] requeue_io 301: inode 4031231 size 329 at 08:07(sda7)
> > [  263.890195] requeue_io 301: inode 4031255 size 177 at 08:07(sda7)
> > [  263.890198] requeue_io 301: inode 4031268 size 94 at 08:07(sda7)
> > [  263.890202] requeue_io 301: inode 4031329 size 88 at 08:07(sda7)
> > [  263.890205] requeue_io 301: inode 4031351 size 74 at 08:07(sda7)
> > [  263.890208] requeue_io 301: inode 4031408 size 175 at 08:07(sda7)
> > [  263.890212] requeue_io 301: inode 4031413 size 129 at 08:07(sda7)
> > [  263.890215] requeue_io 301: inode 4031415 size 391 at 08:07(sda7)
> > [  263.890223] mm/page-writeback.c 655 wb_kupdate: pdflush(248) 24235 global 4593 0 0 wc _M tw 1024 sk 0
> > 
> > It's about sda7, not NFSv4.
> > 
> > Is it a Reiserfs? We have a fresh fix for it: http://lkml.org/lkml/2007/10/23/93
> 
> Yes, it is a Reiserfs.  Incidentally it is the partition that holds
> the kernel sources and build directory.  The message states that the
> same bug exists in 2.6.23 but I do not see the same behavior in
> 2.6.23.  Anyway, I will apply the patch and see what I get.

Fengguang,

This patch does not fix anything for me.  Even such light use of the
reiserfs filesystem as pulling the linux-2.6 git tree updates caused
one CPU to go to 75% iowait.

florin

-- 
Bruce Schneier expects the Spanish Inquisition.
      http://geekz.co.uk/schneierfacts/fact/163

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: pdflush stuck in D state with v2.6.24-rc1-192-gef49c32

[Fengguang Wu]

On Wed, Oct 31, 2007 at 12:53:18PM -0500, Florin Iucha wrote:
> On Wed, Oct 31, 2007 at 07:16:06AM -0500, Florin Iucha wrote:
> > On Wed, Oct 31, 2007 at 02:53:25PM +0800, Fengguang Wu wrote:
> > > On Tue, Oct 30, 2007 at 10:52:45PM -0500, Florin Iucha wrote:
> > > > On Tue, Oct 30, 2007 at 07:02:42PM -0500, Florin Iucha wrote:
> > > > > I have added the patches and started a linux kernel compilation, and
> > > > > something really interesting happens.  I run the build with the
> > > > > equivalent of "make -j3" and in a separate console I am watching the
> > > > > build with 'top'.  The build consumes 98% of both CPUs.  If I stop the
> > > > > output in the build console with "Ctrl-S", one core goes to idle,
> > > > > while the other is in 50% waiting, then goes to 75% waiting.  When I
> > > > > resume the build with "Ctrl-Q", the build starts to use both CPUs at
> > > > > 98-99%.  The NFS4 use was minimal, as I did not login with Gnome, but
> > > > > just logged on the console.  Also, the CPU that is in 75% waiting
> > > > > state changes occasionally.  'Top' shows pdflush in D state, using
> > > > > 5-6% of CPU.
> > > > 
> > > > I forgot the traces:
> > > > 
> > > >    http://iucha.net/2.6.24-rc1/fw.1.gz
> > > >    http://iucha.net/2.6.24-rc1/fw.2.gz
> > > >    http://iucha.net/2.6.24-rc1/fw.3.gz
> > > 
> > > Sorry for the delay - I've been fixing our server today.
> > > 
> > > [  263.685691] mm/page-writeback.c 655 wb_kupdate: pdflush(248) 24235 global 4593 0 0 wc _M tw 1024 sk 0
> > > [  263.789648] requeue_io 301: inode 4031199 size 562 at 08:07(sda7)
> > > [  263.789656] requeue_io 301: inode 4031231 size 329 at 08:07(sda7)
> > > [  263.789660] requeue_io 301: inode 4031255 size 177 at 08:07(sda7)
> > > [  263.789664] requeue_io 301: inode 4031268 size 94 at 08:07(sda7)
> > > [  263.789667] requeue_io 301: inode 4031329 size 88 at 08:07(sda7)
> > > [  263.789671] requeue_io 301: inode 4031351 size 74 at 08:07(sda7)
> > > [  263.789674] requeue_io 301: inode 4031408 size 175 at 08:07(sda7)
> > > [  263.789678] requeue_io 301: inode 4031413 size 129 at 08:07(sda7)
> > > [  263.789681] requeue_io 301: inode 4031415 size 391 at 08:07(sda7)
> > > [  263.789690] mm/page-writeback.c 655 wb_kupdate: pdflush(248) 24235 global 4593 0 0 wc _M tw 1024 sk 0
> > > [  263.890184] requeue_io 301: inode 4031199 size 562 at 08:07(sda7)
> > > [  263.890191] requeue_io 301: inode 4031231 size 329 at 08:07(sda7)
> > > [  263.890195] requeue_io 301: inode 4031255 size 177 at 08:07(sda7)
> > > [  263.890198] requeue_io 301: inode 4031268 size 94 at 08:07(sda7)
> > > [  263.890202] requeue_io 301: inode 4031329 size 88 at 08:07(sda7)
> > > [  263.890205] requeue_io 301: inode 4031351 size 74 at 08:07(sda7)
> > > [  263.890208] requeue_io 301: inode 4031408 size 175 at 08:07(sda7)
> > > [  263.890212] requeue_io 301: inode 4031413 size 129 at 08:07(sda7)
> > > [  263.890215] requeue_io 301: inode 4031415 size 391 at 08:07(sda7)
> > > [  263.890223] mm/page-writeback.c 655 wb_kupdate: pdflush(248) 24235 global 4593 0 0 wc _M tw 1024 sk 0
> > > 
> > > It's about sda7, not NFSv4.
> > > 
> > > Is it a Reiserfs? We have a fresh fix for it: http://lkml.org/lkml/2007/10/23/93
> > 
> > Yes, it is a Reiserfs.  Incidentally it is the partition that holds
> > the kernel sources and build directory.  The message states that the
> > same bug exists in 2.6.23 but I do not see the same behavior in
> > 2.6.23.  Anyway, I will apply the patch and see what I get.
> 
> Fengguang,
> 
> This patch does not fix anything for me.  Even such light use of the
> reiserfs filesystem as pulling the linux-2.6 git tree updates caused
> one CPU to go to 75% iowait.
 
Thank you, Florin. Could you provide more details about sda7, such as
the mount option and output of `reiserfstune /dev/sda7`? I'll try to
reproduce it before asking for your help.

Fengguang

Re: pdflush stuck in D state with v2.6.24-rc1-192-gef49c32

[Florin Iucha]

[-- Attachment #1: Type: text/plain, Size: 2668 bytes --]

On Thu, Nov 01, 2007 at 03:15:32PM +0800, Fengguang Wu wrote:
> On Wed, Oct 31, 2007 at 12:53:18PM -0500, Florin Iucha wrote:
> > This patch does not fix anything for me.  Even such light use of the
> > reiserfs filesystem as pulling the linux-2.6 git tree updates caused
> > one CPU to go to 75% iowait.
>  
> Thank you, Florin. Could you provide more details about sda7, such as
> the mount option and output of `reiserfstune /dev/sda7`? I'll try to
> reproduce it before asking for your help.

Fengguang,

root@zeus:~# mount | grep sda7
/dev/sda7 on /scratch type reiserfs (rw,noatime)
root@zeus:~# df -h /scratch/
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda7              38G   32G  5.7G  85% /scratch
root@zeus:~# umount /dev/sda7
root@zeus:~# reiserfstune /dev/sda7
reiserfstune: Journal device has not been specified. Assuming journal is on the main device (/dev/sda7).

Current parameters:

Filesystem state: consistent

/scratch: Reiserfs super block in block 16 on 0x807 of format 3.6 with standard journal
Count of blocks on the device: 9765504
Number of bitmaps: 299
Blocksize: 4096
Free blocks (count of blocks - used [journal, bitmaps, data, reserved] blocks): 1471399
Root block: 2359332
Filesystem is clean
Tree height: 5
Hash function used to sort names: "r5"
Objectid map size 916, max 972
Journal parameters:
        Device [0x0]
        Magic [0x31037e64]
        Size 8193 blocks (including 1 for journal header) (first block 18)
        Max transaction length 1024 blocks
        Max batch size 900 blocks
        Max commit age 30
Blocks reserved by journal: 0
Fs state field: 0x0:
sb_version: 2
inode generation number: 5856766
UUID: a0191e80-be6e-47f6-8fd0-047e2d763a4a
LABEL: /scratch
Set flags in SB:
        ATTRIBUTES CLEAN

And for bonus points:
###########
reiserfsck --check started at Thu Nov  1 07:09:56 2007
###########
Replaying journal..
Reiserfs journal '/dev/sda7' in blocks [18..8211]: 0 transactions replayed
Checking internal tree..finished                               
Comparing bitmaps..finished
Checking Semantic tree:
finished                                                                       
No corruptions found
There are on the filesystem:
        Leaves 247231
        Internal nodes 1570
        Directories 11330
        Other files 722878
        Data block pointers 8040675 (3880 of them are zero)
        Safe links 0
###########
reiserfsck finished at Thu Nov  1 07:18:43 2007
###########

Cheers,
florin

-- 
Bruce Schneier expects the Spanish Inquisition.
      http://geekz.co.uk/schneierfacts/fact/163

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: pdflush stuck in D state with v2.6.24-rc1-192-gef49c32

[Fengguang Wu]

On Thu, Nov 01, 2007 at 07:25:58AM -0500, Florin Iucha wrote:
> On Thu, Nov 01, 2007 at 03:15:32PM +0800, Fengguang Wu wrote:
> > On Wed, Oct 31, 2007 at 12:53:18PM -0500, Florin Iucha wrote:
> > > This patch does not fix anything for me.  Even such light use of the
> > > reiserfs filesystem as pulling the linux-2.6 git tree updates caused
> > > one CPU to go to 75% iowait.
> >  
> > Thank you, Florin. Could you provide more details about sda7, such as
> > the mount option and output of `reiserfstune /dev/sda7`? I'll try to
> > reproduce it before asking for your help.
> 
> Fengguang,
> 
> root@zeus:~# mount | grep sda7
> /dev/sda7 on /scratch type reiserfs (rw,noatime)
> root@zeus:~# df -h /scratch/
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/sda7              38G   32G  5.7G  85% /scratch
[...]

Thank you. It seems the only difference with mine reiserfs is about
the 'noatime' - which I tried and saw no difference.

Or will the system or fs size/age make any difference? If you happen
to have a spare/swap partition, could you make a new reiserfs and
mount it and copy several less-than-4KB files into it and wait for 30s
and see what happen to pdflush?

btw, what's the exact kernel version you are running?

Thank you,
Fengguang

Re: pdflush stuck in D state with v2.6.24-rc1-192-gef49c32

[Florin Iucha]

[-- Attachment #1: Type: text/plain, Size: 687 bytes --]

On Thu, Nov 01, 2007 at 09:03:33PM +0800, Fengguang Wu wrote:
> Or will the system or fs size/age make any difference? If you happen
> to have a spare/swap partition, could you make a new reiserfs and
> mount it and copy several less-than-4KB files into it and wait for 30s
> and see what happen to pdflush?

I will try that with a USB disk - I hope that won't make a difference.

> btw, what's the exact kernel version you are running?

I noticed it with the kernel in the $SUBJECT, as reported by 'git
describe'.  I have pulled in new changesets since then.

florin

-- 
Bruce Schneier expects the Spanish Inquisition.
      http://geekz.co.uk/schneierfacts/fact/163

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: pdflush stuck in D state with v2.6.24-rc1-192-gef49c32

[Fengguang Wu]

On Thu, Nov 01, 2007 at 09:14:14AM -0500, Florin Iucha wrote:
> On Thu, Nov 01, 2007 at 09:03:33PM +0800, Fengguang Wu wrote:
> > Or will the system or fs size/age make any difference? If you happen
> > to have a spare/swap partition, could you make a new reiserfs and
> > mount it and copy several less-than-4KB files into it and wait for 30s
> > and see what happen to pdflush?
> 
> I will try that with a USB disk - I hope that won't make a difference.

Thank you. I guess a reiserfs on loop file would also be OK.

> > btw, what's the exact kernel version you are running?
> 
> I noticed it with the kernel in the $SUBJECT, as reported by 'git
> describe'.  I have pulled in new changesets since then.

And with the following patch applied?

---
 fs/reiserfs/stree.c |    3 ---
 1 file changed, 3 deletions(-)

--- linux-2.6.24-git17.orig/fs/reiserfs/stree.c
+++ linux-2.6.24-git17/fs/reiserfs/stree.c
@@ -1458,9 +1458,6 @@ static void unmap_buffers(struct page *p
 				}
 				bh = next;
 			} while (bh != head);
-			if (PAGE_SIZE == bh->b_size) {
-				cancel_dirty_page(page, PAGE_CACHE_SIZE);
-			}
 		}
 	}
 }

Re: pdflush stuck in D state with v2.6.24-rc1-192-gef49c32

[Florin Iucha]

[-- Attachment #1.1: Type: text/plain, Size: 1293 bytes --]

On Fri, Nov 02, 2007 at 09:33:21AM +0800, Fengguang Wu wrote:
> > I will try that with a USB disk - I hope that won't make a difference.
> 
> Thank you. I guess a reiserfs on loop file would also be OK.
> 
> > > btw, what's the exact kernel version you are running?
> > 
> > I noticed it with the kernel in the $SUBJECT, as reported by 'git
> > describe'.  I have pulled in new changesets since then.
> 
> And with the following patch applied?
> 
> ---
>  fs/reiserfs/stree.c |    3 ---
>  1 file changed, 3 deletions(-)
> 
> --- linux-2.6.24-git17.orig/fs/reiserfs/stree.c
> +++ linux-2.6.24-git17/fs/reiserfs/stree.c
> @@ -1458,9 +1458,6 @@ static void unmap_buffers(struct page *p
>  				}
>  				bh = next;
>  			} while (bh != head);
> -			if (PAGE_SIZE == bh->b_size) {
> -				cancel_dirty_page(page, PAGE_CACHE_SIZE);
> -			}
>  		}
>  	}
>  }

... and with the above patch applied.

Copying 300 MB from root (ext3) to the new file system did not trigger
the pdflush condition.  But then I did a
   cd $MOUNTPOINT && find . -exec md5sum {} \;
and that brought one cpu to 75% iowait.

I have attached my .config, if it helps.

Cheers,
florin

-- 
Bruce Schneier expects the Spanish Inquisition.
      http://geekz.co.uk/schneierfacts/fact/163

[-- Attachment #1.2: config-2.6.24-rc1-7-fw2 --]
[-- Type: text/plain, Size: 45147 bytes --]

#
# Automatically generated make config: don't edit
# Linux kernel version: 2.6.24-rc1-7-fw2
# Wed Oct 31 07:27:14 2007
#
CONFIG_X86_64=y
CONFIG_64BIT=y
CONFIG_X86=y
CONFIG_GENERIC_TIME=y
CONFIG_GENERIC_TIME_VSYSCALL=y
CONFIG_GENERIC_CMOS_UPDATE=y
CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_ZONE_DMA32=y
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_SEMAPHORE_SLEEPERS=y
CONFIG_MMU=y
CONFIG_ZONE_DMA=y
CONFIG_RWSEM_GENERIC_SPINLOCK=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_X86_CMPXCHG=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
CONFIG_ARCH_POPULATES_NODE_MAP=y
CONFIG_DMI=y
CONFIG_AUDIT_ARCH=y
CONFIG_GENERIC_BUG=y
# CONFIG_ARCH_HAS_ILOG2_U32 is not set
# CONFIG_ARCH_HAS_ILOG2_U64 is not set
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"

#
# General setup
#
CONFIG_EXPERIMENTAL=y
CONFIG_LOCK_KERNEL=y
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_LOCALVERSION=""
# CONFIG_LOCALVERSION_AUTO is not set
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
CONFIG_POSIX_MQUEUE=y
CONFIG_BSD_PROCESS_ACCT=y
CONFIG_BSD_PROCESS_ACCT_V3=y
# CONFIG_TASKSTATS is not set
# CONFIG_USER_NS is not set
# CONFIG_AUDIT is not set
CONFIG_IKCONFIG=y
# CONFIG_IKCONFIG_PROC is not set
CONFIG_LOG_BUF_SHIFT=19
CONFIG_CGROUPS=y
# CONFIG_CGROUP_DEBUG is not set
CONFIG_CGROUP_NS=y
CONFIG_CGROUP_CPUACCT=y
# CONFIG_CPUSETS is not set
CONFIG_FAIR_GROUP_SCHED=y
CONFIG_FAIR_USER_SCHED=y
# CONFIG_FAIR_CGROUP_SCHED is not set
# CONFIG_SYSFS_DEPRECATED is not set
CONFIG_RELAY=y
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
CONFIG_CC_OPTIMIZE_FOR_SIZE=y
CONFIG_SYSCTL=y
# CONFIG_EMBEDDED is not set
CONFIG_UID16=y
CONFIG_SYSCTL_SYSCALL=y
CONFIG_KALLSYMS=y
# CONFIG_KALLSYMS_ALL is not set
# CONFIG_KALLSYMS_EXTRA_PASS is not set
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_ANON_INODES=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_SLUB_DEBUG=y
# CONFIG_SLAB is not set
CONFIG_SLUB=y
# CONFIG_SLOB is not set
CONFIG_RT_MUTEXES=y
# CONFIG_TINY_SHMEM is not set
CONFIG_BASE_SMALL=0
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
CONFIG_MODULE_FORCE_UNLOAD=y
CONFIG_MODVERSIONS=y
# CONFIG_MODULE_SRCVERSION_ALL is not set
CONFIG_KMOD=y
CONFIG_STOP_MACHINE=y
CONFIG_BLOCK=y
# CONFIG_BLK_DEV_IO_TRACE is not set
CONFIG_BLK_DEV_BSG=y
CONFIG_BLOCK_COMPAT=y

#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
# CONFIG_DEFAULT_AS is not set
# CONFIG_DEFAULT_DEADLINE is not set
CONFIG_DEFAULT_CFQ=y
# CONFIG_DEFAULT_NOOP is not set
CONFIG_DEFAULT_IOSCHED="cfq"

#
# Processor type and features
#
CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y
CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
CONFIG_X86_PC=y
# CONFIG_X86_VSMP is not set
CONFIG_MK8=y
# CONFIG_MPSC is not set
# CONFIG_MCORE2 is not set
# CONFIG_GENERIC_CPU is not set
CONFIG_X86_L1_CACHE_BYTES=64
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_X86_INTERNODE_CACHE_BYTES=64
CONFIG_X86_TSC=y
CONFIG_X86_GOOD_APIC=y
CONFIG_MICROCODE=m
CONFIG_MICROCODE_OLD_INTERFACE=y
CONFIG_X86_MSR=y
CONFIG_X86_CPUID=m
CONFIG_X86_IO_APIC=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_MTRR=y
CONFIG_SMP=y
# CONFIG_SCHED_SMT is not set
CONFIG_SCHED_MC=y
# CONFIG_PREEMPT_NONE is not set
CONFIG_PREEMPT_VOLUNTARY=y
# CONFIG_PREEMPT is not set
CONFIG_PREEMPT_BKL=y
# CONFIG_NUMA is not set
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_FLATMEM_ENABLE=y
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_FLATMEM_MANUAL=y
# CONFIG_DISCONTIGMEM_MANUAL is not set
# CONFIG_SPARSEMEM_MANUAL is not set
CONFIG_FLATMEM=y
CONFIG_FLAT_NODE_MEM_MAP=y
# CONFIG_SPARSEMEM_STATIC is not set
CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
CONFIG_SPLIT_PTLOCK_CPUS=4
CONFIG_RESOURCES_64BIT=y
CONFIG_ZONE_DMA_FLAG=1
CONFIG_BOUNCE=y
CONFIG_VIRT_TO_BUS=y
CONFIG_NR_CPUS=2
CONFIG_PHYSICAL_ALIGN=0x200000
CONFIG_HOTPLUG_CPU=y
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
CONFIG_HPET_TIMER=y
CONFIG_GART_IOMMU=y
# CONFIG_CALGARY_IOMMU is not set
CONFIG_SWIOTLB=y
CONFIG_X86_MCE=y
# CONFIG_X86_MCE_INTEL is not set
CONFIG_X86_MCE_AMD=y
# CONFIG_KEXEC is not set
# CONFIG_CRASH_DUMP is not set
# CONFIG_RELOCATABLE is not set
CONFIG_PHYSICAL_START=0x200000
CONFIG_SECCOMP=y
# CONFIG_CC_STACKPROTECTOR is not set
# CONFIG_HZ_100 is not set
CONFIG_HZ_250=y
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=250
CONFIG_K8_NB=y
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_ISA_DMA_API=y
CONFIG_GENERIC_PENDING_IRQ=y

#
# Power management options
#
CONFIG_PM=y
# CONFIG_PM_LEGACY is not set
# CONFIG_PM_DEBUG is not set
CONFIG_SUSPEND_SMP_POSSIBLE=y
# CONFIG_SUSPEND is not set
CONFIG_HIBERNATION_SMP_POSSIBLE=y
# CONFIG_HIBERNATION is not set
CONFIG_ACPI=y
# CONFIG_ACPI_PROCFS is not set
# CONFIG_ACPI_PROC_EVENT is not set
# CONFIG_ACPI_AC is not set
# CONFIG_ACPI_BATTERY is not set
CONFIG_ACPI_BUTTON=y
CONFIG_ACPI_FAN=y
# CONFIG_ACPI_DOCK is not set
CONFIG_ACPI_PROCESSOR=y
CONFIG_ACPI_HOTPLUG_CPU=y
CONFIG_ACPI_THERMAL=y
# CONFIG_ACPI_ASUS is not set
# CONFIG_ACPI_TOSHIBA is not set
CONFIG_ACPI_BLACKLIST_YEAR=0
# CONFIG_ACPI_DEBUG is not set
CONFIG_ACPI_EC=y
CONFIG_ACPI_POWER=y
CONFIG_ACPI_SYSTEM=y
CONFIG_X86_PM_TIMER=y
CONFIG_ACPI_CONTAINER=y
# CONFIG_ACPI_SBS is not set

#
# CPU Frequency scaling
#
CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_TABLE=y
# CONFIG_CPU_FREQ_DEBUG is not set
CONFIG_CPU_FREQ_STAT=y
# CONFIG_CPU_FREQ_STAT_DETAILS is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE is not set
CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_CONSERVATIVE is not set
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_POWERSAVE=y
CONFIG_CPU_FREQ_GOV_USERSPACE=y
CONFIG_CPU_FREQ_GOV_ONDEMAND=y
CONFIG_CPU_FREQ_GOV_CONSERVATIVE=y

#
# CPUFreq processor drivers
#
CONFIG_X86_POWERNOW_K8=y
CONFIG_X86_POWERNOW_K8_ACPI=y
# CONFIG_X86_SPEEDSTEP_CENTRINO is not set
CONFIG_X86_ACPI_CPUFREQ=m

#
# shared options
#
# CONFIG_X86_ACPI_CPUFREQ_PROC_INTF is not set
# CONFIG_X86_SPEEDSTEP_LIB is not set
CONFIG_CPU_IDLE=y
CONFIG_CPU_IDLE_GOV_LADDER=y
CONFIG_CPU_IDLE_GOV_MENU=y

#
# Bus options (PCI etc.)
#
CONFIG_PCI=y
CONFIG_PCI_DIRECT=y
# CONFIG_PCI_MMCONFIG is not set
CONFIG_PCI_DOMAINS=y
# CONFIG_DMAR is not set
CONFIG_PCIEPORTBUS=y
CONFIG_PCIEAER=y
CONFIG_ARCH_SUPPORTS_MSI=y
CONFIG_PCI_MSI=y
# CONFIG_PCI_DEBUG is not set
CONFIG_HT_IRQ=y
# CONFIG_PCCARD is not set
# CONFIG_HOTPLUG_PCI is not set

#
# Executable file formats / Emulations
#
CONFIG_BINFMT_ELF=y
# CONFIG_BINFMT_MISC is not set
CONFIG_IA32_EMULATION=y
# CONFIG_IA32_AOUT is not set
CONFIG_COMPAT=y
CONFIG_COMPAT_FOR_U64_ALIGNMENT=y
CONFIG_SYSVIPC_COMPAT=y

#
# Networking
#
CONFIG_NET=y

#
# Networking options
#
CONFIG_PACKET=y
CONFIG_PACKET_MMAP=y
CONFIG_UNIX=y
CONFIG_XFRM=y
# CONFIG_XFRM_USER is not set
# CONFIG_XFRM_SUB_POLICY is not set
# CONFIG_XFRM_MIGRATE is not set
# CONFIG_NET_KEY is not set
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
# CONFIG_IP_ADVANCED_ROUTER is not set
CONFIG_IP_FIB_HASH=y
# CONFIG_IP_PNP is not set
CONFIG_NET_IPIP=m
CONFIG_NET_IPGRE=m
# CONFIG_NET_IPGRE_BROADCAST is not set
# CONFIG_IP_MROUTE is not set
# CONFIG_ARPD is not set
# CONFIG_SYN_COOKIES is not set
CONFIG_INET_AH=m
CONFIG_INET_ESP=m
CONFIG_INET_IPCOMP=m
CONFIG_INET_XFRM_TUNNEL=m
CONFIG_INET_TUNNEL=m
# CONFIG_INET_XFRM_MODE_TRANSPORT is not set
# CONFIG_INET_XFRM_MODE_TUNNEL is not set
CONFIG_INET_XFRM_MODE_BEET=y
# CONFIG_INET_LRO is not set
CONFIG_INET_DIAG=y
CONFIG_INET_TCP_DIAG=y
# CONFIG_TCP_CONG_ADVANCED is not set
CONFIG_TCP_CONG_CUBIC=y
CONFIG_DEFAULT_TCP_CONG="cubic"
# CONFIG_TCP_MD5SIG is not set
# CONFIG_IPV6 is not set
# CONFIG_INET6_XFRM_TUNNEL is not set
# CONFIG_INET6_TUNNEL is not set
# CONFIG_NETWORK_SECMARK is not set
# CONFIG_NETFILTER is not set
CONFIG_IP_DCCP=m
CONFIG_INET_DCCP_DIAG=m
CONFIG_IP_DCCP_ACKVEC=y

#
# DCCP CCIDs Configuration (EXPERIMENTAL)
#
CONFIG_IP_DCCP_CCID2=m
# CONFIG_IP_DCCP_CCID2_DEBUG is not set
# CONFIG_IP_DCCP_CCID3 is not set

#
# DCCP Kernel Hacking
#
# CONFIG_IP_DCCP_DEBUG is not set
CONFIG_NET_DCCPPROBE=m
CONFIG_IP_SCTP=m
# CONFIG_SCTP_DBG_MSG is not set
# CONFIG_SCTP_DBG_OBJCNT is not set
# CONFIG_SCTP_HMAC_NONE is not set
# CONFIG_SCTP_HMAC_SHA1 is not set
CONFIG_SCTP_HMAC_MD5=y
# CONFIG_TIPC is not set
# CONFIG_ATM is not set
# CONFIG_BRIDGE is not set
# CONFIG_VLAN_8021Q is not set
# CONFIG_DECNET is not set
# CONFIG_LLC2 is not set
# CONFIG_IPX is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_ECONET is not set
# CONFIG_WAN_ROUTER is not set
# CONFIG_NET_SCHED is not set

#
# Network testing
#
# CONFIG_NET_PKTGEN is not set
CONFIG_NET_TCPPROBE=m
# CONFIG_HAMRADIO is not set
# CONFIG_IRDA is not set
CONFIG_BT=y
# CONFIG_BT_L2CAP is not set
# CONFIG_BT_SCO is not set

#
# Bluetooth device drivers
#
# CONFIG_BT_HCIUSB is not set
# CONFIG_BT_HCIBTUSB is not set
# CONFIG_BT_HCIUART is not set
# CONFIG_BT_HCIBCM203X is not set
# CONFIG_BT_HCIBPA10X is not set
# CONFIG_BT_HCIBFUSB is not set
# CONFIG_BT_HCIVHCI is not set
# CONFIG_AF_RXRPC is not set

#
# Wireless
#
# CONFIG_CFG80211 is not set
# CONFIG_WIRELESS_EXT is not set
# CONFIG_MAC80211 is not set
# CONFIG_IEEE80211 is not set
# CONFIG_RFKILL is not set
# CONFIG_NET_9P is not set

#
# Device Drivers
#

#
# Generic Driver Options
#
CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
CONFIG_STANDALONE=y
# CONFIG_PREVENT_FIRMWARE_BUILD is not set
CONFIG_FW_LOADER=y
# CONFIG_DEBUG_DRIVER is not set
# CONFIG_DEBUG_DEVRES is not set
# CONFIG_SYS_HYPERVISOR is not set
CONFIG_CONNECTOR=y
# CONFIG_PROC_EVENTS is not set
# CONFIG_MTD is not set
CONFIG_PARPORT=y
CONFIG_PARPORT_PC=y
# CONFIG_PARPORT_SERIAL is not set
# CONFIG_PARPORT_PC_FIFO is not set
# CONFIG_PARPORT_PC_SUPERIO is not set
# CONFIG_PARPORT_GSC is not set
# CONFIG_PARPORT_AX88796 is not set
# CONFIG_PARPORT_1284 is not set
CONFIG_PNP=y
# CONFIG_PNP_DEBUG is not set

#
# Protocols
#
CONFIG_PNPACPI=y
CONFIG_BLK_DEV=y
CONFIG_BLK_DEV_FD=y
# CONFIG_PARIDE is not set
# CONFIG_BLK_CPQ_DA is not set
# CONFIG_BLK_CPQ_CISS_DA is not set
# CONFIG_BLK_DEV_DAC960 is not set
# CONFIG_BLK_DEV_UMEM is not set
# CONFIG_BLK_DEV_COW_COMMON is not set
CONFIG_BLK_DEV_LOOP=m
# CONFIG_BLK_DEV_CRYPTOLOOP is not set
CONFIG_BLK_DEV_NBD=m
# CONFIG_BLK_DEV_SX8 is not set
# CONFIG_BLK_DEV_UB is not set
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_BLK_DEV_RAM_SIZE=4096
CONFIG_BLK_DEV_RAM_BLOCKSIZE=1024
CONFIG_CDROM_PKTCDVD=y
CONFIG_CDROM_PKTCDVD_BUFFERS=8
# CONFIG_CDROM_PKTCDVD_WCACHE is not set
# CONFIG_ATA_OVER_ETH is not set
# CONFIG_MISC_DEVICES is not set
CONFIG_IDE=y
CONFIG_BLK_DEV_IDE=y

#
# Please see Documentation/ide.txt for help/info on IDE drives
#
# CONFIG_BLK_DEV_IDE_SATA is not set
# CONFIG_BLK_DEV_HD_IDE is not set
CONFIG_BLK_DEV_IDEDISK=y
# CONFIG_IDEDISK_MULTI_MODE is not set
CONFIG_BLK_DEV_IDECD=y
# CONFIG_BLK_DEV_IDETAPE is not set
# CONFIG_BLK_DEV_IDEFLOPPY is not set
# CONFIG_BLK_DEV_IDESCSI is not set
CONFIG_BLK_DEV_IDEACPI=y
# CONFIG_IDE_TASK_IOCTL is not set
# CONFIG_IDE_PROC_FS is not set

#
# IDE chipset support/bugfixes
#
CONFIG_IDE_GENERIC=y
# CONFIG_BLK_DEV_PLATFORM is not set
# CONFIG_BLK_DEV_CMD640 is not set
# CONFIG_BLK_DEV_IDEPNP is not set

#
# PCI IDE chipsets support
#
CONFIG_BLK_DEV_IDEPCI=y
# CONFIG_IDEPCI_SHARE_IRQ is not set
CONFIG_IDEPCI_PCIBUS_ORDER=y
# CONFIG_BLK_DEV_OFFBOARD is not set
# CONFIG_BLK_DEV_GENERIC is not set
# CONFIG_BLK_DEV_OPTI621 is not set
# CONFIG_BLK_DEV_RZ1000 is not set
CONFIG_BLK_DEV_IDEDMA_PCI=y
# CONFIG_BLK_DEV_AEC62XX is not set
# CONFIG_BLK_DEV_ALI15X3 is not set
CONFIG_BLK_DEV_AMD74XX=y
# CONFIG_BLK_DEV_ATIIXP is not set
# CONFIG_BLK_DEV_CMD64X is not set
# CONFIG_BLK_DEV_TRIFLEX is not set
# CONFIG_BLK_DEV_CY82C693 is not set
# CONFIG_BLK_DEV_CS5520 is not set
# CONFIG_BLK_DEV_CS5530 is not set
# CONFIG_BLK_DEV_HPT34X is not set
# CONFIG_BLK_DEV_HPT366 is not set
# CONFIG_BLK_DEV_JMICRON is not set
# CONFIG_BLK_DEV_SC1200 is not set
# CONFIG_BLK_DEV_PIIX is not set
# CONFIG_BLK_DEV_IT8213 is not set
# CONFIG_BLK_DEV_IT821X is not set
# CONFIG_BLK_DEV_NS87415 is not set
# CONFIG_BLK_DEV_PDC202XX_OLD is not set
# CONFIG_BLK_DEV_PDC202XX_NEW is not set
# CONFIG_BLK_DEV_SVWKS is not set
# CONFIG_BLK_DEV_SIIMAGE is not set
# CONFIG_BLK_DEV_SIS5513 is not set
# CONFIG_BLK_DEV_SLC90E66 is not set
# CONFIG_BLK_DEV_TRM290 is not set
# CONFIG_BLK_DEV_VIA82CXXX is not set
# CONFIG_BLK_DEV_TC86C001 is not set
# CONFIG_IDE_ARM is not set
CONFIG_BLK_DEV_IDEDMA=y
CONFIG_IDE_ARCH_OBSOLETE_INIT=y
# CONFIG_BLK_DEV_HD is not set

#
# SCSI device support
#
# CONFIG_RAID_ATTRS is not set
CONFIG_SCSI=y
CONFIG_SCSI_DMA=y
# CONFIG_SCSI_TGT is not set
# CONFIG_SCSI_NETLINK is not set
# CONFIG_SCSI_PROC_FS is not set

#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=y
# CONFIG_CHR_DEV_ST is not set
# CONFIG_CHR_DEV_OSST is not set
CONFIG_BLK_DEV_SR=y
# CONFIG_BLK_DEV_SR_VENDOR is not set
CONFIG_CHR_DEV_SG=y
# CONFIG_CHR_DEV_SCH is not set

#
# Some SCSI devices (e.g. CD jukebox) support multiple LUNs
#
# CONFIG_SCSI_MULTI_LUN is not set
# CONFIG_SCSI_CONSTANTS is not set
# CONFIG_SCSI_LOGGING is not set
# CONFIG_SCSI_SCAN_ASYNC is not set
CONFIG_SCSI_WAIT_SCAN=m

#
# SCSI Transports
#
# CONFIG_SCSI_SPI_ATTRS is not set
# CONFIG_SCSI_FC_ATTRS is not set
CONFIG_SCSI_ISCSI_ATTRS=m
# CONFIG_SCSI_SAS_ATTRS is not set
# CONFIG_SCSI_SAS_LIBSAS is not set
# CONFIG_SCSI_SRP_ATTRS is not set
CONFIG_SCSI_LOWLEVEL=y
CONFIG_ISCSI_TCP=m
# CONFIG_BLK_DEV_3W_XXXX_RAID is not set
# CONFIG_SCSI_3W_9XXX is not set
# CONFIG_SCSI_ACARD is not set
# CONFIG_SCSI_AACRAID is not set
# CONFIG_SCSI_AIC7XXX is not set
# CONFIG_SCSI_AIC7XXX_OLD is not set
# CONFIG_SCSI_AIC79XX is not set
# CONFIG_SCSI_AIC94XX is not set
# CONFIG_SCSI_ADVANSYS is not set
# CONFIG_SCSI_ARCMSR is not set
# CONFIG_MEGARAID_NEWGEN is not set
# CONFIG_MEGARAID_LEGACY is not set
# CONFIG_MEGARAID_SAS is not set
# CONFIG_SCSI_HPTIOP is not set
# CONFIG_SCSI_BUSLOGIC is not set
# CONFIG_SCSI_DMX3191D is not set
# CONFIG_SCSI_EATA is not set
# CONFIG_SCSI_FUTURE_DOMAIN is not set
# CONFIG_SCSI_GDTH is not set
# CONFIG_SCSI_IPS is not set
# CONFIG_SCSI_INITIO is not set
# CONFIG_SCSI_INIA100 is not set
# CONFIG_SCSI_PPA is not set
# CONFIG_SCSI_IMM is not set
# CONFIG_SCSI_STEX is not set
# CONFIG_SCSI_SYM53C8XX_2 is not set
# CONFIG_SCSI_IPR is not set
# CONFIG_SCSI_QLOGIC_1280 is not set
# CONFIG_SCSI_QLA_FC is not set
# CONFIG_SCSI_QLA_ISCSI is not set
# CONFIG_SCSI_LPFC is not set
# CONFIG_SCSI_DC395x is not set
# CONFIG_SCSI_DC390T is not set
# CONFIG_SCSI_DEBUG is not set
# CONFIG_SCSI_SRP is not set
CONFIG_ATA=y
# CONFIG_ATA_NONSTANDARD is not set
CONFIG_ATA_ACPI=y
# CONFIG_SATA_AHCI is not set
# CONFIG_SATA_SVW is not set
# CONFIG_ATA_PIIX is not set
# CONFIG_SATA_MV is not set
CONFIG_SATA_NV=y
# CONFIG_PDC_ADMA is not set
# CONFIG_SATA_QSTOR is not set
# CONFIG_SATA_PROMISE is not set
# CONFIG_SATA_SX4 is not set
CONFIG_SATA_SIL=y
CONFIG_SATA_SIL24=y
# CONFIG_SATA_SIS is not set
# CONFIG_SATA_ULI is not set
# CONFIG_SATA_VIA is not set
# CONFIG_SATA_VITESSE is not set
# CONFIG_SATA_INIC162X is not set
# CONFIG_PATA_ACPI is not set
# CONFIG_PATA_ALI is not set
CONFIG_PATA_AMD=y
# CONFIG_PATA_ARTOP is not set
# CONFIG_PATA_ATIIXP is not set
# CONFIG_PATA_CMD640_PCI is not set
# CONFIG_PATA_CMD64X is not set
# CONFIG_PATA_CS5520 is not set
# CONFIG_PATA_CS5530 is not set
# CONFIG_PATA_CYPRESS is not set
# CONFIG_PATA_EFAR is not set
# CONFIG_ATA_GENERIC is not set
# CONFIG_PATA_HPT366 is not set
# CONFIG_PATA_HPT37X is not set
# CONFIG_PATA_HPT3X2N is not set
# CONFIG_PATA_HPT3X3 is not set
# CONFIG_PATA_IT821X is not set
# CONFIG_PATA_IT8213 is not set
# CONFIG_PATA_JMICRON is not set
# CONFIG_PATA_TRIFLEX is not set
# CONFIG_PATA_MARVELL is not set
# CONFIG_PATA_MPIIX is not set
# CONFIG_PATA_OLDPIIX is not set
# CONFIG_PATA_NETCELL is not set
# CONFIG_PATA_NS87410 is not set
# CONFIG_PATA_NS87415 is not set
# CONFIG_PATA_OPTI is not set
# CONFIG_PATA_OPTIDMA is not set
# CONFIG_PATA_PDC_OLD is not set
# CONFIG_PATA_RADISYS is not set
# CONFIG_PATA_RZ1000 is not set
# CONFIG_PATA_SC1200 is not set
# CONFIG_PATA_SERVERWORKS is not set
# CONFIG_PATA_PDC2027X is not set
# CONFIG_PATA_SIL680 is not set
# CONFIG_PATA_SIS is not set
# CONFIG_PATA_VIA is not set
# CONFIG_PATA_WINBOND is not set
CONFIG_MD=y
CONFIG_BLK_DEV_MD=m
CONFIG_MD_LINEAR=m
CONFIG_MD_RAID0=m
CONFIG_MD_RAID1=m
# CONFIG_MD_RAID10 is not set
# CONFIG_MD_RAID456 is not set
# CONFIG_MD_MULTIPATH is not set
# CONFIG_MD_FAULTY is not set
CONFIG_BLK_DEV_DM=m
# CONFIG_DM_DEBUG is not set
CONFIG_DM_CRYPT=m
CONFIG_DM_SNAPSHOT=m
CONFIG_DM_MIRROR=m
# CONFIG_DM_ZERO is not set
# CONFIG_DM_MULTIPATH is not set
# CONFIG_DM_DELAY is not set
# CONFIG_DM_UEVENT is not set
# CONFIG_FUSION is not set

#
# IEEE 1394 (FireWire) support
#
# CONFIG_FIREWIRE is not set
CONFIG_IEEE1394=y

#
# Subsystem Options
#
# CONFIG_IEEE1394_VERBOSEDEBUG is not set

#
# Controllers
#
# CONFIG_IEEE1394_PCILYNX is not set
# CONFIG_IEEE1394_OHCI1394 is not set

#
# Protocols
#
CONFIG_IEEE1394_SBP2=m
# CONFIG_IEEE1394_SBP2_PHYS_DMA is not set
# CONFIG_IEEE1394_ETH1394_ROM_ENTRY is not set
# CONFIG_IEEE1394_ETH1394 is not set
CONFIG_IEEE1394_RAWIO=m
# CONFIG_I2O is not set
# CONFIG_MACINTOSH_DRIVERS is not set
CONFIG_NETDEVICES=y
# CONFIG_NETDEVICES_MULTIQUEUE is not set
# CONFIG_DUMMY is not set
CONFIG_BONDING=m
# CONFIG_MACVLAN is not set
CONFIG_EQUALIZER=m
CONFIG_TUN=m
# CONFIG_VETH is not set
# CONFIG_NET_SB1000 is not set
# CONFIG_IP1000 is not set
# CONFIG_ARCNET is not set
# CONFIG_PHYLIB is not set
CONFIG_NET_ETHERNET=y
CONFIG_MII=m
# CONFIG_HAPPYMEAL is not set
# CONFIG_SUNGEM is not set
# CONFIG_CASSINI is not set
# CONFIG_NET_VENDOR_3COM is not set
# CONFIG_NET_TULIP is not set
# CONFIG_HP100 is not set
# CONFIG_IBM_NEW_EMAC_ZMII is not set
# CONFIG_IBM_NEW_EMAC_RGMII is not set
# CONFIG_IBM_NEW_EMAC_TAH is not set
# CONFIG_IBM_NEW_EMAC_EMAC4 is not set
CONFIG_NET_PCI=y
# CONFIG_PCNET32 is not set
# CONFIG_AMD8111_ETH is not set
# CONFIG_ADAPTEC_STARFIRE is not set
# CONFIG_B44 is not set
CONFIG_FORCEDETH=m
# CONFIG_FORCEDETH_NAPI is not set
# CONFIG_EEPRO100 is not set
CONFIG_E100=m
# CONFIG_FEALNX is not set
# CONFIG_NATSEMI is not set
# CONFIG_NE2K_PCI is not set
# CONFIG_8139CP is not set
# CONFIG_8139TOO is not set
# CONFIG_SIS900 is not set
# CONFIG_EPIC100 is not set
# CONFIG_SUNDANCE is not set
# CONFIG_VIA_RHINE is not set
# CONFIG_SC92031 is not set
# CONFIG_NET_POCKET is not set
CONFIG_NETDEV_1000=y
# CONFIG_ACENIC is not set
# CONFIG_DL2K is not set
CONFIG_E1000=m
# CONFIG_E1000_NAPI is not set
CONFIG_E1000_DISABLE_PACKET_SPLIT=y
# CONFIG_E1000E is not set
# CONFIG_NS83820 is not set
# CONFIG_HAMACHI is not set
# CONFIG_YELLOWFIN is not set
# CONFIG_R8169 is not set
# CONFIG_SIS190 is not set
CONFIG_SKGE=m
# CONFIG_SKGE_DEBUG is not set
# CONFIG_SKY2 is not set
# CONFIG_SK98LIN is not set
# CONFIG_VIA_VELOCITY is not set
# CONFIG_TIGON3 is not set
# CONFIG_BNX2 is not set
# CONFIG_QLA3XXX is not set
# CONFIG_ATL1 is not set
# CONFIG_NETDEV_10000 is not set
# CONFIG_TR is not set

#
# Wireless LAN
#
# CONFIG_WLAN_PRE80211 is not set
# CONFIG_WLAN_80211 is not set

#
# USB Network Adapters
#
# CONFIG_USB_CATC is not set
# CONFIG_USB_KAWETH is not set
# CONFIG_USB_PEGASUS is not set
# CONFIG_USB_RTL8150 is not set
# CONFIG_USB_USBNET_MII is not set
# CONFIG_USB_USBNET is not set
# CONFIG_WAN is not set
# CONFIG_FDDI is not set
# CONFIG_HIPPI is not set
# CONFIG_PLIP is not set
# CONFIG_PPP is not set
# CONFIG_SLIP is not set
# CONFIG_NET_FC is not set
# CONFIG_SHAPER is not set
CONFIG_NETCONSOLE=y
# CONFIG_NETCONSOLE_DYNAMIC is not set
CONFIG_NETPOLL=y
# CONFIG_NETPOLL_TRAP is not set
CONFIG_NET_POLL_CONTROLLER=y
# CONFIG_ISDN is not set
# CONFIG_PHONE is not set

#
# Input device support
#
CONFIG_INPUT=y
# CONFIG_INPUT_FF_MEMLESS is not set
# CONFIG_INPUT_POLLDEV is not set

#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
# CONFIG_INPUT_MOUSEDEV_PSAUX is not set
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
# CONFIG_INPUT_JOYDEV is not set
CONFIG_INPUT_EVDEV=m
# CONFIG_INPUT_EVBUG is not set

#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_LKKBD is not set
# CONFIG_KEYBOARD_XTKBD is not set
# CONFIG_KEYBOARD_NEWTON is not set
# CONFIG_KEYBOARD_STOWAWAY is not set
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
CONFIG_MOUSE_PS2_ALPS=y
CONFIG_MOUSE_PS2_LOGIPS2PP=y
CONFIG_MOUSE_PS2_SYNAPTICS=y
CONFIG_MOUSE_PS2_LIFEBOOK=y
CONFIG_MOUSE_PS2_TRACKPOINT=y
# CONFIG_MOUSE_PS2_TOUCHKIT is not set
# CONFIG_MOUSE_SERIAL is not set
# CONFIG_MOUSE_APPLETOUCH is not set
# CONFIG_MOUSE_VSXXXAA is not set
# CONFIG_INPUT_JOYSTICK is not set
# CONFIG_INPUT_TABLET is not set
# CONFIG_INPUT_TOUCHSCREEN is not set
CONFIG_INPUT_MISC=y
CONFIG_INPUT_PCSPKR=y
# CONFIG_INPUT_ATLAS_BTNS is not set
# CONFIG_INPUT_ATI_REMOTE is not set
# CONFIG_INPUT_ATI_REMOTE2 is not set
# CONFIG_INPUT_KEYSPAN_REMOTE is not set
# CONFIG_INPUT_POWERMATE is not set
# CONFIG_INPUT_YEALINK is not set
# CONFIG_INPUT_UINPUT is not set

#
# Hardware I/O ports
#
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_SERPORT=y
# CONFIG_SERIO_CT82C710 is not set
# CONFIG_SERIO_PARKBD is not set
# CONFIG_SERIO_PCIPS2 is not set
CONFIG_SERIO_LIBPS2=y
# CONFIG_SERIO_RAW is not set
# CONFIG_GAMEPORT is not set

#
# Character devices
#
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
# CONFIG_VT_HW_CONSOLE_BINDING is not set
# CONFIG_SERIAL_NONSTANDARD is not set

#
# Serial drivers
#
CONFIG_SERIAL_8250=y
# CONFIG_SERIAL_8250_CONSOLE is not set
CONFIG_FIX_EARLYCON_MEM=y
CONFIG_SERIAL_8250_PCI=y
CONFIG_SERIAL_8250_PNP=y
CONFIG_SERIAL_8250_NR_UARTS=4
CONFIG_SERIAL_8250_RUNTIME_UARTS=4
# CONFIG_SERIAL_8250_EXTENDED is not set

#
# Non-8250 serial port support
#
CONFIG_SERIAL_CORE=y
# CONFIG_SERIAL_JSM is not set
CONFIG_UNIX98_PTYS=y
# CONFIG_LEGACY_PTYS is not set
CONFIG_PRINTER=m
# CONFIG_LP_CONSOLE is not set
# CONFIG_PPDEV is not set
# CONFIG_TIPAR is not set
# CONFIG_IPMI_HANDLER is not set
CONFIG_HW_RANDOM=y
# CONFIG_HW_RANDOM_INTEL is not set
CONFIG_HW_RANDOM_AMD=y
CONFIG_NVRAM=m
CONFIG_RTC=m
CONFIG_GEN_RTC=m
# CONFIG_GEN_RTC_X is not set
# CONFIG_R3964 is not set
# CONFIG_APPLICOM is not set
# CONFIG_MWAVE is not set
# CONFIG_PC8736x_GPIO is not set
CONFIG_RAW_DRIVER=m
CONFIG_MAX_RAW_DEVS=256
CONFIG_HPET=y
# CONFIG_HPET_RTC_IRQ is not set
CONFIG_HPET_MMAP=y
CONFIG_HANGCHECK_TIMER=m
# CONFIG_TCG_TPM is not set
# CONFIG_TELCLOCK is not set
CONFIG_DEVPORT=y
CONFIG_I2C=y
CONFIG_I2C_BOARDINFO=y
CONFIG_I2C_CHARDEV=y

#
# I2C Algorithms
#
CONFIG_I2C_ALGOBIT=y
# CONFIG_I2C_ALGOPCF is not set
# CONFIG_I2C_ALGOPCA is not set

#
# I2C Hardware Bus support
#
# CONFIG_I2C_ALI1535 is not set
# CONFIG_I2C_ALI1563 is not set
# CONFIG_I2C_ALI15X3 is not set
CONFIG_I2C_AMD756=m
# CONFIG_I2C_AMD756_S4882 is not set
# CONFIG_I2C_AMD8111 is not set
# CONFIG_I2C_I801 is not set
# CONFIG_I2C_I810 is not set
# CONFIG_I2C_PIIX4 is not set
CONFIG_I2C_NFORCE2=m
# CONFIG_I2C_OCORES is not set
# CONFIG_I2C_PARPORT is not set
# CONFIG_I2C_PARPORT_LIGHT is not set
# CONFIG_I2C_PROSAVAGE is not set
# CONFIG_I2C_SAVAGE4 is not set
# CONFIG_I2C_SIMTEC is not set
# CONFIG_I2C_SIS5595 is not set
# CONFIG_I2C_SIS630 is not set
# CONFIG_I2C_SIS96X is not set
# CONFIG_I2C_TAOS_EVM is not set
# CONFIG_I2C_STUB is not set
# CONFIG_I2C_TINY_USB is not set
# CONFIG_I2C_VIA is not set
# CONFIG_I2C_VIAPRO is not set
# CONFIG_I2C_VOODOO3 is not set

#
# Miscellaneous I2C Chip support
#
# CONFIG_SENSORS_DS1337 is not set
# CONFIG_SENSORS_DS1374 is not set
# CONFIG_DS1682 is not set
# CONFIG_SENSORS_EEPROM is not set
# CONFIG_SENSORS_PCF8574 is not set
# CONFIG_SENSORS_PCA9539 is not set
# CONFIG_SENSORS_PCF8591 is not set
# CONFIG_SENSORS_MAX6875 is not set
# CONFIG_SENSORS_TSL2550 is not set
# CONFIG_I2C_DEBUG_CORE is not set
# CONFIG_I2C_DEBUG_ALGO is not set
# CONFIG_I2C_DEBUG_BUS is not set
# CONFIG_I2C_DEBUG_CHIP is not set

#
# SPI support
#
# CONFIG_SPI is not set
# CONFIG_SPI_MASTER is not set
# CONFIG_W1 is not set
# CONFIG_POWER_SUPPLY is not set
CONFIG_HWMON=y
CONFIG_HWMON_VID=y
# CONFIG_SENSORS_ABITUGURU is not set
# CONFIG_SENSORS_ABITUGURU3 is not set
# CONFIG_SENSORS_AD7418 is not set
# CONFIG_SENSORS_ADM1021 is not set
# CONFIG_SENSORS_ADM1025 is not set
# CONFIG_SENSORS_ADM1026 is not set
# CONFIG_SENSORS_ADM1029 is not set
# CONFIG_SENSORS_ADM1031 is not set
# CONFIG_SENSORS_ADM9240 is not set
# CONFIG_SENSORS_ADT7470 is not set
CONFIG_SENSORS_K8TEMP=y
# CONFIG_SENSORS_ASB100 is not set
# CONFIG_SENSORS_ATXP1 is not set
# CONFIG_SENSORS_DS1621 is not set
# CONFIG_SENSORS_F71805F is not set
# CONFIG_SENSORS_F71882FG is not set
# CONFIG_SENSORS_F75375S is not set
# CONFIG_SENSORS_FSCHER is not set
# CONFIG_SENSORS_FSCPOS is not set
# CONFIG_SENSORS_FSCHMD is not set
# CONFIG_SENSORS_GL518SM is not set
# CONFIG_SENSORS_GL520SM is not set
# CONFIG_SENSORS_CORETEMP is not set
CONFIG_SENSORS_IT87=y
# CONFIG_SENSORS_LM63 is not set
# CONFIG_SENSORS_LM75 is not set
# CONFIG_SENSORS_LM77 is not set
# CONFIG_SENSORS_LM78 is not set
# CONFIG_SENSORS_LM80 is not set
# CONFIG_SENSORS_LM83 is not set
# CONFIG_SENSORS_LM85 is not set
CONFIG_SENSORS_LM87=m
# CONFIG_SENSORS_LM90 is not set
# CONFIG_SENSORS_LM92 is not set
# CONFIG_SENSORS_LM93 is not set
# CONFIG_SENSORS_MAX1619 is not set
# CONFIG_SENSORS_MAX6650 is not set
# CONFIG_SENSORS_PC87360 is not set
# CONFIG_SENSORS_PC87427 is not set
# CONFIG_SENSORS_SIS5595 is not set
# CONFIG_SENSORS_DME1737 is not set
# CONFIG_SENSORS_SMSC47M1 is not set
# CONFIG_SENSORS_SMSC47M192 is not set
# CONFIG_SENSORS_SMSC47B397 is not set
# CONFIG_SENSORS_THMC50 is not set
# CONFIG_SENSORS_VIA686A is not set
# CONFIG_SENSORS_VT1211 is not set
# CONFIG_SENSORS_VT8231 is not set
# CONFIG_SENSORS_W83781D is not set
# CONFIG_SENSORS_W83791D is not set
# CONFIG_SENSORS_W83792D is not set
# CONFIG_SENSORS_W83793 is not set
# CONFIG_SENSORS_W83L785TS is not set
# CONFIG_SENSORS_W83627HF is not set
# CONFIG_SENSORS_W83627EHF is not set
# CONFIG_SENSORS_HDAPS is not set
# CONFIG_SENSORS_APPLESMC is not set
# CONFIG_HWMON_DEBUG_CHIP is not set
# CONFIG_WATCHDOG is not set

#
# Sonics Silicon Backplane
#
CONFIG_SSB_POSSIBLE=y
# CONFIG_SSB is not set

#
# Multifunction device drivers
#
# CONFIG_MFD_SM501 is not set

#
# Multimedia devices
#
CONFIG_VIDEO_DEV=y
# CONFIG_VIDEO_V4L1 is not set
CONFIG_VIDEO_V4L1_COMPAT=y
CONFIG_VIDEO_V4L2=y
CONFIG_VIDEO_CAPTURE_DRIVERS=y
# CONFIG_VIDEO_ADV_DEBUG is not set
CONFIG_VIDEO_HELPER_CHIPS_AUTO=y
# CONFIG_VIDEO_VIVI is not set
# CONFIG_VIDEO_SAA5246A is not set
# CONFIG_VIDEO_SAA5249 is not set
# CONFIG_VIDEO_SAA7134 is not set
# CONFIG_VIDEO_HEXIUM_ORION is not set
# CONFIG_VIDEO_HEXIUM_GEMINI is not set
CONFIG_VIDEO_CX88=m
CONFIG_VIDEO_CX88_ALSA=m
# CONFIG_VIDEO_CX88_BLACKBIRD is not set
CONFIG_VIDEO_CX88_DVB=m
CONFIG_VIDEO_CX88_VP3054=m
CONFIG_VIDEO_CX23885=m
# CONFIG_VIDEO_CAFE_CCIC is not set
# CONFIG_V4L_USB_DRIVERS is not set
# CONFIG_RADIO_ADAPTERS is not set
CONFIG_DVB_CORE=y
CONFIG_DVB_CORE_ATTACH=y
CONFIG_DVB_CAPTURE_DRIVERS=y

#
# Supported SAA7146 based PCI Adapters
#

#
# Supported USB Adapters
#
# CONFIG_DVB_USB is not set
# CONFIG_DVB_TTUSB_BUDGET is not set
# CONFIG_DVB_TTUSB_DEC is not set
# CONFIG_DVB_CINERGYT2 is not set

#
# Supported FlexCopII (B2C2) Adapters
#
# CONFIG_DVB_B2C2_FLEXCOP is not set

#
# Supported BT878 Adapters
#

#
# Supported Pluto2 Adapters
#
# CONFIG_DVB_PLUTO2 is not set

#
# Supported DVB Frontends
#

#
# Customise DVB Frontends
#
# CONFIG_DVB_FE_CUSTOMISE is not set

#
# DVB-S (satellite) frontends
#
# CONFIG_DVB_STV0299 is not set
# CONFIG_DVB_CX24110 is not set
CONFIG_DVB_CX24123=m
# CONFIG_DVB_TDA8083 is not set
# CONFIG_DVB_MT312 is not set
# CONFIG_DVB_VES1X93 is not set
# CONFIG_DVB_S5H1420 is not set
# CONFIG_DVB_TDA10086 is not set

#
# DVB-T (terrestrial) frontends
#
# CONFIG_DVB_SP8870 is not set
# CONFIG_DVB_SP887X is not set
# CONFIG_DVB_CX22700 is not set
CONFIG_DVB_CX22702=m
# CONFIG_DVB_L64781 is not set
# CONFIG_DVB_TDA1004X is not set
# CONFIG_DVB_NXT6000 is not set
CONFIG_DVB_MT352=m
CONFIG_DVB_ZL10353=m
# CONFIG_DVB_DIB3000MB is not set
# CONFIG_DVB_DIB3000MC is not set
# CONFIG_DVB_DIB7000M is not set
# CONFIG_DVB_DIB7000P is not set

#
# DVB-C (cable) frontends
#
# CONFIG_DVB_VES1820 is not set
# CONFIG_DVB_TDA10021 is not set
# CONFIG_DVB_TDA10023 is not set
# CONFIG_DVB_STV0297 is not set

#
# ATSC (North American/Korean Terrestrial/Cable DTV) frontends
#
CONFIG_DVB_NXT200X=m
CONFIG_DVB_OR51211=m
CONFIG_DVB_OR51132=m
CONFIG_DVB_BCM3510=m
CONFIG_DVB_LGDT330X=m
CONFIG_DVB_S5H1409=m

#
# Tuners/PLL support
#
CONFIG_DVB_PLL=m
CONFIG_DVB_TDA826X=m
CONFIG_DVB_TDA827X=m
CONFIG_DVB_TUNER_QT1010=m
CONFIG_DVB_TUNER_MT2060=m
CONFIG_DVB_TUNER_MT2266=m
CONFIG_DVB_TUNER_MT2131=m
CONFIG_DVB_TUNER_DIB0070=m

#
# Miscellaneous devices
#
# CONFIG_DVB_LNBP21 is not set
CONFIG_DVB_ISL6421=m
# CONFIG_DVB_TUA6100 is not set
CONFIG_VIDEO_TUNER=m
# CONFIG_VIDEO_TUNER_CUSTOMIZE is not set
CONFIG_TUNER_MT20XX=m
CONFIG_TUNER_TDA8290=m
CONFIG_TUNER_TEA5761=m
CONFIG_TUNER_TEA5767=m
CONFIG_TUNER_SIMPLE=m
CONFIG_VIDEOBUF_GEN=m
CONFIG_VIDEOBUF_DMA_SG=m
CONFIG_VIDEOBUF_DVB=m
CONFIG_VIDEO_BTCX=m
CONFIG_VIDEO_IR_I2C=m
CONFIG_VIDEO_IR=m
CONFIG_VIDEO_TVEEPROM=m
# CONFIG_DAB is not set

#
# Graphics support
#
CONFIG_AGP=y
CONFIG_AGP_AMD64=y
# CONFIG_AGP_INTEL is not set
# CONFIG_AGP_SIS is not set
# CONFIG_AGP_VIA is not set
CONFIG_DRM=y
# CONFIG_DRM_TDFX is not set
# CONFIG_DRM_R128 is not set
CONFIG_DRM_RADEON=m
# CONFIG_DRM_MGA is not set
# CONFIG_DRM_SIS is not set
# CONFIG_DRM_VIA is not set
# CONFIG_DRM_SAVAGE is not set
# CONFIG_VGASTATE is not set
# CONFIG_VIDEO_OUTPUT_CONTROL is not set
# CONFIG_FB is not set
# CONFIG_BACKLIGHT_LCD_SUPPORT is not set

#
# Display device support
#
# CONFIG_DISPLAY_SUPPORT is not set

#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
# CONFIG_VGACON_SOFT_SCROLLBACK is not set
# CONFIG_VIDEO_SELECT is not set
CONFIG_DUMMY_CONSOLE=y

#
# Sound
#
CONFIG_SOUND=y

#
# Advanced Linux Sound Architecture
#
CONFIG_SND=y
CONFIG_SND_TIMER=y
CONFIG_SND_PCM=y
CONFIG_SND_RAWMIDI=y
CONFIG_SND_SEQUENCER=y
# CONFIG_SND_SEQ_DUMMY is not set
CONFIG_SND_OSSEMUL=y
CONFIG_SND_MIXER_OSS=y
CONFIG_SND_PCM_OSS=y
# CONFIG_SND_PCM_OSS_PLUGINS is not set
CONFIG_SND_SEQUENCER_OSS=y
# CONFIG_SND_RTCTIMER is not set
# CONFIG_SND_DYNAMIC_MINORS is not set
# CONFIG_SND_SUPPORT_OLD_API is not set
# CONFIG_SND_VERBOSE_PROCFS is not set
# CONFIG_SND_VERBOSE_PRINTK is not set
# CONFIG_SND_DEBUG is not set

#
# Generic devices
#
CONFIG_SND_MPU401_UART=y
CONFIG_SND_AC97_CODEC=y
# CONFIG_SND_DUMMY is not set
# CONFIG_SND_VIRMIDI is not set
# CONFIG_SND_MTPAV is not set
# CONFIG_SND_MTS64 is not set
CONFIG_SND_SERIAL_U16550=m
CONFIG_SND_MPU401=m
# CONFIG_SND_PORTMAN2X4 is not set

#
# PCI devices
#
# CONFIG_SND_AD1889 is not set
# CONFIG_SND_ALS300 is not set
# CONFIG_SND_ALS4000 is not set
# CONFIG_SND_ALI5451 is not set
# CONFIG_SND_ATIIXP is not set
# CONFIG_SND_ATIIXP_MODEM is not set
# CONFIG_SND_AU8810 is not set
# CONFIG_SND_AU8820 is not set
# CONFIG_SND_AU8830 is not set
# CONFIG_SND_AZT3328 is not set
# CONFIG_SND_BT87X is not set
# CONFIG_SND_CA0106 is not set
# CONFIG_SND_CMIPCI is not set
# CONFIG_SND_CS4281 is not set
CONFIG_SND_CS46XX=m
CONFIG_SND_CS46XX_NEW_DSP=y
# CONFIG_SND_CS5530 is not set
# CONFIG_SND_DARLA20 is not set
# CONFIG_SND_GINA20 is not set
# CONFIG_SND_LAYLA20 is not set
# CONFIG_SND_DARLA24 is not set
# CONFIG_SND_GINA24 is not set
# CONFIG_SND_LAYLA24 is not set
# CONFIG_SND_MONA is not set
# CONFIG_SND_MIA is not set
# CONFIG_SND_ECHO3G is not set
# CONFIG_SND_INDIGO is not set
# CONFIG_SND_INDIGOIO is not set
# CONFIG_SND_INDIGODJ is not set
# CONFIG_SND_EMU10K1 is not set
# CONFIG_SND_EMU10K1X is not set
# CONFIG_SND_ENS1370 is not set
# CONFIG_SND_ENS1371 is not set
# CONFIG_SND_ES1938 is not set
# CONFIG_SND_ES1968 is not set
# CONFIG_SND_FM801 is not set
# CONFIG_SND_HDA_INTEL is not set
# CONFIG_SND_HDSP is not set
# CONFIG_SND_HDSPM is not set
# CONFIG_SND_ICE1712 is not set
CONFIG_SND_ICE1724=y
CONFIG_SND_INTEL8X0=m
# CONFIG_SND_INTEL8X0M is not set
# CONFIG_SND_KORG1212 is not set
# CONFIG_SND_MAESTRO3 is not set
# CONFIG_SND_MIXART is not set
# CONFIG_SND_NM256 is not set
# CONFIG_SND_PCXHR is not set
# CONFIG_SND_RIPTIDE is not set
# CONFIG_SND_RME32 is not set
# CONFIG_SND_RME96 is not set
# CONFIG_SND_RME9652 is not set
# CONFIG_SND_SONICVIBES is not set
# CONFIG_SND_TRIDENT is not set
# CONFIG_SND_VIA82XX is not set
# CONFIG_SND_VIA82XX_MODEM is not set
# CONFIG_SND_VX222 is not set
# CONFIG_SND_YMFPCI is not set
# CONFIG_SND_AC97_POWER_SAVE is not set

#
# USB devices
#
# CONFIG_SND_USB_AUDIO is not set
# CONFIG_SND_USB_USX2Y is not set
# CONFIG_SND_USB_CAIAQ is not set

#
# System on Chip audio support
#
# CONFIG_SND_SOC is not set

#
# SoC Audio support for SuperH
#

#
# Open Sound System
#
# CONFIG_SOUND_PRIME is not set
CONFIG_AC97_BUS=y
CONFIG_HID_SUPPORT=y
CONFIG_HID=y
CONFIG_HID_DEBUG=y
CONFIG_HIDRAW=y

#
# USB Input Devices
#
CONFIG_USB_HID=y
# CONFIG_USB_HIDINPUT_POWERBOOK is not set
# CONFIG_HID_FF is not set
CONFIG_USB_HIDDEV=y
CONFIG_USB_SUPPORT=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB_ARCH_HAS_EHCI=y
CONFIG_USB=y
# CONFIG_USB_DEBUG is not set

#
# Miscellaneous USB options
#
CONFIG_USB_DEVICEFS=y
# CONFIG_USB_DEVICE_CLASS is not set
# CONFIG_USB_DYNAMIC_MINORS is not set
# CONFIG_USB_SUSPEND is not set
# CONFIG_USB_PERSIST is not set
# CONFIG_USB_OTG is not set

#
# USB Host Controller Drivers
#
CONFIG_USB_EHCI_HCD=m
# CONFIG_USB_EHCI_SPLIT_ISO is not set
# CONFIG_USB_EHCI_ROOT_HUB_TT is not set
# CONFIG_USB_EHCI_TT_NEWSCHED is not set
# CONFIG_USB_ISP116X_HCD is not set
CONFIG_USB_OHCI_HCD=y
# CONFIG_USB_OHCI_BIG_ENDIAN_DESC is not set
# CONFIG_USB_OHCI_BIG_ENDIAN_MMIO is not set
CONFIG_USB_OHCI_LITTLE_ENDIAN=y
# CONFIG_USB_UHCI_HCD is not set
# CONFIG_USB_SL811_HCD is not set
# CONFIG_USB_R8A66597_HCD is not set

#
# USB Device Class drivers
#
# CONFIG_USB_ACM is not set
# CONFIG_USB_PRINTER is not set

#
# NOTE: USB_STORAGE enables SCSI, and 'SCSI disk support'
#

#
# may also be needed; see USB_STORAGE Help for more information
#
CONFIG_USB_STORAGE=y
# CONFIG_USB_STORAGE_DEBUG is not set
# CONFIG_USB_STORAGE_DATAFAB is not set
# CONFIG_USB_STORAGE_FREECOM is not set
# CONFIG_USB_STORAGE_ISD200 is not set
# CONFIG_USB_STORAGE_DPCM is not set
# CONFIG_USB_STORAGE_USBAT is not set
# CONFIG_USB_STORAGE_SDDR09 is not set
# CONFIG_USB_STORAGE_SDDR55 is not set
# CONFIG_USB_STORAGE_JUMPSHOT is not set
# CONFIG_USB_STORAGE_ALAUDA is not set
# CONFIG_USB_STORAGE_KARMA is not set
# CONFIG_USB_LIBUSUAL is not set

#
# USB Imaging devices
#
# CONFIG_USB_MDC800 is not set
# CONFIG_USB_MICROTEK is not set
CONFIG_USB_MON=y

#
# USB port drivers
#
# CONFIG_USB_USS720 is not set

#
# USB Serial Converter support
#
CONFIG_USB_SERIAL=m
CONFIG_USB_SERIAL_GENERIC=y
# CONFIG_USB_SERIAL_AIRCABLE is not set
# CONFIG_USB_SERIAL_AIRPRIME is not set
# CONFIG_USB_SERIAL_ARK3116 is not set
CONFIG_USB_SERIAL_BELKIN=m
# CONFIG_USB_SERIAL_CH341 is not set
# CONFIG_USB_SERIAL_WHITEHEAT is not set
# CONFIG_USB_SERIAL_DIGI_ACCELEPORT is not set
# CONFIG_USB_SERIAL_CP2101 is not set
# CONFIG_USB_SERIAL_CYPRESS_M8 is not set
# CONFIG_USB_SERIAL_EMPEG is not set
CONFIG_USB_SERIAL_FTDI_SIO=m
CONFIG_USB_SERIAL_FUNSOFT=m
# CONFIG_USB_SERIAL_VISOR is not set
# CONFIG_USB_SERIAL_IPAQ is not set
# CONFIG_USB_SERIAL_IR is not set
# CONFIG_USB_SERIAL_EDGEPORT is not set
# CONFIG_USB_SERIAL_EDGEPORT_TI is not set
# CONFIG_USB_SERIAL_GARMIN is not set
# CONFIG_USB_SERIAL_IPW is not set
# CONFIG_USB_SERIAL_KEYSPAN_PDA is not set
# CONFIG_USB_SERIAL_KEYSPAN is not set
# CONFIG_USB_SERIAL_KLSI is not set
# CONFIG_USB_SERIAL_KOBIL_SCT is not set
# CONFIG_USB_SERIAL_MCT_U232 is not set
# CONFIG_USB_SERIAL_MOS7720 is not set
# CONFIG_USB_SERIAL_MOS7840 is not set
# CONFIG_USB_SERIAL_NAVMAN is not set
# CONFIG_USB_SERIAL_PL2303 is not set
# CONFIG_USB_SERIAL_OTI6858 is not set
# CONFIG_USB_SERIAL_HP4X is not set
CONFIG_USB_SERIAL_SAFE=m
# CONFIG_USB_SERIAL_SAFE_PADDED is not set
# CONFIG_USB_SERIAL_SIERRAWIRELESS is not set
# CONFIG_USB_SERIAL_TI is not set
# CONFIG_USB_SERIAL_CYBERJACK is not set
# CONFIG_USB_SERIAL_XIRCOM is not set
# CONFIG_USB_SERIAL_OPTION is not set
# CONFIG_USB_SERIAL_OMNINET is not set
# CONFIG_USB_SERIAL_DEBUG is not set

#
# USB Miscellaneous drivers
#
# CONFIG_USB_EMI62 is not set
# CONFIG_USB_EMI26 is not set
# CONFIG_USB_ADUTUX is not set
# CONFIG_USB_AUERSWALD is not set
# CONFIG_USB_RIO500 is not set
# CONFIG_USB_LEGOTOWER is not set
# CONFIG_USB_LCD is not set
# CONFIG_USB_BERRY_CHARGE is not set
# CONFIG_USB_LED is not set
# CONFIG_USB_CYPRESS_CY7C63 is not set
# CONFIG_USB_CYTHERM is not set
# CONFIG_USB_PHIDGET is not set
# CONFIG_USB_IDMOUSE is not set
# CONFIG_USB_FTDI_ELAN is not set
# CONFIG_USB_APPLEDISPLAY is not set
# CONFIG_USB_SISUSBVGA is not set
# CONFIG_USB_LD is not set
# CONFIG_USB_TRANCEVIBRATOR is not set
# CONFIG_USB_IOWARRIOR is not set
# CONFIG_USB_TEST is not set

#
# USB DSL modem support
#

#
# USB Gadget Support
#
# CONFIG_USB_GADGET is not set
# CONFIG_MMC is not set
# CONFIG_NEW_LEDS is not set
# CONFIG_INFINIBAND is not set
# CONFIG_EDAC is not set
# CONFIG_RTC_CLASS is not set
# CONFIG_DMADEVICES is not set
# CONFIG_AUXDISPLAY is not set
CONFIG_VIRTUALIZATION=y
# CONFIG_KVM is not set

#
# Userspace I/O
#
# CONFIG_UIO is not set

#
# Firmware Drivers
#
# CONFIG_EDD is not set
# CONFIG_DELL_RBU is not set
# CONFIG_DCDBAS is not set
CONFIG_DMIID=y

#
# File systems
#
# CONFIG_EXT2_FS is not set
CONFIG_EXT3_FS=y
CONFIG_EXT3_FS_XATTR=y
CONFIG_EXT3_FS_POSIX_ACL=y
# CONFIG_EXT3_FS_SECURITY is not set
CONFIG_EXT4DEV_FS=m
CONFIG_EXT4DEV_FS_XATTR=y
CONFIG_EXT4DEV_FS_POSIX_ACL=y
CONFIG_EXT4DEV_FS_SECURITY=y
CONFIG_JBD=y
# CONFIG_JBD_DEBUG is not set
CONFIG_JBD2=m
# CONFIG_JBD2_DEBUG is not set
CONFIG_FS_MBCACHE=y
CONFIG_REISERFS_FS=y
# CONFIG_REISERFS_CHECK is not set
# CONFIG_REISERFS_PROC_INFO is not set
CONFIG_REISERFS_FS_XATTR=y
CONFIG_REISERFS_FS_POSIX_ACL=y
# CONFIG_REISERFS_FS_SECURITY is not set
CONFIG_JFS_FS=m
CONFIG_JFS_POSIX_ACL=y
# CONFIG_JFS_SECURITY is not set
# CONFIG_JFS_DEBUG is not set
# CONFIG_JFS_STATISTICS is not set
CONFIG_FS_POSIX_ACL=y
CONFIG_XFS_FS=m
# CONFIG_XFS_QUOTA is not set
CONFIG_XFS_SECURITY=y
CONFIG_XFS_POSIX_ACL=y
# CONFIG_XFS_RT is not set
# CONFIG_GFS2_FS is not set
CONFIG_OCFS2_FS=m
# CONFIG_OCFS2_DEBUG_MASKLOG is not set
# CONFIG_MINIX_FS is not set
# CONFIG_ROMFS_FS is not set
CONFIG_INOTIFY=y
CONFIG_INOTIFY_USER=y
# CONFIG_QUOTA is not set
CONFIG_DNOTIFY=y
# CONFIG_AUTOFS_FS is not set
CONFIG_AUTOFS4_FS=y
CONFIG_FUSE_FS=m

#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=y
CONFIG_JOLIET=y
# CONFIG_ZISOFS is not set
CONFIG_UDF_FS=y
CONFIG_UDF_NLS=y

#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=y
CONFIG_MSDOS_FS=y
CONFIG_VFAT_FS=y
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
CONFIG_NTFS_FS=m
# CONFIG_NTFS_DEBUG is not set
# CONFIG_NTFS_RW is not set

#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
# CONFIG_PROC_KCORE is not set
CONFIG_PROC_SYSCTL=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
# CONFIG_TMPFS_POSIX_ACL is not set
# CONFIG_HUGETLBFS is not set
# CONFIG_HUGETLB_PAGE is not set
CONFIG_CONFIGFS_FS=m

#
# Miscellaneous filesystems
#
# CONFIG_ADFS_FS is not set
# CONFIG_AFFS_FS is not set
# CONFIG_HFS_FS is not set
# CONFIG_HFSPLUS_FS is not set
# CONFIG_BEFS_FS is not set
# CONFIG_BFS_FS is not set
# CONFIG_EFS_FS is not set
CONFIG_CRAMFS=y
# CONFIG_VXFS_FS is not set
# CONFIG_HPFS_FS is not set
# CONFIG_QNX4FS_FS is not set
# CONFIG_SYSV_FS is not set
# CONFIG_UFS_FS is not set
CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_NFS_FS=y
CONFIG_NFS_V3=y
CONFIG_NFS_V3_ACL=y
CONFIG_NFS_V4=y
# CONFIG_NFS_DIRECTIO is not set
# CONFIG_NFSD is not set
CONFIG_LOCKD=y
CONFIG_LOCKD_V4=y
CONFIG_NFS_ACL_SUPPORT=y
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=y
CONFIG_SUNRPC_GSS=y
# CONFIG_SUNRPC_BIND34 is not set
CONFIG_RPCSEC_GSS_KRB5=y
# CONFIG_RPCSEC_GSS_SPKM3 is not set
# CONFIG_SMB_FS is not set
CONFIG_CIFS=y
# CONFIG_CIFS_STATS is not set
# CONFIG_CIFS_WEAK_PW_HASH is not set
CONFIG_CIFS_XATTR=y
CONFIG_CIFS_POSIX=y
# CONFIG_CIFS_DEBUG2 is not set
# CONFIG_CIFS_EXPERIMENTAL is not set
# CONFIG_NCP_FS is not set
# CONFIG_CODA_FS is not set
# CONFIG_AFS_FS is not set

#
# Partition Types
#
# CONFIG_PARTITION_ADVANCED is not set
CONFIG_MSDOS_PARTITION=y
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso8859-1"
CONFIG_NLS_CODEPAGE_437=y
# CONFIG_NLS_CODEPAGE_737 is not set
# CONFIG_NLS_CODEPAGE_775 is not set
CONFIG_NLS_CODEPAGE_850=m
CONFIG_NLS_CODEPAGE_852=m
# CONFIG_NLS_CODEPAGE_855 is not set
# CONFIG_NLS_CODEPAGE_857 is not set
# CONFIG_NLS_CODEPAGE_860 is not set
# CONFIG_NLS_CODEPAGE_861 is not set
# CONFIG_NLS_CODEPAGE_862 is not set
# CONFIG_NLS_CODEPAGE_863 is not set
# CONFIG_NLS_CODEPAGE_864 is not set
# CONFIG_NLS_CODEPAGE_865 is not set
# CONFIG_NLS_CODEPAGE_866 is not set
# CONFIG_NLS_CODEPAGE_869 is not set
# CONFIG_NLS_CODEPAGE_936 is not set
# CONFIG_NLS_CODEPAGE_950 is not set
# CONFIG_NLS_CODEPAGE_932 is not set
# CONFIG_NLS_CODEPAGE_949 is not set
# CONFIG_NLS_CODEPAGE_874 is not set
# CONFIG_NLS_ISO8859_8 is not set
# CONFIG_NLS_CODEPAGE_1250 is not set
# CONFIG_NLS_CODEPAGE_1251 is not set
CONFIG_NLS_ASCII=y
CONFIG_NLS_ISO8859_1=y
CONFIG_NLS_ISO8859_2=y
# CONFIG_NLS_ISO8859_3 is not set
# CONFIG_NLS_ISO8859_4 is not set
# CONFIG_NLS_ISO8859_5 is not set
# CONFIG_NLS_ISO8859_6 is not set
# CONFIG_NLS_ISO8859_7 is not set
# CONFIG_NLS_ISO8859_9 is not set
# CONFIG_NLS_ISO8859_13 is not set
# CONFIG_NLS_ISO8859_14 is not set
# CONFIG_NLS_ISO8859_15 is not set
# CONFIG_NLS_KOI8_R is not set
# CONFIG_NLS_KOI8_U is not set
CONFIG_NLS_UTF8=y
# CONFIG_DLM is not set
CONFIG_INSTRUMENTATION=y
CONFIG_PROFILING=y
CONFIG_OPROFILE=m
CONFIG_KPROBES=y
CONFIG_MARKERS=y

#
# Kernel hacking
#
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
CONFIG_PRINTK_TIME=y
CONFIG_ENABLE_WARN_DEPRECATED=y
CONFIG_ENABLE_MUST_CHECK=y
CONFIG_MAGIC_SYSRQ=y
# CONFIG_UNUSED_SYMBOLS is not set
CONFIG_DEBUG_FS=y
# CONFIG_HEADERS_CHECK is not set
CONFIG_DEBUG_KERNEL=y
# CONFIG_DEBUG_SHIRQ is not set
CONFIG_DETECT_SOFTLOCKUP=y
# CONFIG_SCHED_DEBUG is not set
# CONFIG_SCHEDSTATS is not set
# CONFIG_TIMER_STATS is not set
# CONFIG_SLUB_DEBUG_ON is not set
# CONFIG_DEBUG_RT_MUTEXES is not set
# CONFIG_RT_MUTEX_TESTER is not set
CONFIG_DEBUG_SPINLOCK=y
CONFIG_DEBUG_MUTEXES=y
# CONFIG_DEBUG_LOCK_ALLOC is not set
# CONFIG_PROVE_LOCKING is not set
# CONFIG_LOCK_STAT is not set
# CONFIG_DEBUG_SPINLOCK_SLEEP is not set
# CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
# CONFIG_DEBUG_KOBJECT is not set
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_VM is not set
# CONFIG_DEBUG_LIST is not set
# CONFIG_DEBUG_SG is not set
CONFIG_FRAME_POINTER=y
# CONFIG_FORCED_INLINING is not set
# CONFIG_BOOT_PRINTK_DELAY is not set
# CONFIG_RCU_TORTURE_TEST is not set
# CONFIG_LKDTM is not set
# CONFIG_FAULT_INJECTION is not set
# CONFIG_SAMPLES is not set
CONFIG_EARLY_PRINTK=y
CONFIG_DEBUG_STACKOVERFLOW=y
# CONFIG_DEBUG_STACK_USAGE is not set
CONFIG_DEBUG_RODATA=y
# CONFIG_IOMMU_DEBUG is not set

#
# Security options
#
# CONFIG_KEYS is not set
# CONFIG_SECURITY is not set
# CONFIG_SECURITY_FILE_CAPABILITIES is not set
CONFIG_CRYPTO=y
CONFIG_CRYPTO_ALGAPI=y
CONFIG_CRYPTO_BLKCIPHER=y
CONFIG_CRYPTO_HASH=y
CONFIG_CRYPTO_MANAGER=y
CONFIG_CRYPTO_HMAC=y
# CONFIG_CRYPTO_XCBC is not set
CONFIG_CRYPTO_NULL=y
# CONFIG_CRYPTO_MD4 is not set
CONFIG_CRYPTO_MD5=y
CONFIG_CRYPTO_SHA1=y
CONFIG_CRYPTO_SHA256=y
CONFIG_CRYPTO_SHA512=y
# CONFIG_CRYPTO_WP512 is not set
# CONFIG_CRYPTO_TGR192 is not set
# CONFIG_CRYPTO_GF128MUL is not set
CONFIG_CRYPTO_ECB=m
CONFIG_CRYPTO_CBC=y
# CONFIG_CRYPTO_PCBC is not set
# CONFIG_CRYPTO_LRW is not set
# CONFIG_CRYPTO_XTS is not set
# CONFIG_CRYPTO_CRYPTD is not set
CONFIG_CRYPTO_DES=y
# CONFIG_CRYPTO_FCRYPT is not set
# CONFIG_CRYPTO_BLOWFISH is not set
# CONFIG_CRYPTO_TWOFISH is not set
# CONFIG_CRYPTO_TWOFISH_X86_64 is not set
# CONFIG_CRYPTO_SERPENT is not set
CONFIG_CRYPTO_AES=m
CONFIG_CRYPTO_AES_X86_64=y
# CONFIG_CRYPTO_CAST5 is not set
# CONFIG_CRYPTO_CAST6 is not set
# CONFIG_CRYPTO_TEA is not set
# CONFIG_CRYPTO_ARC4 is not set
# CONFIG_CRYPTO_KHAZAD is not set
# CONFIG_CRYPTO_ANUBIS is not set
# CONFIG_CRYPTO_SEED is not set
CONFIG_CRYPTO_DEFLATE=y
# CONFIG_CRYPTO_MICHAEL_MIC is not set
CONFIG_CRYPTO_CRC32C=y
# CONFIG_CRYPTO_CAMELLIA is not set
# CONFIG_CRYPTO_TEST is not set
# CONFIG_CRYPTO_AUTHENC is not set
# CONFIG_CRYPTO_HW is not set

#
# Library routines
#
CONFIG_BITREVERSE=y
CONFIG_CRC_CCITT=y
CONFIG_CRC16=y
# CONFIG_CRC_ITU_T is not set
CONFIG_CRC32=y
CONFIG_CRC7=y
CONFIG_LIBCRC32C=y
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=y
CONFIG_PLIST=y
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT=y
CONFIG_HAS_DMA=y

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: pdflush stuck in D state with v2.6.24-rc1-192-gef49c32

[Fengguang Wu]

On Thu, Nov 01, 2007 at 09:10:02PM -0500, Florin Iucha wrote:
> On Fri, Nov 02, 2007 at 09:33:21AM +0800, Fengguang Wu wrote:
> > > I will try that with a USB disk - I hope that won't make a difference.
> > 
> > Thank you. I guess a reiserfs on loop file would also be OK.
> > 
> > > > btw, what's the exact kernel version you are running?
> > > 
> > > I noticed it with the kernel in the $SUBJECT, as reported by 'git
> > > describe'.  I have pulled in new changesets since then.
> > 
> > And with the following patch applied?
> > 
> > ---
> >  fs/reiserfs/stree.c |    3 ---
> >  1 file changed, 3 deletions(-)
> > 
> > --- linux-2.6.24-git17.orig/fs/reiserfs/stree.c
> > +++ linux-2.6.24-git17/fs/reiserfs/stree.c
> > @@ -1458,9 +1458,6 @@ static void unmap_buffers(struct page *p
> >  				}
> >  				bh = next;
> >  			} while (bh != head);
> > -			if (PAGE_SIZE == bh->b_size) {
> > -				cancel_dirty_page(page, PAGE_CACHE_SIZE);
> > -			}
> >  		}
> >  	}
> >  }
> 
> ... and with the above patch applied.
> 
> Copying 300 MB from root (ext3) to the new file system did not trigger
> the pdflush condition.  But then I did a
>    cd $MOUNTPOINT && find . -exec md5sum {} \;
> and that brought one cpu to 75% iowait.

Immediately? Do you have the debug printk messages this time(with the
above patch)?

> I have attached my .config, if it helps.

It's really curious - I tried your .config and commands, and still
could not trigger the high iowait. I'm running 64bit Intel Core 2,
and kernel 2.6.24-rc1-git6 with the above patch.

Fengguang

Re: pdflush stuck in D state with v2.6.24-rc1-192-gef49c32

[Florin Iucha]

[-- Attachment #1: Type: text/plain, Size: 1543 bytes --]

On Fri, Nov 02, 2007 at 08:56:55PM +0800, Fengguang Wu wrote:
> > > ---
> > >  fs/reiserfs/stree.c |    3 ---
> > >  1 file changed, 3 deletions(-)
> > > 
> > > --- linux-2.6.24-git17.orig/fs/reiserfs/stree.c
> > > +++ linux-2.6.24-git17/fs/reiserfs/stree.c
> > > @@ -1458,9 +1458,6 @@ static void unmap_buffers(struct page *p
> > >  				}
> > >  				bh = next;
> > >  			} while (bh != head);
> > > -			if (PAGE_SIZE == bh->b_size) {
> > > -				cancel_dirty_page(page, PAGE_CACHE_SIZE);
> > > -			}
> > >  		}
> > >  	}
> > >  }
> > 
> > ... and with the above patch applied.
> > 
> > Copying 300 MB from root (ext3) to the new file system did not trigger
> > the pdflush condition.  But then I did a
> >    cd $MOUNTPOINT && find . -exec md5sum {} \;
> > and that brought one cpu to 75% iowait.
> 
> Immediately? Do you have the debug printk messages this time(with the
> above patch)?

No, but I will add them this afternoon.

> > I have attached my .config, if it helps.
> 
> It's really curious - I tried your .config and commands, and still
> could not trigger the high iowait. I'm running 64bit Intel Core 2,
> and kernel 2.6.24-rc1-git6 with the above patch.

Curious but 100% reproducible, at least on my box.  What I'm going to
try is booting into the kernel with your patch and just doing the find
/ md5sum.  It would be really interesting if the read-only access
triggers it.

florin

-- 
Bruce Schneier expects the Spanish Inquisition.
      http://geekz.co.uk/schneierfacts/fact/163

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: pdflush stuck in D state with v2.6.24-rc1-192-gef49c32

[Thomas]

Florin Iucha <florin <at> iucha.net> writes:

> > It's really curious - I tried your .config and commands, and still
> > could not trigger the high iowait. I'm running 64bit Intel Core 2,
> > and kernel 2.6.24-rc1-git6 with the above patch.
> 
> Curious but 100% reproducible, at least on my box.  What I'm going to
> try is booting into the kernel with your patch and just doing the find
> / md5sum.  It would be really interesting if the read-only access
> triggers it.
> 
> florin
> 

I can confirm this issue too on any .24-rc. I'm also using reiserfs on a LVM.

And there is one more user on Gentoo forums having the same issue.
http://forums.gentoo.org/viewtopic-t-612959.html

So you are not alone, florian.

[Bug 9291] pdflush stuck in D state with v2.6.24-rc1-192-gef49c32

[Ingo Molnar]

* Thomas <gimpel@sonnenkinder.org> wrote:

> I can confirm this issue too on any .24-rc. I'm also using reiserfs on 
> a LVM.
> 
> And there is one more user on Gentoo forums having the same issue. 
> http://forums.gentoo.org/viewtopic-t-612959.html
> 
> So you are not alone, florian.

any progress on this issue? Seems a bit stalled.

	Ingo

Re: [Bug 9291] pdflush stuck in D state with v2.6.24-rc1-192-gef49c32

[Thomas Kuther]

[-- Attachment #1: Type: text/plain, Size: 752 bytes --]

On Di, 04.12.07 11:28 Ingo Molnar <mingo@elte.hu> wrote:

> 
> * Thomas <gimpel@sonnenkinder.org> wrote:
> 
> > I can confirm this issue too on any .24-rc. I'm also using reiserfs
> > on a LVM.
> > 
> > And there is one more user on Gentoo forums having the same issue. 
> > http://forums.gentoo.org/viewtopic-t-612959.html
> > 
> > So you are not alone, florian.
> 
> any progress on this issue? Seems a bit stalled.
> 
> 	Ingo

For me the two patches
 * mm-speed-up-writeback-ramp-up-on-clean-systems.patch 
 * reiserfs-writeback-fix.patch
solved the issue.

IIRC one was from this thread, the other from
http://lkml.org/lkml/2007/10/23/93

So since 2.6.24-rc2-git5 all is fine again. No problems since.

Regards,
Thomas

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]