-mm: xfs lockdep warning

-mm: xfs lockdep warning

[Yang Ruirui]

Hi,

I got following lockdep warning, xfs related?

[  604.416384] =================================
[  604.416625] [ INFO: inconsistent lock state ]
[  604.416625] 2.6.36-rc4-mm1 #2
[  604.416625] ---------------------------------
[  604.416625] inconsistent {RECLAIM_FS-ON-W} -> {IN-RECLAIM_FS-W} usage.
[  604.416625] kswapd0/418 [HC0[0]:SC0[0]:HE1:SE1] takes:
[  604.416625]  (&(&ip->i_iolock)->mr_lock#2){++++?+}, at: [<ffffffff812046df>] xfs_ilock+0x94/0x137
[  604.416625] {RECLAIM_FS-ON-W} state was registered at:
[  604.416625]   [<ffffffff81065571>] mark_held_locks+0x4d/0x6b
[  604.416625]   [<ffffffff81065641>] lockdep_trace_alloc+0xb2/0xd7
[  604.416625]   [<ffffffff810ee215>] kmem_cache_alloc+0x2a/0x126
[  604.416625]   [<ffffffff81225d5b>] kmem_zone_alloc+0x67/0xaf
[  604.416625]   [<ffffffff81225db2>] kmem_zone_zalloc+0xf/0x30
[  604.416625]   [<ffffffff8121dd73>] _xfs_trans_alloc+0x22/0x5f
[  604.416625]   [<ffffffff8121ef9f>] xfs_trans_alloc+0x9d/0xaa
[  604.416625]   [<ffffffff8122520c>] xfs_setattr+0x3d2/0x8b4
[  604.416625]   [<ffffffff8122eea4>] xfs_vn_setattr+0x16/0x1a
[  604.416625]   [<ffffffff81106ec7>] notify_change+0x18f/0x27d
[  604.416625]   [<ffffffff810f1de1>] do_truncate+0x6a/0x88
[  604.416625]   [<ffffffff810fd7ff>] do_last+0x588/0x58f
[  604.416625]   [<ffffffff810fda43>] do_filp_open+0x23d/0x5db
[  604.416625]   [<ffffffff810f1055>] do_sys_open+0x5a/0xf0
[  604.416625]   [<ffffffff810f1114>] sys_open+0x1b/0x1d
[  604.416625]   [<ffffffff81002b82>] system_call_fastpath+0x16/0x1b
[  604.416625] irq event stamp: 144829
[  604.416625] hardirqs last  enabled at (144829): [<ffffffff815c96ba>] _raw_spin_unlock_irqrestore+0x46/0x55
[  604.416625] hardirqs last disabled at (144828): [<ffffffff815c910b>] _raw_spin_lock_irqsave+0x24/0x58
[  604.416625] softirqs last  enabled at (142796): [<ffffffff8103fc97>] __do_softirq+0x1b6/0x1c7
[  604.416625] softirqs last disabled at (142791): [<ffffffff81003afc>] call_softirq+0x1c/0x28
[  604.416625] 
[  604.416625] other info that might help us debug this:
[  604.416625] 1 lock held by kswapd0/418:
[  604.416625]  #0:  (shrinker_rwsem){++++..}, at: [<ffffffff810c682c>] shrink_slab+0x38/0x164
[  604.416625] 
[  604.416625] stack backtrace:
[  604.416625] Pid: 418, comm: kswapd0 Not tainted 2.6.36-rc4-mm1 #2
[  604.416625] Call Trace:
[  604.416625]  [<ffffffff810652b0>] valid_state+0x18b/0x19e
[  604.416625]  [<ffffffff8100db7b>] ? save_stack_trace+0x2a/0x48
[  604.416625]  [<ffffffff81065ca9>] ? check_usage_forwards+0x0/0x7e
[  604.416625]  [<ffffffff810653c9>] mark_lock+0x106/0x261
[  604.416625]  [<ffffffff81364cef>] ? radix_tree_tag_clear+0xa5/0x108
[  604.416625]  [<ffffffff8106676f>] __lock_acquire+0x3bb/0xe1f
[  604.416625]  [<ffffffff810671c4>] ? __lock_acquire+0xe10/0xe1f
[  604.416625]  [<ffffffff8100db7b>] ? save_stack_trace+0x2a/0x48
[  604.416625]  [<ffffffff810671c4>] ? __lock_acquire+0xe10/0xe1f
[  604.416625]  [<ffffffff81364ecd>] ? radix_tree_delete+0xad/0x1b7
[  604.416625]  [<ffffffff810672ab>] lock_acquire+0xd8/0x104
[  604.416625]  [<ffffffff812046df>] ? xfs_ilock+0x94/0x137
[  604.416625]  [<ffffffff810586cf>] down_write_nested+0x4a/0x6d
[  604.416625]  [<ffffffff812046df>] ? xfs_ilock+0x94/0x137
[  604.416625]  [<ffffffff812046df>] xfs_ilock+0x94/0x137
[  604.416625]  [<ffffffff81231c9f>] xfs_reclaim_inode+0x277/0x2c1
[  604.416625]  [<ffffffff81232608>] xfs_inode_ag_walk+0x8e/0xe9
[  604.416625]  [<ffffffff81231a28>] ? xfs_reclaim_inode+0x0/0x2c1
[  604.416625]  [<ffffffff812326c7>] xfs_inode_ag_iterator+0x64/0xc3
[  604.416625]  [<ffffffff81231a28>] ? xfs_reclaim_inode+0x0/0x2c1
[  604.416625]  [<ffffffff81232762>] xfs_reclaim_inode_shrink+0x3c/0x83
[  604.416625]  [<ffffffff810c68d5>] shrink_slab+0xe1/0x164
[  604.416625]  [<ffffffff810c6f3c>] kswapd+0x5e4/0x864
[  604.416625]  [<ffffffff8105499e>] ? autoremove_wake_function+0x0/0x38
[  604.416625]  [<ffffffff810c6958>] ? kswapd+0x0/0x864
[  604.416625]  [<ffffffff8105452d>] kthread+0x81/0x89
[  604.416625]  [<ffffffff81003a04>] kernel_thread_helper+0x4/0x10
[  604.416625]  [<ffffffff81096ee2>] ? watchdog+0x0/0x281
[  604.416625]  [<ffffffff815c9ad0>] ? restore_args+0x0/0x30
[  604.416625]  [<ffffffff810544ac>] ? kthread+0x0/0x89
[  604.416625]  [<ffffffff81003a00>] ? kernel_thread_helper+0x0/0x10
--
Thanks
Yang Ruirui

-mm: xfs lockdep warning

[Yang Ruirui]

Hi,

I got following lockdep warning, xfs related?

[  604.416384] =================================
[  604.416625] [ INFO: inconsistent lock state ]
[  604.416625] 2.6.36-rc4-mm1 #2
[  604.416625] ---------------------------------
[  604.416625] inconsistent {RECLAIM_FS-ON-W} -> {IN-RECLAIM_FS-W} usage.
[  604.416625] kswapd0/418 [HC0[0]:SC0[0]:HE1:SE1] takes:
[  604.416625]  (&(&ip->i_iolock)->mr_lock#2){++++?+}, at: [<ffffffff812046df>] xfs_ilock+0x94/0x137
[  604.416625] {RECLAIM_FS-ON-W} state was registered at:
[  604.416625]   [<ffffffff81065571>] mark_held_locks+0x4d/0x6b
[  604.416625]   [<ffffffff81065641>] lockdep_trace_alloc+0xb2/0xd7
[  604.416625]   [<ffffffff810ee215>] kmem_cache_alloc+0x2a/0x126
[  604.416625]   [<ffffffff81225d5b>] kmem_zone_alloc+0x67/0xaf
[  604.416625]   [<ffffffff81225db2>] kmem_zone_zalloc+0xf/0x30
[  604.416625]   [<ffffffff8121dd73>] _xfs_trans_alloc+0x22/0x5f
[  604.416625]   [<ffffffff8121ef9f>] xfs_trans_alloc+0x9d/0xaa
[  604.416625]   [<ffffffff8122520c>] xfs_setattr+0x3d2/0x8b4
[  604.416625]   [<ffffffff8122eea4>] xfs_vn_setattr+0x16/0x1a
[  604.416625]   [<ffffffff81106ec7>] notify_change+0x18f/0x27d
[  604.416625]   [<ffffffff810f1de1>] do_truncate+0x6a/0x88
[  604.416625]   [<ffffffff810fd7ff>] do_last+0x588/0x58f
[  604.416625]   [<ffffffff810fda43>] do_filp_open+0x23d/0x5db
[  604.416625]   [<ffffffff810f1055>] do_sys_open+0x5a/0xf0
[  604.416625]   [<ffffffff810f1114>] sys_open+0x1b/0x1d
[  604.416625]   [<ffffffff81002b82>] system_call_fastpath+0x16/0x1b
[  604.416625] irq event stamp: 144829
[  604.416625] hardirqs last  enabled at (144829): [<ffffffff815c96ba>] _raw_spin_unlock_irqrestore+0x46/0x55
[  604.416625] hardirqs last disabled at (144828): [<ffffffff815c910b>] _raw_spin_lock_irqsave+0x24/0x58
[  604.416625] softirqs last  enabled at (142796): [<ffffffff8103fc97>] __do_softirq+0x1b6/0x1c7
[  604.416625] softirqs last disabled at (142791): [<ffffffff81003afc>] call_softirq+0x1c/0x28
[  604.416625] 
[  604.416625] other info that might help us debug this:
[  604.416625] 1 lock held by kswapd0/418:
[  604.416625]  #0:  (shrinker_rwsem){++++..}, at: [<ffffffff810c682c>] shrink_slab+0x38/0x164
[  604.416625] 
[  604.416625] stack backtrace:
[  604.416625] Pid: 418, comm: kswapd0 Not tainted 2.6.36-rc4-mm1 #2
[  604.416625] Call Trace:
[  604.416625]  [<ffffffff810652b0>] valid_state+0x18b/0x19e
[  604.416625]  [<ffffffff8100db7b>] ? save_stack_trace+0x2a/0x48
[  604.416625]  [<ffffffff81065ca9>] ? check_usage_forwards+0x0/0x7e
[  604.416625]  [<ffffffff810653c9>] mark_lock+0x106/0x261
[  604.416625]  [<ffffffff81364cef>] ? radix_tree_tag_clear+0xa5/0x108
[  604.416625]  [<ffffffff8106676f>] __lock_acquire+0x3bb/0xe1f
[  604.416625]  [<ffffffff810671c4>] ? __lock_acquire+0xe10/0xe1f
[  604.416625]  [<ffffffff8100db7b>] ? save_stack_trace+0x2a/0x48
[  604.416625]  [<ffffffff810671c4>] ? __lock_acquire+0xe10/0xe1f
[  604.416625]  [<ffffffff81364ecd>] ? radix_tree_delete+0xad/0x1b7
[  604.416625]  [<ffffffff810672ab>] lock_acquire+0xd8/0x104
[  604.416625]  [<ffffffff812046df>] ? xfs_ilock+0x94/0x137
[  604.416625]  [<ffffffff810586cf>] down_write_nested+0x4a/0x6d
[  604.416625]  [<ffffffff812046df>] ? xfs_ilock+0x94/0x137
[  604.416625]  [<ffffffff812046df>] xfs_ilock+0x94/0x137
[  604.416625]  [<ffffffff81231c9f>] xfs_reclaim_inode+0x277/0x2c1
[  604.416625]  [<ffffffff81232608>] xfs_inode_ag_walk+0x8e/0xe9
[  604.416625]  [<ffffffff81231a28>] ? xfs_reclaim_inode+0x0/0x2c1
[  604.416625]  [<ffffffff812326c7>] xfs_inode_ag_iterator+0x64/0xc3
[  604.416625]  [<ffffffff81231a28>] ? xfs_reclaim_inode+0x0/0x2c1
[  604.416625]  [<ffffffff81232762>] xfs_reclaim_inode_shrink+0x3c/0x83
[  604.416625]  [<ffffffff810c68d5>] shrink_slab+0xe1/0x164
[  604.416625]  [<ffffffff810c6f3c>] kswapd+0x5e4/0x864
[  604.416625]  [<ffffffff8105499e>] ? autoremove_wake_function+0x0/0x38
[  604.416625]  [<ffffffff810c6958>] ? kswapd+0x0/0x864
[  604.416625]  [<ffffffff8105452d>] kthread+0x81/0x89
[  604.416625]  [<ffffffff81003a04>] kernel_thread_helper+0x4/0x10
[  604.416625]  [<ffffffff81096ee2>] ? watchdog+0x0/0x281
[  604.416625]  [<ffffffff815c9ad0>] ? restore_args+0x0/0x30
[  604.416625]  [<ffffffff810544ac>] ? kthread+0x0/0x89
[  604.416625]  [<ffffffff81003a00>] ? kernel_thread_helper+0x0/0x10
--
Thanks
Yang Ruirui

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: -mm: xfs lockdep warning

[Dave Chinner]

On Thu, Sep 16, 2010 at 03:46:16PM +0800, Yang Ruirui wrote:
> Hi,
> 
> I got following lockdep warning, xfs related?

It's a false positive.

> [  604.416384] =================================
> [  604.416625] [ INFO: inconsistent lock state ]
> [  604.416625] 2.6.36-rc4-mm1 #2
> [  604.416625] ---------------------------------
> [  604.416625] inconsistent {RECLAIM_FS-ON-W} -> {IN-RECLAIM_FS-W} usage.
> [  604.416625] kswapd0/418 [HC0[0]:SC0[0]:HE1:SE1] takes:
> [  604.416625]  (&(&ip->i_iolock)->mr_lock#2){++++?+}, at: [<ffffffff812046df>] xfs_ilock+0x94/0x137
> [  604.416625] {RECLAIM_FS-ON-W} state was registered at:
> [  604.416625]   [<ffffffff81065571>] mark_held_locks+0x4d/0x6b
> [  604.416625]   [<ffffffff81065641>] lockdep_trace_alloc+0xb2/0xd7
> [  604.416625]   [<ffffffff810ee215>] kmem_cache_alloc+0x2a/0x126
> [  604.416625]   [<ffffffff81225d5b>] kmem_zone_alloc+0x67/0xaf
> [  604.416625]   [<ffffffff81225db2>] kmem_zone_zalloc+0xf/0x30
> [  604.416625]   [<ffffffff8121dd73>] _xfs_trans_alloc+0x22/0x5f
> [  604.416625]   [<ffffffff8121ef9f>] xfs_trans_alloc+0x9d/0xaa
> [  604.416625]   [<ffffffff8122520c>] xfs_setattr+0x3d2/0x8b4
> [  604.416625]   [<ffffffff8122eea4>] xfs_vn_setattr+0x16/0x1a
> [  604.416625]   [<ffffffff81106ec7>] notify_change+0x18f/0x27d
> [  604.416625]   [<ffffffff810f1de1>] do_truncate+0x6a/0x88
> [  604.416625]   [<ffffffff810fd7ff>] do_last+0x588/0x58f
> [  604.416625]   [<ffffffff810fda43>] do_filp_open+0x23d/0x5db
> [  604.416625]   [<ffffffff810f1055>] do_sys_open+0x5a/0xf0
> [  604.416625]   [<ffffffff810f1114>] sys_open+0x1b/0x1d
> [  604.416625]   [<ffffffff81002b82>] system_call_fastpath+0x16/0x1b
> [  604.416625] irq event stamp: 144829
> [  604.416625] hardirqs last  enabled at (144829): [<ffffffff815c96ba>] _raw_spin_unlock_irqrestore+0x46/0x55
> [  604.416625] hardirqs last disabled at (144828): [<ffffffff815c910b>] _raw_spin_lock_irqsave+0x24/0x58
> [  604.416625] softirqs last  enabled at (142796): [<ffffffff8103fc97>] __do_softirq+0x1b6/0x1c7
> [  604.416625] softirqs last disabled at (142791): [<ffffffff81003afc>] call_softirq+0x1c/0x28
> [  604.416625] 
> [  604.416625] other info that might help us debug this:
> [  604.416625] 1 lock held by kswapd0/418:
> [  604.416625]  #0:  (shrinker_rwsem){++++..}, at: [<ffffffff810c682c>] shrink_slab+0x38/0x164
> [  604.416625] 
> [  604.416625] stack backtrace:
> [  604.416625] Pid: 418, comm: kswapd0 Not tainted 2.6.36-rc4-mm1 #2
> [  604.416625] Call Trace:
> [  604.416625]  [<ffffffff810652b0>] valid_state+0x18b/0x19e
> [  604.416625]  [<ffffffff8100db7b>] ? save_stack_trace+0x2a/0x48
> [  604.416625]  [<ffffffff81065ca9>] ? check_usage_forwards+0x0/0x7e
> [  604.416625]  [<ffffffff810653c9>] mark_lock+0x106/0x261
> [  604.416625]  [<ffffffff81364cef>] ? radix_tree_tag_clear+0xa5/0x108
> [  604.416625]  [<ffffffff8106676f>] __lock_acquire+0x3bb/0xe1f
> [  604.416625]  [<ffffffff810671c4>] ? __lock_acquire+0xe10/0xe1f
> [  604.416625]  [<ffffffff8100db7b>] ? save_stack_trace+0x2a/0x48
> [  604.416625]  [<ffffffff810671c4>] ? __lock_acquire+0xe10/0xe1f
> [  604.416625]  [<ffffffff81364ecd>] ? radix_tree_delete+0xad/0x1b7
> [  604.416625]  [<ffffffff810672ab>] lock_acquire+0xd8/0x104
> [  604.416625]  [<ffffffff812046df>] ? xfs_ilock+0x94/0x137
> [  604.416625]  [<ffffffff810586cf>] down_write_nested+0x4a/0x6d
> [  604.416625]  [<ffffffff812046df>] ? xfs_ilock+0x94/0x137
> [  604.416625]  [<ffffffff812046df>] xfs_ilock+0x94/0x137
> [  604.416625]  [<ffffffff81231c9f>] xfs_reclaim_inode+0x277/0x2c1
> [  604.416625]  [<ffffffff81232608>] xfs_inode_ag_walk+0x8e/0xe9
> [  604.416625]  [<ffffffff81231a28>] ? xfs_reclaim_inode+0x0/0x2c1
> [  604.416625]  [<ffffffff812326c7>] xfs_inode_ag_iterator+0x64/0xc3
> [  604.416625]  [<ffffffff81231a28>] ? xfs_reclaim_inode+0x0/0x2c1
> [  604.416625]  [<ffffffff81232762>] xfs_reclaim_inode_shrink+0x3c/0x83
> [  604.416625]  [<ffffffff810c68d5>] shrink_slab+0xe1/0x164
> [  604.416625]  [<ffffffff810c6f3c>] kswapd+0x5e4/0x864
> [  604.416625]  [<ffffffff8105499e>] ? autoremove_wake_function+0x0/0x38
> [  604.416625]  [<ffffffff810c6958>] ? kswapd+0x0/0x864
> [  604.416625]  [<ffffffff8105452d>] kthread+0x81/0x89
> [  604.416625]  [<ffffffff81003a04>] kernel_thread_helper+0x4/0x10
> [  604.416625]  [<ffffffff81096ee2>] ? watchdog+0x0/0x281
> [  604.416625]  [<ffffffff815c9ad0>] ? restore_args+0x0/0x30
> [  604.416625]  [<ffffffff810544ac>] ? kthread+0x0/0x89
> [  604.416625]  [<ffffffff81003a00>] ? kernel_thread_helper+0x0/0x10

Christoph, this implies an inode that has been marked for reclaim
that has not passed through xfs_fs_evict_inode() after being
initialised. If it went through the eviction process, the iolock
would have been re-initialised to a different context. Can you think
of any path that can get here without going through ->evict? I can't
off the top of my head...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: -mm: xfs lockdep warning

[Dave Chinner]

On Thu, Sep 16, 2010 at 03:46:16PM +0800, Yang Ruirui wrote:
> Hi,
> 
> I got following lockdep warning, xfs related?

It's a false positive.

> [  604.416384] =================================
> [  604.416625] [ INFO: inconsistent lock state ]
> [  604.416625] 2.6.36-rc4-mm1 #2
> [  604.416625] ---------------------------------
> [  604.416625] inconsistent {RECLAIM_FS-ON-W} -> {IN-RECLAIM_FS-W} usage.
> [  604.416625] kswapd0/418 [HC0[0]:SC0[0]:HE1:SE1] takes:
> [  604.416625]  (&(&ip->i_iolock)->mr_lock#2){++++?+}, at: [<ffffffff812046df>] xfs_ilock+0x94/0x137
> [  604.416625] {RECLAIM_FS-ON-W} state was registered at:
> [  604.416625]   [<ffffffff81065571>] mark_held_locks+0x4d/0x6b
> [  604.416625]   [<ffffffff81065641>] lockdep_trace_alloc+0xb2/0xd7
> [  604.416625]   [<ffffffff810ee215>] kmem_cache_alloc+0x2a/0x126
> [  604.416625]   [<ffffffff81225d5b>] kmem_zone_alloc+0x67/0xaf
> [  604.416625]   [<ffffffff81225db2>] kmem_zone_zalloc+0xf/0x30
> [  604.416625]   [<ffffffff8121dd73>] _xfs_trans_alloc+0x22/0x5f
> [  604.416625]   [<ffffffff8121ef9f>] xfs_trans_alloc+0x9d/0xaa
> [  604.416625]   [<ffffffff8122520c>] xfs_setattr+0x3d2/0x8b4
> [  604.416625]   [<ffffffff8122eea4>] xfs_vn_setattr+0x16/0x1a
> [  604.416625]   [<ffffffff81106ec7>] notify_change+0x18f/0x27d
> [  604.416625]   [<ffffffff810f1de1>] do_truncate+0x6a/0x88
> [  604.416625]   [<ffffffff810fd7ff>] do_last+0x588/0x58f
> [  604.416625]   [<ffffffff810fda43>] do_filp_open+0x23d/0x5db
> [  604.416625]   [<ffffffff810f1055>] do_sys_open+0x5a/0xf0
> [  604.416625]   [<ffffffff810f1114>] sys_open+0x1b/0x1d
> [  604.416625]   [<ffffffff81002b82>] system_call_fastpath+0x16/0x1b
> [  604.416625] irq event stamp: 144829
> [  604.416625] hardirqs last  enabled at (144829): [<ffffffff815c96ba>] _raw_spin_unlock_irqrestore+0x46/0x55
> [  604.416625] hardirqs last disabled at (144828): [<ffffffff815c910b>] _raw_spin_lock_irqsave+0x24/0x58
> [  604.416625] softirqs last  enabled at (142796): [<ffffffff8103fc97>] __do_softirq+0x1b6/0x1c7
> [  604.416625] softirqs last disabled at (142791): [<ffffffff81003afc>] call_softirq+0x1c/0x28
> [  604.416625] 
> [  604.416625] other info that might help us debug this:
> [  604.416625] 1 lock held by kswapd0/418:
> [  604.416625]  #0:  (shrinker_rwsem){++++..}, at: [<ffffffff810c682c>] shrink_slab+0x38/0x164
> [  604.416625] 
> [  604.416625] stack backtrace:
> [  604.416625] Pid: 418, comm: kswapd0 Not tainted 2.6.36-rc4-mm1 #2
> [  604.416625] Call Trace:
> [  604.416625]  [<ffffffff810652b0>] valid_state+0x18b/0x19e
> [  604.416625]  [<ffffffff8100db7b>] ? save_stack_trace+0x2a/0x48
> [  604.416625]  [<ffffffff81065ca9>] ? check_usage_forwards+0x0/0x7e
> [  604.416625]  [<ffffffff810653c9>] mark_lock+0x106/0x261
> [  604.416625]  [<ffffffff81364cef>] ? radix_tree_tag_clear+0xa5/0x108
> [  604.416625]  [<ffffffff8106676f>] __lock_acquire+0x3bb/0xe1f
> [  604.416625]  [<ffffffff810671c4>] ? __lock_acquire+0xe10/0xe1f
> [  604.416625]  [<ffffffff8100db7b>] ? save_stack_trace+0x2a/0x48
> [  604.416625]  [<ffffffff810671c4>] ? __lock_acquire+0xe10/0xe1f
> [  604.416625]  [<ffffffff81364ecd>] ? radix_tree_delete+0xad/0x1b7
> [  604.416625]  [<ffffffff810672ab>] lock_acquire+0xd8/0x104
> [  604.416625]  [<ffffffff812046df>] ? xfs_ilock+0x94/0x137
> [  604.416625]  [<ffffffff810586cf>] down_write_nested+0x4a/0x6d
> [  604.416625]  [<ffffffff812046df>] ? xfs_ilock+0x94/0x137
> [  604.416625]  [<ffffffff812046df>] xfs_ilock+0x94/0x137
> [  604.416625]  [<ffffffff81231c9f>] xfs_reclaim_inode+0x277/0x2c1
> [  604.416625]  [<ffffffff81232608>] xfs_inode_ag_walk+0x8e/0xe9
> [  604.416625]  [<ffffffff81231a28>] ? xfs_reclaim_inode+0x0/0x2c1
> [  604.416625]  [<ffffffff812326c7>] xfs_inode_ag_iterator+0x64/0xc3
> [  604.416625]  [<ffffffff81231a28>] ? xfs_reclaim_inode+0x0/0x2c1
> [  604.416625]  [<ffffffff81232762>] xfs_reclaim_inode_shrink+0x3c/0x83
> [  604.416625]  [<ffffffff810c68d5>] shrink_slab+0xe1/0x164
> [  604.416625]  [<ffffffff810c6f3c>] kswapd+0x5e4/0x864
> [  604.416625]  [<ffffffff8105499e>] ? autoremove_wake_function+0x0/0x38
> [  604.416625]  [<ffffffff810c6958>] ? kswapd+0x0/0x864
> [  604.416625]  [<ffffffff8105452d>] kthread+0x81/0x89
> [  604.416625]  [<ffffffff81003a04>] kernel_thread_helper+0x4/0x10
> [  604.416625]  [<ffffffff81096ee2>] ? watchdog+0x0/0x281
> [  604.416625]  [<ffffffff815c9ad0>] ? restore_args+0x0/0x30
> [  604.416625]  [<ffffffff810544ac>] ? kthread+0x0/0x89
> [  604.416625]  [<ffffffff81003a00>] ? kernel_thread_helper+0x0/0x10

Christoph, this implies an inode that has been marked for reclaim
that has not passed through xfs_fs_evict_inode() after being
initialised. If it went through the eviction process, the iolock
would have been re-initialised to a different context. Can you think
of any path that can get here without going through ->evict? I can't
off the top of my head...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: -mm: xfs lockdep warning

[Christoph Hellwig]

On Fri, Sep 17, 2010 at 10:52:27AM +1000, Dave Chinner wrote:
> Christoph, this implies an inode that has been marked for reclaim
> that has not passed through xfs_fs_evict_inode() after being
> initialised. If it went through the eviction process, the iolock
> would have been re-initialised to a different context. Can you think
> of any path that can get here without going through ->evict? I can't
> off the top of my head...

I think this could happen if the init_inode_always during
re-initialization of an inode in reclaim fails in iget.  I have a patch
to add that I'll run through xfsqa.  It should only happen very rarely.

Re: -mm: xfs lockdep warning

[Christoph Hellwig]

On Fri, Sep 17, 2010 at 10:52:27AM +1000, Dave Chinner wrote:
> Christoph, this implies an inode that has been marked for reclaim
> that has not passed through xfs_fs_evict_inode() after being
> initialised. If it went through the eviction process, the iolock
> would have been re-initialised to a different context. Can you think
> of any path that can get here without going through ->evict? I can't
> off the top of my head...

I think this could happen if the init_inode_always during
re-initialization of an inode in reclaim fails in iget.  I have a patch
to add that I'll run through xfsqa.  It should only happen very rarely.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: -mm: xfs lockdep warning

[Torsten Kaiser]

[adding Tejun Heo, because I'm partly blaming his workqueues]

On Mon, Sep 20, 2010 at 9:13 PM, Christoph Hellwig <hch@infradead.org> wrote:
> On Fri, Sep 17, 2010 at 10:52:27AM +1000, Dave Chinner wrote:
>> Christoph, this implies an inode that has been marked for reclaim
>> that has not passed through xfs_fs_evict_inode() after being
>> initialised. If it went through the eviction process, the iolock
>> would have been re-initialised to a different context. Can you think
>> of any path that can get here without going through ->evict? I can't
>> off the top of my head...
>
> I think this could happen if the init_inode_always during
> re-initialization of an inode in reclaim fails in iget.  I have a patch
> to add that I'll run through xfsqa.  It should only happen very rarely.

I had the same lockdep report in mainline 2.6.36-rc5, when I enabled
lockdep to try to debug a hang under high load and some memory
pressure.
Do you have a patch I could try to get rid of this lockdep report to
see, if there is another one lurking behind it?

As for the hang: Today I wanted to install updates on my gentoo system
and that included an update of koffice. Because I allow emerge
(gentoos package manager) to process 4 packages in parallel and
specify -j5 for each of the compiles the result was, that emerge
decided to build 4 koffice programs in parallel which resulted in a
load ~18 on my 4 core system. With all these gcc running and the
compiles happening on a tmpfs this also resulted in some memory
pressure. (But not in a swap storm)

The first time I got this hang, I was not able to capture useful
informations, the second time it happened I got a report from the hung
task timeout:
3 processes where blocked in:
[ 1203.056532]  [<ffffffff810801c0>] ? sync_page+0x0/0x50
[ 1203.061836]  [<ffffffff8156078d>] ? io_schedule+0x3d/0x60
[ 1203.067390]  [<ffffffff810801fd>] ? sync_page+0x3d/0x50
probably because of these:
[ 1202.872424] INFO: task plasma-desktop:3169 blocked for more than 120 seconds.
[ 1202.879883] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 1202.888118]  ffff88011ef860b0 0000000000000086 000000011ef860b0
ffff880114faff48
[ 1202.895936]  ffff8800071cf1c0 0000000000012780 ffff880114faffd8
0000000000012780
[ 1202.903777]  ffff880114faffd8 ffff88011ef86310 ffff880114faffd8
ffff88011ef86308
[ 1202.911586] Call Trace:
[ 1202.914140]  [<ffffffff815612c1>] ? __mutex_lock_slowpath+0xe1/0x160
[ 1202.920693]  [<ffffffff810d1e57>] ? __d_lookup+0x97/0x120
[ 1202.926272]  [<ffffffff81560d8a>] ? mutex_lock+0x1a/0x40
[ 1202.931783]  [<ffffffff810c9dbf>] ? do_lookup+0x10f/0x190
[ 1202.937426]  [<ffffffff810c7e24>] ? acl_permission_check+0x54/0xb0
[ 1202.943898]  [<ffffffff810ca8d9>] ? link_path_walk+0x4c9/0x9a0
[ 1202.950028]  [<ffffffff810d65f4>] ? mnt_want_write+0x34/0x70
[ 1202.955947]  [<ffffffff810caecf>] ? path_walk+0x5f/0xe0
[ 1202.961416]  [<ffffffff810caf9b>] ? do_path_lookup+0x4b/0x50
[ 1202.967325]  [<ffffffff810cbc35>] ? user_path_at+0x55/0xb0
[ 1202.973020]  [<ffffffff8102e860>] ? __dequeue_entity+0x40/0x50
[ 1202.979054]  [<ffffffff8107bcf4>] ? perf_event_task_sched_out+0x44/0x260
[ 1202.985995]  [<ffffffff810c2e46>] ? vfs_fstatat+0x36/0x80
[ 1202.991600]  [<ffffffff815602ac>] ? schedule+0x24c/0x6f0
[ 1202.997157]  [<ffffffff810c2f8f>] ? sys_newstat+0x1f/0x50
[ 1203.002784]  [<ffffffff81003035>] ? device_not_available+0x15/0x20
[ 1203.009204]  [<ffffffff8100246b>] ? system_call_fastpath+0x16/0x1b

[ 1202.400046] INFO: task kworker/u:8:845 blocked for more than 120 seconds.
[ 1202.407128] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 1202.415372]  ffff88007ffa8b60 0000000000000046 ffff88007f74fe98
0000000000000000
[ 1202.423229]  ffff8800071cf770 0000000000012780 ffff8800070e9fd8
0000000000012780
[ 1202.431054]  ffff8800070e9fd8 ffff88007ffa8dc0 ffff8800070e9fd8
ffff88007ffa8db8
[ 1202.438898] Call Trace:
[ 1202.441495]  [<ffffffff814361ed>] ? md_write_start+0x9d/0x190
[ 1202.447536]  [<ffffffff81052e70>] ? autoremove_wake_function+0x0/0x30
[ 1202.454311]  [<ffffffff810925ca>] ? zone_nr_free_pages+0x9a/0xb0
[ 1202.460639]  [<ffffffff8142c423>] ? make_request+0x23/0x800
[ 1202.466496]  [<ffffffff811d90ea>] ? blkcipher_walk_done+0x8a/0x230
[ 1202.472990]  [<ffffffff811df4c4>] ? crypto_cbc_encrypt+0xe4/0x180
[ 1202.479327]  [<ffffffff8102a150>] ? twofish_encrypt+0x0/0x10
[ 1202.485276]  [<ffffffff81431ac3>] ? md_make_request+0xc3/0x220
[ 1202.491412]  [<ffffffff811eafca>] ? generic_make_request+0x18a/0x330
[ 1202.498096]  [<ffffffff814484fc>] ? crypt_convert+0x25c/0x310
[ 1202.504090]  [<ffffffff81448881>] ? kcryptd_crypt+0x2d1/0x3f0
[ 1202.510031]  [<ffffffff814485b0>] ? kcryptd_crypt+0x0/0x3f0
[ 1202.516081]  [<ffffffff8104d2eb>] ? process_one_work+0xfb/0x370
[ 1202.522123]  [<ffffffff8104ee4c>] ? worker_thread+0x16c/0x360
[ 1202.527935]  [<ffffffff8104ece0>] ? worker_thread+0x0/0x360
[ 1202.533541]  [<ffffffff8104ece0>] ? worker_thread+0x0/0x360
[ 1202.539131]  [<ffffffff810529e6>] ? kthread+0x96/0xa0
[ 1202.544245]  [<ffffffff81003194>] ? kernel_thread_helper+0x4/0x10
[ 1202.550363]  [<ffffffff81052950>] ? kthread+0x0/0xa0
[ 1202.555346]  [<ffffffff81003190>] ? kernel_thread_helper+0x0/0x10

After that I enabled lockdep and retried the same update: This time no
hang, only the same lockdep report that Yang Ruirui had.

So I'm currently at a loss how I should continue from here. The XFS
false positive drowns any other lockdep problems, but XFS had some
hang problems with the new workqueues. (Personally I had no other hang
with earlier 2.6.36-rcs)
As seen in the call trace from the kworker, there are more workqueues
involved, thats why I added Tejun to the CC. My root filesystem is XFS
on dm-crypt on a md/raid1, so it might have been something there.

If you need more information, just ask, I will try to provide it.

Torsten

Re: -mm: xfs lockdep warning

[Torsten Kaiser]

[adding Tejun Heo, because I'm partly blaming his workqueues]

On Mon, Sep 20, 2010 at 9:13 PM, Christoph Hellwig <hch@infradead.org> wrote:
> On Fri, Sep 17, 2010 at 10:52:27AM +1000, Dave Chinner wrote:
>> Christoph, this implies an inode that has been marked for reclaim
>> that has not passed through xfs_fs_evict_inode() after being
>> initialised. If it went through the eviction process, the iolock
>> would have been re-initialised to a different context. Can you think
>> of any path that can get here without going through ->evict? I can't
>> off the top of my head...
>
> I think this could happen if the init_inode_always during
> re-initialization of an inode in reclaim fails in iget.  I have a patch
> to add that I'll run through xfsqa.  It should only happen very rarely.

I had the same lockdep report in mainline 2.6.36-rc5, when I enabled
lockdep to try to debug a hang under high load and some memory
pressure.
Do you have a patch I could try to get rid of this lockdep report to
see, if there is another one lurking behind it?

As for the hang: Today I wanted to install updates on my gentoo system
and that included an update of koffice. Because I allow emerge
(gentoos package manager) to process 4 packages in parallel and
specify -j5 for each of the compiles the result was, that emerge
decided to build 4 koffice programs in parallel which resulted in a
load ~18 on my 4 core system. With all these gcc running and the
compiles happening on a tmpfs this also resulted in some memory
pressure. (But not in a swap storm)

The first time I got this hang, I was not able to capture useful
informations, the second time it happened I got a report from the hung
task timeout:
3 processes where blocked in:
[ 1203.056532]  [<ffffffff810801c0>] ? sync_page+0x0/0x50
[ 1203.061836]  [<ffffffff8156078d>] ? io_schedule+0x3d/0x60
[ 1203.067390]  [<ffffffff810801fd>] ? sync_page+0x3d/0x50
probably because of these:
[ 1202.872424] INFO: task plasma-desktop:3169 blocked for more than 120 seconds.
[ 1202.879883] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 1202.888118]  ffff88011ef860b0 0000000000000086 000000011ef860b0
ffff880114faff48
[ 1202.895936]  ffff8800071cf1c0 0000000000012780 ffff880114faffd8
0000000000012780
[ 1202.903777]  ffff880114faffd8 ffff88011ef86310 ffff880114faffd8
ffff88011ef86308
[ 1202.911586] Call Trace:
[ 1202.914140]  [<ffffffff815612c1>] ? __mutex_lock_slowpath+0xe1/0x160
[ 1202.920693]  [<ffffffff810d1e57>] ? __d_lookup+0x97/0x120
[ 1202.926272]  [<ffffffff81560d8a>] ? mutex_lock+0x1a/0x40
[ 1202.931783]  [<ffffffff810c9dbf>] ? do_lookup+0x10f/0x190
[ 1202.937426]  [<ffffffff810c7e24>] ? acl_permission_check+0x54/0xb0
[ 1202.943898]  [<ffffffff810ca8d9>] ? link_path_walk+0x4c9/0x9a0
[ 1202.950028]  [<ffffffff810d65f4>] ? mnt_want_write+0x34/0x70
[ 1202.955947]  [<ffffffff810caecf>] ? path_walk+0x5f/0xe0
[ 1202.961416]  [<ffffffff810caf9b>] ? do_path_lookup+0x4b/0x50
[ 1202.967325]  [<ffffffff810cbc35>] ? user_path_at+0x55/0xb0
[ 1202.973020]  [<ffffffff8102e860>] ? __dequeue_entity+0x40/0x50
[ 1202.979054]  [<ffffffff8107bcf4>] ? perf_event_task_sched_out+0x44/0x260
[ 1202.985995]  [<ffffffff810c2e46>] ? vfs_fstatat+0x36/0x80
[ 1202.991600]  [<ffffffff815602ac>] ? schedule+0x24c/0x6f0
[ 1202.997157]  [<ffffffff810c2f8f>] ? sys_newstat+0x1f/0x50
[ 1203.002784]  [<ffffffff81003035>] ? device_not_available+0x15/0x20
[ 1203.009204]  [<ffffffff8100246b>] ? system_call_fastpath+0x16/0x1b

[ 1202.400046] INFO: task kworker/u:8:845 blocked for more than 120 seconds.
[ 1202.407128] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 1202.415372]  ffff88007ffa8b60 0000000000000046 ffff88007f74fe98
0000000000000000
[ 1202.423229]  ffff8800071cf770 0000000000012780 ffff8800070e9fd8
0000000000012780
[ 1202.431054]  ffff8800070e9fd8 ffff88007ffa8dc0 ffff8800070e9fd8
ffff88007ffa8db8
[ 1202.438898] Call Trace:
[ 1202.441495]  [<ffffffff814361ed>] ? md_write_start+0x9d/0x190
[ 1202.447536]  [<ffffffff81052e70>] ? autoremove_wake_function+0x0/0x30
[ 1202.454311]  [<ffffffff810925ca>] ? zone_nr_free_pages+0x9a/0xb0
[ 1202.460639]  [<ffffffff8142c423>] ? make_request+0x23/0x800
[ 1202.466496]  [<ffffffff811d90ea>] ? blkcipher_walk_done+0x8a/0x230
[ 1202.472990]  [<ffffffff811df4c4>] ? crypto_cbc_encrypt+0xe4/0x180
[ 1202.479327]  [<ffffffff8102a150>] ? twofish_encrypt+0x0/0x10
[ 1202.485276]  [<ffffffff81431ac3>] ? md_make_request+0xc3/0x220
[ 1202.491412]  [<ffffffff811eafca>] ? generic_make_request+0x18a/0x330
[ 1202.498096]  [<ffffffff814484fc>] ? crypt_convert+0x25c/0x310
[ 1202.504090]  [<ffffffff81448881>] ? kcryptd_crypt+0x2d1/0x3f0
[ 1202.510031]  [<ffffffff814485b0>] ? kcryptd_crypt+0x0/0x3f0
[ 1202.516081]  [<ffffffff8104d2eb>] ? process_one_work+0xfb/0x370
[ 1202.522123]  [<ffffffff8104ee4c>] ? worker_thread+0x16c/0x360
[ 1202.527935]  [<ffffffff8104ece0>] ? worker_thread+0x0/0x360
[ 1202.533541]  [<ffffffff8104ece0>] ? worker_thread+0x0/0x360
[ 1202.539131]  [<ffffffff810529e6>] ? kthread+0x96/0xa0
[ 1202.544245]  [<ffffffff81003194>] ? kernel_thread_helper+0x4/0x10
[ 1202.550363]  [<ffffffff81052950>] ? kthread+0x0/0xa0
[ 1202.555346]  [<ffffffff81003190>] ? kernel_thread_helper+0x0/0x10

After that I enabled lockdep and retried the same update: This time no
hang, only the same lockdep report that Yang Ruirui had.

So I'm currently at a loss how I should continue from here. The XFS
false positive drowns any other lockdep problems, but XFS had some
hang problems with the new workqueues. (Personally I had no other hang
with earlier 2.6.36-rcs)
As seen in the call trace from the kworker, there are more workqueues
involved, thats why I added Tejun to the CC. My root filesystem is XFS
on dm-crypt on a md/raid1, so it might have been something there.

If you need more information, just ask, I will try to provide it.

Torsten

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: -mm: xfs lockdep warning

[Tejun Heo]

Hello,

On 09/25/2010 03:08 PM, Torsten Kaiser wrote:
> [adding Tejun Heo, because I'm partly blaming his workqueues]

:-) Sorry about the delay.  I was away the last week.

> After that I enabled lockdep and retried the same update: This time no
> hang, only the same lockdep report that Yang Ruirui had.
> 
> So I'm currently at a loss how I should continue from here. The XFS
> false positive drowns any other lockdep problems, but XFS had some
> hang problems with the new workqueues. (Personally I had no other hang
> with earlier 2.6.36-rcs)

If workqueue lockdep annotations are triggering spuriously, just nuke
all lockdep annotations in workqueue.c.  Currently the annotations are
the same as before and trigger even when an actual deadlock won't
happen anymore around flushes (but then again all the current users
should be operating under the same constraints as before).  I'm
looking into how to improve the lockdep annotations but in the end it
might be best to leave it as it is.  I'll try to analyze the warnings.

Thanks.

-- 
tejun

Re: -mm: xfs lockdep warning

[Tejun Heo]

Hello,

On 09/25/2010 03:08 PM, Torsten Kaiser wrote:
> [adding Tejun Heo, because I'm partly blaming his workqueues]

:-) Sorry about the delay.  I was away the last week.

> After that I enabled lockdep and retried the same update: This time no
> hang, only the same lockdep report that Yang Ruirui had.
> 
> So I'm currently at a loss how I should continue from here. The XFS
> false positive drowns any other lockdep problems, but XFS had some
> hang problems with the new workqueues. (Personally I had no other hang
> with earlier 2.6.36-rcs)

If workqueue lockdep annotations are triggering spuriously, just nuke
all lockdep annotations in workqueue.c.  Currently the annotations are
the same as before and trigger even when an actual deadlock won't
happen anymore around flushes (but then again all the current users
should be operating under the same constraints as before).  I'm
looking into how to improve the lockdep annotations but in the end it
might be best to leave it as it is.  I'll try to analyze the warnings.

Thanks.

-- 
tejun

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: -mm: xfs lockdep warning

[Dave Chinner]

On Mon, Oct 04, 2010 at 11:01:19AM +0200, Tejun Heo wrote:
> Hello,
> 
> On 09/25/2010 03:08 PM, Torsten Kaiser wrote:
> > [adding Tejun Heo, because I'm partly blaming his workqueues]
> 
> :-) Sorry about the delay.  I was away the last week.
> 
> > After that I enabled lockdep and retried the same update: This time no
> > hang, only the same lockdep report that Yang Ruirui had.
> > 
> > So I'm currently at a loss how I should continue from here. The XFS
> > false positive drowns any other lockdep problems, but XFS had some
> > hang problems with the new workqueues. (Personally I had no other hang
> > with earlier 2.6.36-rcs)

What XFS hang problem? I've seen temporary livelocks from the new
workqueue code, but thay have been fixed. I haven't come across any
deadlocks or hangs. Do you have a pointer to a bug report in case my
memory is failing me?

> If workqueue lockdep annotations are triggering spuriously, just nuke
> all lockdep annotations in workqueue.c.  Currently the annotations are
> the same as before and trigger even when an actual deadlock won't
> happen anymore around flushes (but then again all the current users
> should be operating under the same constraints as before).  I'm
> looking into how to improve the lockdep annotations but in the end it
> might be best to leave it as it is.  I'll try to analyze the warnings.

It looks like there's been some discussion that has been private so
I'm clueless as to the context being discussed here.  Can someone
fill me in on whatever is going on?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: -mm: xfs lockdep warning

[Dave Chinner]

On Mon, Oct 04, 2010 at 11:01:19AM +0200, Tejun Heo wrote:
> Hello,
> 
> On 09/25/2010 03:08 PM, Torsten Kaiser wrote:
> > [adding Tejun Heo, because I'm partly blaming his workqueues]
> 
> :-) Sorry about the delay.  I was away the last week.
> 
> > After that I enabled lockdep and retried the same update: This time no
> > hang, only the same lockdep report that Yang Ruirui had.
> > 
> > So I'm currently at a loss how I should continue from here. The XFS
> > false positive drowns any other lockdep problems, but XFS had some
> > hang problems with the new workqueues. (Personally I had no other hang
> > with earlier 2.6.36-rcs)

What XFS hang problem? I've seen temporary livelocks from the new
workqueue code, but thay have been fixed. I haven't come across any
deadlocks or hangs. Do you have a pointer to a bug report in case my
memory is failing me?

> If workqueue lockdep annotations are triggering spuriously, just nuke
> all lockdep annotations in workqueue.c.  Currently the annotations are
> the same as before and trigger even when an actual deadlock won't
> happen anymore around flushes (but then again all the current users
> should be operating under the same constraints as before).  I'm
> looking into how to improve the lockdep annotations but in the end it
> might be best to leave it as it is.  I'll try to analyze the warnings.

It looks like there's been some discussion that has been private so
I'm clueless as to the context being discussed here.  Can someone
fill me in on whatever is going on?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: -mm: xfs lockdep warning

[Torsten Kaiser]

On Mon, Oct 4, 2010 at 11:21 AM, Dave Chinner <david@fromorbit.com> wrote:
> On Mon, Oct 04, 2010 at 11:01:19AM +0200, Tejun Heo wrote:
>> Hello,
>>
>> On 09/25/2010 03:08 PM, Torsten Kaiser wrote:
>> > [adding Tejun Heo, because I'm partly blaming his workqueues]
>>
>> :-) Sorry about the delay.  I was away the last week.
>>
>> > After that I enabled lockdep and retried the same update: This time no
>> > hang, only the same lockdep report that Yang Ruirui had.
>> >
>> > So I'm currently at a loss how I should continue from here. The XFS
>> > false positive drowns any other lockdep problems, but XFS had some
>> > hang problems with the new workqueues. (Personally I had no other hang
>> > with earlier 2.6.36-rcs)
>
> What XFS hang problem? I've seen temporary livelocks from the new
> workqueue code, but thay have been fixed. I haven't come across any
> deadlocks or hangs. Do you have a pointer to a bug report in case my
> memory is failing me?

I was referring to the hang that prompted to increase of the priority
of the xfslogd thread. Its commit
51749e47e191db8e588ad5cebea731caf7b705d7 in Linus' tree. But it seems
I have misremembered that one as a hang.

The hang I have seen is describe here:
http://marc.info/?l=linux-kernel&m=128542018414916&w=2

As 2.6.36-rc5 already includes this priority fix, I believe this hang
is something different and wrote that mail.

>> If workqueue lockdep annotations are triggering spuriously, just nuke
>> all lockdep annotations in workqueue.c.  Currently the annotations are
>> the same as before and trigger even when an actual deadlock won't
>> happen anymore around flushes (but then again all the current users
>> should be operating under the same constraints as before).  I'm
>> looking into how to improve the lockdep annotations but in the end it
>> might be best to leave it as it is.  I'll try to analyze the warnings.
>
> It looks like there's been some discussion that has been private so
> I'm clueless as to the context being discussed here.  Can someone
> fill me in on whatever is going on?

I only send the above message both to the lkml and the XFS mailing list.

But it could have been wrong to make it a reply in this thread,
because now there are three different things mixed up.

1) the old livelock, that I was referring too -> That is already fixed
in mainline, I just wanted to note, that this is unrelated to the hang
I have seen.
2) the hang with 2.6.35-rc5 I have seen twice (triggered probably by
high load while building KOffice on an tmpfs)
3) the lockdep issue that started this thread and that I am seeing in
mainline 2.6.36-rc5 and -rc6 and that seems to be rather easy to
trigger for me.

Because 3) is regarded as a false positive, it should not be the cause
of 2). But because 3) triggers on my system I can't see if lockdep
might give better pointers to what might be the real cause of 2).

So, I would like to test the patch Christoph was testing in
http://marc.info/?l=linux-kernel&m=128501006509743&w=2 to see, if that
helps to debug my hang. (And because I did not see that patch on the
lkml, I replied to Christophs mail.)

I CC'ed Tejun, because I believe the primary suspect for my hang was
the stuck kworker/u:8:845 and I was hoping he might see something
obvious wrong in that stacktrace.

Torsten

Re: -mm: xfs lockdep warning

[Torsten Kaiser]

On Mon, Oct 4, 2010 at 11:21 AM, Dave Chinner <david@fromorbit.com> wrote:
> On Mon, Oct 04, 2010 at 11:01:19AM +0200, Tejun Heo wrote:
>> Hello,
>>
>> On 09/25/2010 03:08 PM, Torsten Kaiser wrote:
>> > [adding Tejun Heo, because I'm partly blaming his workqueues]
>>
>> :-) Sorry about the delay.  I was away the last week.
>>
>> > After that I enabled lockdep and retried the same update: This time no
>> > hang, only the same lockdep report that Yang Ruirui had.
>> >
>> > So I'm currently at a loss how I should continue from here. The XFS
>> > false positive drowns any other lockdep problems, but XFS had some
>> > hang problems with the new workqueues. (Personally I had no other hang
>> > with earlier 2.6.36-rcs)
>
> What XFS hang problem? I've seen temporary livelocks from the new
> workqueue code, but thay have been fixed. I haven't come across any
> deadlocks or hangs. Do you have a pointer to a bug report in case my
> memory is failing me?

I was referring to the hang that prompted to increase of the priority
of the xfslogd thread. Its commit
51749e47e191db8e588ad5cebea731caf7b705d7 in Linus' tree. But it seems
I have misremembered that one as a hang.

The hang I have seen is describe here:
http://marc.info/?l=linux-kernel&m=128542018414916&w=2

As 2.6.36-rc5 already includes this priority fix, I believe this hang
is something different and wrote that mail.

>> If workqueue lockdep annotations are triggering spuriously, just nuke
>> all lockdep annotations in workqueue.c.  Currently the annotations are
>> the same as before and trigger even when an actual deadlock won't
>> happen anymore around flushes (but then again all the current users
>> should be operating under the same constraints as before).  I'm
>> looking into how to improve the lockdep annotations but in the end it
>> might be best to leave it as it is.  I'll try to analyze the warnings.
>
> It looks like there's been some discussion that has been private so
> I'm clueless as to the context being discussed here.  Can someone
> fill me in on whatever is going on?

I only send the above message both to the lkml and the XFS mailing list.

But it could have been wrong to make it a reply in this thread,
because now there are three different things mixed up.

1) the old livelock, that I was referring too -> That is already fixed
in mainline, I just wanted to note, that this is unrelated to the hang
I have seen.
2) the hang with 2.6.35-rc5 I have seen twice (triggered probably by
high load while building KOffice on an tmpfs)
3) the lockdep issue that started this thread and that I am seeing in
mainline 2.6.36-rc5 and -rc6 and that seems to be rather easy to
trigger for me.

Because 3) is regarded as a false positive, it should not be the cause
of 2). But because 3) triggers on my system I can't see if lockdep
might give better pointers to what might be the real cause of 2).

So, I would like to test the patch Christoph was testing in
http://marc.info/?l=linux-kernel&m=128501006509743&w=2 to see, if that
helps to debug my hang. (And because I did not see that patch on the
lkml, I replied to Christophs mail.)

I CC'ed Tejun, because I believe the primary suspect for my hang was
the stuck kworker/u:8:845 and I was hoping he might see something
obvious wrong in that stacktrace.

Torsten

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: -mm: xfs lockdep warning

[Tejun Heo]

Hello,

This is getting confusing.

On 10/05/2010 12:09 PM, Torsten Kaiser wrote:
> 2) the hang with 2.6.35-rc5 I have seen twice (triggered probably by
> high load while building KOffice on an tmpfs)

The trace doesn't seem to firmly point at xfs locking up.  There are
md and dm crypt involved.  Are you sure this is locking up inside xfs?
Also, can you please enable frame pointers so that we can get more
reliable backtrace?

> 3) the lockdep issue that started this thread and that I am seeing in
> mainline 2.6.36-rc5 and -rc6 and that seems to be rather easy to
> trigger for me.
> 
> Because 3) is regarded as a false positive, it should not be the cause
> of 2)

For now, let's not draw any conclusion or mix the issues.  I'll look
up the original thread and look into what the lockdep warning means.

Thanks.

-- 
tejun

Re: -mm: xfs lockdep warning

[Tejun Heo]

Hello,

This is getting confusing.

On 10/05/2010 12:09 PM, Torsten Kaiser wrote:
> 2) the hang with 2.6.35-rc5 I have seen twice (triggered probably by
> high load while building KOffice on an tmpfs)

The trace doesn't seem to firmly point at xfs locking up.  There are
md and dm crypt involved.  Are you sure this is locking up inside xfs?
Also, can you please enable frame pointers so that we can get more
reliable backtrace?

> 3) the lockdep issue that started this thread and that I am seeing in
> mainline 2.6.36-rc5 and -rc6 and that seems to be rather easy to
> trigger for me.
> 
> Because 3) is regarded as a false positive, it should not be the cause
> of 2)

For now, let's not draw any conclusion or mix the issues.  I'll look
up the original thread and look into what the lockdep warning means.

Thanks.

-- 
tejun

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: -mm: xfs lockdep warning

[Tejun Heo]

On 10/05/2010 06:51 PM, Tejun Heo wrote:
> On 10/05/2010 12:09 PM, Torsten Kaiser wrote:
>> 2) the hang with 2.6.35-rc5 I have seen twice (triggered probably by
>> high load while building KOffice on an tmpfs)
> 
> The trace doesn't seem to firmly point at xfs locking up.  There are
> md and dm crypt involved.  Are you sure this is locking up inside xfs?
> Also, can you please enable frame pointers so that we can get more
> reliable backtrace?
> 
>> 3) the lockdep issue that started this thread and that I am seeing in
>> mainline 2.6.36-rc5 and -rc6 and that seems to be rather easy to
>> trigger for me.
>>
>> Because 3) is regarded as a false positive, it should not be the cause
>> of 2)
> 
> For now, let's not draw any conclusion or mix the issues.  I'll look
> up the original thread and look into what the lockdep warning means.

Ah, okay, David already looked into it and concluded it to be a
spurious one.  Can you please attach full kernel message of the
lockup?

Thanks.

-- 
tejun

Re: -mm: xfs lockdep warning

[Tejun Heo]

On 10/05/2010 06:51 PM, Tejun Heo wrote:
> On 10/05/2010 12:09 PM, Torsten Kaiser wrote:
>> 2) the hang with 2.6.35-rc5 I have seen twice (triggered probably by
>> high load while building KOffice on an tmpfs)
> 
> The trace doesn't seem to firmly point at xfs locking up.  There are
> md and dm crypt involved.  Are you sure this is locking up inside xfs?
> Also, can you please enable frame pointers so that we can get more
> reliable backtrace?
> 
>> 3) the lockdep issue that started this thread and that I am seeing in
>> mainline 2.6.36-rc5 and -rc6 and that seems to be rather easy to
>> trigger for me.
>>
>> Because 3) is regarded as a false positive, it should not be the cause
>> of 2)
> 
> For now, let's not draw any conclusion or mix the issues.  I'll look
> up the original thread and look into what the lockdep warning means.

Ah, okay, David already looked into it and concluded it to be a
spurious one.  Can you please attach full kernel message of the
lockup?

Thanks.

-- 
tejun

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: -mm: xfs lockdep warning

[Torsten Kaiser]

On Tue, Oct 5, 2010 at 6:51 PM, Tejun Heo <tj@kernel.org> wrote:
> Hello,
>
> This is getting confusing.
>
> On 10/05/2010 12:09 PM, Torsten Kaiser wrote:
>> 2) the hang with 2.6.35-rc5 I have seen twice (triggered probably by
>> high load while building KOffice on an tmpfs)
>
> The trace doesn't seem to firmly point at xfs locking up.  There are
> md and dm crypt involved.  Are you sure this is locking up inside xfs?

Not 100%, but apart from the md/dm-worker the other prcesses got stuck
in sync_page() and do_lockup() and these function look filesystem
related. And the only filesystem I'm using on that system is XFS.

And having just read about some problems (that were fixed in
2.6.36-rc5) regarding XFS vs the new workqueues, I "feared" that I
somehow triggered some funny interaction between the workqueues of
XFS, md-raid1 and dm-crypt.

> Also, can you please enable frame pointers so that we can get more
> reliable backtrace?

I will do that and see if -rc6 gets stuck the same way.
But it might take until tomorrow to test this.

>> 3) the lockdep issue that started this thread and that I am seeing in
>> mainline 2.6.36-rc5 and -rc6 and that seems to be rather easy to
>> trigger for me.
>>
>> Because 3) is regarded as a false positive, it should not be the cause
>> of 2)
>
> For now, let's not draw any conclusion or mix the issues.  I'll look
> up the original thread and look into what the lockdep warning means.
>
> Thanks.
>
> --
> tejun
>

Re: -mm: xfs lockdep warning

[Torsten Kaiser]

On Tue, Oct 5, 2010 at 6:51 PM, Tejun Heo <tj@kernel.org> wrote:
> Hello,
>
> This is getting confusing.
>
> On 10/05/2010 12:09 PM, Torsten Kaiser wrote:
>> 2) the hang with 2.6.35-rc5 I have seen twice (triggered probably by
>> high load while building KOffice on an tmpfs)
>
> The trace doesn't seem to firmly point at xfs locking up.  There are
> md and dm crypt involved.  Are you sure this is locking up inside xfs?

Not 100%, but apart from the md/dm-worker the other prcesses got stuck
in sync_page() and do_lockup() and these function look filesystem
related. And the only filesystem I'm using on that system is XFS.

And having just read about some problems (that were fixed in
2.6.36-rc5) regarding XFS vs the new workqueues, I "feared" that I
somehow triggered some funny interaction between the workqueues of
XFS, md-raid1 and dm-crypt.

> Also, can you please enable frame pointers so that we can get more
> reliable backtrace?

I will do that and see if -rc6 gets stuck the same way.
But it might take until tomorrow to test this.

>> 3) the lockdep issue that started this thread and that I am seeing in
>> mainline 2.6.36-rc5 and -rc6 and that seems to be rather easy to
>> trigger for me.
>>
>> Because 3) is regarded as a false positive, it should not be the cause
>> of 2)
>
> For now, let's not draw any conclusion or mix the issues.  I'll look
> up the original thread and look into what the lockdep warning means.
>
> Thanks.
>
> --
> tejun
>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: -mm: xfs lockdep warning

[Torsten Kaiser]

[-- Attachment #1: Type: text/plain, Size: 1636 bytes --]

On Tue, Oct 5, 2010 at 6:55 PM, Tejun Heo <tj@kernel.org> wrote:
> On 10/05/2010 06:51 PM, Tejun Heo wrote:
>> On 10/05/2010 12:09 PM, Torsten Kaiser wrote:
>>> 2) the hang with 2.6.35-rc5 I have seen twice (triggered probably by
>>> high load while building KOffice on an tmpfs)
>>
>> The trace doesn't seem to firmly point at xfs locking up.  There are
>> md and dm crypt involved.  Are you sure this is locking up inside xfs?
>> Also, can you please enable frame pointers so that we can get more
>> reliable backtrace?
>>
>>> 3) the lockdep issue that started this thread and that I am seeing in
>>> mainline 2.6.36-rc5 and -rc6 and that seems to be rather easy to
>>> trigger for me.
>>>
>>> Because 3) is regarded as a false positive, it should not be the cause
>>> of 2)
>>
>> For now, let's not draw any conclusion or mix the issues.  I'll look
>> up the original thread and look into what the lockdep warning means.
>
> Ah, okay, David already looked into it and concluded it to be a
> spurious one.  Can you please attach full kernel message of the
> lockup?

Here is the complete log from bootup until the hang happened the second time.
I just logged into KDE and then started the upgrade of KOffice, which
meant that the package manager started to compile 4 programs each with
-j5. After a few minutes the load got to ~18 and probably at the point
where filecache and GCCs filled up my 4GB RAM the system got stuck.
The mouse so longer moved and SysRq+S did not work. The log was
captured via serial console on a second system.

I will try tomorrow with frame pointers enabled...

Torsten

[-- Attachment #2: xfs-lockup.txt --]
[-- Type: text/plain, Size: 71116 bytes --]

[    0.000000]  BIOS-e820: 0000000000100000 - 00000000dffd0000 (usable)
[    0.000000]  BIOS-e820: 00000000dffd0000 - 00000000dffde000 (ACPI data)
[    0.000000]  BIOS-e820: 00000000dffde000 - 00000000e0000000 (ACPI NVS)
[    0.000000]  BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
[    0.000000]  BIOS-e820: 00000000fee00000 - 00000000fef00000 (reserved)
[    0.000000]  BIOS-e820: 00000000ff700000 - 0000000100000000 (reserved)
[    0.000000]  BIOS-e820: 0000000100000000 - 0000000120000000 (usable)
[    0.000000] bootconsole [earlyser0] enabled
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] DMI present.
[    0.000000] No AGP bridge found
[    0.000000] last_pfn = 0x120000 max_arch_pfn = 0x400000000
[    0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
[    0.000000] last_pfn = 0xdffd0 max_arch_pfn = 0x400000000
[    0.000000] found SMP MP-table at [ffff8800000ff780] ff780
[    0.000000] init_memory_mapping: 0000000000000000-00000000dffd0000
[    0.000000] init_memory_mapping: 0000000100000000-0000000120000000
[    0.000000] RAMDISK: 37ec8000 - 37ff0000
[    0.000000] ACPI: RSDP 00000000000fb080 00024 (v02 ACPIAM)
[    0.000000] ACPI: XSDT 00000000dffd0100 00064 (v01 A_M_I_ OEMXSDT  04000713 MSFT 00000097)
[    0.000000] ACPI: FACP 00000000dffd0290 000F4 (v03 A_M_I_ OEMFACP  04000713 MSFT 00000097)
[    0.000000] ACPI: DSDT 00000000dffd0450 04FC5 (v01  S0027 S0027000 00000000 INTL 20051117)
[    0.000000] ACPI: FACS 00000000dffde000 00040
[    0.000000] ACPI: APIC 00000000dffd0390 00080 (v01 A_M_I_ OEMAPIC  04000713 MSFT 00000097)
[    0.000000] ACPI: MCFG 00000000dffd0410 0003C (v01 A_M_I_ OEMMCFG  04000713 MSFT 00000097)
[    0.000000] ACPI: OEMB 00000000dffde040 00060 (v01 A_M_I_ AMI_OEM  04000713 MSFT 00000097)
[    0.000000] ACPI: HPET 00000000dffd5420 00038 (v01 A_M_I_ OEMHPET0 04000713 MSFT 00000097)
[    0.000000] ACPI: MCFG 00000000dffd5460 0003C (v01 A_M_I_ OEMMCFG  04000713 MSFT 00000097)
[    0.000000] ACPI: SRAT 00000000dffd54a0 00110 (v01 AMD    HAMMER   00000001 AMD  00000001)
[    0.000000] ACPI: SSDT 00000000dffd55b0 004F0 (v01 A_M_I_ POWERNOW 00000001 AMD  00000001)
[    0.000000] SRAT: PXM 0 -> APIC 0x00 -> Node 0
[    0.000000] SRAT: PXM 0 -> APIC 0x01 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x02 -> Node 1
[    0.000000] SRAT: PXM 1 -> APIC 0x03 -> Node 1
[    0.000000] SRAT: Node 0 PXM 0 0-a0000
[    0.000000] SRAT: Node 0 PXM 0 100000-80000000
[    0.000000] SRAT: Node 1 PXM 1 80000000-e0000000
[    0.000000] SRAT: Node 1 PXM 1 100000000-120000000
[    0.000000] SRAT: Node 0 [0,a0000) + [100000,80000000) -> [0,80000000)
[    0.000000] SRAT: Node 1 [80000000,e0000000) + [100000000,120000000) -> [80000000,120000000)
[    0.000000] Initmem setup node 0 0000000000000000-0000000080000000
[    0.000000]   NODE_DATA [0000000001bb1240 - 0000000001bb323f]
[    0.000000] Initmem setup node 1 0000000080000000-0000000120000000
[    0.000000]   NODE_DATA [0000000100000000 - 0000000100001fff]
[    0.000000] Zone PFN ranges:
[    0.000000]   DMA      0x00000001 -> 0x00001000
[    0.000000]   DMA32    0x00001000 -> 0x00100000
[    0.000000]   Normal   0x00100000 -> 0x00120000
[    0.000000] Movable zone start PFN for each node
[    0.000000] early_node_map[4] active PFN ranges
[    0.000000]     0: 0x00000001 -> 0x0000009f
[    0.000000]     0: 0x00000100 -> 0x00080000
[    0.000000]     1: 0x00080000 -> 0x000dffd0
[    0.000000]     1: 0x00100000 -> 0x00120000
[    0.000000] Detected use of extended apic ids on hypertransport bus
[    0.000000] Detected use of extended apic ids on hypertransport bus
[    0.000000] ACPI: PM-Timer IO Port: 0x2008
[    0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x02] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x03] enabled)
[    0.000000] ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0])
[    0.000000] IOAPIC[0]: apic_id 4, version 17, address 0xfec00000, GSI 0-23
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 14 global_irq 14 high edge)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 15 global_irq 15 high edge)
[    0.000000] Using ACPI (MADT) for SMP configuration information
[    0.000000] ACPI: HPET id: 0x10de8201 base: 0xfed00000
[    0.000000] SMP: Allowing 4 CPUs, 0 hotplug CPUs
[    0.000000] Allocating PCI resources starting at e0000000 (gap: e0000000:1ec00000)
[    0.000000] setup_percpu: NR_CPUS:4 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:2
[    0.000000] PERCPU: Embedded 27 pages/cpu @ffff880001c00000 s78592 r8192 d23808 u1048576
[    0.000000] pcpu-alloc: s78592 r8192 d23808 u1048576 alloc=1*2097152
[    0.000000] pcpu-alloc: [0] 0 1 [1] 2 3 
[    0.000000] Built 2 zonelists in Node order, mobility grouping on.  Total pages: 1032302
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: fastboot earlyprintk=serial,ttyS0,115200 console=ttyS0,115200 console=tty1 crypt_root=/dev/md3 radeon.modeset=1 video=1280x1024
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] Checking aperture...
[    0.000000] No AGP bridge found
[    0.000000] Node 0: aperture @ 20000000 size 32 MB
[    0.000000] Aperture pointing to e820 RAM. Ignoring.
[    0.000000] Your BIOS doesn't leave a aperture memory hole
[    0.000000] Please enable the IOMMU option in the BIOS setup
[    0.000000] This costs you 64 MB of RAM
[    0.000000] Mapping aperture over 65536 KB of RAM @ 20000000
[    0.000000] Subtract (58 early reservations)
[    0.000000]   #1 [0001000000 - 0001bb03c8]   TEXT DATA BSS
[    0.000000]   #2 [0037ec8000 - 0037ff0000]         RAMDISK
[    0.000000]   #3 [0001bb1000 - 0001bb1238]             BRK
[    0.000000]   #4 [00000ff790 - 0000100000]   BIOS reserved
[    0.000000]   #5 [00000ff780 - 00000ff790]    MP-table mpf
[    0.000000]   #6 [000009fc00 - 00000f16f0]   BIOS reserved
[    0.000000]   #7 [00000f188c - 00000ff780]   BIOS reserved
[    0.000000]   #8 [00000f16f0 - 00000f188c]    MP-table mpc
[    0.000000]   #9 [0000001000 - 0000003000]      TRAMPOLINE
[    0.000000]   #10 [0000008000 - 000000c000]         PGTABLE
[    0.000000]   #11 [000000c000 - 000000d000]         PGTABLE
[    0.000000]   #12 [0001bb1240 - 0001bb3240]       NODE_DATA
[    0.000000]   #13 [0100000000 - 0100002000]       NODE_DATA
[    0.000000]   #14 [0001bb3240 - 0001bb4240]         BOOTMEM
[    0.000000]   #15 [0001bb0400 - 0001bb0580]         BOOTMEM
[    0.000000]   #16 [0080000000 - 0080000180]         BOOTMEM
[    0.000000]   #17 [00023b5000 - 00023b6000]         BOOTMEM
[    0.000000]   #18 [00023b6000 - 00023b7000]         BOOTMEM
[    0.000000]   #19 [0002400000 - 0004000000]        MEMMAP 0
[    0.000000]   #20 [0100200000 - 0102000000]        MEMMAP 1
[    0.000000]   #21 [0001bb0580 - 0001bb0700]         BOOTMEM
[    0.000000]   #22 [0001bb4240 - 0001bc0240]         BOOTMEM
[    0.000000]   #23 [0080000180 - 008000c180]         BOOTMEM
[    0.000000]   #24 [008000c180 - 008000f180]         BOOTMEM
[    0.000000]   #25 [0001bc1000 - 0001bc2000]         BOOTMEM
[    0.000000]   #26 [0001bb0700 - 0001bb0741]         BOOTMEM
[    0.000000]   #27 [0001bb0780 - 0001bb07c3]         BOOTMEM
[    0.000000]   #28 [0001bb0800 - 0001bb0a68]         BOOTMEM
[    0.000000]   #29 [0001bb0a80 - 0001bb0ae8]         BOOTMEM
[    0.000000]   #30 [0001bb0b00 - 0001bb0b68]         BOOTMEM
[    0.000000]   #31 [0001bb0b80 - 0001bb0be8]         BOOTMEM
[    0.000000]   #32 [0001bb0c00 - 0001bb0c68]         BOOTMEM
[    0.000000]   #33 [0001bb0c80 - 0001bb0ce8]         BOOTMEM
[    0.000000]   #34 [0001bb0d00 - 0001bb0d68]         BOOTMEM
[    0.000000]   #35 [0001bb0d80 - 0001bb0de8]         BOOTMEM
[    0.000000]   #36 [0001bb0e00 - 0001bb0e68]         BOOTMEM
[    0.000000]   #37 [0001bb0e80 - 0001bb0ee8]         BOOTMEM
[    0.000000]   #38 [0001bb0f00 - 0001bb0f68]         BOOTMEM
[    0.000000]   #39 [0001bb0f80 - 0001bb1000]         BOOTMEM
[    0.000000]   #40 [0001bc0240 - 0001bc02c0]         BOOTMEM
[    0.000000]   #41 [0001c00000 - 0001c1b000]         BOOTMEM
[    0.000000]   #42 [0001d00000 - 0001d1b000]         BOOTMEM
[    0.000000]   #43 [0080200000 - 008021b000]         BOOTMEM
[    0.000000]   #44 [0080300000 - 008031b000]         BOOTMEM
[    0.000000]   #45 [0001bc02c0 - 0001bc02d0]         BOOTMEM
[    0.000000]   #46 [0001bc0300 - 0001bc0310]         BOOTMEM
[    0.000000]   #47 [0001bc0340 - 0001bc0350]         BOOTMEM
[    0.000000]   #48 [0001bc0380 - 0001bc03a0]         BOOTMEM
[    0.000000]   #49 [0001bc03c0 - 0001bc0500]         BOOTMEM
[    0.000000]   #50 [0001bc0500 - 0001bc0560]         BOOTMEM
[    0.000000]   #51 [0001bc0580 - 0001bc05e0]         BOOTMEM
[    0.000000]   #52 [0001bc2000 - 0001bca000]         BOOTMEM
[    0.000000]   #53 [0020000000 - 0024000000]         BOOTMEM
[    0.000000]   #54 [0004000000 - 0008000000]         BOOTMEM
[    0.000000]   #55 [0001bca000 - 0001bea000]         BOOTMEM
[    0.000000]   #56 [0001c1b000 - 0001c5b000]         BOOTMEM
[    0.000000]   #57 [000000d000 - 0000015000]         BOOTMEM
[    0.000000] Memory: 3989044k/4718592k available (5525k kernel code, 524872k absent, 204676k reserved, 5367k data, 564k init)
[    0.000000] SLUB: Genslabs=14, HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=2
[    0.000000] Hierarchical RCU implementation.
[    0.000000] 	CONFIG_RCU_FANOUT set to non-default value of 4
[    0.000000] 	RCU dyntick-idle grace-period acceleration is enabled.
[    0.000000] 	Verbose stalled-CPUs detection is disabled.
[    0.000000] NR_IRQS:4352 nr_irqs:712
[    0.000000] Extended CMOS year: 2000
[    0.000000] Console: colour VGA+ 80x25
[    0.000000] console [tty1] enabled, bootconsole disabled
[    0.000000] Linux version 2.6.36-rc5 (root@treogen) (gcc version 4.4.4 (Gentoo 4.4.4-r2 p1.2, pie-0.4.5) ) #1 SMP Tue Sep 21 16:24:03 CEST 2010
[    0.000000] Command line: fastboot earlyprintk=serial,ttyS0,115200 console=ttyS0,115200 console=tty1 crypt_root=/dev/md3 radeon.modeset=1 video=1280x1024
[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
[    0.000000]  BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
[    0.000000]  BIOS-e820: 00000000000e4000 - 0000000000100000 (reserved)
[    0.000000]  BIOS-e820: 0000000000100000 - 00000000dffd0000 (usable)
[    0.000000]  BIOS-e820: 00000000dffd0000 - 00000000dffde000 (ACPI data)
[    0.000000]  BIOS-e820: 00000000dffde000 - 00000000e0000000 (ACPI NVS)
[    0.000000]  BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
[    0.000000]  BIOS-e820: 00000000fee00000 - 00000000fef00000 (reserved)
[    0.000000]  BIOS-e820: 00000000ff700000 - 0000000100000000 (reserved)
[    0.000000]  BIOS-e820: 0000000100000000 - 0000000120000000 (usable)
[    0.000000] bootconsole [earlyser0] enabled
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] DMI present.
[    0.000000] No AGP bridge found
[    0.000000] last_pfn = 0x120000 max_arch_pfn = 0x400000000
[    0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
[    0.000000] last_pfn = 0xdffd0 max_arch_pfn = 0x400000000
[    0.000000] found SMP MP-table at [ffff8800000ff780] ff780
[    0.000000] init_memory_mapping: 0000000000000000-00000000dffd0000
[    0.000000] init_memory_mapping: 0000000100000000-0000000120000000
[    0.000000] RAMDISK: 37ec8000 - 37ff0000
[    0.000000] ACPI: RSDP 00000000000fb080 00024 (v02 ACPIAM)
[    0.000000] ACPI: XSDT 00000000dffd0100 00064 (v01 A_M_I_ OEMXSDT  04000713 MSFT 00000097)
[    0.000000] ACPI: FACP 00000000dffd0290 000F4 (v03 A_M_I_ OEMFACP  04000713 MSFT 00000097)
[    0.000000] ACPI: DSDT 00000000dffd0450 04FC5 (v01  S0027 S0027000 00000000 INTL 20051117)
[    0.000000] ACPI: FACS 00000000dffde000 00040
[    0.000000] ACPI: APIC 00000000dffd0390 00080 (v01 A_M_I_ OEMAPIC  04000713 MSFT 00000097)
[    0.000000] ACPI: MCFG 00000000dffd0410 0003C (v01 A_M_I_ OEMMCFG  04000713 MSFT 00000097)
[    0.000000] ACPI: OEMB 00000000dffde040 00060 (v01 A_M_I_ AMI_OEM  04000713 MSFT 00000097)
[    0.000000] ACPI: HPET 00000000dffd5420 00038 (v01 A_M_I_ OEMHPET0 04000713 MSFT 00000097)
[    0.000000] ACPI: MCFG 00000000dffd5460 0003C (v01 A_M_I_ OEMMCFG  04000713 MSFT 00000097)
[    0.000000] ACPI: SRAT 00000000dffd54a0 00110 (v01 AMD    HAMMER   00000001 AMD  00000001)
[    0.000000] ACPI: SSDT 00000000dffd55b0 004F0 (v01 A_M_I_ POWERNOW 00000001 AMD  00000001)
[    0.000000] SRAT: PXM 0 -> APIC 0x00 -> Node 0
[    0.000000] SRAT: PXM 0 -> APIC 0x01 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x02 -> Node 1
[    0.000000] SRAT: PXM 1 -> APIC 0x03 -> Node 1
[    0.000000] SRAT: Node 0 PXM 0 0-a0000
[    0.000000] SRAT: Node 0 PXM 0 100000-80000000
[    0.000000] SRAT: Node 1 PXM 1 80000000-e0000000
[    0.000000] SRAT: Node 1 PXM 1 100000000-120000000
[    0.000000] SRAT: Node 0 [0,a0000) + [100000,80000000) -> [0,80000000)
[    0.000000] SRAT: Node 1 [80000000,e0000000) + [100000000,120000000) -> [80000000,120000000)
[    0.000000] Initmem setup node 0 0000000000000000-0000000080000000
[    0.000000]   NODE_DATA [0000000001bb1240 - 0000000001bb323f]
[    0.000000] Initmem setup node 1 0000000080000000-0000000120000000
[    0.000000]   NODE_DATA [0000000100000000 - 0000000100001fff]
[    0.000000] Zone PFN ranges:
[    0.000000]   DMA      0x00000001 -> 0x00001000
[    0.000000]   DMA32    0x00001000 -> 0x00100000
[    0.000000]   Normal   0x00100000 -> 0x00120000
[    0.000000] Movable zone start PFN for each node
[    0.000000] early_node_map[4] active PFN ranges
[    0.000000]     0: 0x00000001 -> 0x0000009f
[    0.000000]     0: 0x00000100 -> 0x00080000
[    0.000000]     1: 0x00080000 -> 0x000dffd0
[    0.000000]     1: 0x00100000 -> 0x00120000
[    0.000000] Detected use of extended apic ids on hypertransport bus
[    0.000000] Detected use of extended apic ids on hypertransport bus
[    0.000000] ACPI: PM-Timer IO Port: 0x2008
[    0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x02] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x03] enabled)
[    0.000000] ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0])
[    0.000000] IOAPIC[0]: apic_id 4, version 17, address 0xfec00000, GSI 0-23
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 14 global_irq 14 high edge)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 15 global_irq 15 high edge)
[    0.000000] Using ACPI (MADT) for SMP configuration information
[    0.000000] ACPI: HPET id: 0x10de8201 base: 0xfed00000
[    0.000000] SMP: Allowing 4 CPUs, 0 hotplug CPUs
[    0.000000] Allocating PCI resources starting at e0000000 (gap: e0000000:1ec00000)
[    0.000000] setup_percpu: NR_CPUS:4 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:2
[    0.000000] PERCPU: Embedded 27 pages/cpu @ffff880001c00000 s78592 r8192 d23808 u1048576
[    0.000000] pcpu-alloc: s78592 r8192 d23808 u1048576 alloc=1*2097152
[    0.000000] pcpu-alloc: [0] 0 1 [1] 2 3 
[    0.000000] Built 2 zonelists in Node order, mobility grouping on.  Total pages: 1032302
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: fastboot earlyprintk=serial,ttyS0,115200 console=ttyS0,115200 console=tty1 crypt_root=/dev/md3 radeon.modeset=1 video=1280x1024
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] Checking aperture...
[    0.000000] No AGP bridge found
[    0.000000] Node 0: aperture @ 20000000 size 32 MB
[    0.000000] Aperture pointing to e820 RAM. Ignoring.
[    0.000000] Your BIOS doesn't leave a aperture memory hole
[    0.000000] Please enable the IOMMU option in the BIOS setup
[    0.000000] This costs you 64 MB of RAM
[    0.000000] Mapping aperture over 65536 KB of RAM @ 20000000
[    0.000000] Subtract (58 early reservations)
[    0.000000]   #1 [0001000000 - 0001bb03c8]   TEXT DATA BSS
[    0.000000]   #2 [0037ec8000 - 0037ff0000]         RAMDISK
[    0.000000]   #3 [0001bb1000 - 0001bb1238]             BRK
[    0.000000]   #4 [00000ff790 - 0000100000]   BIOS reserved
[    0.000000]   #5 [00000ff780 - 00000ff790]    MP-table mpf
[    0.000000]   #6 [000009fc00 - 00000f16f0]   BIOS reserved
[    0.000000]   #7 [00000f188c - 00000ff780]   BIOS reserved
[    0.000000]   #8 [00000f16f0 - 00000f188c]    MP-table mpc
[    0.000000]   #9 [0000001000 - 0000003000]      TRAMPOLINE
[    0.000000]   #10 [0000008000 - 000000c000]         PGTABLE
[    0.000000]   #11 [000000c000 - 000000d000]         PGTABLE
[    0.000000]   #12 [0001bb1240 - 0001bb3240]       NODE_DATA
[    0.000000]   #13 [0100000000 - 0100002000]       NODE_DATA
[    0.000000]   #14 [0001bb3240 - 0001bb4240]         BOOTMEM
[    0.000000]   #15 [0001bb0400 - 0001bb0580]         BOOTMEM
[    0.000000]   #16 [0080000000 - 0080000180]         BOOTMEM
[    0.000000]   #17 [00023b5000 - 00023b6000]         BOOTMEM
[    0.000000]   #18 [00023b6000 - 00023b7000]         BOOTMEM
[    0.000000]   #19 [0002400000 - 0004000000]        MEMMAP 0
[    0.000000]   #20 [0100200000 - 0102000000]        MEMMAP 1
[    0.000000]   #21 [0001bb0580 - 0001bb0700]         BOOTMEM
[    0.000000]   #22 [0001bb4240 - 0001bc0240]         BOOTMEM
[    0.000000]   #23 [0080000180 - 008000c180]         BOOTMEM
[    0.000000]   #24 [008000c180 - 008000f180]         BOOTMEM
[    0.000000]   #25 [0001bc1000 - 0001bc2000]         BOOTMEM
[    0.000000]   #26 [0001bb0700 - 0001bb0741]         BOOTMEM
[    0.000000]   #27 [0001bb0780 - 0001bb07c3]         BOOTMEM
[    0.000000]   #28 [0001bb0800 - 0001bb0a68]         BOOTMEM
[    0.000000]   #29 [0001bb0a80 - 0001bb0ae8]         BOOTMEM
[    0.000000]   #30 [0001bb0b00 - 0001bb0b68]         BOOTMEM
[    0.000000]   #31 [0001bb0b80 - 0001bb0be8]         BOOTMEM
[    0.000000]   #32 [0001bb0c00 - 0001bb0c68]         BOOTMEM
[    0.000000]   #33 [0001bb0c80 - 0001bb0ce8]         BOOTMEM
[    0.000000]   #34 [0001bb0d00 - 0001bb0d68]         BOOTMEM
[    0.000000]   #35 [0001bb0d80 - 0001bb0de8]         BOOTMEM
[    0.000000]   #36 [0001bb0e00 - 0001bb0e68]         BOOTMEM
[    0.000000]   #37 [0001bb0e80 - 0001bb0ee8]         BOOTMEM
[    0.000000]   #38 [0001bb0f00 - 0001bb0f68]         BOOTMEM
[    0.000000]   #39 [0001bb0f80 - 0001bb1000]         BOOTMEM
[    0.000000]   #40 [0001bc0240 - 0001bc02c0]         BOOTMEM
[    0.000000]   #41 [0001c00000 - 0001c1b000]         BOOTMEM
[    0.000000]   #42 [0001d00000 - 0001d1b000]         BOOTMEM
[    0.000000]   #43 [0080200000 - 008021b000]         BOOTMEM
[    0.000000]   #44 [0080300000 - 008031b000]         BOOTMEM
[    0.000000]   #45 [0001bc02c0 - 0001bc02d0]         BOOTMEM
[    0.000000]   #46 [0001bc0300 - 0001bc0310]         BOOTMEM
[    0.000000]   #47 [0001bc0340 - 0001bc0350]         BOOTMEM
[    0.000000]   #48 [0001bc0380 - 0001bc03a0]         BOOTMEM
[    0.000000]   #49 [0001bc03c0 - 0001bc0500]         BOOTMEM
[    0.000000]   #50 [0001bc0500 - 0001bc0560]         BOOTMEM
[    0.000000]   #51 [0001bc0580 - 0001bc05e0]         BOOTMEM
[    0.000000]   #52 [0001bc2000 - 0001bca000]         BOOTMEM
[    0.000000]   #53 [0020000000 - 0024000000]         BOOTMEM
[    0.000000]   #54 [0004000000 - 0008000000]         BOOTMEM
[    0.000000]   #55 [0001bca000 - 0001bea000]         BOOTMEM
[    0.000000]   #56 [0001c1b000 - 0001c5b000]         BOOTMEM
[    0.000000]   #57 [000000d000 - 0000015000]         BOOTMEM
[    0.000000] Memory: 3989044k/4718592k available (5525k kernel code, 524872k absent, 204676k reserved, 5367k data, 564k init)
[    0.000000] SLUB: Genslabs=14, HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=2
[    0.000000] Hierarchical RCU implementation.
[    0.000000] 	CONFIG_RCU_FANOUT set to non-default value of 4
[    0.000000] 	RCU dyntick-idle grace-period acceleration is enabled.
[    0.000000] 	Verbose stalled-CPUs detection is disabled.
[    0.000000] NR_IRQS:4352 nr_irqs:712
[    0.000000] Extended CMOS year: 2000
[    0.000000] Console: colour VGA+ 80x25
[    0.000000] console [tty1] enabled, bootconsole disabled
[    0.000000] console [ttyS0] enabled
[    0.000000] Fast TSC calibration using PIT
[    0.000000] Detected 2599.982 MHz processor.
[    0.020011] Calibrating delay loop (skipped), value calculated using timer frequency.. 5199.96 BogoMIPS (lpj=25999820)
[    0.030002] pid_max: default: 32768 minimum: 301
[    0.035197] Security Framework initialized
[    0.040261] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes)
[    0.050399] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes)
[    0.058566] Mount-cache hash table entries: 256
[    0.060161] CPU: Physical Processor ID: 0
[    0.064176] CPU: Processor Core ID: 0
[    0.070002] mce: CPU supports 5 MCE banks
[    0.074022] using C1E aware idle routine
[    0.077950] Performance Events: AMD PMU driver.
[    0.081501] ... version:                0
[    0.085511] ... bit width:              48
[    0.090001] ... generic registers:      4
[    0.094016] ... value mask:             0000ffffffffffff
[    0.100001] ... max period:             00007fffffffffff
[    0.105469] ... fixed-purpose events:   0
[    0.110001] ... event mask:             000000000000000f
[    0.115336] Freeing SMP alternatives: 20k freed
[    0.120012] ACPI: Core revision 20100702
[    0.130878] Setting APIC routing to flat
[    0.135231] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.241267] CPU0: Dual-Core AMD Opteron(tm) Processor 2218 stepping 02
[    0.250000] NMI watchdog enabled, takes one hw-pmu counter.
[    0.250000] Booting Node   0, Processors  #1
[    0.410108] NMI watchdog enabled, takes one hw-pmu counter.
[    0.415953]  Ok.
[    0.417806] Booting Node   1, Processors  #2
[    0.580082] NMI watchdog enabled, takes one hw-pmu counter.
[    0.585920]  #3 Ok.
[    0.740132] NMI watchdog enabled, takes one hw-pmu counter.
[    0.745720] Brought up 4 CPUs
[    0.748697] Total of 4 processors activated (20800.19 BogoMIPS).
[    0.750392] NET: Registered protocol family 16
[    0.760023] TOM: 00000000e0000000 aka 3584M
[    0.764207] TOM2: 0000000120000000 aka 4608M
[    0.768495] ACPI: bus type pci registered
[    0.770026] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xf0000000-0xffffffff] (base 0xf0000000)
[    0.780002] PCI: not using MMCONFIG
[    0.783499] PCI: Using configuration type 1 for base access
[    0.800066] bio: create slab <bio-0> at 0
[    0.804043] ACPI: Executed 1 blocks of module-level executable AML code
[    0.813216] ACPI: Interpreter enabled
[    0.820001] ACPI: (supports S0 S5)
[    0.823476] ACPI: Using IOAPIC for interrupt routing
[    0.828474] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xf0000000-0xffffffff] (base 0xf0000000)
[    0.844447] [Firmware Bug]: PCI: MMCONFIG at [mem 0xf0000000-0xffffffff] not reserved in ACPI motherboard resources
[    0.850001] PCI: not using MMCONFIG
[    0.860438] ACPI Warning: Incorrect checksum in table [OEMB] - 0xEC, should be 0xE7 (20100702/tbutils-314)
[    0.872254] PCI: Ignoring host bridge windows from ACPI; if necessary, use "pci=use_crs" and report a bug
[    0.880227] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
[    0.891149] pci 0000:00:06.0: PCI bridge to [bus 05-05] (subtractive decode)
[    0.900166] pci 0000:04:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
[    0.910007] pci 0000:00:0b.0: PCI bridge to [bus 04-04]
[    0.930008] pci 0000:00:0c.0: PCI bridge to [bus 03-03]
[    0.950007] pci 0000:00:0d.0: PCI bridge to [bus 02-02]
[    0.955365] pci 0000:01:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
[    0.960005] pci 0000:00:0f.0: PCI bridge to [bus 01-01]
[    0.970724] ACPI: PCI Interrupt Link [LNKA] (IRQs 16 17 18 19) *11
[    0.981976] ACPI: PCI Interrupt Link [LNKB] (IRQs 16 17 18 19) *0, disabled.
[    0.991713] ACPI: PCI Interrupt Link [LNKC] (IRQs 16 17 18 19) *10
[    0.998216] ACPI: PCI Interrupt Link [LNKD] (IRQs 16 17 18 19) *0, disabled.
[    1.002878] ACPI: PCI Interrupt Link [LNEA] (IRQs 16 17 18 19) *10
[    1.011982] ACPI: PCI Interrupt Link [LNEB] (IRQs 16 17 18 19) *10
[    1.018501] ACPI: PCI Interrupt Link [LNEC] (IRQs 16 17 18 19) *0, disabled.
[    1.022874] ACPI: PCI Interrupt Link [LNED] (IRQs 16 17 18 19) *5
[    1.031907] ACPI: PCI Interrupt Link [LUB0] (IRQs 20 21 22 23) *10
[    1.038425] ACPI: PCI Interrupt Link [LMAD] (IRQs 20 21 22 23) *0, disabled.
[    1.042890] ACPI: PCI Interrupt Link [LUB2] (IRQs 20 21 22 23) *11
[    1.051988] ACPI: PCI Interrupt Link [LMAC] (IRQs 20 21 22 23) *0, disabled.
[    1.060312] ACPI: PCI Interrupt Link [LAZA] (IRQs 20 21 22 23) *11
[    1.066815] ACPI: PCI Interrupt Link [LSMB] (IRQs 20 21 22 23) *11
[    1.071974] ACPI: PCI Interrupt Link [LPMU] (IRQs 20 21 22 23) *5
[    1.080108] ACPI: PCI Interrupt Link [LSA0] (IRQs 20 21 22 23) *5
[    1.086540] ACPI: PCI Interrupt Link [LSA1] (IRQs 20 21 22 23) *10
[    1.091983] ACPI: PCI Interrupt Link [LATA] (IRQs 20 21 22 23) *0, disabled.
[    1.100315] ACPI: PCI Interrupt Link [LSA2] (IRQs 20 21 22 23) *10
[    1.106750] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none
[    1.110002] vgaarb: loaded
[    1.112789] SCSI subsystem initialized
[    1.120017] usbcore: registered new interface driver usbfs
[    1.125498] usbcore: registered new interface driver hub
[    1.130012] usbcore: registered new device driver usb
[    1.135240] Advanced Linux Sound Architecture Driver Version 1.0.23.
[    1.140001] PCI: Using ACPI for IRQ routing
[    1.144265] NetLabel: Initializing
[    1.150002] NetLabel:  domain hash size = 128
[    1.154366] NetLabel:  protocols = UNLABELED CIPSOv4
[    1.160011] NetLabel:  unlabeled traffic allowed by default
[    1.165589] HPET: 3 timers in total, 0 timers will be used for per-cpu timer
[    1.170005] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 31
[    1.175111] hpet0: 3 comparators, 32-bit 25.000000 MHz counter
[    1.210030] Switching to clocksource hpet
[    1.214052] FS-Cache: Loaded
[    1.214054] CacheFiles: Loaded
[    1.216188] pnp: PnP ACPI init
[    1.219260] ACPI: bus type pnp registered
[    1.227219] pnp: PnP ACPI: found 14 devices
[    1.231424] ACPI: ACPI bus type pnp unregistered
[    1.236057] system 00:06: [io  0x04d0-0x04d1] has been reserved
[    1.242008] system 00:06: [io  0x07b0-0x07df] has been reserved
[    1.247926] system 00:06: [io  0x0800-0x080f] has been reserved
[    1.253879] system 00:06: [io  0x0bb0-0x0bdf] has been reserved
[    1.259979] system 00:06: [io  0x2000-0x207f] has been reserved
[    1.265936] system 00:06: [io  0x2080-0x20ff] has been reserved
[    1.271872] system 00:06: [io  0x2400-0x247f] has been reserved
[    1.277790] system 00:06: [io  0x2480-0x24ff] has been reserved
[    1.283746] system 00:06: [io  0x2800-0x287f] has been reserved
[    1.289664] system 00:06: [io  0x2880-0x28ff] has been reserved
[    1.295618] system 00:06: [io  0x2c00-0x2c7f] has been reserved
[    1.301556] system 00:06: [io  0x2c80-0x2cff] has been reserved
[    1.307475] system 00:06: [mem 0x000d0000-0x000d3fff window] has been reserved
[    1.314727] system 00:06: [mem 0x000d4000-0x000d7fff window] has been reserved
[    1.322095] system 00:06: [mem 0x000de000-0x000dffff window] has been reserved
[    1.329321] system 00:06: [mem 0xfee01000-0xfeefffff] has been reserved
[    1.335971] system 00:06: [mem 0xefa80000-0xefabffff] has been reserved
[    1.342601] system 00:06: [mem 0xffb00000-0xffbfffff] has been reserved
[    1.349211] system 00:06: [mem 0xfff00000-0xffffffff] has been reserved
[    1.355864] system 00:08: [mem 0xfec00000-0xfec00fff] could not be reserved
[    1.362845] system 00:08: [mem 0xfee00000-0xfee00fff] has been reserved
[    1.369458] system 00:0c: [io  0x0290-0x0297] has been reserved
[    1.375414] system 00:0d: [mem 0x00000000-0x0009ffff] could not be reserved
[    1.382398] system 00:0d: [mem 0x000c0000-0x000cffff] has been reserved
[    1.389190] system 00:0d: [mem 0x000e0000-0x000fffff] could not be reserved
[    1.396187] system 00:0d: [mem 0x00100000-0xdfffffff] could not be reserved
[    1.403164] system 00:0d: [mem 0xfec00000-0xffffffff] could not be reserved
[    1.419451] pci 0000:00:06.0: PCI bridge to [bus 05-05]
[    1.424709] pci 0000:00:06.0:   bridge window [io  disabled]
[    1.430385] pci 0000:00:06.0:   bridge window [mem 0xeff00000-0xefffffff]
[    1.437168] pci 0000:00:06.0:   bridge window [mem 0xeef00000-0xeeffffff pref]
[    1.444427] pci 0000:00:0b.0: PCI bridge to [bus 04-04]
[    1.449653] pci 0000:00:0b.0:   bridge window [io  0xe000-0xefff]
[    1.455902] pci 0000:00:0b.0:   bridge window [mem 0xefe00000-0xefefffff]
[    1.462713] pci 0000:00:0b.0:   bridge window [mem pref disabled]
[    1.468804] pci 0000:00:0c.0: PCI bridge to [bus 03-03]
[    1.474065] pci 0000:00:0c.0:   bridge window [io  disabled]
[    1.479725] pci 0000:00:0c.0:   bridge window [mem 0xefd00000-0xefdfffff]
[    1.486546] pci 0000:00:0c.0:   bridge window [mem pref disabled]
[    1.492657] pci 0000:00:0d.0: PCI bridge to [bus 02-02]
[    1.497882] pci 0000:00:0d.0:   bridge window [io  disabled]
[    1.503575] pci 0000:00:0d.0:   bridge window [mem 0xefc00000-0xefcfffff]
[    1.510380] Clocksource tsc unstable (delta = 631633489 ns)
[    1.516068] pci 0000:00:0d.0:   bridge window [mem pref disabled]
[    1.522195] pci 0000:00:0f.0: PCI bridge to [bus 01-01]
[    1.527427] pci 0000:00:0f.0:   bridge window [io  0xd000-0xdfff]
[    1.533555] pci 0000:00:0f.0:   bridge window [mem 0xefb00000-0xefbfffff]
[    1.540368] pci 0000:00:0f.0:   bridge window [mem 0xe0000000-0xe7ffffff 64bit pref]
[    1.548211] NET: Registered protocol family 2
[    1.552764] IP route cache hash table entries: 131072 (order: 8, 1048576 bytes)
[    1.561709] TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
[    1.573951] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[    1.581402] TCP: Hash tables configured (established 524288 bind 65536)
[    1.588019] TCP reno registered
[    1.591214] UDP hash table entries: 2048 (order: 4, 65536 bytes)
[    1.597278] UDP-Lite hash table entries: 2048 (order: 4, 65536 bytes)
[    1.603887] NET: Registered protocol family 1
[    1.608385] RPC: Registered udp transport module.
[    1.613115] RPC: Registered tcp transport module.
[    1.617830] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    1.646008] pci 0000:00:00.0: Found enabled HT MSI Mapping
[    1.651559] pci 0000:00:00.0: Found enabled HT MSI Mapping
[    1.657086] pci 0000:00:00.0: Found enabled HT MSI Mapping
[    1.662642] pci 0000:00:00.0: Found enabled HT MSI Mapping
[    1.668175] pci 0000:00:00.0: Found enabled HT MSI Mapping
[    1.673742] pci 0000:00:00.0: Found enabled HT MSI Mapping
[    1.679284] pci 0000:00:00.0: Found enabled HT MSI Mapping
[    1.685475] pci 0000:00:00.0: Found enabled HT MSI Mapping
[    1.691043] pci 0000:00:00.0: Found enabled HT MSI Mapping
[    1.696623] Unpacking initramfs...
[    1.722701] Freeing initrd memory: 1184k freed
[    1.727901] PCI-DMA: Disabling AGP.
[    1.731664] PCI-DMA: aperture base @ 20000000 size 65536 KB
[    1.737235] PCI-DMA: using GART IOMMU.
[    1.741019] PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture
[    1.754121] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    1.764006] VFS: Disk quotas dquot_6.5.2
[    1.768036] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    1.774611] FS-Cache: Netfs 'nfs' registered for caching
[    1.780155] SGI XFS with ACLs, security attributes, large block/inode numbers, no debug enabled
[    1.789679] SGI XFS Quota Management subsystem
[    1.794183] msgmni has been set to 7922
[    1.798769] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 254)
[    1.806191] io scheduler noop registered
[    1.810208] io scheduler cfq registered (default)
[    1.815342] pcieport 0000:00:0b.0: ACPI _OSC control granted for 0x1d
[    1.821942] pcieport 0000:00:0c.0: ACPI _OSC control granted for 0x1d
[    1.828516] pcieport 0000:00:0d.0: ACPI _OSC control granted for 0x1d
[    1.835096] pcieport 0000:00:0f.0: ACPI _OSC control granted for 0x1d
[    1.842053] input: Power Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0C:00/input/input0
[    1.850438] ACPI: Power Button [PWRB]
[    1.854203] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input1
[    1.861623] ACPI: Power Button [PWRF]
[    1.865968] Real Time Clock Driver v1.12b
[    1.870232] Hangcheck: starting hangcheck timer 0.9.1 (tick is 180 seconds, margin is 60 seconds).
[    1.879180] Hangcheck: Using getrawmonotonic().
[    1.883806] [drm] Initialized drm 1.1.0 20060810
[    1.888487] [drm] radeon kernel modesetting enabled.
[    1.893721] ACPI: PCI Interrupt Link [LNEB] enabled at IRQ 19
[    1.899478] radeon 0000:01:00.0: PCI INT A -> Link[LNEB] -> GSI 19 (level, low) -> IRQ 19
[    1.909871] [drm] initializing kernel modesetting (RV380 0x1002:0x5B60).
[    1.916673] [drm] register mmio base: 0xEFBF0000
[    1.921322] [drm] register mmio size: 65536
[    1.926142] [drm] Generation 2 PCI interface, using max accessible memory
[    1.932959] radeon 0000:01:00.0: VRAM: 128M 0xE0000000 - 0xE7FFFFFF (128M used)
[    1.940288] radeon 0000:01:00.0: GTT: 512M 0xC0000000 - 0xDFFFFFFF
[    1.946513] radeon 0000:01:00.0: radeon: using MSI.
[    1.951443] [drm] radeon: irq initialized.
[    1.956254] [drm] Detected VRAM RAM=128M, BAR=128M
[    1.961100] [drm] RAM width 64bits DDR
[    1.964935] [TTM] Zone  kernel: Available graphics memory: 2028100 kiB.
[    1.971566] [TTM] Initializing pool allocator.
[    1.976038] [drm] radeon: 128M of VRAM memory ready
[    1.980945] [drm] radeon: 512M of GTT memory ready.
[    1.985841] [drm] GART: num cpu pages 131072, num gpu pages 131072
[    1.993082] [drm] radeon: 1 quad pipes, 1 Z pipes initialized.
[    1.999658] [drm] PCIE GART of 512M enabled (table at 0xE0040000).
[    2.005989] [drm] Loading R300 Microcode
[    2.010098] [drm] radeon: ring at 0x00000000C0000000
[    2.015083] [drm] ring test succeeded in 1 usecs
[    2.019782] [drm] radeon: ib pool ready.
[    2.023812] [drm] ib test succeeded in 0 usecs
[    2.028621] [drm] Default TV standard: NTSC
[    2.032823] [drm] 27.000000000 MHz TV ref clk
[    2.037185] [drm] DFP table revision: 4
[    2.041146] [drm] Default TV standard: NTSC
[    2.045330] [drm] 27.000000000 MHz TV ref clk
[    2.049804] [drm] Radeon Display Connectors
[    2.054004] [drm] Connector 0:
[    2.057068] [drm]   VGA
[    2.059526] [drm]   DDC: 0x60 0x60 0x60 0x60 0x60 0x60 0x60 0x60
[    2.065556] [drm]   Encoders:
[    2.068536] [drm]     CRT1: INTERNAL_DAC1
[    2.072568] [drm] Connector 1:
[    2.075632] [drm]   DVI-I
[    2.078261] [drm]   HPD1
[    2.080826] [drm]   DDC: 0x64 0x64 0x64 0x64 0x64 0x64 0x64 0x64
[    2.086832] [drm]   Encoders:
[    2.089807] [drm]     CRT2: INTERNAL_DAC2
[    2.093853] [drm]     DFP1: INTERNAL_TMDS1
[    2.097955] [drm] Connector 2:
[    2.101037] [drm]   S-video
[    2.103843] [drm]   Encoders:
[    2.106818] [drm]     TV1: INTERNAL_DAC2
[    2.242976] [drm] fb mappable at 0xE00C0000
[    2.247162] [drm] vram apper at 0xE0000000
[    2.251290] [drm] size 5242880
[    2.254357] [drm] fb depth is 24
[    2.257592] [drm]    pitch is 5120
[    2.261107] fbcon: radeondrmfb (fb0) is primary device
[    2.381151] Console: switching to colour frame buffer device 160x64
[    2.393850] fb0: radeondrmfb frame buffer device
[    2.398488] drm: registered panic notifier
[    2.402632] [drm] Initialized radeon 2.6.0 20080528 for 0000:01:00.0 on minor 0
[    2.410120] Serial: 8250/16550 driver, 1 ports, IRQ sharing disabled
[    2.437083] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[    2.463858] 00:09: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[    2.469955] sata_sil24 0000:04:00.0: PCI INT A -> Link[LNEB] -> GSI 19 (level, low) -> IRQ 19
[    2.478822] scsi0 : sata_sil24
[    2.482089] scsi1 : sata_sil24
[    2.485287] ata1: SATA max UDMA/100 host m128@0xefeffc00 port 0xefef8000 irq 19
[    2.492650] ata2: SATA max UDMA/100 host m128@0xefeffc00 port 0xefefa000 irq 19
[    2.500476] ACPI: PCI Interrupt Link [LSA0] enabled at IRQ 23
[    2.506257] sata_nv 0000:00:05.0: PCI INT A -> Link[LSA0] -> GSI 23 (level, low) -> IRQ 23
[    2.514594] sata_nv 0000:00:05.0: Using SWNCQ mode
[    2.519577] scsi2 : sata_nv
[    2.522540] scsi3 : sata_nv
[    2.525518] ata3: SATA max UDMA/133 cmd 0xcc00 ctl 0xc880 bmdma 0xc400 irq 23
[    2.532706] ata4: SATA max UDMA/133 cmd 0xc800 ctl 0xc480 bmdma 0xc408 irq 23
[    2.540126] ACPI: PCI Interrupt Link [LSA1] enabled at IRQ 22
[    2.545906] sata_nv 0000:00:05.1: PCI INT B -> Link[LSA1] -> GSI 22 (level, low) -> IRQ 22
[    2.554244] sata_nv 0000:00:05.1: Using SWNCQ mode
[    2.559214] scsi4 : sata_nv
[    2.562195] scsi5 : sata_nv
[    2.565156] ata5: SATA max UDMA/133 cmd 0xc080 ctl 0xc000 bmdma 0xb800 irq 22
[    2.572340] ata6: SATA max UDMA/133 cmd 0xbc00 ctl 0xb880 bmdma 0xb808 irq 22
[    2.579742] ACPI: PCI Interrupt Link [LSA2] enabled at IRQ 21
[    2.585543] sata_nv 0000:00:05.2: PCI INT C -> Link[LSA2] -> GSI 21 (level, low) -> IRQ 21
[    2.593867] sata_nv 0000:00:05.2: Using SWNCQ mode
[    2.598857] scsi6 : sata_nv
[    2.601843] scsi7 : sata_nv
[    2.604817] ata7: SATA max UDMA/133 cmd 0xb480 ctl 0xb400 bmdma 0xac00 irq 21
[    2.611999] ata8: SATA max UDMA/133 cmd 0xb080 ctl 0xb000 bmdma 0xac08 irq 21
[    2.619457] scsi8 : pata_amd
[    2.622499] scsi9 : pata_amd
[    2.627895] ata9: PATA max UDMA/133 cmd 0x1f0 ctl 0x3f6 bmdma 0xffa0 irq 14
[    2.636448] ata10: PATA max UDMA/133 cmd 0x170 ctl 0x376 bmdma 0xffa8 irq 15
[    2.645289] tg3.c:v3.113 (August 2, 2010)
[    2.651160] ACPI: PCI Interrupt Link [LNEA] enabled at IRQ 18
[    2.658412] tg3 0000:03:00.0: PCI INT A -> Link[LNEA] -> GSI 18 (level, low) -> IRQ 18
[    2.705556] tg3 0000:03:00.0: eth0: Tigon3 [partno(BCM95700A6) rev b002] (PCI Express) MAC address 00:17:31:f9:cc:c5
[    2.717828] tg3 0000:03:00.0: eth0: attached PHY is 5787 (10/100/1000Base-T Ethernet) (WireSpeed[1])
[    2.728773] tg3 0000:03:00.0: eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] TSOcap[1]
[    2.738767] tg3 0000:03:00.0: eth0: dma_rwctrl[76180000] dma_mask[64-bit]
[    2.747648] ACPI: PCI Interrupt Link [LNED] enabled at IRQ 17
[    2.755492] tg3 0000:02:00.0: PCI INT A -> Link[LNED] -> GSI 17 (level, low) -> IRQ 17
[    2.804492] tg3 0000:02:00.0: eth1: Tigon3 [partno(BCM95700A6) rev b002] (PCI Express) MAC address 00:18:f3:44:df:ea
[    2.817081] tg3 0000:02:00.0: eth1: attached PHY is 5787 (10/100/1000Base-T Ethernet) (WireSpeed[1])
[    2.828016] tg3 0000:02:00.0: eth1: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] TSOcap[1]
[    2.838052] tg3 0000:02:00.0: eth1: dma_rwctrl[76180000] dma_mask[64-bit]
[    2.847417] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 16
[    2.855170] ohci1394 0000:05:08.0: PCI INT A -> Link[LNKA] -> GSI 16 (level, low) -> IRQ 16
[    2.900036] ata5: SATA link down (SStatus 0 SControl 300)
[    2.920053] ohci1394: fw-host0: OHCI-1394 1.1 (PCI): IRQ=[16]  MMIO=[effff800-efffffff]  Max Packet=[2048]  IR/IT contexts=[4/8]
[    2.934544] eth1394: eth2: IPv4 over IEEE 1394 (fw-host0)
[    2.942300] ata7: SATA link down (SStatus 0 SControl 300)
[    2.942441] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    2.942656] ACPI: PCI Interrupt Link [LUB2] enabled at IRQ 20
[    2.942669] ehci_hcd 0000:00:02.1: PCI INT B -> Link[LUB2] -> GSI 20 (level, low) -> IRQ 20
[    2.942683] ehci_hcd 0000:00:02.1: EHCI Host Controller
[    2.942691] ehci_hcd 0000:00:02.1: new USB bus registered, assigned bus number 1
[    2.942714] ehci_hcd 0000:00:02.1: debug port 1
[    2.942744] ehci_hcd 0000:00:02.1: irq 20, io mem 0xefaebc00
[    2.970014] ehci_hcd 0000:00:02.1: USB 2.0 started, EHCI 1.00
[    2.970032] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[    2.970034] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.970036] usb usb1: Product: EHCI Host Controller
[    2.970037] usb usb1: Manufacturer: Linux 2.6.36-rc5 ehci_hcd
[    2.970039] usb usb1: SerialNumber: 0000:00:02.1
[    2.970203] hub 1-0:1.0: USB hub found
[    2.970207] hub 1-0:1.0: 10 ports detected
[    2.970326] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    2.970542] ACPI: PCI Interrupt Link [LUB0] enabled at IRQ 23
[    2.970545] ohci_hcd 0000:00:02.0: PCI INT A -> Link[LUB0] -> GSI 23 (level, low) -> IRQ 23
[    2.970560] ohci_hcd 0000:00:02.0: OHCI Host Controller
[    2.970566] ohci_hcd 0000:00:02.0: new USB bus registered, assigned bus number 2
[    2.970578] ohci_hcd 0000:00:02.0: irq 23, io mem 0xefaf8000
[    3.042026] usb usb2: New USB device found, idVendor=1d6b, idProduct=0001
[    3.042028] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    3.042030] usb usb2: Product: OHCI Host Controller
[    3.042032] usb usb2: Manufacturer: Linux 2.6.36-rc5 ohci_hcd
[    3.042033] usb usb2: SerialNumber: 0000:00:02.0
[    3.042159] hub 2-0:1.0: USB hub found
[    3.042164] hub 2-0:1.0: 10 ports detected
[    3.042271] Initializing USB Mass Storage driver...
[    3.042338] usbcore: registered new interface driver usb-storage
[    3.042340] USB Mass Storage support registered.
[    3.042404] usbcore: registered new interface driver libusual
[    3.042456] usbcore: registered new interface driver ums-alauda
[    3.042511] usbcore: registered new interface driver ums-cypress
[    3.042563] usbcore: registered new interface driver ums-datafab
[    3.042618] usbcore: registered new interface driver ums-freecom
[    3.042669] usbcore: registered new interface driver ums-isd200
[    3.042721] usbcore: registered new interface driver ums-jumpshot
[    3.042777] usbcore: registered new interface driver ums-karma
[    3.042829] usbcore: registered new interface driver ums-sddr09
[    3.042880] usbcore: registered new interface driver ums-sddr55
[    3.042934] usbcore: registered new interface driver ums-usbat
[    3.043059] PNP: PS/2 Controller [PNP0303:PS2K,PNP0f03:PS2M] at 0x60,0x64 irq 1,12
[    3.045588] serio: i8042 KBD port at 0x60,0x64 irq 1
[    3.045595] serio: i8042 AUX port at 0x60,0x64 irq 12
[    3.045855] mice: PS/2 mouse device common for all mice
[    3.046185] input: PC Speaker as /devices/platform/pcspkr/input/input2
[    3.046284] I2O subsystem v1.325
[    3.046285] i2o: max drivers = 8
[    3.046470] i2c /dev entries driver
[    3.046933] i2c i2c-4: nForce2 SMBus adapter at 0x2a00
[    3.047038] i2c i2c-5: nForce2 SMBus adapter at 0x2a80
[    3.047196] lirc_dev: IR Remote Control driver registered, major 253 
[    3.047198] IR NEC protocol handler initialized
[    3.047199] IR RC5(x) protocol handler initialized
[    3.047200] IR RC6 protocol handler initialized
[    3.047202] IR JVC protocol handler initialized
[    3.047203] IR Sony protocol handler initialized
[    3.047204] IR LIRC bridge handler initialized
[    3.047205] Linux video capture interface: v2.00
[    3.047570] bttv: driver version 0.9.18 loaded
[    3.047572] bttv: using 8 buffers with 2080k (520 pages) each for capture
[    3.047666] bttv: Bt8xx card found (0).
[    3.047857] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 19
[    3.047860] bttv 0000:05:06.0: PCI INT A -> Link[LNKC] -> GSI 19 (level, low) -> IRQ 19
[    3.047867] bttv0: Bt878 (rev 17) at 0000:05:06.0, irq: 19, latency: 64, mmio: 0xeefff000
[    3.047881] bttv0: detected: Hauppauge WinTV [card=10], PCI subsystem ID is 0070:13eb
[    3.047883] bttv0: using: Hauppauge (bt878) [card=10,autodetected]
[    3.050412] bttv0: Hauppauge/Voodoo msp34xx: reset line init [5]
[    3.066621] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input3
[    3.084419] tveeprom 6-0050: Hauppauge model 61344, rev D421, serial# 3902813
[    3.084421] tveeprom 6-0050: tuner model is Philips FM1216 (idx 21, type 5)
[    3.084424] tveeprom 6-0050: TV standards PAL(B/G) (eeprom 0x04)
[    3.084426] tveeprom 6-0050: audio processor is MSP3415 (idx 6)
[    3.084427] tveeprom 6-0050: has radio
[    3.084429] bttv0: Hauppauge eeprom indicates model#61344
[    3.084430] bttv0: tuner type=5
[    3.097785] msp3400 6-0040: MSP3415D-B3 found @ 0x80 (bt878 #0 [sw])
[    3.097787] msp3400 6-0040: msp3400 supports nicam, mode is autodetect
[    3.109810] tuner 6-0061: chip found @ 0xc2 (bt878 #0 [sw])
[    3.110103] tuner-simple 6-0061: creating new instance
[    3.110106] tuner-simple 6-0061: type set to 5 (Philips PAL_BG (FI1216 and compatibles))
[    3.110811] bttv0: registered device video0
[    3.110866] bttv0: registered device vbi0
[    3.110919] bttv0: registered device radio0
[    3.112429] bttv0: PLL: 28636363 => 35468950 .
[    3.603026] ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[    3.610075] . ok
[    3.635040] Registered IR keymap rc-rc5-tv
[    3.641059] input: i2c IR (Hauppauge) as /devices/virtual/rc/rc0/input4
[    3.649554] rc0: i2c IR (Hauppauge) as /devices/virtual/rc/rc0
[    3.656914] ir-kbd-i2c: i2c IR (Hauppauge) detected at i2c-6/6-0018/ir0 [bt878 #0 [sw]]
[    3.675611] ata3.00: ATA-7: MAXTOR STM3320820AS, 3.AAE, max UDMA/133
[    3.683771] ata3.00: 625142448 sectors, multi 16: LBA48 NCQ (depth 31/32)
[    3.758899] ata3.00: configured for UDMA/133
[    4.070019] usb 2-6: new low speed USB device using ohci_hcd and address 2
[    4.258036] usb 2-6: New USB device found, idVendor=062a, idProduct=0000
[    4.266620] usb 2-6: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[    4.500081] k8temp 0000:00:18.3: Temperature readouts might be wrong - check erratum #141
[    4.510043] k8temp 0000:00:19.3: Temperature readouts might be wrong - check erratum #141
[    4.520230] md: raid1 personality registered for level 1
[    4.527821] device-mapper: uevent: version 1.0.3
[    4.534492] device-mapper: ioctl: 4.18.0-ioctl (2010-06-29) initialised: dm-devel@redhat.com
[    4.544918] EDAC MC: Ver: 2.1.0 Sep 21 2010
[    4.551330] EDAC amd64_edac:  Ver: 3.3.0 Sep 21 2010
[    4.558278] EDAC amd64: ECC is enabled by BIOS.
[    4.564784] EDAC amd64: ECC is enabled by BIOS.
[    4.571822] EDAC MC0: Giving out device to 'amd64_edac' 'RevF': DEV 0000:00:18.2
[    4.581340] EDAC MC1: Giving out device to 'amd64_edac' 'RevF': DEV 0000:00:19.2
[    4.591090] EDAC PCI0: Giving out device to module 'amd64_edac' controller 'EDAC PCI controller': DEV '0000:00:18.2' (POLLED)
[    4.604470] cpuidle: using governor ladder
[    4.610553] cpuidle: using governor menu
[    4.618253] usbcore: registered new interface driver hiddev
[    4.631332] input: HID 062a:0000 as /devices/pci0000:00/0000:00:02.0/usb2/2-6/2-6:1.0/input/input5
[    4.642674] generic-usb 0003:062A:0000.0001: input,hidraw0: USB HID v1.10 Mouse [HID 062a:0000] on usb-0000:00:02.0-6/input0
[    4.656445] usbcore: registered new interface driver usbhid
[    4.664405] usbhid: USB HID core driver
[    4.670639] Bt87x 0000:05:06.1: PCI INT A -> Link[LNKC] -> GSI 19 (level, low) -> IRQ 19
[    4.681301] bt87x0: Using board 1, analog, digital (rate 32000 Hz)
[    4.690294] ACPI: PCI Interrupt Link [LAZA] enabled at IRQ 22
[    4.698571] HDA Intel 0000:00:06.1: PCI INT B -> Link[LAZA] -> GSI 22 (level, low) -> IRQ 22
[    4.709150] hda_intel: Disable MSI for Nvidia chipset
[    4.730054] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 0)
[    4.744574] ata1.00: ATA-7: SAMSUNG HD642JJ, 1AA01113, max UDMA7
[    4.752850] ata1.00: 1250263728 sectors, multi 0: LBA48 NCQ (depth 31/32)
[    4.768610] ata1.00: configured for UDMA/100
[    4.775591] scsi 0:0:0:0: Direct-Access     ATA      SAMSUNG HD642JJ  1AA0 PQ: 0 ANSI: 5
[    4.786182] sd 0:0:0:0: [sda] 1250263728 512-byte logical blocks: (640 GB/596 GiB)
[    4.796285] sd 0:0:0:0: [sda] Write Protect is off
[    4.803438] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    4.834245]  sda: sda1 sda2 sda3
[    4.839996] sd 0:0:0:0: [sda] Attached SCSI disk
[    6.260528] ALSA device list:
[    6.265987]   #0: HDA NVidia at 0xefafc000 irq 22
[    6.272887]   #1: Brooktree Bt878 at 0xeeffe000, irq 19
[    6.280987] Initializing XFRM netlink socket
[    6.287498] NET: Registered protocol family 17
[    6.294482] NET: Registered protocol family 15
[    6.301255] Registering the dns_resolver key type
[    6.308132] powernow-k8: Found 2 Dual-Core AMD Opteron(tm) Processor 2218 (4 cpu cores) (version 2.20.00)
[    6.320221] powernow-k8:    0 : fid 0x12 (2600 MHz), vid 0x8
[    6.328449] powernow-k8:    1 : fid 0x10 (2400 MHz), vid 0xa
[    6.336387] powernow-k8:    2 : fid 0xe (2200 MHz), vid 0xc
[    6.344569] powernow-k8:    3 : fid 0xc (2000 MHz), vid 0xe
[    6.352350] powernow-k8:    4 : fid 0xa (1800 MHz), vid 0x10
[    6.360550] powernow-k8:    5 : fid 0x2 (1000 MHz), vid 0x12
[    6.368862] powernow-k8:    0 : fid 0x12 (2600 MHz), vid 0x8
[    6.377476] powernow-k8:    1 : fid 0x10 (2400 MHz), vid 0xa
[    6.385197] powernow-k8:    2 : fid 0xe (2200 MHz), vid 0xc
[    6.393015] powernow-k8:    3 : fid 0xc (2000 MHz), vid 0xe
[    6.400598] powernow-k8:    4 : fid 0xa (1800 MHz), vid 0x10
[    6.408116] powernow-k8:    5 : fid 0x2 (1000 MHz), vid 0x12
[    7.000051] ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 0)
[    7.016333] ata2.00: ATA-7: SAMSUNG HD642JJ, 1AA01113, max UDMA7
[    7.026243] ata2.00: 1250263728 sectors, multi 0: LBA48 NCQ (depth 31/32)
[    7.043365] ata2.00: configured for UDMA/100
[    7.051789] scsi 1:0:0:0: Direct-Access     ATA      SAMSUNG HD642JJ  1AA0 PQ: 0 ANSI: 5
[    7.064244] sd 1:0:0:0: [sdb] 1250263728 512-byte logical blocks: (640 GB/596 GiB)
[    7.064583] scsi 2:0:0:0: Direct-Access     ATA      MAXTOR STM332082 3.AA PQ: 0 ANSI: 5
[    7.065024] sd 2:0:0:0: [sdc] 625142448 512-byte logical blocks: (320 GB/298 GiB)
[    7.065129] sd 2:0:0:0: [sdc] Write Protect is off
[    7.065183] sd 2:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    7.080725]  sdc: sdc1 sdc2
[    7.127963] sd 1:0:0:0: [sdb] Write Protect is off
[    7.128309] sd 2:0:0:0: [sdc] Attached SCSI disk
[    7.145445] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    7.177685]  sdb: sdb1 sdb2 sdb3
[    7.185858] sd 1:0:0:0: [sdb] Attached SCSI disk
[    7.400022] ata4: SATA link down (SStatus 0 SControl 300)
[    7.730028] ata6: SATA link down (SStatus 0 SControl 300)
[    8.060027] ata8: SATA link down (SStatus 0 SControl 300)
[    8.069502] Freeing unused kernel memory: 564k freed
[    8.078937] Write protecting the kernel read-only data: 10240k
[    8.088937] Freeing unused kernel memory: 596k freed
[    8.098543] Freeing unused kernel memory: 1728k freed
Sep 25 07:44:57 treogen syslog-ng[2716]: syslog-ng starting up; version='3.1.2'
Sep 25 07:45:02 treogen mdadm[2864]: NewArray event detected on md device /dev/md2
Sep 25 07:45:02 treogen mdadm[2864]: NewArray event detected on md device /dev/md3
Sep 25 07:45:09 treogen ntpdate[2875]: step time server 192.168.3.2 offset -1.399604 sec
Sep 25 07:45:09 treogen rpcbind: cannot bind 127.0.0.1 on udp: Address already in use
Sep 25 07:45:09 treogen rpcbind: cannot create socket for udp6
Sep 25 07:45:09 treogen rpcbind: cannot create socket for tcp6
Sep 25 07:45:10 treogen kdm: :0[2637]: pam_unix(kde:session): session opened for user tk13 by (uid=0)
Sep 25 07:45:10 treogen sm-notify[2944]: Version 1.2.2 starting
Sep 25 07:45:12 treogen postfix/postfix-script[3069]: starting the Postfix mail system
Sep 25 07:45:12 treogen postfix/master[3070]: daemon started -- version 2.6.7, configuration /etc/postfix
Sep 25 07:45:12 treogen smartd[3087]: smartd 5.39.1 2010-01-28 r3054 [x86_64-pc-linux-gnu] (local build)
Sep 25 07:45:12 treogen Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net
Sep 25 07:45:12 treogen smartd[3087]: Opened configuration file /etc/smartd.conf
Sep 25 07:45:12 treogen smartd[3087]: Drive: /dev/sda, implied '-a' Directive on line 24 of file /etc/smartd.conf
Sep 25 07:45:12 treogen smartd[3087]: Drive: /dev/sdb, implied '-a' Directive on line 25 of file /etc/smartd.conf
Sep 25 07:45:12 treogen smartd[3087]: Drive: /dev/sdc, implied '-a' Directive on line 26 of file /etc/smartd.conf
Sep 25 07:45:12 treogen smartd[3087]: Configuration file /etc/smartd.conf parsed.
Sep 25 07:45:12 treogen smartd[3087]: Device: /dev/sda, opened
Sep 25 07:45:12 treogen smartd[3087]: Device: /dev/sda, found in smartd database.
Sep 25 07:45:13 treogen smartd[3087]: Device: /dev/sda, is SMART capable. Adding to "monitor" list.
Sep 25 07:45:13 treogen smartd[3087]: Device: /dev/sdb, opened
Sep 25 07:45:13 treogen smartd[3087]: Device: /dev/sdb, found in smartd database.
Sep 25 07:45:13 treogen smartd[3087]: Device: /dev/sdb, is SMART capable. Adding to "monitor" list.
Sep 25 07:45:13 treogen smartd[3087]: Device: /dev/sdc, opened
Sep 25 07:45:13 treogen smartd[3087]: Device: /dev/sdc, found in smartd database.
Sep 25 07:45:13 treogen smartd[3087]: Device: /dev/sdc, is SMART capable. Adding to "monitor" list.
Sep 25 07:45:13 treogen smartd[3087]: Monitoring 3 ATA and 0 SCSI devices
Sep 25 07:45:13 treogen smartd[3100]: smartd has fork()ed into background mode. New PID=3100.
Sep 25 07:45:13 treogen smartd[3100]: file /var/run/smartd.pid written containing PID 3100
Sep 25 07:45:15 treogen cron[3128]: (CRON) STARTUP (V5.0)
Sep 25 07:46:08 treogen polkitd[3277]: started daemon version 0.96 using authority implementation `local' version `0.96'
Sep 25 07:46:08 treogen polkitd(authority=local): Registered Authentication Agent for session /org/freedesktop/ConsoleKit/Session1 (system bus name :1.11 [/usr/lib64/kde4/libexec/polkit-kde-authentication-agent-1 -session 10dbd7d36f000127743216200000067470009_1285365114_26164], object path /org/kde/PolicyKit1/AuthenticationAgent, locale en_GB.utf8)
Sep 25 07:46:14 treogen pulseaudio[3342]: pid.c: Daemon already running.
Sep 25 07:46:14 treogen pulseaudio[3343]: pid.c: Daemon already running.
Sep 25 07:46:14 treogen pulseaudio[3345]: pid.c: Daemon already running.
Sep 25 07:46:32 treogen su[3385]: Successful su for root by tk13
Sep 25 07:46:32 treogen su[3385]: + /dev/pts/3 tk13:root
Sep 25 07:46:32 treogen su[3385]: pam_unix(su:session): session opened for user root by tk13(uid=1000)
Sep 25 07:50:01 treogen cron[3444]: (root) CMD (test -x /usr/sbin/run-crons && /usr/sbin/run-crons )
[ 1033.312407] SysRq : Show Blocked State
[ 1035.876602] SysRq : Changing Loglevel
[ 1035.880405] Loglevel set to 6
[ 1081.760056] INFO: task X:2585 blocked for more than 120 seconds.
[ 1081.766302] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1081.774410]  ffff88011f2b9c70 0000000000003082 ffff880100000001 ffff880100000003
[ 1081.782146]  ffff8800078dd550 0000000000012780 ffff88011e031fd8 0000000000012780
[ 1081.789850]  ffff88011e031fd8 ffff88011f2b9ed0 ffff88011e031fd8 ffff88011f2b9ec8
[ 1081.797606] Call Trace:
[ 1081.800166]  [<ffffffff810801c0>] ? sync_page+0x0/0x50
[ 1081.805509]  [<ffffffff8156078d>] ? io_schedule+0x3d/0x60
[ 1081.811129]  [<ffffffff810801fd>] ? sync_page+0x3d/0x50
[ 1081.816556]  [<ffffffff81560b7a>] ? __wait_on_bit_lock+0x4a/0xa0
[ 1081.822802]  [<ffffffff8108019e>] ? __lock_page+0x5e/0x70
[ 1081.828420]  [<ffffffff81052ea0>] ? wake_bit_function+0x0/0x40
[ 1081.834488]  [<ffffffff81099ce7>] ? handle_mm_fault+0xa47/0xa60
[ 1081.840641]  [<ffffffff813d4b0c>] ? evdev_read+0xec/0x270
[ 1081.846258]  [<ffffffff810cd91f>] ? do_vfs_ioctl+0x9f/0x530
[ 1081.852077]  [<ffffffff8102404d>] ? do_page_fault+0x10d/0x320
[ 1081.858052]  [<ffffffff8156265f>] ? page_fault+0x1f/0x30
[ 1081.863591] INFO: task hald-addon-inpu:2805 blocked for more than 120 seconds.
[ 1081.871107] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1081.879245]  ffff8800071cf1c0 0000000000000082 ffff880100000001 ffff880000000003
[ 1081.887012]  ffff88007f71c440 0000000000012780 ffff88000417bfd8 0000000000012780
[ 1081.894777]  ffff88000417bfd8 ffff8800071cf420 ffff88000417bfd8 ffff8800071cf418
[ 1081.902517] Call Trace:
[ 1081.905078]  [<ffffffff810801c0>] ? sync_page+0x0/0x50
[ 1081.910443]  [<ffffffff8156078d>] ? io_schedule+0x3d/0x60
[ 1081.916048]  [<ffffffff810801fd>] ? sync_page+0x3d/0x50
[ 1081.921492]  [<ffffffff81560b7a>] ? __wait_on_bit_lock+0x4a/0xa0
[ 1081.927731]  [<ffffffff8108019e>] ? __lock_page+0x5e/0x70
[ 1081.933346]  [<ffffffff81052ea0>] ? wake_bit_function+0x0/0x40
[ 1081.939414]  [<ffffffff81099ce7>] ? handle_mm_fault+0xa47/0xa60
[ 1081.945575]  [<ffffffff815602ac>] ? schedule+0x24c/0x6f0
[ 1081.951277]  [<ffffffff8102404d>] ? do_page_fault+0x10d/0x320
[ 1081.957103]  [<ffffffff8156265f>] ? page_fault+0x1f/0x30
[ 1081.962494]  [<ffffffff810cf29e>] ? do_sys_poll+0x2be/0x4b0
[ 1081.968126]  [<ffffffff810cf285>] ? do_sys_poll+0x2a5/0x4b0
[ 1081.973787]  [<ffffffff810cecf0>] ? __pollwait+0x0/0x110
[ 1081.979167]  [<ffffffff810cee00>] ? pollwake+0x0/0x60
[ 1081.984298]  [<ffffffff810cee00>] ? pollwake+0x0/0x60
[ 1081.989421]  [<ffffffff810cee00>] ? pollwake+0x0/0x60
[ 1081.994541]  [<ffffffff810cee00>] ? pollwake+0x0/0x60
[ 1081.999665]  [<ffffffff810cee00>] ? pollwake+0x0/0x60
[ 1082.004779]  [<ffffffff810c22ed>] ? chrdev_open+0xdd/0x180
[ 1082.010346]  [<ffffffff81080e68>] ? find_get_page+0x18/0xa0
[ 1082.015992]  [<ffffffff810819e9>] ? filemap_fault+0x99/0x3f0
[ 1082.021733]  [<ffffffff814a3efa>] ? verify_iovec+0x6a/0xc0
[ 1082.027277]  [<ffffffff810973ea>] ? __do_fault+0x3da/0x4b0
[ 1082.032848]  [<ffffffff8156244e>] ? common_interrupt+0xe/0x13
[ 1082.038674]  [<ffffffff813d4b63>] ? evdev_read+0x143/0x270
[ 1082.044236]  [<ffffffff813d28e2>] ? input_event_to_user+0x62/0x70
[ 1082.050416]  [<ffffffff810cf691>] ? sys_poll+0x71/0x100
[ 1082.055712]  [<ffffffff8156265f>] ? page_fault+0x1f/0x30
[ 1082.061165]  [<ffffffff8100246b>] ? system_call_fastpath+0x16/0x1b
[ 1082.067654] INFO: task plasma-desktop:3169 blocked for more than 120 seconds.
[ 1082.075166] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1082.083425]  ffff88011ef860b0 0000000000000086 000000011ef860b0 ffff880114faff48
[ 1082.091234]  ffff8800071cf1c0 0000000000012780 ffff880114faffd8 0000000000012780
[ 1082.099051]  ffff880114faffd8 ffff88011ef86310 ffff880114faffd8 ffff88011ef86308
[ 1082.106887] Call Trace:
[ 1082.109467]  [<ffffffff815612c1>] ? __mutex_lock_slowpath+0xe1/0x160
[ 1082.116165]  [<ffffffff810d1e57>] ? __d_lookup+0x97/0x120
[ 1082.121849]  [<ffffffff81560d8a>] ? mutex_lock+0x1a/0x40
[ 1082.127439]  [<ffffffff810c9dbf>] ? do_lookup+0x10f/0x190
[ 1082.133099]  [<ffffffff810c7e24>] ? acl_permission_check+0x54/0xb0
[ 1082.139598]  [<ffffffff810ca8d9>] ? link_path_walk+0x4c9/0x9a0
[ 1082.145733]  [<ffffffff810d65f4>] ? mnt_want_write+0x34/0x70
[ 1082.151678]  [<ffffffff810caecf>] ? path_walk+0x5f/0xe0
[ 1082.157174]  [<ffffffff810caf9b>] ? do_path_lookup+0x4b/0x50
[ 1082.163110]  [<ffffffff810cbc35>] ? user_path_at+0x55/0xb0
[ 1082.168884]  [<ffffffff8102e860>] ? __dequeue_entity+0x40/0x50
[ 1082.175022]  [<ffffffff8107bcf4>] ? perf_event_task_sched_out+0x44/0x260
[ 1082.182056]  [<ffffffff810c2e46>] ? vfs_fstatat+0x36/0x80
[ 1082.187742]  [<ffffffff815602ac>] ? schedule+0x24c/0x6f0
[ 1082.193331]  [<ffffffff810c2f8f>] ? sys_newstat+0x1f/0x50
[ 1082.199019]  [<ffffffff81003035>] ? device_not_available+0x15/0x20
[ 1082.205535]  [<ffffffff8100246b>] ? system_call_fastpath+0x16/0x1b
[ 1082.212034] INFO: task virtuoso-t:3362 blocked for more than 120 seconds.
[ 1082.219163] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1082.227412]  ffff8800df8ac440 0000000000000082 ffff880100000001 ffff880000000003
[ 1082.235231]  ffff8800ba6e60b0 0000000000012780 ffff8800de265fd8 0000000000012780
[ 1082.243065]  ffff8800de265fd8 ffff8800df8ac6a0 ffff8800de265fd8 ffff8800df8ac698
[ 1082.250864] Call Trace:
[ 1082.253445]  [<ffffffff810801c0>] ? sync_page+0x0/0x50
[ 1082.258844]  [<ffffffff8156078d>] ? io_schedule+0x3d/0x60
[ 1082.264508]  [<ffffffff810801fd>] ? sync_page+0x3d/0x50
[ 1082.269989]  [<ffffffff81560b7a>] ? __wait_on_bit_lock+0x4a/0xa0
[ 1082.276296]  [<ffffffff8108019e>] ? __lock_page+0x5e/0x70
[ 1082.281984]  [<ffffffff81052ea0>] ? wake_bit_function+0x0/0x40
[ 1082.288119]  [<ffffffff81099ce7>] ? handle_mm_fault+0xa47/0xa60
[ 1082.294333]  [<ffffffff815602ac>] ? schedule+0x24c/0x6f0
[ 1082.299905]  [<ffffffff8102404d>] ? do_page_fault+0x10d/0x320
[ 1082.305936]  [<ffffffff8156265f>] ? page_fault+0x1f/0x30
[ 1082.311541] INFO: task cc1plus:12626 blocked for more than 120 seconds.
[ 1082.318473] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1082.326698]  ffff88003ecf2d80 0000000000000082 ffff880100000001 ffff880000000003
[ 1082.334552]  ffff88001f95b8e0 0000000000012780 ffff88000ec89fd8 0000000000012780
[ 1082.342384]  ffff88000ec89fd8 ffff88003ecf2fe0 ffff88000ec89fd8 ffff88003ecf2fd8
[ 1082.350246] Call Trace:
[ 1082.352825]  [<ffffffff810801c0>] ? sync_page+0x0/0x50
[ 1082.358213]  [<ffffffff8156078d>] ? io_schedule+0x3d/0x60
[ 1082.363882]  [<ffffffff810801fd>] ? sync_page+0x3d/0x50
[ 1082.369375]  [<ffffffff81560b7a>] ? __wait_on_bit_lock+0x4a/0xa0
[ 1082.375701]  [<ffffffff8108019e>] ? __lock_page+0x5e/0x70
[ 1082.381397]  [<ffffffff81052ea0>] ? wake_bit_function+0x0/0x40
[ 1082.387526]  [<ffffffff81099ce7>] ? handle_mm_fault+0xa47/0xa60
[ 1082.393765]  [<ffffffff8102404d>] ? do_page_fault+0x10d/0x320
[ 1082.399786]  [<ffffffff8156265f>] ? page_fault+0x1f/0x30
[ 1202.400046] INFO: task kworker/u:8:845 blocked for more than 120 seconds.
[ 1202.407128] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1202.415372]  ffff88007ffa8b60 0000000000000046 ffff88007f74fe98 0000000000000000
[ 1202.423229]  ffff8800071cf770 0000000000012780 ffff8800070e9fd8 0000000000012780
[ 1202.431054]  ffff8800070e9fd8 ffff88007ffa8dc0 ffff8800070e9fd8 ffff88007ffa8db8
[ 1202.438898] Call Trace:
[ 1202.441495]  [<ffffffff814361ed>] ? md_write_start+0x9d/0x190
[ 1202.447536]  [<ffffffff81052e70>] ? autoremove_wake_function+0x0/0x30
[ 1202.454311]  [<ffffffff810925ca>] ? zone_nr_free_pages+0x9a/0xb0
[ 1202.460639]  [<ffffffff8142c423>] ? make_request+0x23/0x800
[ 1202.466496]  [<ffffffff811d90ea>] ? blkcipher_walk_done+0x8a/0x230
[ 1202.472990]  [<ffffffff811df4c4>] ? crypto_cbc_encrypt+0xe4/0x180
[ 1202.479327]  [<ffffffff8102a150>] ? twofish_encrypt+0x0/0x10
[ 1202.485276]  [<ffffffff81431ac3>] ? md_make_request+0xc3/0x220
[ 1202.491412]  [<ffffffff811eafca>] ? generic_make_request+0x18a/0x330
[ 1202.498096]  [<ffffffff814484fc>] ? crypt_convert+0x25c/0x310
[ 1202.504090]  [<ffffffff81448881>] ? kcryptd_crypt+0x2d1/0x3f0
[ 1202.510031]  [<ffffffff814485b0>] ? kcryptd_crypt+0x0/0x3f0
[ 1202.516081]  [<ffffffff8104d2eb>] ? process_one_work+0xfb/0x370
[ 1202.522123]  [<ffffffff8104ee4c>] ? worker_thread+0x16c/0x360
[ 1202.527935]  [<ffffffff8104ece0>] ? worker_thread+0x0/0x360
[ 1202.533541]  [<ffffffff8104ece0>] ? worker_thread+0x0/0x360
[ 1202.539131]  [<ffffffff810529e6>] ? kthread+0x96/0xa0
[ 1202.544245]  [<ffffffff81003194>] ? kernel_thread_helper+0x4/0x10
[ 1202.550363]  [<ffffffff81052950>] ? kthread+0x0/0xa0
[ 1202.555346]  [<ffffffff81003190>] ? kernel_thread_helper+0x0/0x10
[ 1202.561479] INFO: task X:2585 blocked for more than 120 seconds.
[ 1202.567503] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1202.575468]  ffff88011f2b9c70 0000000000003082 ffff880100000001 ffff880100000003
[ 1202.583060]  ffff8800078dd550 0000000000012780 ffff88011e031fd8 0000000000012780
[ 1202.590757]  ffff88011e031fd8 ffff88011f2b9ed0 ffff88011e031fd8 ffff88011f2b9ec8
[ 1202.598538] Call Trace:
[ 1202.601059]  [<ffffffff810801c0>] ? sync_page+0x0/0x50
[ 1202.606381]  [<ffffffff8156078d>] ? io_schedule+0x3d/0x60
[ 1202.611854]  [<ffffffff810801fd>] ? sync_page+0x3d/0x50
[ 1202.617210]  [<ffffffff81560b7a>] ? __wait_on_bit_lock+0x4a/0xa0
[ 1202.623684]  [<ffffffff8108019e>] ? __lock_page+0x5e/0x70
[ 1202.629300]  [<ffffffff81052ea0>] ? wake_bit_function+0x0/0x40
[ 1202.635396]  [<ffffffff81099ce7>] ? handle_mm_fault+0xa47/0xa60
[ 1202.641606]  [<ffffffff813d4b0c>] ? evdev_read+0xec/0x270
[ 1202.647275]  [<ffffffff810cd91f>] ? do_vfs_ioctl+0x9f/0x530
[ 1202.653125]  [<ffffffff8102404d>] ? do_page_fault+0x10d/0x320
[ 1202.659105]  [<ffffffff8156265f>] ? page_fault+0x1f/0x30
[ 1202.664690] INFO: task hald-addon-inpu:2805 blocked for more than 120 seconds.
[ 1202.672282] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1202.680507]  ffff8800071cf1c0 0000000000000082 ffff880100000001 ffff880000000003
[ 1202.688289]  ffff88007f71c440 0000000000012780 ffff88000417bfd8 0000000000012780
[ 1202.696100]  ffff88000417bfd8 ffff8800071cf420 ffff88000417bfd8 ffff8800071cf418
[ 1202.703924] Call Trace:
[ 1202.706503]  [<ffffffff810801c0>] ? sync_page+0x0/0x50
[ 1202.711901]  [<ffffffff8156078d>] ? io_schedule+0x3d/0x60
[ 1202.717577]  [<ffffffff810801fd>] ? sync_page+0x3d/0x50
[ 1202.723081]  [<ffffffff81560b7a>] ? __wait_on_bit_lock+0x4a/0xa0
[ 1202.729381]  [<ffffffff8108019e>] ? __lock_page+0x5e/0x70
[ 1202.735031]  [<ffffffff81052ea0>] ? wake_bit_function+0x0/0x40
[ 1202.741116]  [<ffffffff81099ce7>] ? handle_mm_fault+0xa47/0xa60
[ 1202.747340]  [<ffffffff815602ac>] ? schedule+0x24c/0x6f0
[ 1202.752902]  [<ffffffff8102404d>] ? do_page_fault+0x10d/0x320
[ 1202.758893]  [<ffffffff8156265f>] ? page_fault+0x1f/0x30
[ 1202.764428]  [<ffffffff810cf29e>] ? do_sys_poll+0x2be/0x4b0
[ 1202.770338]  [<ffffffff810cf285>] ? do_sys_poll+0x2a5/0x4b0
[ 1202.776113]  [<ffffffff810cecf0>] ? __pollwait+0x0/0x110
[ 1202.781591]  [<ffffffff810cee00>] ? pollwake+0x0/0x60
[ 1202.786764]  [<ffffffff810cee00>] ? pollwake+0x0/0x60
[ 1202.791973]  [<ffffffff810cee00>] ? pollwake+0x0/0x60
[ 1202.797181]  [<ffffffff810cee00>] ? pollwake+0x0/0x60
[ 1202.802468]  [<ffffffff810cee00>] ? pollwake+0x0/0x60
[ 1202.807773]  [<ffffffff810c22ed>] ? chrdev_open+0xdd/0x180
[ 1202.813545]  [<ffffffff81080e68>] ? find_get_page+0x18/0xa0
[ 1202.819384]  [<ffffffff810819e9>] ? filemap_fault+0x99/0x3f0
[ 1202.825326]  [<ffffffff814a3efa>] ? verify_iovec+0x6a/0xc0
[ 1202.831095]  [<ffffffff810973ea>] ? __do_fault+0x3da/0x4b0
[ 1202.836847]  [<ffffffff8156244e>] ? common_interrupt+0xe/0x13
[ 1202.842865]  [<ffffffff813d4b63>] ? evdev_read+0x143/0x270
[ 1202.848626]  [<ffffffff813d28e2>] ? input_event_to_user+0x62/0x70
[ 1202.855032]  [<ffffffff810cf691>] ? sys_poll+0x71/0x100
[ 1202.860499]  [<ffffffff8156265f>] ? page_fault+0x1f/0x30
[ 1202.866005]  [<ffffffff8100246b>] ? system_call_fastpath+0x16/0x1b
[ 1202.872424] INFO: task plasma-desktop:3169 blocked for more than 120 seconds.
[ 1202.879883] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1202.888118]  ffff88011ef860b0 0000000000000086 000000011ef860b0 ffff880114faff48
[ 1202.895936]  ffff8800071cf1c0 0000000000012780 ffff880114faffd8 0000000000012780
[ 1202.903777]  ffff880114faffd8 ffff88011ef86310 ffff880114faffd8 ffff88011ef86308
[ 1202.911586] Call Trace:
[ 1202.914140]  [<ffffffff815612c1>] ? __mutex_lock_slowpath+0xe1/0x160
[ 1202.920693]  [<ffffffff810d1e57>] ? __d_lookup+0x97/0x120
[ 1202.926272]  [<ffffffff81560d8a>] ? mutex_lock+0x1a/0x40
[ 1202.931783]  [<ffffffff810c9dbf>] ? do_lookup+0x10f/0x190
[ 1202.937426]  [<ffffffff810c7e24>] ? acl_permission_check+0x54/0xb0
[ 1202.943898]  [<ffffffff810ca8d9>] ? link_path_walk+0x4c9/0x9a0
[ 1202.950028]  [<ffffffff810d65f4>] ? mnt_want_write+0x34/0x70
[ 1202.955947]  [<ffffffff810caecf>] ? path_walk+0x5f/0xe0
[ 1202.961416]  [<ffffffff810caf9b>] ? do_path_lookup+0x4b/0x50
[ 1202.967325]  [<ffffffff810cbc35>] ? user_path_at+0x55/0xb0
[ 1202.973020]  [<ffffffff8102e860>] ? __dequeue_entity+0x40/0x50
[ 1202.979054]  [<ffffffff8107bcf4>] ? perf_event_task_sched_out+0x44/0x260
[ 1202.985995]  [<ffffffff810c2e46>] ? vfs_fstatat+0x36/0x80
[ 1202.991600]  [<ffffffff815602ac>] ? schedule+0x24c/0x6f0
[ 1202.997157]  [<ffffffff810c2f8f>] ? sys_newstat+0x1f/0x50
[ 1203.002784]  [<ffffffff81003035>] ? device_not_available+0x15/0x20
[ 1203.009204]  [<ffffffff8100246b>] ? system_call_fastpath+0x16/0x1b
[ 1203.015608] INFO: task virtuoso-t:3362 blocked for more than 120 seconds.
[ 1203.022633] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1203.030726]  ffff8800df8ac440 0000000000000082 ffff880100000001 ffff880000000003
[ 1203.038449]  ffff8800ba6e60b0 0000000000012780 ffff8800de265fd8 0000000000012780
[ 1203.046215]  ffff8800de265fd8 ffff8800df8ac6a0 ffff8800de265fd8 ffff8800df8ac698
[ 1203.053988] Call Trace:
[ 1203.056532]  [<ffffffff810801c0>] ? sync_page+0x0/0x50
[ 1203.061836]  [<ffffffff8156078d>] ? io_schedule+0x3d/0x60
[ 1203.067390]  [<ffffffff810801fd>] ? sync_page+0x3d/0x50
[ 1203.072775]  [<ffffffff81560b7a>] ? __wait_on_bit_lock+0x4a/0xa0
[ 1203.078926]  [<ffffffff8108019e>] ? __lock_page+0x5e/0x70
[ 1203.084552]  [<ffffffff81052ea0>] ? wake_bit_function+0x0/0x40
[ 1203.090670]  [<ffffffff81099ce7>] ? handle_mm_fault+0xa47/0xa60
[ 1203.096867]  [<ffffffff815602ac>] ? schedule+0x24c/0x6f0
[ 1203.102439]  [<ffffffff8102404d>] ? do_page_fault+0x10d/0x320
[ 1203.108462]  [<ffffffff8156265f>] ? page_fault+0x1f/0x30
[ 1229.421480] SysRq : Emergency Sync
[ 1237.014575] SysRq : Emergency Sync
[ 1239.121784] SysRq : Emergency Remount R/O
[ 1243.234840] SysRq : Emergency Sync
[ 1246.327060] SysRq : Power Off

Re: -mm: xfs lockdep warning

[Torsten Kaiser]

[-- Attachment #1: Type: text/plain, Size: 1636 bytes --]

On Tue, Oct 5, 2010 at 6:55 PM, Tejun Heo <tj@kernel.org> wrote:
> On 10/05/2010 06:51 PM, Tejun Heo wrote:
>> On 10/05/2010 12:09 PM, Torsten Kaiser wrote:
>>> 2) the hang with 2.6.35-rc5 I have seen twice (triggered probably by
>>> high load while building KOffice on an tmpfs)
>>
>> The trace doesn't seem to firmly point at xfs locking up.  There are
>> md and dm crypt involved.  Are you sure this is locking up inside xfs?
>> Also, can you please enable frame pointers so that we can get more
>> reliable backtrace?
>>
>>> 3) the lockdep issue that started this thread and that I am seeing in
>>> mainline 2.6.36-rc5 and -rc6 and that seems to be rather easy to
>>> trigger for me.
>>>
>>> Because 3) is regarded as a false positive, it should not be the cause
>>> of 2)
>>
>> For now, let's not draw any conclusion or mix the issues.  I'll look
>> up the original thread and look into what the lockdep warning means.
>
> Ah, okay, David already looked into it and concluded it to be a
> spurious one.  Can you please attach full kernel message of the
> lockup?

Here is the complete log from bootup until the hang happened the second time.
I just logged into KDE and then started the upgrade of KOffice, which
meant that the package manager started to compile 4 programs each with
-j5. After a few minutes the load got to ~18 and probably at the point
where filecache and GCCs filled up my 4GB RAM the system got stuck.
The mouse so longer moved and SysRq+S did not work. The log was
captured via serial console on a second system.

I will try tomorrow with frame pointers enabled...

Torsten

[-- Attachment #2: xfs-lockup.txt --]
[-- Type: text/plain, Size: 71116 bytes --]

[    0.000000]  BIOS-e820: 0000000000100000 - 00000000dffd0000 (usable)
[    0.000000]  BIOS-e820: 00000000dffd0000 - 00000000dffde000 (ACPI data)
[    0.000000]  BIOS-e820: 00000000dffde000 - 00000000e0000000 (ACPI NVS)
[    0.000000]  BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
[    0.000000]  BIOS-e820: 00000000fee00000 - 00000000fef00000 (reserved)
[    0.000000]  BIOS-e820: 00000000ff700000 - 0000000100000000 (reserved)
[    0.000000]  BIOS-e820: 0000000100000000 - 0000000120000000 (usable)
[    0.000000] bootconsole [earlyser0] enabled
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] DMI present.
[    0.000000] No AGP bridge found
[    0.000000] last_pfn = 0x120000 max_arch_pfn = 0x400000000
[    0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
[    0.000000] last_pfn = 0xdffd0 max_arch_pfn = 0x400000000
[    0.000000] found SMP MP-table at [ffff8800000ff780] ff780
[    0.000000] init_memory_mapping: 0000000000000000-00000000dffd0000
[    0.000000] init_memory_mapping: 0000000100000000-0000000120000000
[    0.000000] RAMDISK: 37ec8000 - 37ff0000
[    0.000000] ACPI: RSDP 00000000000fb080 00024 (v02 ACPIAM)
[    0.000000] ACPI: XSDT 00000000dffd0100 00064 (v01 A_M_I_ OEMXSDT  04000713 MSFT 00000097)
[    0.000000] ACPI: FACP 00000000dffd0290 000F4 (v03 A_M_I_ OEMFACP  04000713 MSFT 00000097)
[    0.000000] ACPI: DSDT 00000000dffd0450 04FC5 (v01  S0027 S0027000 00000000 INTL 20051117)
[    0.000000] ACPI: FACS 00000000dffde000 00040
[    0.000000] ACPI: APIC 00000000dffd0390 00080 (v01 A_M_I_ OEMAPIC  04000713 MSFT 00000097)
[    0.000000] ACPI: MCFG 00000000dffd0410 0003C (v01 A_M_I_ OEMMCFG  04000713 MSFT 00000097)
[    0.000000] ACPI: OEMB 00000000dffde040 00060 (v01 A_M_I_ AMI_OEM  04000713 MSFT 00000097)
[    0.000000] ACPI: HPET 00000000dffd5420 00038 (v01 A_M_I_ OEMHPET0 04000713 MSFT 00000097)
[    0.000000] ACPI: MCFG 00000000dffd5460 0003C (v01 A_M_I_ OEMMCFG  04000713 MSFT 00000097)
[    0.000000] ACPI: SRAT 00000000dffd54a0 00110 (v01 AMD    HAMMER   00000001 AMD  00000001)
[    0.000000] ACPI: SSDT 00000000dffd55b0 004F0 (v01 A_M_I_ POWERNOW 00000001 AMD  00000001)
[    0.000000] SRAT: PXM 0 -> APIC 0x00 -> Node 0
[    0.000000] SRAT: PXM 0 -> APIC 0x01 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x02 -> Node 1
[    0.000000] SRAT: PXM 1 -> APIC 0x03 -> Node 1
[    0.000000] SRAT: Node 0 PXM 0 0-a0000
[    0.000000] SRAT: Node 0 PXM 0 100000-80000000
[    0.000000] SRAT: Node 1 PXM 1 80000000-e0000000
[    0.000000] SRAT: Node 1 PXM 1 100000000-120000000
[    0.000000] SRAT: Node 0 [0,a0000) + [100000,80000000) -> [0,80000000)
[    0.000000] SRAT: Node 1 [80000000,e0000000) + [100000000,120000000) -> [80000000,120000000)
[    0.000000] Initmem setup node 0 0000000000000000-0000000080000000
[    0.000000]   NODE_DATA [0000000001bb1240 - 0000000001bb323f]
[    0.000000] Initmem setup node 1 0000000080000000-0000000120000000
[    0.000000]   NODE_DATA [0000000100000000 - 0000000100001fff]
[    0.000000] Zone PFN ranges:
[    0.000000]   DMA      0x00000001 -> 0x00001000
[    0.000000]   DMA32    0x00001000 -> 0x00100000
[    0.000000]   Normal   0x00100000 -> 0x00120000
[    0.000000] Movable zone start PFN for each node
[    0.000000] early_node_map[4] active PFN ranges
[    0.000000]     0: 0x00000001 -> 0x0000009f
[    0.000000]     0: 0x00000100 -> 0x00080000
[    0.000000]     1: 0x00080000 -> 0x000dffd0
[    0.000000]     1: 0x00100000 -> 0x00120000
[    0.000000] Detected use of extended apic ids on hypertransport bus
[    0.000000] Detected use of extended apic ids on hypertransport bus
[    0.000000] ACPI: PM-Timer IO Port: 0x2008
[    0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x02] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x03] enabled)
[    0.000000] ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0])
[    0.000000] IOAPIC[0]: apic_id 4, version 17, address 0xfec00000, GSI 0-23
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 14 global_irq 14 high edge)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 15 global_irq 15 high edge)
[    0.000000] Using ACPI (MADT) for SMP configuration information
[    0.000000] ACPI: HPET id: 0x10de8201 base: 0xfed00000
[    0.000000] SMP: Allowing 4 CPUs, 0 hotplug CPUs
[    0.000000] Allocating PCI resources starting at e0000000 (gap: e0000000:1ec00000)
[    0.000000] setup_percpu: NR_CPUS:4 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:2
[    0.000000] PERCPU: Embedded 27 pages/cpu @ffff880001c00000 s78592 r8192 d23808 u1048576
[    0.000000] pcpu-alloc: s78592 r8192 d23808 u1048576 alloc=1*2097152
[    0.000000] pcpu-alloc: [0] 0 1 [1] 2 3 
[    0.000000] Built 2 zonelists in Node order, mobility grouping on.  Total pages: 1032302
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: fastboot earlyprintk=serial,ttyS0,115200 console=ttyS0,115200 console=tty1 crypt_root=/dev/md3 radeon.modeset=1 video=1280x1024
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] Checking aperture...
[    0.000000] No AGP bridge found
[    0.000000] Node 0: aperture @ 20000000 size 32 MB
[    0.000000] Aperture pointing to e820 RAM. Ignoring.
[    0.000000] Your BIOS doesn't leave a aperture memory hole
[    0.000000] Please enable the IOMMU option in the BIOS setup
[    0.000000] This costs you 64 MB of RAM
[    0.000000] Mapping aperture over 65536 KB of RAM @ 20000000
[    0.000000] Subtract (58 early reservations)
[    0.000000]   #1 [0001000000 - 0001bb03c8]   TEXT DATA BSS
[    0.000000]   #2 [0037ec8000 - 0037ff0000]         RAMDISK
[    0.000000]   #3 [0001bb1000 - 0001bb1238]             BRK
[    0.000000]   #4 [00000ff790 - 0000100000]   BIOS reserved
[    0.000000]   #5 [00000ff780 - 00000ff790]    MP-table mpf
[    0.000000]   #6 [000009fc00 - 00000f16f0]   BIOS reserved
[    0.000000]   #7 [00000f188c - 00000ff780]   BIOS reserved
[    0.000000]   #8 [00000f16f0 - 00000f188c]    MP-table mpc
[    0.000000]   #9 [0000001000 - 0000003000]      TRAMPOLINE
[    0.000000]   #10 [0000008000 - 000000c000]         PGTABLE
[    0.000000]   #11 [000000c000 - 000000d000]         PGTABLE
[    0.000000]   #12 [0001bb1240 - 0001bb3240]       NODE_DATA
[    0.000000]   #13 [0100000000 - 0100002000]       NODE_DATA
[    0.000000]   #14 [0001bb3240 - 0001bb4240]         BOOTMEM
[    0.000000]   #15 [0001bb0400 - 0001bb0580]         BOOTMEM
[    0.000000]   #16 [0080000000 - 0080000180]         BOOTMEM
[    0.000000]   #17 [00023b5000 - 00023b6000]         BOOTMEM
[    0.000000]   #18 [00023b6000 - 00023b7000]         BOOTMEM
[    0.000000]   #19 [0002400000 - 0004000000]        MEMMAP 0
[    0.000000]   #20 [0100200000 - 0102000000]        MEMMAP 1
[    0.000000]   #21 [0001bb0580 - 0001bb0700]         BOOTMEM
[    0.000000]   #22 [0001bb4240 - 0001bc0240]         BOOTMEM
[    0.000000]   #23 [0080000180 - 008000c180]         BOOTMEM
[    0.000000]   #24 [008000c180 - 008000f180]         BOOTMEM
[    0.000000]   #25 [0001bc1000 - 0001bc2000]         BOOTMEM
[    0.000000]   #26 [0001bb0700 - 0001bb0741]         BOOTMEM
[    0.000000]   #27 [0001bb0780 - 0001bb07c3]         BOOTMEM
[    0.000000]   #28 [0001bb0800 - 0001bb0a68]         BOOTMEM
[    0.000000]   #29 [0001bb0a80 - 0001bb0ae8]         BOOTMEM
[    0.000000]   #30 [0001bb0b00 - 0001bb0b68]         BOOTMEM
[    0.000000]   #31 [0001bb0b80 - 0001bb0be8]         BOOTMEM
[    0.000000]   #32 [0001bb0c00 - 0001bb0c68]         BOOTMEM
[    0.000000]   #33 [0001bb0c80 - 0001bb0ce8]         BOOTMEM
[    0.000000]   #34 [0001bb0d00 - 0001bb0d68]         BOOTMEM
[    0.000000]   #35 [0001bb0d80 - 0001bb0de8]         BOOTMEM
[    0.000000]   #36 [0001bb0e00 - 0001bb0e68]         BOOTMEM
[    0.000000]   #37 [0001bb0e80 - 0001bb0ee8]         BOOTMEM
[    0.000000]   #38 [0001bb0f00 - 0001bb0f68]         BOOTMEM
[    0.000000]   #39 [0001bb0f80 - 0001bb1000]         BOOTMEM
[    0.000000]   #40 [0001bc0240 - 0001bc02c0]         BOOTMEM
[    0.000000]   #41 [0001c00000 - 0001c1b000]         BOOTMEM
[    0.000000]   #42 [0001d00000 - 0001d1b000]         BOOTMEM
[    0.000000]   #43 [0080200000 - 008021b000]         BOOTMEM
[    0.000000]   #44 [0080300000 - 008031b000]         BOOTMEM
[    0.000000]   #45 [0001bc02c0 - 0001bc02d0]         BOOTMEM
[    0.000000]   #46 [0001bc0300 - 0001bc0310]         BOOTMEM
[    0.000000]   #47 [0001bc0340 - 0001bc0350]         BOOTMEM
[    0.000000]   #48 [0001bc0380 - 0001bc03a0]         BOOTMEM
[    0.000000]   #49 [0001bc03c0 - 0001bc0500]         BOOTMEM
[    0.000000]   #50 [0001bc0500 - 0001bc0560]         BOOTMEM
[    0.000000]   #51 [0001bc0580 - 0001bc05e0]         BOOTMEM
[    0.000000]   #52 [0001bc2000 - 0001bca000]         BOOTMEM
[    0.000000]   #53 [0020000000 - 0024000000]         BOOTMEM
[    0.000000]   #54 [0004000000 - 0008000000]         BOOTMEM
[    0.000000]   #55 [0001bca000 - 0001bea000]         BOOTMEM
[    0.000000]   #56 [0001c1b000 - 0001c5b000]         BOOTMEM
[    0.000000]   #57 [000000d000 - 0000015000]         BOOTMEM
[    0.000000] Memory: 3989044k/4718592k available (5525k kernel code, 524872k absent, 204676k reserved, 5367k data, 564k init)
[    0.000000] SLUB: Genslabs=14, HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=2
[    0.000000] Hierarchical RCU implementation.
[    0.000000] 	CONFIG_RCU_FANOUT set to non-default value of 4
[    0.000000] 	RCU dyntick-idle grace-period acceleration is enabled.
[    0.000000] 	Verbose stalled-CPUs detection is disabled.
[    0.000000] NR_IRQS:4352 nr_irqs:712
[    0.000000] Extended CMOS year: 2000
[    0.000000] Console: colour VGA+ 80x25
[    0.000000] console [tty1] enabled, bootconsole disabled
[    0.000000] Linux version 2.6.36-rc5 (root@treogen) (gcc version 4.4.4 (Gentoo 4.4.4-r2 p1.2, pie-0.4.5) ) #1 SMP Tue Sep 21 16:24:03 CEST 2010
[    0.000000] Command line: fastboot earlyprintk=serial,ttyS0,115200 console=ttyS0,115200 console=tty1 crypt_root=/dev/md3 radeon.modeset=1 video=1280x1024
[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
[    0.000000]  BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
[    0.000000]  BIOS-e820: 00000000000e4000 - 0000000000100000 (reserved)
[    0.000000]  BIOS-e820: 0000000000100000 - 00000000dffd0000 (usable)
[    0.000000]  BIOS-e820: 00000000dffd0000 - 00000000dffde000 (ACPI data)
[    0.000000]  BIOS-e820: 00000000dffde000 - 00000000e0000000 (ACPI NVS)
[    0.000000]  BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
[    0.000000]  BIOS-e820: 00000000fee00000 - 00000000fef00000 (reserved)
[    0.000000]  BIOS-e820: 00000000ff700000 - 0000000100000000 (reserved)
[    0.000000]  BIOS-e820: 0000000100000000 - 0000000120000000 (usable)
[    0.000000] bootconsole [earlyser0] enabled
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] DMI present.
[    0.000000] No AGP bridge found
[    0.000000] last_pfn = 0x120000 max_arch_pfn = 0x400000000
[    0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
[    0.000000] last_pfn = 0xdffd0 max_arch_pfn = 0x400000000
[    0.000000] found SMP MP-table at [ffff8800000ff780] ff780
[    0.000000] init_memory_mapping: 0000000000000000-00000000dffd0000
[    0.000000] init_memory_mapping: 0000000100000000-0000000120000000
[    0.000000] RAMDISK: 37ec8000 - 37ff0000
[    0.000000] ACPI: RSDP 00000000000fb080 00024 (v02 ACPIAM)
[    0.000000] ACPI: XSDT 00000000dffd0100 00064 (v01 A_M_I_ OEMXSDT  04000713 MSFT 00000097)
[    0.000000] ACPI: FACP 00000000dffd0290 000F4 (v03 A_M_I_ OEMFACP  04000713 MSFT 00000097)
[    0.000000] ACPI: DSDT 00000000dffd0450 04FC5 (v01  S0027 S0027000 00000000 INTL 20051117)
[    0.000000] ACPI: FACS 00000000dffde000 00040
[    0.000000] ACPI: APIC 00000000dffd0390 00080 (v01 A_M_I_ OEMAPIC  04000713 MSFT 00000097)
[    0.000000] ACPI: MCFG 00000000dffd0410 0003C (v01 A_M_I_ OEMMCFG  04000713 MSFT 00000097)
[    0.000000] ACPI: OEMB 00000000dffde040 00060 (v01 A_M_I_ AMI_OEM  04000713 MSFT 00000097)
[    0.000000] ACPI: HPET 00000000dffd5420 00038 (v01 A_M_I_ OEMHPET0 04000713 MSFT 00000097)
[    0.000000] ACPI: MCFG 00000000dffd5460 0003C (v01 A_M_I_ OEMMCFG  04000713 MSFT 00000097)
[    0.000000] ACPI: SRAT 00000000dffd54a0 00110 (v01 AMD    HAMMER   00000001 AMD  00000001)
[    0.000000] ACPI: SSDT 00000000dffd55b0 004F0 (v01 A_M_I_ POWERNOW 00000001 AMD  00000001)
[    0.000000] SRAT: PXM 0 -> APIC 0x00 -> Node 0
[    0.000000] SRAT: PXM 0 -> APIC 0x01 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 0x02 -> Node 1
[    0.000000] SRAT: PXM 1 -> APIC 0x03 -> Node 1
[    0.000000] SRAT: Node 0 PXM 0 0-a0000
[    0.000000] SRAT: Node 0 PXM 0 100000-80000000
[    0.000000] SRAT: Node 1 PXM 1 80000000-e0000000
[    0.000000] SRAT: Node 1 PXM 1 100000000-120000000
[    0.000000] SRAT: Node 0 [0,a0000) + [100000,80000000) -> [0,80000000)
[    0.000000] SRAT: Node 1 [80000000,e0000000) + [100000000,120000000) -> [80000000,120000000)
[    0.000000] Initmem setup node 0 0000000000000000-0000000080000000
[    0.000000]   NODE_DATA [0000000001bb1240 - 0000000001bb323f]
[    0.000000] Initmem setup node 1 0000000080000000-0000000120000000
[    0.000000]   NODE_DATA [0000000100000000 - 0000000100001fff]
[    0.000000] Zone PFN ranges:
[    0.000000]   DMA      0x00000001 -> 0x00001000
[    0.000000]   DMA32    0x00001000 -> 0x00100000
[    0.000000]   Normal   0x00100000 -> 0x00120000
[    0.000000] Movable zone start PFN for each node
[    0.000000] early_node_map[4] active PFN ranges
[    0.000000]     0: 0x00000001 -> 0x0000009f
[    0.000000]     0: 0x00000100 -> 0x00080000
[    0.000000]     1: 0x00080000 -> 0x000dffd0
[    0.000000]     1: 0x00100000 -> 0x00120000
[    0.000000] Detected use of extended apic ids on hypertransport bus
[    0.000000] Detected use of extended apic ids on hypertransport bus
[    0.000000] ACPI: PM-Timer IO Port: 0x2008
[    0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x02] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x03] enabled)
[    0.000000] ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0])
[    0.000000] IOAPIC[0]: apic_id 4, version 17, address 0xfec00000, GSI 0-23
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 14 global_irq 14 high edge)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 15 global_irq 15 high edge)
[    0.000000] Using ACPI (MADT) for SMP configuration information
[    0.000000] ACPI: HPET id: 0x10de8201 base: 0xfed00000
[    0.000000] SMP: Allowing 4 CPUs, 0 hotplug CPUs
[    0.000000] Allocating PCI resources starting at e0000000 (gap: e0000000:1ec00000)
[    0.000000] setup_percpu: NR_CPUS:4 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:2
[    0.000000] PERCPU: Embedded 27 pages/cpu @ffff880001c00000 s78592 r8192 d23808 u1048576
[    0.000000] pcpu-alloc: s78592 r8192 d23808 u1048576 alloc=1*2097152
[    0.000000] pcpu-alloc: [0] 0 1 [1] 2 3 
[    0.000000] Built 2 zonelists in Node order, mobility grouping on.  Total pages: 1032302
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: fastboot earlyprintk=serial,ttyS0,115200 console=ttyS0,115200 console=tty1 crypt_root=/dev/md3 radeon.modeset=1 video=1280x1024
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] Checking aperture...
[    0.000000] No AGP bridge found
[    0.000000] Node 0: aperture @ 20000000 size 32 MB
[    0.000000] Aperture pointing to e820 RAM. Ignoring.
[    0.000000] Your BIOS doesn't leave a aperture memory hole
[    0.000000] Please enable the IOMMU option in the BIOS setup
[    0.000000] This costs you 64 MB of RAM
[    0.000000] Mapping aperture over 65536 KB of RAM @ 20000000
[    0.000000] Subtract (58 early reservations)
[    0.000000]   #1 [0001000000 - 0001bb03c8]   TEXT DATA BSS
[    0.000000]   #2 [0037ec8000 - 0037ff0000]         RAMDISK
[    0.000000]   #3 [0001bb1000 - 0001bb1238]             BRK
[    0.000000]   #4 [00000ff790 - 0000100000]   BIOS reserved
[    0.000000]   #5 [00000ff780 - 00000ff790]    MP-table mpf
[    0.000000]   #6 [000009fc00 - 00000f16f0]   BIOS reserved
[    0.000000]   #7 [00000f188c - 00000ff780]   BIOS reserved
[    0.000000]   #8 [00000f16f0 - 00000f188c]    MP-table mpc
[    0.000000]   #9 [0000001000 - 0000003000]      TRAMPOLINE
[    0.000000]   #10 [0000008000 - 000000c000]         PGTABLE
[    0.000000]   #11 [000000c000 - 000000d000]         PGTABLE
[    0.000000]   #12 [0001bb1240 - 0001bb3240]       NODE_DATA
[    0.000000]   #13 [0100000000 - 0100002000]       NODE_DATA
[    0.000000]   #14 [0001bb3240 - 0001bb4240]         BOOTMEM
[    0.000000]   #15 [0001bb0400 - 0001bb0580]         BOOTMEM
[    0.000000]   #16 [0080000000 - 0080000180]         BOOTMEM
[    0.000000]   #17 [00023b5000 - 00023b6000]         BOOTMEM
[    0.000000]   #18 [00023b6000 - 00023b7000]         BOOTMEM
[    0.000000]   #19 [0002400000 - 0004000000]        MEMMAP 0
[    0.000000]   #20 [0100200000 - 0102000000]        MEMMAP 1
[    0.000000]   #21 [0001bb0580 - 0001bb0700]         BOOTMEM
[    0.000000]   #22 [0001bb4240 - 0001bc0240]         BOOTMEM
[    0.000000]   #23 [0080000180 - 008000c180]         BOOTMEM
[    0.000000]   #24 [008000c180 - 008000f180]         BOOTMEM
[    0.000000]   #25 [0001bc1000 - 0001bc2000]         BOOTMEM
[    0.000000]   #26 [0001bb0700 - 0001bb0741]         BOOTMEM
[    0.000000]   #27 [0001bb0780 - 0001bb07c3]         BOOTMEM
[    0.000000]   #28 [0001bb0800 - 0001bb0a68]         BOOTMEM
[    0.000000]   #29 [0001bb0a80 - 0001bb0ae8]         BOOTMEM
[    0.000000]   #30 [0001bb0b00 - 0001bb0b68]         BOOTMEM
[    0.000000]   #31 [0001bb0b80 - 0001bb0be8]         BOOTMEM
[    0.000000]   #32 [0001bb0c00 - 0001bb0c68]         BOOTMEM
[    0.000000]   #33 [0001bb0c80 - 0001bb0ce8]         BOOTMEM
[    0.000000]   #34 [0001bb0d00 - 0001bb0d68]         BOOTMEM
[    0.000000]   #35 [0001bb0d80 - 0001bb0de8]         BOOTMEM
[    0.000000]   #36 [0001bb0e00 - 0001bb0e68]         BOOTMEM
[    0.000000]   #37 [0001bb0e80 - 0001bb0ee8]         BOOTMEM
[    0.000000]   #38 [0001bb0f00 - 0001bb0f68]         BOOTMEM
[    0.000000]   #39 [0001bb0f80 - 0001bb1000]         BOOTMEM
[    0.000000]   #40 [0001bc0240 - 0001bc02c0]         BOOTMEM
[    0.000000]   #41 [0001c00000 - 0001c1b000]         BOOTMEM
[    0.000000]   #42 [0001d00000 - 0001d1b000]         BOOTMEM
[    0.000000]   #43 [0080200000 - 008021b000]         BOOTMEM
[    0.000000]   #44 [0080300000 - 008031b000]         BOOTMEM
[    0.000000]   #45 [0001bc02c0 - 0001bc02d0]         BOOTMEM
[    0.000000]   #46 [0001bc0300 - 0001bc0310]         BOOTMEM
[    0.000000]   #47 [0001bc0340 - 0001bc0350]         BOOTMEM
[    0.000000]   #48 [0001bc0380 - 0001bc03a0]         BOOTMEM
[    0.000000]   #49 [0001bc03c0 - 0001bc0500]         BOOTMEM
[    0.000000]   #50 [0001bc0500 - 0001bc0560]         BOOTMEM
[    0.000000]   #51 [0001bc0580 - 0001bc05e0]         BOOTMEM
[    0.000000]   #52 [0001bc2000 - 0001bca000]         BOOTMEM
[    0.000000]   #53 [0020000000 - 0024000000]         BOOTMEM
[    0.000000]   #54 [0004000000 - 0008000000]         BOOTMEM
[    0.000000]   #55 [0001bca000 - 0001bea000]         BOOTMEM
[    0.000000]   #56 [0001c1b000 - 0001c5b000]         BOOTMEM
[    0.000000]   #57 [000000d000 - 0000015000]         BOOTMEM
[    0.000000] Memory: 3989044k/4718592k available (5525k kernel code, 524872k absent, 204676k reserved, 5367k data, 564k init)
[    0.000000] SLUB: Genslabs=14, HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=2
[    0.000000] Hierarchical RCU implementation.
[    0.000000] 	CONFIG_RCU_FANOUT set to non-default value of 4
[    0.000000] 	RCU dyntick-idle grace-period acceleration is enabled.
[    0.000000] 	Verbose stalled-CPUs detection is disabled.
[    0.000000] NR_IRQS:4352 nr_irqs:712
[    0.000000] Extended CMOS year: 2000
[    0.000000] Console: colour VGA+ 80x25
[    0.000000] console [tty1] enabled, bootconsole disabled
[    0.000000] console [ttyS0] enabled
[    0.000000] Fast TSC calibration using PIT
[    0.000000] Detected 2599.982 MHz processor.
[    0.020011] Calibrating delay loop (skipped), value calculated using timer frequency.. 5199.96 BogoMIPS (lpj=25999820)
[    0.030002] pid_max: default: 32768 minimum: 301
[    0.035197] Security Framework initialized
[    0.040261] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes)
[    0.050399] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes)
[    0.058566] Mount-cache hash table entries: 256
[    0.060161] CPU: Physical Processor ID: 0
[    0.064176] CPU: Processor Core ID: 0
[    0.070002] mce: CPU supports 5 MCE banks
[    0.074022] using C1E aware idle routine
[    0.077950] Performance Events: AMD PMU driver.
[    0.081501] ... version:                0
[    0.085511] ... bit width:              48
[    0.090001] ... generic registers:      4
[    0.094016] ... value mask:             0000ffffffffffff
[    0.100001] ... max period:             00007fffffffffff
[    0.105469] ... fixed-purpose events:   0
[    0.110001] ... event mask:             000000000000000f
[    0.115336] Freeing SMP alternatives: 20k freed
[    0.120012] ACPI: Core revision 20100702
[    0.130878] Setting APIC routing to flat
[    0.135231] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.241267] CPU0: Dual-Core AMD Opteron(tm) Processor 2218 stepping 02
[    0.250000] NMI watchdog enabled, takes one hw-pmu counter.
[    0.250000] Booting Node   0, Processors  #1
[    0.410108] NMI watchdog enabled, takes one hw-pmu counter.
[    0.415953]  Ok.
[    0.417806] Booting Node   1, Processors  #2
[    0.580082] NMI watchdog enabled, takes one hw-pmu counter.
[    0.585920]  #3 Ok.
[    0.740132] NMI watchdog enabled, takes one hw-pmu counter.
[    0.745720] Brought up 4 CPUs
[    0.748697] Total of 4 processors activated (20800.19 BogoMIPS).
[    0.750392] NET: Registered protocol family 16
[    0.760023] TOM: 00000000e0000000 aka 3584M
[    0.764207] TOM2: 0000000120000000 aka 4608M
[    0.768495] ACPI: bus type pci registered
[    0.770026] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xf0000000-0xffffffff] (base 0xf0000000)
[    0.780002] PCI: not using MMCONFIG
[    0.783499] PCI: Using configuration type 1 for base access
[    0.800066] bio: create slab <bio-0> at 0
[    0.804043] ACPI: Executed 1 blocks of module-level executable AML code
[    0.813216] ACPI: Interpreter enabled
[    0.820001] ACPI: (supports S0 S5)
[    0.823476] ACPI: Using IOAPIC for interrupt routing
[    0.828474] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xf0000000-0xffffffff] (base 0xf0000000)
[    0.844447] [Firmware Bug]: PCI: MMCONFIG at [mem 0xf0000000-0xffffffff] not reserved in ACPI motherboard resources
[    0.850001] PCI: not using MMCONFIG
[    0.860438] ACPI Warning: Incorrect checksum in table [OEMB] - 0xEC, should be 0xE7 (20100702/tbutils-314)
[    0.872254] PCI: Ignoring host bridge windows from ACPI; if necessary, use "pci=use_crs" and report a bug
[    0.880227] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
[    0.891149] pci 0000:00:06.0: PCI bridge to [bus 05-05] (subtractive decode)
[    0.900166] pci 0000:04:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
[    0.910007] pci 0000:00:0b.0: PCI bridge to [bus 04-04]
[    0.930008] pci 0000:00:0c.0: PCI bridge to [bus 03-03]
[    0.950007] pci 0000:00:0d.0: PCI bridge to [bus 02-02]
[    0.955365] pci 0000:01:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
[    0.960005] pci 0000:00:0f.0: PCI bridge to [bus 01-01]
[    0.970724] ACPI: PCI Interrupt Link [LNKA] (IRQs 16 17 18 19) *11
[    0.981976] ACPI: PCI Interrupt Link [LNKB] (IRQs 16 17 18 19) *0, disabled.
[    0.991713] ACPI: PCI Interrupt Link [LNKC] (IRQs 16 17 18 19) *10
[    0.998216] ACPI: PCI Interrupt Link [LNKD] (IRQs 16 17 18 19) *0, disabled.
[    1.002878] ACPI: PCI Interrupt Link [LNEA] (IRQs 16 17 18 19) *10
[    1.011982] ACPI: PCI Interrupt Link [LNEB] (IRQs 16 17 18 19) *10
[    1.018501] ACPI: PCI Interrupt Link [LNEC] (IRQs 16 17 18 19) *0, disabled.
[    1.022874] ACPI: PCI Interrupt Link [LNED] (IRQs 16 17 18 19) *5
[    1.031907] ACPI: PCI Interrupt Link [LUB0] (IRQs 20 21 22 23) *10
[    1.038425] ACPI: PCI Interrupt Link [LMAD] (IRQs 20 21 22 23) *0, disabled.
[    1.042890] ACPI: PCI Interrupt Link [LUB2] (IRQs 20 21 22 23) *11
[    1.051988] ACPI: PCI Interrupt Link [LMAC] (IRQs 20 21 22 23) *0, disabled.
[    1.060312] ACPI: PCI Interrupt Link [LAZA] (IRQs 20 21 22 23) *11
[    1.066815] ACPI: PCI Interrupt Link [LSMB] (IRQs 20 21 22 23) *11
[    1.071974] ACPI: PCI Interrupt Link [LPMU] (IRQs 20 21 22 23) *5
[    1.080108] ACPI: PCI Interrupt Link [LSA0] (IRQs 20 21 22 23) *5
[    1.086540] ACPI: PCI Interrupt Link [LSA1] (IRQs 20 21 22 23) *10
[    1.091983] ACPI: PCI Interrupt Link [LATA] (IRQs 20 21 22 23) *0, disabled.
[    1.100315] ACPI: PCI Interrupt Link [LSA2] (IRQs 20 21 22 23) *10
[    1.106750] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none
[    1.110002] vgaarb: loaded
[    1.112789] SCSI subsystem initialized
[    1.120017] usbcore: registered new interface driver usbfs
[    1.125498] usbcore: registered new interface driver hub
[    1.130012] usbcore: registered new device driver usb
[    1.135240] Advanced Linux Sound Architecture Driver Version 1.0.23.
[    1.140001] PCI: Using ACPI for IRQ routing
[    1.144265] NetLabel: Initializing
[    1.150002] NetLabel:  domain hash size = 128
[    1.154366] NetLabel:  protocols = UNLABELED CIPSOv4
[    1.160011] NetLabel:  unlabeled traffic allowed by default
[    1.165589] HPET: 3 timers in total, 0 timers will be used for per-cpu timer
[    1.170005] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 31
[    1.175111] hpet0: 3 comparators, 32-bit 25.000000 MHz counter
[    1.210030] Switching to clocksource hpet
[    1.214052] FS-Cache: Loaded
[    1.214054] CacheFiles: Loaded
[    1.216188] pnp: PnP ACPI init
[    1.219260] ACPI: bus type pnp registered
[    1.227219] pnp: PnP ACPI: found 14 devices
[    1.231424] ACPI: ACPI bus type pnp unregistered
[    1.236057] system 00:06: [io  0x04d0-0x04d1] has been reserved
[    1.242008] system 00:06: [io  0x07b0-0x07df] has been reserved
[    1.247926] system 00:06: [io  0x0800-0x080f] has been reserved
[    1.253879] system 00:06: [io  0x0bb0-0x0bdf] has been reserved
[    1.259979] system 00:06: [io  0x2000-0x207f] has been reserved
[    1.265936] system 00:06: [io  0x2080-0x20ff] has been reserved
[    1.271872] system 00:06: [io  0x2400-0x247f] has been reserved
[    1.277790] system 00:06: [io  0x2480-0x24ff] has been reserved
[    1.283746] system 00:06: [io  0x2800-0x287f] has been reserved
[    1.289664] system 00:06: [io  0x2880-0x28ff] has been reserved
[    1.295618] system 00:06: [io  0x2c00-0x2c7f] has been reserved
[    1.301556] system 00:06: [io  0x2c80-0x2cff] has been reserved
[    1.307475] system 00:06: [mem 0x000d0000-0x000d3fff window] has been reserved
[    1.314727] system 00:06: [mem 0x000d4000-0x000d7fff window] has been reserved
[    1.322095] system 00:06: [mem 0x000de000-0x000dffff window] has been reserved
[    1.329321] system 00:06: [mem 0xfee01000-0xfeefffff] has been reserved
[    1.335971] system 00:06: [mem 0xefa80000-0xefabffff] has been reserved
[    1.342601] system 00:06: [mem 0xffb00000-0xffbfffff] has been reserved
[    1.349211] system 00:06: [mem 0xfff00000-0xffffffff] has been reserved
[    1.355864] system 00:08: [mem 0xfec00000-0xfec00fff] could not be reserved
[    1.362845] system 00:08: [mem 0xfee00000-0xfee00fff] has been reserved
[    1.369458] system 00:0c: [io  0x0290-0x0297] has been reserved
[    1.375414] system 00:0d: [mem 0x00000000-0x0009ffff] could not be reserved
[    1.382398] system 00:0d: [mem 0x000c0000-0x000cffff] has been reserved
[    1.389190] system 00:0d: [mem 0x000e0000-0x000fffff] could not be reserved
[    1.396187] system 00:0d: [mem 0x00100000-0xdfffffff] could not be reserved
[    1.403164] system 00:0d: [mem 0xfec00000-0xffffffff] could not be reserved
[    1.419451] pci 0000:00:06.0: PCI bridge to [bus 05-05]
[    1.424709] pci 0000:00:06.0:   bridge window [io  disabled]
[    1.430385] pci 0000:00:06.0:   bridge window [mem 0xeff00000-0xefffffff]
[    1.437168] pci 0000:00:06.0:   bridge window [mem 0xeef00000-0xeeffffff pref]
[    1.444427] pci 0000:00:0b.0: PCI bridge to [bus 04-04]
[    1.449653] pci 0000:00:0b.0:   bridge window [io  0xe000-0xefff]
[    1.455902] pci 0000:00:0b.0:   bridge window [mem 0xefe00000-0xefefffff]
[    1.462713] pci 0000:00:0b.0:   bridge window [mem pref disabled]
[    1.468804] pci 0000:00:0c.0: PCI bridge to [bus 03-03]
[    1.474065] pci 0000:00:0c.0:   bridge window [io  disabled]
[    1.479725] pci 0000:00:0c.0:   bridge window [mem 0xefd00000-0xefdfffff]
[    1.486546] pci 0000:00:0c.0:   bridge window [mem pref disabled]
[    1.492657] pci 0000:00:0d.0: PCI bridge to [bus 02-02]
[    1.497882] pci 0000:00:0d.0:   bridge window [io  disabled]
[    1.503575] pci 0000:00:0d.0:   bridge window [mem 0xefc00000-0xefcfffff]
[    1.510380] Clocksource tsc unstable (delta = 631633489 ns)
[    1.516068] pci 0000:00:0d.0:   bridge window [mem pref disabled]
[    1.522195] pci 0000:00:0f.0: PCI bridge to [bus 01-01]
[    1.527427] pci 0000:00:0f.0:   bridge window [io  0xd000-0xdfff]
[    1.533555] pci 0000:00:0f.0:   bridge window [mem 0xefb00000-0xefbfffff]
[    1.540368] pci 0000:00:0f.0:   bridge window [mem 0xe0000000-0xe7ffffff 64bit pref]
[    1.548211] NET: Registered protocol family 2
[    1.552764] IP route cache hash table entries: 131072 (order: 8, 1048576 bytes)
[    1.561709] TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
[    1.573951] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[    1.581402] TCP: Hash tables configured (established 524288 bind 65536)
[    1.588019] TCP reno registered
[    1.591214] UDP hash table entries: 2048 (order: 4, 65536 bytes)
[    1.597278] UDP-Lite hash table entries: 2048 (order: 4, 65536 bytes)
[    1.603887] NET: Registered protocol family 1
[    1.608385] RPC: Registered udp transport module.
[    1.613115] RPC: Registered tcp transport module.
[    1.617830] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    1.646008] pci 0000:00:00.0: Found enabled HT MSI Mapping
[    1.651559] pci 0000:00:00.0: Found enabled HT MSI Mapping
[    1.657086] pci 0000:00:00.0: Found enabled HT MSI Mapping
[    1.662642] pci 0000:00:00.0: Found enabled HT MSI Mapping
[    1.668175] pci 0000:00:00.0: Found enabled HT MSI Mapping
[    1.673742] pci 0000:00:00.0: Found enabled HT MSI Mapping
[    1.679284] pci 0000:00:00.0: Found enabled HT MSI Mapping
[    1.685475] pci 0000:00:00.0: Found enabled HT MSI Mapping
[    1.691043] pci 0000:00:00.0: Found enabled HT MSI Mapping
[    1.696623] Unpacking initramfs...
[    1.722701] Freeing initrd memory: 1184k freed
[    1.727901] PCI-DMA: Disabling AGP.
[    1.731664] PCI-DMA: aperture base @ 20000000 size 65536 KB
[    1.737235] PCI-DMA: using GART IOMMU.
[    1.741019] PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture
[    1.754121] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    1.764006] VFS: Disk quotas dquot_6.5.2
[    1.768036] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    1.774611] FS-Cache: Netfs 'nfs' registered for caching
[    1.780155] SGI XFS with ACLs, security attributes, large block/inode numbers, no debug enabled
[    1.789679] SGI XFS Quota Management subsystem
[    1.794183] msgmni has been set to 7922
[    1.798769] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 254)
[    1.806191] io scheduler noop registered
[    1.810208] io scheduler cfq registered (default)
[    1.815342] pcieport 0000:00:0b.0: ACPI _OSC control granted for 0x1d
[    1.821942] pcieport 0000:00:0c.0: ACPI _OSC control granted for 0x1d
[    1.828516] pcieport 0000:00:0d.0: ACPI _OSC control granted for 0x1d
[    1.835096] pcieport 0000:00:0f.0: ACPI _OSC control granted for 0x1d
[    1.842053] input: Power Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0C:00/input/input0
[    1.850438] ACPI: Power Button [PWRB]
[    1.854203] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input1
[    1.861623] ACPI: Power Button [PWRF]
[    1.865968] Real Time Clock Driver v1.12b
[    1.870232] Hangcheck: starting hangcheck timer 0.9.1 (tick is 180 seconds, margin is 60 seconds).
[    1.879180] Hangcheck: Using getrawmonotonic().
[    1.883806] [drm] Initialized drm 1.1.0 20060810
[    1.888487] [drm] radeon kernel modesetting enabled.
[    1.893721] ACPI: PCI Interrupt Link [LNEB] enabled at IRQ 19
[    1.899478] radeon 0000:01:00.0: PCI INT A -> Link[LNEB] -> GSI 19 (level, low) -> IRQ 19
[    1.909871] [drm] initializing kernel modesetting (RV380 0x1002:0x5B60).
[    1.916673] [drm] register mmio base: 0xEFBF0000
[    1.921322] [drm] register mmio size: 65536
[    1.926142] [drm] Generation 2 PCI interface, using max accessible memory
[    1.932959] radeon 0000:01:00.0: VRAM: 128M 0xE0000000 - 0xE7FFFFFF (128M used)
[    1.940288] radeon 0000:01:00.0: GTT: 512M 0xC0000000 - 0xDFFFFFFF
[    1.946513] radeon 0000:01:00.0: radeon: using MSI.
[    1.951443] [drm] radeon: irq initialized.
[    1.956254] [drm] Detected VRAM RAM=128M, BAR=128M
[    1.961100] [drm] RAM width 64bits DDR
[    1.964935] [TTM] Zone  kernel: Available graphics memory: 2028100 kiB.
[    1.971566] [TTM] Initializing pool allocator.
[    1.976038] [drm] radeon: 128M of VRAM memory ready
[    1.980945] [drm] radeon: 512M of GTT memory ready.
[    1.985841] [drm] GART: num cpu pages 131072, num gpu pages 131072
[    1.993082] [drm] radeon: 1 quad pipes, 1 Z pipes initialized.
[    1.999658] [drm] PCIE GART of 512M enabled (table at 0xE0040000).
[    2.005989] [drm] Loading R300 Microcode
[    2.010098] [drm] radeon: ring at 0x00000000C0000000
[    2.015083] [drm] ring test succeeded in 1 usecs
[    2.019782] [drm] radeon: ib pool ready.
[    2.023812] [drm] ib test succeeded in 0 usecs
[    2.028621] [drm] Default TV standard: NTSC
[    2.032823] [drm] 27.000000000 MHz TV ref clk
[    2.037185] [drm] DFP table revision: 4
[    2.041146] [drm] Default TV standard: NTSC
[    2.045330] [drm] 27.000000000 MHz TV ref clk
[    2.049804] [drm] Radeon Display Connectors
[    2.054004] [drm] Connector 0:
[    2.057068] [drm]   VGA
[    2.059526] [drm]   DDC: 0x60 0x60 0x60 0x60 0x60 0x60 0x60 0x60
[    2.065556] [drm]   Encoders:
[    2.068536] [drm]     CRT1: INTERNAL_DAC1
[    2.072568] [drm] Connector 1:
[    2.075632] [drm]   DVI-I
[    2.078261] [drm]   HPD1
[    2.080826] [drm]   DDC: 0x64 0x64 0x64 0x64 0x64 0x64 0x64 0x64
[    2.086832] [drm]   Encoders:
[    2.089807] [drm]     CRT2: INTERNAL_DAC2
[    2.093853] [drm]     DFP1: INTERNAL_TMDS1
[    2.097955] [drm] Connector 2:
[    2.101037] [drm]   S-video
[    2.103843] [drm]   Encoders:
[    2.106818] [drm]     TV1: INTERNAL_DAC2
[    2.242976] [drm] fb mappable at 0xE00C0000
[    2.247162] [drm] vram apper at 0xE0000000
[    2.251290] [drm] size 5242880
[    2.254357] [drm] fb depth is 24
[    2.257592] [drm]    pitch is 5120
[    2.261107] fbcon: radeondrmfb (fb0) is primary device
[    2.381151] Console: switching to colour frame buffer device 160x64
[    2.393850] fb0: radeondrmfb frame buffer device
[    2.398488] drm: registered panic notifier
[    2.402632] [drm] Initialized radeon 2.6.0 20080528 for 0000:01:00.0 on minor 0
[    2.410120] Serial: 8250/16550 driver, 1 ports, IRQ sharing disabled
[    2.437083] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[    2.463858] 00:09: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[    2.469955] sata_sil24 0000:04:00.0: PCI INT A -> Link[LNEB] -> GSI 19 (level, low) -> IRQ 19
[    2.478822] scsi0 : sata_sil24
[    2.482089] scsi1 : sata_sil24
[    2.485287] ata1: SATA max UDMA/100 host m128@0xefeffc00 port 0xefef8000 irq 19
[    2.492650] ata2: SATA max UDMA/100 host m128@0xefeffc00 port 0xefefa000 irq 19
[    2.500476] ACPI: PCI Interrupt Link [LSA0] enabled at IRQ 23
[    2.506257] sata_nv 0000:00:05.0: PCI INT A -> Link[LSA0] -> GSI 23 (level, low) -> IRQ 23
[    2.514594] sata_nv 0000:00:05.0: Using SWNCQ mode
[    2.519577] scsi2 : sata_nv
[    2.522540] scsi3 : sata_nv
[    2.525518] ata3: SATA max UDMA/133 cmd 0xcc00 ctl 0xc880 bmdma 0xc400 irq 23
[    2.532706] ata4: SATA max UDMA/133 cmd 0xc800 ctl 0xc480 bmdma 0xc408 irq 23
[    2.540126] ACPI: PCI Interrupt Link [LSA1] enabled at IRQ 22
[    2.545906] sata_nv 0000:00:05.1: PCI INT B -> Link[LSA1] -> GSI 22 (level, low) -> IRQ 22
[    2.554244] sata_nv 0000:00:05.1: Using SWNCQ mode
[    2.559214] scsi4 : sata_nv
[    2.562195] scsi5 : sata_nv
[    2.565156] ata5: SATA max UDMA/133 cmd 0xc080 ctl 0xc000 bmdma 0xb800 irq 22
[    2.572340] ata6: SATA max UDMA/133 cmd 0xbc00 ctl 0xb880 bmdma 0xb808 irq 22
[    2.579742] ACPI: PCI Interrupt Link [LSA2] enabled at IRQ 21
[    2.585543] sata_nv 0000:00:05.2: PCI INT C -> Link[LSA2] -> GSI 21 (level, low) -> IRQ 21
[    2.593867] sata_nv 0000:00:05.2: Using SWNCQ mode
[    2.598857] scsi6 : sata_nv
[    2.601843] scsi7 : sata_nv
[    2.604817] ata7: SATA max UDMA/133 cmd 0xb480 ctl 0xb400 bmdma 0xac00 irq 21
[    2.611999] ata8: SATA max UDMA/133 cmd 0xb080 ctl 0xb000 bmdma 0xac08 irq 21
[    2.619457] scsi8 : pata_amd
[    2.622499] scsi9 : pata_amd
[    2.627895] ata9: PATA max UDMA/133 cmd 0x1f0 ctl 0x3f6 bmdma 0xffa0 irq 14
[    2.636448] ata10: PATA max UDMA/133 cmd 0x170 ctl 0x376 bmdma 0xffa8 irq 15
[    2.645289] tg3.c:v3.113 (August 2, 2010)
[    2.651160] ACPI: PCI Interrupt Link [LNEA] enabled at IRQ 18
[    2.658412] tg3 0000:03:00.0: PCI INT A -> Link[LNEA] -> GSI 18 (level, low) -> IRQ 18
[    2.705556] tg3 0000:03:00.0: eth0: Tigon3 [partno(BCM95700A6) rev b002] (PCI Express) MAC address 00:17:31:f9:cc:c5
[    2.717828] tg3 0000:03:00.0: eth0: attached PHY is 5787 (10/100/1000Base-T Ethernet) (WireSpeed[1])
[    2.728773] tg3 0000:03:00.0: eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] TSOcap[1]
[    2.738767] tg3 0000:03:00.0: eth0: dma_rwctrl[76180000] dma_mask[64-bit]
[    2.747648] ACPI: PCI Interrupt Link [LNED] enabled at IRQ 17
[    2.755492] tg3 0000:02:00.0: PCI INT A -> Link[LNED] -> GSI 17 (level, low) -> IRQ 17
[    2.804492] tg3 0000:02:00.0: eth1: Tigon3 [partno(BCM95700A6) rev b002] (PCI Express) MAC address 00:18:f3:44:df:ea
[    2.817081] tg3 0000:02:00.0: eth1: attached PHY is 5787 (10/100/1000Base-T Ethernet) (WireSpeed[1])
[    2.828016] tg3 0000:02:00.0: eth1: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] TSOcap[1]
[    2.838052] tg3 0000:02:00.0: eth1: dma_rwctrl[76180000] dma_mask[64-bit]
[    2.847417] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 16
[    2.855170] ohci1394 0000:05:08.0: PCI INT A -> Link[LNKA] -> GSI 16 (level, low) -> IRQ 16
[    2.900036] ata5: SATA link down (SStatus 0 SControl 300)
[    2.920053] ohci1394: fw-host0: OHCI-1394 1.1 (PCI): IRQ=[16]  MMIO=[effff800-efffffff]  Max Packet=[2048]  IR/IT contexts=[4/8]
[    2.934544] eth1394: eth2: IPv4 over IEEE 1394 (fw-host0)
[    2.942300] ata7: SATA link down (SStatus 0 SControl 300)
[    2.942441] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    2.942656] ACPI: PCI Interrupt Link [LUB2] enabled at IRQ 20
[    2.942669] ehci_hcd 0000:00:02.1: PCI INT B -> Link[LUB2] -> GSI 20 (level, low) -> IRQ 20
[    2.942683] ehci_hcd 0000:00:02.1: EHCI Host Controller
[    2.942691] ehci_hcd 0000:00:02.1: new USB bus registered, assigned bus number 1
[    2.942714] ehci_hcd 0000:00:02.1: debug port 1
[    2.942744] ehci_hcd 0000:00:02.1: irq 20, io mem 0xefaebc00
[    2.970014] ehci_hcd 0000:00:02.1: USB 2.0 started, EHCI 1.00
[    2.970032] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[    2.970034] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    2.970036] usb usb1: Product: EHCI Host Controller
[    2.970037] usb usb1: Manufacturer: Linux 2.6.36-rc5 ehci_hcd
[    2.970039] usb usb1: SerialNumber: 0000:00:02.1
[    2.970203] hub 1-0:1.0: USB hub found
[    2.970207] hub 1-0:1.0: 10 ports detected
[    2.970326] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    2.970542] ACPI: PCI Interrupt Link [LUB0] enabled at IRQ 23
[    2.970545] ohci_hcd 0000:00:02.0: PCI INT A -> Link[LUB0] -> GSI 23 (level, low) -> IRQ 23
[    2.970560] ohci_hcd 0000:00:02.0: OHCI Host Controller
[    2.970566] ohci_hcd 0000:00:02.0: new USB bus registered, assigned bus number 2
[    2.970578] ohci_hcd 0000:00:02.0: irq 23, io mem 0xefaf8000
[    3.042026] usb usb2: New USB device found, idVendor=1d6b, idProduct=0001
[    3.042028] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    3.042030] usb usb2: Product: OHCI Host Controller
[    3.042032] usb usb2: Manufacturer: Linux 2.6.36-rc5 ohci_hcd
[    3.042033] usb usb2: SerialNumber: 0000:00:02.0
[    3.042159] hub 2-0:1.0: USB hub found
[    3.042164] hub 2-0:1.0: 10 ports detected
[    3.042271] Initializing USB Mass Storage driver...
[    3.042338] usbcore: registered new interface driver usb-storage
[    3.042340] USB Mass Storage support registered.
[    3.042404] usbcore: registered new interface driver libusual
[    3.042456] usbcore: registered new interface driver ums-alauda
[    3.042511] usbcore: registered new interface driver ums-cypress
[    3.042563] usbcore: registered new interface driver ums-datafab
[    3.042618] usbcore: registered new interface driver ums-freecom
[    3.042669] usbcore: registered new interface driver ums-isd200
[    3.042721] usbcore: registered new interface driver ums-jumpshot
[    3.042777] usbcore: registered new interface driver ums-karma
[    3.042829] usbcore: registered new interface driver ums-sddr09
[    3.042880] usbcore: registered new interface driver ums-sddr55
[    3.042934] usbcore: registered new interface driver ums-usbat
[    3.043059] PNP: PS/2 Controller [PNP0303:PS2K,PNP0f03:PS2M] at 0x60,0x64 irq 1,12
[    3.045588] serio: i8042 KBD port at 0x60,0x64 irq 1
[    3.045595] serio: i8042 AUX port at 0x60,0x64 irq 12
[    3.045855] mice: PS/2 mouse device common for all mice
[    3.046185] input: PC Speaker as /devices/platform/pcspkr/input/input2
[    3.046284] I2O subsystem v1.325
[    3.046285] i2o: max drivers = 8
[    3.046470] i2c /dev entries driver
[    3.046933] i2c i2c-4: nForce2 SMBus adapter at 0x2a00
[    3.047038] i2c i2c-5: nForce2 SMBus adapter at 0x2a80
[    3.047196] lirc_dev: IR Remote Control driver registered, major 253 
[    3.047198] IR NEC protocol handler initialized
[    3.047199] IR RC5(x) protocol handler initialized
[    3.047200] IR RC6 protocol handler initialized
[    3.047202] IR JVC protocol handler initialized
[    3.047203] IR Sony protocol handler initialized
[    3.047204] IR LIRC bridge handler initialized
[    3.047205] Linux video capture interface: v2.00
[    3.047570] bttv: driver version 0.9.18 loaded
[    3.047572] bttv: using 8 buffers with 2080k (520 pages) each for capture
[    3.047666] bttv: Bt8xx card found (0).
[    3.047857] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 19
[    3.047860] bttv 0000:05:06.0: PCI INT A -> Link[LNKC] -> GSI 19 (level, low) -> IRQ 19
[    3.047867] bttv0: Bt878 (rev 17) at 0000:05:06.0, irq: 19, latency: 64, mmio: 0xeefff000
[    3.047881] bttv0: detected: Hauppauge WinTV [card=10], PCI subsystem ID is 0070:13eb
[    3.047883] bttv0: using: Hauppauge (bt878) [card=10,autodetected]
[    3.050412] bttv0: Hauppauge/Voodoo msp34xx: reset line init [5]
[    3.066621] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input3
[    3.084419] tveeprom 6-0050: Hauppauge model 61344, rev D421, serial# 3902813
[    3.084421] tveeprom 6-0050: tuner model is Philips FM1216 (idx 21, type 5)
[    3.084424] tveeprom 6-0050: TV standards PAL(B/G) (eeprom 0x04)
[    3.084426] tveeprom 6-0050: audio processor is MSP3415 (idx 6)
[    3.084427] tveeprom 6-0050: has radio
[    3.084429] bttv0: Hauppauge eeprom indicates model#61344
[    3.084430] bttv0: tuner type=5
[    3.097785] msp3400 6-0040: MSP3415D-B3 found @ 0x80 (bt878 #0 [sw])
[    3.097787] msp3400 6-0040: msp3400 supports nicam, mode is autodetect
[    3.109810] tuner 6-0061: chip found @ 0xc2 (bt878 #0 [sw])
[    3.110103] tuner-simple 6-0061: creating new instance
[    3.110106] tuner-simple 6-0061: type set to 5 (Philips PAL_BG (FI1216 and compatibles))
[    3.110811] bttv0: registered device video0
[    3.110866] bttv0: registered device vbi0
[    3.110919] bttv0: registered device radio0
[    3.112429] bttv0: PLL: 28636363 => 35468950 .
[    3.603026] ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[    3.610075] . ok
[    3.635040] Registered IR keymap rc-rc5-tv
[    3.641059] input: i2c IR (Hauppauge) as /devices/virtual/rc/rc0/input4
[    3.649554] rc0: i2c IR (Hauppauge) as /devices/virtual/rc/rc0
[    3.656914] ir-kbd-i2c: i2c IR (Hauppauge) detected at i2c-6/6-0018/ir0 [bt878 #0 [sw]]
[    3.675611] ata3.00: ATA-7: MAXTOR STM3320820AS, 3.AAE, max UDMA/133
[    3.683771] ata3.00: 625142448 sectors, multi 16: LBA48 NCQ (depth 31/32)
[    3.758899] ata3.00: configured for UDMA/133
[    4.070019] usb 2-6: new low speed USB device using ohci_hcd and address 2
[    4.258036] usb 2-6: New USB device found, idVendor=062a, idProduct=0000
[    4.266620] usb 2-6: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[    4.500081] k8temp 0000:00:18.3: Temperature readouts might be wrong - check erratum #141
[    4.510043] k8temp 0000:00:19.3: Temperature readouts might be wrong - check erratum #141
[    4.520230] md: raid1 personality registered for level 1
[    4.527821] device-mapper: uevent: version 1.0.3
[    4.534492] device-mapper: ioctl: 4.18.0-ioctl (2010-06-29) initialised: dm-devel@redhat.com
[    4.544918] EDAC MC: Ver: 2.1.0 Sep 21 2010
[    4.551330] EDAC amd64_edac:  Ver: 3.3.0 Sep 21 2010
[    4.558278] EDAC amd64: ECC is enabled by BIOS.
[    4.564784] EDAC amd64: ECC is enabled by BIOS.
[    4.571822] EDAC MC0: Giving out device to 'amd64_edac' 'RevF': DEV 0000:00:18.2
[    4.581340] EDAC MC1: Giving out device to 'amd64_edac' 'RevF': DEV 0000:00:19.2
[    4.591090] EDAC PCI0: Giving out device to module 'amd64_edac' controller 'EDAC PCI controller': DEV '0000:00:18.2' (POLLED)
[    4.604470] cpuidle: using governor ladder
[    4.610553] cpuidle: using governor menu
[    4.618253] usbcore: registered new interface driver hiddev
[    4.631332] input: HID 062a:0000 as /devices/pci0000:00/0000:00:02.0/usb2/2-6/2-6:1.0/input/input5
[    4.642674] generic-usb 0003:062A:0000.0001: input,hidraw0: USB HID v1.10 Mouse [HID 062a:0000] on usb-0000:00:02.0-6/input0
[    4.656445] usbcore: registered new interface driver usbhid
[    4.664405] usbhid: USB HID core driver
[    4.670639] Bt87x 0000:05:06.1: PCI INT A -> Link[LNKC] -> GSI 19 (level, low) -> IRQ 19
[    4.681301] bt87x0: Using board 1, analog, digital (rate 32000 Hz)
[    4.690294] ACPI: PCI Interrupt Link [LAZA] enabled at IRQ 22
[    4.698571] HDA Intel 0000:00:06.1: PCI INT B -> Link[LAZA] -> GSI 22 (level, low) -> IRQ 22
[    4.709150] hda_intel: Disable MSI for Nvidia chipset
[    4.730054] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 0)
[    4.744574] ata1.00: ATA-7: SAMSUNG HD642JJ, 1AA01113, max UDMA7
[    4.752850] ata1.00: 1250263728 sectors, multi 0: LBA48 NCQ (depth 31/32)
[    4.768610] ata1.00: configured for UDMA/100
[    4.775591] scsi 0:0:0:0: Direct-Access     ATA      SAMSUNG HD642JJ  1AA0 PQ: 0 ANSI: 5
[    4.786182] sd 0:0:0:0: [sda] 1250263728 512-byte logical blocks: (640 GB/596 GiB)
[    4.796285] sd 0:0:0:0: [sda] Write Protect is off
[    4.803438] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    4.834245]  sda: sda1 sda2 sda3
[    4.839996] sd 0:0:0:0: [sda] Attached SCSI disk
[    6.260528] ALSA device list:
[    6.265987]   #0: HDA NVidia at 0xefafc000 irq 22
[    6.272887]   #1: Brooktree Bt878 at 0xeeffe000, irq 19
[    6.280987] Initializing XFRM netlink socket
[    6.287498] NET: Registered protocol family 17
[    6.294482] NET: Registered protocol family 15
[    6.301255] Registering the dns_resolver key type
[    6.308132] powernow-k8: Found 2 Dual-Core AMD Opteron(tm) Processor 2218 (4 cpu cores) (version 2.20.00)
[    6.320221] powernow-k8:    0 : fid 0x12 (2600 MHz), vid 0x8
[    6.328449] powernow-k8:    1 : fid 0x10 (2400 MHz), vid 0xa
[    6.336387] powernow-k8:    2 : fid 0xe (2200 MHz), vid 0xc
[    6.344569] powernow-k8:    3 : fid 0xc (2000 MHz), vid 0xe
[    6.352350] powernow-k8:    4 : fid 0xa (1800 MHz), vid 0x10
[    6.360550] powernow-k8:    5 : fid 0x2 (1000 MHz), vid 0x12
[    6.368862] powernow-k8:    0 : fid 0x12 (2600 MHz), vid 0x8
[    6.377476] powernow-k8:    1 : fid 0x10 (2400 MHz), vid 0xa
[    6.385197] powernow-k8:    2 : fid 0xe (2200 MHz), vid 0xc
[    6.393015] powernow-k8:    3 : fid 0xc (2000 MHz), vid 0xe
[    6.400598] powernow-k8:    4 : fid 0xa (1800 MHz), vid 0x10
[    6.408116] powernow-k8:    5 : fid 0x2 (1000 MHz), vid 0x12
[    7.000051] ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 0)
[    7.016333] ata2.00: ATA-7: SAMSUNG HD642JJ, 1AA01113, max UDMA7
[    7.026243] ata2.00: 1250263728 sectors, multi 0: LBA48 NCQ (depth 31/32)
[    7.043365] ata2.00: configured for UDMA/100
[    7.051789] scsi 1:0:0:0: Direct-Access     ATA      SAMSUNG HD642JJ  1AA0 PQ: 0 ANSI: 5
[    7.064244] sd 1:0:0:0: [sdb] 1250263728 512-byte logical blocks: (640 GB/596 GiB)
[    7.064583] scsi 2:0:0:0: Direct-Access     ATA      MAXTOR STM332082 3.AA PQ: 0 ANSI: 5
[    7.065024] sd 2:0:0:0: [sdc] 625142448 512-byte logical blocks: (320 GB/298 GiB)
[    7.065129] sd 2:0:0:0: [sdc] Write Protect is off
[    7.065183] sd 2:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    7.080725]  sdc: sdc1 sdc2
[    7.127963] sd 1:0:0:0: [sdb] Write Protect is off
[    7.128309] sd 2:0:0:0: [sdc] Attached SCSI disk
[    7.145445] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    7.177685]  sdb: sdb1 sdb2 sdb3
[    7.185858] sd 1:0:0:0: [sdb] Attached SCSI disk
[    7.400022] ata4: SATA link down (SStatus 0 SControl 300)
[    7.730028] ata6: SATA link down (SStatus 0 SControl 300)
[    8.060027] ata8: SATA link down (SStatus 0 SControl 300)
[    8.069502] Freeing unused kernel memory: 564k freed
[    8.078937] Write protecting the kernel read-only data: 10240k
[    8.088937] Freeing unused kernel memory: 596k freed
[    8.098543] Freeing unused kernel memory: 1728k freed
Sep 25 07:44:57 treogen syslog-ng[2716]: syslog-ng starting up; version='3.1.2'
Sep 25 07:45:02 treogen mdadm[2864]: NewArray event detected on md device /dev/md2
Sep 25 07:45:02 treogen mdadm[2864]: NewArray event detected on md device /dev/md3
Sep 25 07:45:09 treogen ntpdate[2875]: step time server 192.168.3.2 offset -1.399604 sec
Sep 25 07:45:09 treogen rpcbind: cannot bind 127.0.0.1 on udp: Address already in use
Sep 25 07:45:09 treogen rpcbind: cannot create socket for udp6
Sep 25 07:45:09 treogen rpcbind: cannot create socket for tcp6
Sep 25 07:45:10 treogen kdm: :0[2637]: pam_unix(kde:session): session opened for user tk13 by (uid=0)
Sep 25 07:45:10 treogen sm-notify[2944]: Version 1.2.2 starting
Sep 25 07:45:12 treogen postfix/postfix-script[3069]: starting the Postfix mail system
Sep 25 07:45:12 treogen postfix/master[3070]: daemon started -- version 2.6.7, configuration /etc/postfix
Sep 25 07:45:12 treogen smartd[3087]: smartd 5.39.1 2010-01-28 r3054 [x86_64-pc-linux-gnu] (local build)
Sep 25 07:45:12 treogen Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net
Sep 25 07:45:12 treogen smartd[3087]: Opened configuration file /etc/smartd.conf
Sep 25 07:45:12 treogen smartd[3087]: Drive: /dev/sda, implied '-a' Directive on line 24 of file /etc/smartd.conf
Sep 25 07:45:12 treogen smartd[3087]: Drive: /dev/sdb, implied '-a' Directive on line 25 of file /etc/smartd.conf
Sep 25 07:45:12 treogen smartd[3087]: Drive: /dev/sdc, implied '-a' Directive on line 26 of file /etc/smartd.conf
Sep 25 07:45:12 treogen smartd[3087]: Configuration file /etc/smartd.conf parsed.
Sep 25 07:45:12 treogen smartd[3087]: Device: /dev/sda, opened
Sep 25 07:45:12 treogen smartd[3087]: Device: /dev/sda, found in smartd database.
Sep 25 07:45:13 treogen smartd[3087]: Device: /dev/sda, is SMART capable. Adding to "monitor" list.
Sep 25 07:45:13 treogen smartd[3087]: Device: /dev/sdb, opened
Sep 25 07:45:13 treogen smartd[3087]: Device: /dev/sdb, found in smartd database.
Sep 25 07:45:13 treogen smartd[3087]: Device: /dev/sdb, is SMART capable. Adding to "monitor" list.
Sep 25 07:45:13 treogen smartd[3087]: Device: /dev/sdc, opened
Sep 25 07:45:13 treogen smartd[3087]: Device: /dev/sdc, found in smartd database.
Sep 25 07:45:13 treogen smartd[3087]: Device: /dev/sdc, is SMART capable. Adding to "monitor" list.
Sep 25 07:45:13 treogen smartd[3087]: Monitoring 3 ATA and 0 SCSI devices
Sep 25 07:45:13 treogen smartd[3100]: smartd has fork()ed into background mode. New PID=3100.
Sep 25 07:45:13 treogen smartd[3100]: file /var/run/smartd.pid written containing PID 3100
Sep 25 07:45:15 treogen cron[3128]: (CRON) STARTUP (V5.0)
Sep 25 07:46:08 treogen polkitd[3277]: started daemon version 0.96 using authority implementation `local' version `0.96'
Sep 25 07:46:08 treogen polkitd(authority=local): Registered Authentication Agent for session /org/freedesktop/ConsoleKit/Session1 (system bus name :1.11 [/usr/lib64/kde4/libexec/polkit-kde-authentication-agent-1 -session 10dbd7d36f000127743216200000067470009_1285365114_26164], object path /org/kde/PolicyKit1/AuthenticationAgent, locale en_GB.utf8)
Sep 25 07:46:14 treogen pulseaudio[3342]: pid.c: Daemon already running.
Sep 25 07:46:14 treogen pulseaudio[3343]: pid.c: Daemon already running.
Sep 25 07:46:14 treogen pulseaudio[3345]: pid.c: Daemon already running.
Sep 25 07:46:32 treogen su[3385]: Successful su for root by tk13
Sep 25 07:46:32 treogen su[3385]: + /dev/pts/3 tk13:root
Sep 25 07:46:32 treogen su[3385]: pam_unix(su:session): session opened for user root by tk13(uid=1000)
Sep 25 07:50:01 treogen cron[3444]: (root) CMD (test -x /usr/sbin/run-crons && /usr/sbin/run-crons )
[ 1033.312407] SysRq : Show Blocked State
[ 1035.876602] SysRq : Changing Loglevel
[ 1035.880405] Loglevel set to 6
[ 1081.760056] INFO: task X:2585 blocked for more than 120 seconds.
[ 1081.766302] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1081.774410]  ffff88011f2b9c70 0000000000003082 ffff880100000001 ffff880100000003
[ 1081.782146]  ffff8800078dd550 0000000000012780 ffff88011e031fd8 0000000000012780
[ 1081.789850]  ffff88011e031fd8 ffff88011f2b9ed0 ffff88011e031fd8 ffff88011f2b9ec8
[ 1081.797606] Call Trace:
[ 1081.800166]  [<ffffffff810801c0>] ? sync_page+0x0/0x50
[ 1081.805509]  [<ffffffff8156078d>] ? io_schedule+0x3d/0x60
[ 1081.811129]  [<ffffffff810801fd>] ? sync_page+0x3d/0x50
[ 1081.816556]  [<ffffffff81560b7a>] ? __wait_on_bit_lock+0x4a/0xa0
[ 1081.822802]  [<ffffffff8108019e>] ? __lock_page+0x5e/0x70
[ 1081.828420]  [<ffffffff81052ea0>] ? wake_bit_function+0x0/0x40
[ 1081.834488]  [<ffffffff81099ce7>] ? handle_mm_fault+0xa47/0xa60
[ 1081.840641]  [<ffffffff813d4b0c>] ? evdev_read+0xec/0x270
[ 1081.846258]  [<ffffffff810cd91f>] ? do_vfs_ioctl+0x9f/0x530
[ 1081.852077]  [<ffffffff8102404d>] ? do_page_fault+0x10d/0x320
[ 1081.858052]  [<ffffffff8156265f>] ? page_fault+0x1f/0x30
[ 1081.863591] INFO: task hald-addon-inpu:2805 blocked for more than 120 seconds.
[ 1081.871107] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1081.879245]  ffff8800071cf1c0 0000000000000082 ffff880100000001 ffff880000000003
[ 1081.887012]  ffff88007f71c440 0000000000012780 ffff88000417bfd8 0000000000012780
[ 1081.894777]  ffff88000417bfd8 ffff8800071cf420 ffff88000417bfd8 ffff8800071cf418
[ 1081.902517] Call Trace:
[ 1081.905078]  [<ffffffff810801c0>] ? sync_page+0x0/0x50
[ 1081.910443]  [<ffffffff8156078d>] ? io_schedule+0x3d/0x60
[ 1081.916048]  [<ffffffff810801fd>] ? sync_page+0x3d/0x50
[ 1081.921492]  [<ffffffff81560b7a>] ? __wait_on_bit_lock+0x4a/0xa0
[ 1081.927731]  [<ffffffff8108019e>] ? __lock_page+0x5e/0x70
[ 1081.933346]  [<ffffffff81052ea0>] ? wake_bit_function+0x0/0x40
[ 1081.939414]  [<ffffffff81099ce7>] ? handle_mm_fault+0xa47/0xa60
[ 1081.945575]  [<ffffffff815602ac>] ? schedule+0x24c/0x6f0
[ 1081.951277]  [<ffffffff8102404d>] ? do_page_fault+0x10d/0x320
[ 1081.957103]  [<ffffffff8156265f>] ? page_fault+0x1f/0x30
[ 1081.962494]  [<ffffffff810cf29e>] ? do_sys_poll+0x2be/0x4b0
[ 1081.968126]  [<ffffffff810cf285>] ? do_sys_poll+0x2a5/0x4b0
[ 1081.973787]  [<ffffffff810cecf0>] ? __pollwait+0x0/0x110
[ 1081.979167]  [<ffffffff810cee00>] ? pollwake+0x0/0x60
[ 1081.984298]  [<ffffffff810cee00>] ? pollwake+0x0/0x60
[ 1081.989421]  [<ffffffff810cee00>] ? pollwake+0x0/0x60
[ 1081.994541]  [<ffffffff810cee00>] ? pollwake+0x0/0x60
[ 1081.999665]  [<ffffffff810cee00>] ? pollwake+0x0/0x60
[ 1082.004779]  [<ffffffff810c22ed>] ? chrdev_open+0xdd/0x180
[ 1082.010346]  [<ffffffff81080e68>] ? find_get_page+0x18/0xa0
[ 1082.015992]  [<ffffffff810819e9>] ? filemap_fault+0x99/0x3f0
[ 1082.021733]  [<ffffffff814a3efa>] ? verify_iovec+0x6a/0xc0
[ 1082.027277]  [<ffffffff810973ea>] ? __do_fault+0x3da/0x4b0
[ 1082.032848]  [<ffffffff8156244e>] ? common_interrupt+0xe/0x13
[ 1082.038674]  [<ffffffff813d4b63>] ? evdev_read+0x143/0x270
[ 1082.044236]  [<ffffffff813d28e2>] ? input_event_to_user+0x62/0x70
[ 1082.050416]  [<ffffffff810cf691>] ? sys_poll+0x71/0x100
[ 1082.055712]  [<ffffffff8156265f>] ? page_fault+0x1f/0x30
[ 1082.061165]  [<ffffffff8100246b>] ? system_call_fastpath+0x16/0x1b
[ 1082.067654] INFO: task plasma-desktop:3169 blocked for more than 120 seconds.
[ 1082.075166] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1082.083425]  ffff88011ef860b0 0000000000000086 000000011ef860b0 ffff880114faff48
[ 1082.091234]  ffff8800071cf1c0 0000000000012780 ffff880114faffd8 0000000000012780
[ 1082.099051]  ffff880114faffd8 ffff88011ef86310 ffff880114faffd8 ffff88011ef86308
[ 1082.106887] Call Trace:
[ 1082.109467]  [<ffffffff815612c1>] ? __mutex_lock_slowpath+0xe1/0x160
[ 1082.116165]  [<ffffffff810d1e57>] ? __d_lookup+0x97/0x120
[ 1082.121849]  [<ffffffff81560d8a>] ? mutex_lock+0x1a/0x40
[ 1082.127439]  [<ffffffff810c9dbf>] ? do_lookup+0x10f/0x190
[ 1082.133099]  [<ffffffff810c7e24>] ? acl_permission_check+0x54/0xb0
[ 1082.139598]  [<ffffffff810ca8d9>] ? link_path_walk+0x4c9/0x9a0
[ 1082.145733]  [<ffffffff810d65f4>] ? mnt_want_write+0x34/0x70
[ 1082.151678]  [<ffffffff810caecf>] ? path_walk+0x5f/0xe0
[ 1082.157174]  [<ffffffff810caf9b>] ? do_path_lookup+0x4b/0x50
[ 1082.163110]  [<ffffffff810cbc35>] ? user_path_at+0x55/0xb0
[ 1082.168884]  [<ffffffff8102e860>] ? __dequeue_entity+0x40/0x50
[ 1082.175022]  [<ffffffff8107bcf4>] ? perf_event_task_sched_out+0x44/0x260
[ 1082.182056]  [<ffffffff810c2e46>] ? vfs_fstatat+0x36/0x80
[ 1082.187742]  [<ffffffff815602ac>] ? schedule+0x24c/0x6f0
[ 1082.193331]  [<ffffffff810c2f8f>] ? sys_newstat+0x1f/0x50
[ 1082.199019]  [<ffffffff81003035>] ? device_not_available+0x15/0x20
[ 1082.205535]  [<ffffffff8100246b>] ? system_call_fastpath+0x16/0x1b
[ 1082.212034] INFO: task virtuoso-t:3362 blocked for more than 120 seconds.
[ 1082.219163] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1082.227412]  ffff8800df8ac440 0000000000000082 ffff880100000001 ffff880000000003
[ 1082.235231]  ffff8800ba6e60b0 0000000000012780 ffff8800de265fd8 0000000000012780
[ 1082.243065]  ffff8800de265fd8 ffff8800df8ac6a0 ffff8800de265fd8 ffff8800df8ac698
[ 1082.250864] Call Trace:
[ 1082.253445]  [<ffffffff810801c0>] ? sync_page+0x0/0x50
[ 1082.258844]  [<ffffffff8156078d>] ? io_schedule+0x3d/0x60
[ 1082.264508]  [<ffffffff810801fd>] ? sync_page+0x3d/0x50
[ 1082.269989]  [<ffffffff81560b7a>] ? __wait_on_bit_lock+0x4a/0xa0
[ 1082.276296]  [<ffffffff8108019e>] ? __lock_page+0x5e/0x70
[ 1082.281984]  [<ffffffff81052ea0>] ? wake_bit_function+0x0/0x40
[ 1082.288119]  [<ffffffff81099ce7>] ? handle_mm_fault+0xa47/0xa60
[ 1082.294333]  [<ffffffff815602ac>] ? schedule+0x24c/0x6f0
[ 1082.299905]  [<ffffffff8102404d>] ? do_page_fault+0x10d/0x320
[ 1082.305936]  [<ffffffff8156265f>] ? page_fault+0x1f/0x30
[ 1082.311541] INFO: task cc1plus:12626 blocked for more than 120 seconds.
[ 1082.318473] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1082.326698]  ffff88003ecf2d80 0000000000000082 ffff880100000001 ffff880000000003
[ 1082.334552]  ffff88001f95b8e0 0000000000012780 ffff88000ec89fd8 0000000000012780
[ 1082.342384]  ffff88000ec89fd8 ffff88003ecf2fe0 ffff88000ec89fd8 ffff88003ecf2fd8
[ 1082.350246] Call Trace:
[ 1082.352825]  [<ffffffff810801c0>] ? sync_page+0x0/0x50
[ 1082.358213]  [<ffffffff8156078d>] ? io_schedule+0x3d/0x60
[ 1082.363882]  [<ffffffff810801fd>] ? sync_page+0x3d/0x50
[ 1082.369375]  [<ffffffff81560b7a>] ? __wait_on_bit_lock+0x4a/0xa0
[ 1082.375701]  [<ffffffff8108019e>] ? __lock_page+0x5e/0x70
[ 1082.381397]  [<ffffffff81052ea0>] ? wake_bit_function+0x0/0x40
[ 1082.387526]  [<ffffffff81099ce7>] ? handle_mm_fault+0xa47/0xa60
[ 1082.393765]  [<ffffffff8102404d>] ? do_page_fault+0x10d/0x320
[ 1082.399786]  [<ffffffff8156265f>] ? page_fault+0x1f/0x30
[ 1202.400046] INFO: task kworker/u:8:845 blocked for more than 120 seconds.
[ 1202.407128] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1202.415372]  ffff88007ffa8b60 0000000000000046 ffff88007f74fe98 0000000000000000
[ 1202.423229]  ffff8800071cf770 0000000000012780 ffff8800070e9fd8 0000000000012780
[ 1202.431054]  ffff8800070e9fd8 ffff88007ffa8dc0 ffff8800070e9fd8 ffff88007ffa8db8
[ 1202.438898] Call Trace:
[ 1202.441495]  [<ffffffff814361ed>] ? md_write_start+0x9d/0x190
[ 1202.447536]  [<ffffffff81052e70>] ? autoremove_wake_function+0x0/0x30
[ 1202.454311]  [<ffffffff810925ca>] ? zone_nr_free_pages+0x9a/0xb0
[ 1202.460639]  [<ffffffff8142c423>] ? make_request+0x23/0x800
[ 1202.466496]  [<ffffffff811d90ea>] ? blkcipher_walk_done+0x8a/0x230
[ 1202.472990]  [<ffffffff811df4c4>] ? crypto_cbc_encrypt+0xe4/0x180
[ 1202.479327]  [<ffffffff8102a150>] ? twofish_encrypt+0x0/0x10
[ 1202.485276]  [<ffffffff81431ac3>] ? md_make_request+0xc3/0x220
[ 1202.491412]  [<ffffffff811eafca>] ? generic_make_request+0x18a/0x330
[ 1202.498096]  [<ffffffff814484fc>] ? crypt_convert+0x25c/0x310
[ 1202.504090]  [<ffffffff81448881>] ? kcryptd_crypt+0x2d1/0x3f0
[ 1202.510031]  [<ffffffff814485b0>] ? kcryptd_crypt+0x0/0x3f0
[ 1202.516081]  [<ffffffff8104d2eb>] ? process_one_work+0xfb/0x370
[ 1202.522123]  [<ffffffff8104ee4c>] ? worker_thread+0x16c/0x360
[ 1202.527935]  [<ffffffff8104ece0>] ? worker_thread+0x0/0x360
[ 1202.533541]  [<ffffffff8104ece0>] ? worker_thread+0x0/0x360
[ 1202.539131]  [<ffffffff810529e6>] ? kthread+0x96/0xa0
[ 1202.544245]  [<ffffffff81003194>] ? kernel_thread_helper+0x4/0x10
[ 1202.550363]  [<ffffffff81052950>] ? kthread+0x0/0xa0
[ 1202.555346]  [<ffffffff81003190>] ? kernel_thread_helper+0x0/0x10
[ 1202.561479] INFO: task X:2585 blocked for more than 120 seconds.
[ 1202.567503] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1202.575468]  ffff88011f2b9c70 0000000000003082 ffff880100000001 ffff880100000003
[ 1202.583060]  ffff8800078dd550 0000000000012780 ffff88011e031fd8 0000000000012780
[ 1202.590757]  ffff88011e031fd8 ffff88011f2b9ed0 ffff88011e031fd8 ffff88011f2b9ec8
[ 1202.598538] Call Trace:
[ 1202.601059]  [<ffffffff810801c0>] ? sync_page+0x0/0x50
[ 1202.606381]  [<ffffffff8156078d>] ? io_schedule+0x3d/0x60
[ 1202.611854]  [<ffffffff810801fd>] ? sync_page+0x3d/0x50
[ 1202.617210]  [<ffffffff81560b7a>] ? __wait_on_bit_lock+0x4a/0xa0
[ 1202.623684]  [<ffffffff8108019e>] ? __lock_page+0x5e/0x70
[ 1202.629300]  [<ffffffff81052ea0>] ? wake_bit_function+0x0/0x40
[ 1202.635396]  [<ffffffff81099ce7>] ? handle_mm_fault+0xa47/0xa60
[ 1202.641606]  [<ffffffff813d4b0c>] ? evdev_read+0xec/0x270
[ 1202.647275]  [<ffffffff810cd91f>] ? do_vfs_ioctl+0x9f/0x530
[ 1202.653125]  [<ffffffff8102404d>] ? do_page_fault+0x10d/0x320
[ 1202.659105]  [<ffffffff8156265f>] ? page_fault+0x1f/0x30
[ 1202.664690] INFO: task hald-addon-inpu:2805 blocked for more than 120 seconds.
[ 1202.672282] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1202.680507]  ffff8800071cf1c0 0000000000000082 ffff880100000001 ffff880000000003
[ 1202.688289]  ffff88007f71c440 0000000000012780 ffff88000417bfd8 0000000000012780
[ 1202.696100]  ffff88000417bfd8 ffff8800071cf420 ffff88000417bfd8 ffff8800071cf418
[ 1202.703924] Call Trace:
[ 1202.706503]  [<ffffffff810801c0>] ? sync_page+0x0/0x50
[ 1202.711901]  [<ffffffff8156078d>] ? io_schedule+0x3d/0x60
[ 1202.717577]  [<ffffffff810801fd>] ? sync_page+0x3d/0x50
[ 1202.723081]  [<ffffffff81560b7a>] ? __wait_on_bit_lock+0x4a/0xa0
[ 1202.729381]  [<ffffffff8108019e>] ? __lock_page+0x5e/0x70
[ 1202.735031]  [<ffffffff81052ea0>] ? wake_bit_function+0x0/0x40
[ 1202.741116]  [<ffffffff81099ce7>] ? handle_mm_fault+0xa47/0xa60
[ 1202.747340]  [<ffffffff815602ac>] ? schedule+0x24c/0x6f0
[ 1202.752902]  [<ffffffff8102404d>] ? do_page_fault+0x10d/0x320
[ 1202.758893]  [<ffffffff8156265f>] ? page_fault+0x1f/0x30
[ 1202.764428]  [<ffffffff810cf29e>] ? do_sys_poll+0x2be/0x4b0
[ 1202.770338]  [<ffffffff810cf285>] ? do_sys_poll+0x2a5/0x4b0
[ 1202.776113]  [<ffffffff810cecf0>] ? __pollwait+0x0/0x110
[ 1202.781591]  [<ffffffff810cee00>] ? pollwake+0x0/0x60
[ 1202.786764]  [<ffffffff810cee00>] ? pollwake+0x0/0x60
[ 1202.791973]  [<ffffffff810cee00>] ? pollwake+0x0/0x60
[ 1202.797181]  [<ffffffff810cee00>] ? pollwake+0x0/0x60
[ 1202.802468]  [<ffffffff810cee00>] ? pollwake+0x0/0x60
[ 1202.807773]  [<ffffffff810c22ed>] ? chrdev_open+0xdd/0x180
[ 1202.813545]  [<ffffffff81080e68>] ? find_get_page+0x18/0xa0
[ 1202.819384]  [<ffffffff810819e9>] ? filemap_fault+0x99/0x3f0
[ 1202.825326]  [<ffffffff814a3efa>] ? verify_iovec+0x6a/0xc0
[ 1202.831095]  [<ffffffff810973ea>] ? __do_fault+0x3da/0x4b0
[ 1202.836847]  [<ffffffff8156244e>] ? common_interrupt+0xe/0x13
[ 1202.842865]  [<ffffffff813d4b63>] ? evdev_read+0x143/0x270
[ 1202.848626]  [<ffffffff813d28e2>] ? input_event_to_user+0x62/0x70
[ 1202.855032]  [<ffffffff810cf691>] ? sys_poll+0x71/0x100
[ 1202.860499]  [<ffffffff8156265f>] ? page_fault+0x1f/0x30
[ 1202.866005]  [<ffffffff8100246b>] ? system_call_fastpath+0x16/0x1b
[ 1202.872424] INFO: task plasma-desktop:3169 blocked for more than 120 seconds.
[ 1202.879883] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1202.888118]  ffff88011ef860b0 0000000000000086 000000011ef860b0 ffff880114faff48
[ 1202.895936]  ffff8800071cf1c0 0000000000012780 ffff880114faffd8 0000000000012780
[ 1202.903777]  ffff880114faffd8 ffff88011ef86310 ffff880114faffd8 ffff88011ef86308
[ 1202.911586] Call Trace:
[ 1202.914140]  [<ffffffff815612c1>] ? __mutex_lock_slowpath+0xe1/0x160
[ 1202.920693]  [<ffffffff810d1e57>] ? __d_lookup+0x97/0x120
[ 1202.926272]  [<ffffffff81560d8a>] ? mutex_lock+0x1a/0x40
[ 1202.931783]  [<ffffffff810c9dbf>] ? do_lookup+0x10f/0x190
[ 1202.937426]  [<ffffffff810c7e24>] ? acl_permission_check+0x54/0xb0
[ 1202.943898]  [<ffffffff810ca8d9>] ? link_path_walk+0x4c9/0x9a0
[ 1202.950028]  [<ffffffff810d65f4>] ? mnt_want_write+0x34/0x70
[ 1202.955947]  [<ffffffff810caecf>] ? path_walk+0x5f/0xe0
[ 1202.961416]  [<ffffffff810caf9b>] ? do_path_lookup+0x4b/0x50
[ 1202.967325]  [<ffffffff810cbc35>] ? user_path_at+0x55/0xb0
[ 1202.973020]  [<ffffffff8102e860>] ? __dequeue_entity+0x40/0x50
[ 1202.979054]  [<ffffffff8107bcf4>] ? perf_event_task_sched_out+0x44/0x260
[ 1202.985995]  [<ffffffff810c2e46>] ? vfs_fstatat+0x36/0x80
[ 1202.991600]  [<ffffffff815602ac>] ? schedule+0x24c/0x6f0
[ 1202.997157]  [<ffffffff810c2f8f>] ? sys_newstat+0x1f/0x50
[ 1203.002784]  [<ffffffff81003035>] ? device_not_available+0x15/0x20
[ 1203.009204]  [<ffffffff8100246b>] ? system_call_fastpath+0x16/0x1b
[ 1203.015608] INFO: task virtuoso-t:3362 blocked for more than 120 seconds.
[ 1203.022633] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1203.030726]  ffff8800df8ac440 0000000000000082 ffff880100000001 ffff880000000003
[ 1203.038449]  ffff8800ba6e60b0 0000000000012780 ffff8800de265fd8 0000000000012780
[ 1203.046215]  ffff8800de265fd8 ffff8800df8ac6a0 ffff8800de265fd8 ffff8800df8ac698
[ 1203.053988] Call Trace:
[ 1203.056532]  [<ffffffff810801c0>] ? sync_page+0x0/0x50
[ 1203.061836]  [<ffffffff8156078d>] ? io_schedule+0x3d/0x60
[ 1203.067390]  [<ffffffff810801fd>] ? sync_page+0x3d/0x50
[ 1203.072775]  [<ffffffff81560b7a>] ? __wait_on_bit_lock+0x4a/0xa0
[ 1203.078926]  [<ffffffff8108019e>] ? __lock_page+0x5e/0x70
[ 1203.084552]  [<ffffffff81052ea0>] ? wake_bit_function+0x0/0x40
[ 1203.090670]  [<ffffffff81099ce7>] ? handle_mm_fault+0xa47/0xa60
[ 1203.096867]  [<ffffffff815602ac>] ? schedule+0x24c/0x6f0
[ 1203.102439]  [<ffffffff8102404d>] ? do_page_fault+0x10d/0x320
[ 1203.108462]  [<ffffffff8156265f>] ? page_fault+0x1f/0x30
[ 1229.421480] SysRq : Emergency Sync
[ 1237.014575] SysRq : Emergency Sync
[ 1239.121784] SysRq : Emergency Remount R/O
[ 1243.234840] SysRq : Emergency Sync
[ 1246.327060] SysRq : Power Off

[-- Attachment #3: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: -mm: xfs lockdep warning

[Christoph Hellwig]

On Tue, Oct 05, 2010 at 08:09:58PM +0200, Torsten Kaiser wrote:
> Here is the complete log from bootup until the hang happened the second time.
> I just logged into KDE and then started the upgrade of KOffice, which
> meant that the package manager started to compile 4 programs each with
> -j5. After a few minutes the load got to ~18 and probably at the point
> where filecache and GCCs filled up my 4GB RAM the system got stuck.
> The mouse so longer moved and SysRq+S did not work. The log was
> captured via serial console on a second system.
> 
> I will try tomorrow with frame pointers enabled..

Lots of hangs in sync_page and dm-crypt in the stack is unfortunately
something we've seen quite a few times, and it usually points to that
layer.  In the best case we might just have a similar workqueue priority
issue somewhere in the dm-crypt or maybe just md code. 

Re: -mm: xfs lockdep warning

[Christoph Hellwig]

On Tue, Oct 05, 2010 at 08:09:58PM +0200, Torsten Kaiser wrote:
> Here is the complete log from bootup until the hang happened the second time.
> I just logged into KDE and then started the upgrade of KOffice, which
> meant that the package manager started to compile 4 programs each with
> -j5. After a few minutes the load got to ~18 and probably at the point
> where filecache and GCCs filled up my 4GB RAM the system got stuck.
> The mouse so longer moved and SysRq+S did not work. The log was
> captured via serial console on a second system.
> 
> I will try tomorrow with frame pointers enabled..

Lots of hangs in sync_page and dm-crypt in the stack is unfortunately
something we've seen quite a few times, and it usually points to that
layer.  In the best case we might just have a similar workqueue priority
issue somewhere in the dm-crypt or maybe just md code. 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: -mm: xfs lockdep warning

[Tejun Heo]

(restoring cc to lkml)
Hello,

On 10/06/2010 08:32 PM, Tejun Heo wrote:
> On 10/06/2010 07:20 PM, Torsten Kaiser wrote:
>>> It seems the system isn't completely stuck yet.  The above process is
>>> still trying to free memory.  Maybe memory reclaim is just extremely
>>> slow for some reason?
>>
>> How can I check this? With the hund -rc3 I repeatedly hit SysRq+M
>> every time waiting 10..60 seconds.
>> Can you see from these outputs if reclaim ins making any progress?
> 
> I think I have an idea of what's going on.  I'll prep a patch and send
> it to you tomorrow.

Can you test whether the following patch fix the issue?

Thanks.

diff --git a/block/blk-core.c b/block/blk-core.c
index 32a1c12..622602b 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -2584,7 +2584,7 @@ int __init blk_dev_init(void)
 	BUILD_BUG_ON(__REQ_NR_BITS > 8 *
 			sizeof(((struct request *)0)->cmd_flags));

-	kblockd_workqueue = create_workqueue("kblockd");
+	kblockd_workqueue = alloc_workqueue("kblockd", WQ_MEM_RECLAIM, 1);
 	if (!kblockd_workqueue)
 		panic("Failed to create kblockd\n");

diff --git a/crypto/crypto_wq.c b/crypto/crypto_wq.c
index fdcf624..4c893a1 100644
--- a/crypto/crypto_wq.c
+++ b/crypto/crypto_wq.c
@@ -20,7 +20,7 @@ EXPORT_SYMBOL_GPL(kcrypto_wq);

 static int __init crypto_wq_init(void)
 {
-	kcrypto_wq = create_workqueue("crypto");
+	kcrypto_wq = alloc_workqueue("crypto", WQ_MEM_RECLAIM, 1);
 	if (unlikely(!kcrypto_wq))
 		return -ENOMEM;
 	return 0;
diff --git a/crypto/pcrypt.c b/crypto/pcrypt.c
index de30782..2ee6ba2 100644
--- a/crypto/pcrypt.c
+++ b/crypto/pcrypt.c
@@ -455,7 +455,7 @@ static int pcrypt_init_padata(struct padata_pcrypt *pcrypt,

 	get_online_cpus();

-	pcrypt->wq = create_workqueue(name);
+	pcrypt->wq = alloc_workqueue(name, WQ_MEM_RECLAIM, 1);
 	if (!pcrypt->wq)
 		goto err;

diff --git a/drivers/infiniband/core/cm.c b/drivers/infiniband/core/cm.c
index 64e0903..2631930 100644
--- a/drivers/infiniband/core/cm.c
+++ b/drivers/infiniband/core/cm.c
@@ -3804,7 +3804,7 @@ static int __init ib_cm_init(void)
 	if (ret)
 		return -ENOMEM;

-	cm.wq = create_workqueue("ib_cm");
+	cm.wq = alloc_workqueue("ib_cm", WQ_MEM_RECLAIM, 1);
 	if (!cm.wq) {
 		ret = -ENOMEM;
 		goto error1;
diff --git a/drivers/infiniband/hw/qib/qib_init.c b/drivers/infiniband/hw/qib/qib_init.c
index f1d16d3..0f559f2 100644
--- a/drivers/infiniband/hw/qib/qib_init.c
+++ b/drivers/infiniband/hw/qib/qib_init.c
@@ -1053,7 +1053,7 @@ static int __init qlogic_ib_init(void)
 	 * so flush_scheduled_work() can deadlock during device
 	 * removal.
 	 */
-	qib_wq = create_workqueue("qib");
+	qib_wq = alloc_workqueue("qib", WQ_MEM_RECLAIM, 1);
 	if (!qib_wq) {
 		ret = -ENOMEM;
 		goto bail_dev;
diff --git a/drivers/md/dm-delay.c b/drivers/md/dm-delay.c
index baa1191..1f8d1e4 100644
--- a/drivers/md/dm-delay.c
+++ b/drivers/md/dm-delay.c
@@ -352,7 +352,7 @@ static int __init dm_delay_init(void)
 {
 	int r = -ENOMEM;

-	kdelayd_wq = create_workqueue("kdelayd");
+	kdelayd_wq = alloc_workqueue("kdelayd", WQ_MEM_RECLAIM, 1);
 	if (!kdelayd_wq) {
 		DMERR("Couldn't start kdelayd");
 		goto bad_queue;
diff --git a/drivers/md/dm-mpath.c b/drivers/md/dm-mpath.c
index 487ecda..e60f22d 100644
--- a/drivers/md/dm-mpath.c
+++ b/drivers/md/dm-mpath.c
@@ -1687,7 +1687,7 @@ static int __init dm_multipath_init(void)
 		return -EINVAL;
 	}

-	kmultipathd = create_workqueue("kmpathd");
+	kmultipathd = alloc_workqueue("kmpathd", WQ_MEM_RECLAIM, 1);
 	if (!kmultipathd) {
 		DMERR("failed to create workqueue kmpathd");
 		dm_unregister_target(&multipath_target);
diff --git a/drivers/message/i2o/driver.c b/drivers/message/i2o/driver.c
index a0421ef..8a5b2d8 100644
--- a/drivers/message/i2o/driver.c
+++ b/drivers/message/i2o/driver.c
@@ -84,7 +84,8 @@ int i2o_driver_register(struct i2o_driver *drv)
 	osm_debug("Register driver %s\n", drv->name);

 	if (drv->event) {
-		drv->event_queue = create_workqueue(drv->name);
+		drv->event_queue = alloc_workqueue(drv->name,
+						   WQ_MEM_RECLAIM, 1);
 		if (!drv->event_queue) {
 			osm_err("Could not initialize event queue for driver "
 				"%s\n", drv->name);
diff --git a/drivers/scsi/qla2xxx/qla_os.c b/drivers/scsi/qla2xxx/qla_os.c
index 1e4bff6..a6dd94d 100644
--- a/drivers/scsi/qla2xxx/qla_os.c
+++ b/drivers/scsi/qla2xxx/qla_os.c
@@ -351,7 +351,7 @@ static int qla25xx_setup_mode(struct scsi_qla_host *vha)
 				"Can't create request queue\n");
 			goto fail;
 		}
-		ha->wq = create_workqueue("qla2xxx_wq");
+		ha->wq = alloc_workqueue("qla2xxx_wq", WQ_MEM_RECLAIM, 1);
 		vha->req = ha->req_q_map[req];
 		options |= BIT_1;
 		for (ques = 1; ques < ha->max_rsp_queues; ques++) {
diff --git a/fs/bio-integrity.c b/fs/bio-integrity.c
index 4d0ff5e..b37653f 100644
--- a/fs/bio-integrity.c
+++ b/fs/bio-integrity.c
@@ -782,7 +782,7 @@ void __init bio_integrity_init(void)
 {
 	unsigned int i;

-	kintegrityd_wq = create_workqueue("kintegrityd");
+	kintegrityd_wq = alloc_workqueue("kintegrityd", WQ_MEM_RECLAIM, 1);
 	if (!kintegrityd_wq)
 		panic("Failed to create kintegrityd\n");

diff --git a/fs/ext4/super.c b/fs/ext4/super.c
index 2614774..5efccc6 100644
--- a/fs/ext4/super.c
+++ b/fs/ext4/super.c
@@ -3038,7 +3038,8 @@ no_journal:
 		goto failed_mount_wq;
 	}

-	EXT4_SB(sb)->dio_unwritten_wq = create_workqueue("ext4-dio-unwritten");
+	EXT4_SB(sb)->dio_unwritten_wq = alloc_workqueue("ext4-dio-unwritten",
+							WQ_MEM_RECLAIM, 1);
 	if (!EXT4_SB(sb)->dio_unwritten_wq) {
 		printk(KERN_ERR "EXT4-fs: failed to create DIO workqueue\n");
 		goto failed_mount_wq;
diff --git a/fs/reiserfs/journal.c b/fs/reiserfs/journal.c
index 812e2c0..03275f1 100644
--- a/fs/reiserfs/journal.c
+++ b/fs/reiserfs/journal.c
@@ -2950,7 +2950,7 @@ int journal_init(struct super_block *sb, const char *j_dev_name,
 	reiserfs_mounted_fs_count++;
 	if (reiserfs_mounted_fs_count <= 1) {
 		reiserfs_write_unlock(sb);
-		commit_wq = create_workqueue("reiserfs");
+		commit_wq = alloc_workqueue("reiserfs", WQ_MEM_RECLAIM, 1);
 		reiserfs_write_lock(sb);
 	}

diff --git a/include/linux/workqueue.h b/include/linux/workqueue.h
index 25e02c9..77cb9d9 100644
--- a/include/linux/workqueue.h
+++ b/include/linux/workqueue.h
@@ -247,6 +247,8 @@ enum {
 	WQ_HIGHPRI		= 1 << 4, /* high priority */
 	WQ_CPU_INTENSIVE	= 1 << 5, /* cpu instensive workqueue */

+	WQ_MEM_RECLAIM		= WQ_RESCUER | WQ_HIGHPRI,
+
 	WQ_DYING		= 1 << 6, /* internal: workqueue is dying */

 	WQ_MAX_ACTIVE		= 512,	  /* I like 512, better ideas? */

Re: -mm: xfs lockdep warning

[Torsten Kaiser]

On Thu, Oct 7, 2010 at 1:14 PM, Tejun Heo <tj@kernel.org> wrote:
> (restoring cc to lkml)
> Hello,
>
> On 10/06/2010 08:32 PM, Tejun Heo wrote:
>> On 10/06/2010 07:20 PM, Torsten Kaiser wrote:
>>>> It seems the system isn't completely stuck yet.  The above process is
>>>> still trying to free memory.  Maybe memory reclaim is just extremely
>>>> slow for some reason?
>>>
>>> How can I check this? With the hund -rc3 I repeatedly hit SysRq+M
>>> every time waiting 10..60 seconds.
>>> Can you see from these outputs if reclaim ins making any progress?
>>
>> I think I have an idea of what's going on.  I'll prep a patch and send
>> it to you tomorrow.
>
> Can you test whether the following patch fix the issue?

Yes. After adding this patch to 2.6.36-rc6 it survived my testcase. It
first seems to get stuck again, but after less then a minute the
system recovered and all compiles finished successfully.

I suspect the combination of CFLAGs, the gcc version and maybe even
changes in KOffice from 2.2.1 to 2.2.2 are import for my testcase. It
looks like at the start of the complies something is really eating an
enormous amount of memory and I did not hit such a case with earlier
kernels.

output from vmstat 60 during the start of the testcase:
 1  0      0 2442196   1060 698180    0    0   242    19 6847 15039 26  7 66  1
12  1      0 1786212   1060 1040988    0    0  1248    15 6401 21721 56 20 19  5
18  0      0 787880   1060 1097136    0    0   270     9  742 1614 72 28  0  0
18  0      0 322680   1060 1110036    0    0    47     4  611 1234 71 29  0  0
18  1 161208 975064     96 412960   38 2709   757  2722 1616 2958 79 21  0  0
27 28 1000036  14728      0 135520   12 13991   503 14023 1808 3781 25  8  0 66
18  4 1296652 815832      0 131852 1824 8245  3268  8266 2037 4365 42 16  0 42
21  0 857012 903888      0 185292 1351    0  1810    65  846 1678 87 13  0  0
16  0 817320 1890752      0 217404  102    0   275    18  530 1142 89 11  0  0
11  0 652620 2441520      0 270528 1132    0  1424    38  934 2783 79 18  1  2
11  1 636148 1629492      0 318020   43    0   413   144  613 1417 84 16  0  0

As the system recovered from that, I think your patch is the correct
solution for my problem.
I will run my testcase later again, but if you do not hear from me, I
was not able to get it stuck with your patch.

Thanks for looking into this and providing this patch!

Torsten

> diff --git a/block/blk-core.c b/block/blk-core.c
> index 32a1c12..622602b 100644
> --- a/block/blk-core.c
> +++ b/block/blk-core.c
> @@ -2584,7 +2584,7 @@ int __init blk_dev_init(void)
>        BUILD_BUG_ON(__REQ_NR_BITS > 8 *
>                        sizeof(((struct request *)0)->cmd_flags));
>
> -       kblockd_workqueue = create_workqueue("kblockd");
> +       kblockd_workqueue = alloc_workqueue("kblockd", WQ_MEM_RECLAIM, 1);
>        if (!kblockd_workqueue)
>                panic("Failed to create kblockd\n");
>
> diff --git a/crypto/crypto_wq.c b/crypto/crypto_wq.c
> index fdcf624..4c893a1 100644
> --- a/crypto/crypto_wq.c
> +++ b/crypto/crypto_wq.c
> @@ -20,7 +20,7 @@ EXPORT_SYMBOL_GPL(kcrypto_wq);
>
>  static int __init crypto_wq_init(void)
>  {
> -       kcrypto_wq = create_workqueue("crypto");
> +       kcrypto_wq = alloc_workqueue("crypto", WQ_MEM_RECLAIM, 1);
>        if (unlikely(!kcrypto_wq))
>                return -ENOMEM;
>        return 0;
> diff --git a/crypto/pcrypt.c b/crypto/pcrypt.c
> index de30782..2ee6ba2 100644
> --- a/crypto/pcrypt.c
> +++ b/crypto/pcrypt.c
> @@ -455,7 +455,7 @@ static int pcrypt_init_padata(struct padata_pcrypt *pcrypt,
>
>        get_online_cpus();
>
> -       pcrypt->wq = create_workqueue(name);
> +       pcrypt->wq = alloc_workqueue(name, WQ_MEM_RECLAIM, 1);
>        if (!pcrypt->wq)
>                goto err;
>
> diff --git a/drivers/infiniband/core/cm.c b/drivers/infiniband/core/cm.c
> index 64e0903..2631930 100644
> --- a/drivers/infiniband/core/cm.c
> +++ b/drivers/infiniband/core/cm.c
> @@ -3804,7 +3804,7 @@ static int __init ib_cm_init(void)
>        if (ret)
>                return -ENOMEM;
>
> -       cm.wq = create_workqueue("ib_cm");
> +       cm.wq = alloc_workqueue("ib_cm", WQ_MEM_RECLAIM, 1);
>        if (!cm.wq) {
>                ret = -ENOMEM;
>                goto error1;
> diff --git a/drivers/infiniband/hw/qib/qib_init.c b/drivers/infiniband/hw/qib/qib_init.c
> index f1d16d3..0f559f2 100644
> --- a/drivers/infiniband/hw/qib/qib_init.c
> +++ b/drivers/infiniband/hw/qib/qib_init.c
> @@ -1053,7 +1053,7 @@ static int __init qlogic_ib_init(void)
>         * so flush_scheduled_work() can deadlock during device
>         * removal.
>         */
> -       qib_wq = create_workqueue("qib");
> +       qib_wq = alloc_workqueue("qib", WQ_MEM_RECLAIM, 1);
>        if (!qib_wq) {
>                ret = -ENOMEM;
>                goto bail_dev;
> diff --git a/drivers/md/dm-delay.c b/drivers/md/dm-delay.c
> index baa1191..1f8d1e4 100644
> --- a/drivers/md/dm-delay.c
> +++ b/drivers/md/dm-delay.c
> @@ -352,7 +352,7 @@ static int __init dm_delay_init(void)
>  {
>        int r = -ENOMEM;
>
> -       kdelayd_wq = create_workqueue("kdelayd");
> +       kdelayd_wq = alloc_workqueue("kdelayd", WQ_MEM_RECLAIM, 1);
>        if (!kdelayd_wq) {
>                DMERR("Couldn't start kdelayd");
>                goto bad_queue;
> diff --git a/drivers/md/dm-mpath.c b/drivers/md/dm-mpath.c
> index 487ecda..e60f22d 100644
> --- a/drivers/md/dm-mpath.c
> +++ b/drivers/md/dm-mpath.c
> @@ -1687,7 +1687,7 @@ static int __init dm_multipath_init(void)
>                return -EINVAL;
>        }
>
> -       kmultipathd = create_workqueue("kmpathd");
> +       kmultipathd = alloc_workqueue("kmpathd", WQ_MEM_RECLAIM, 1);
>        if (!kmultipathd) {
>                DMERR("failed to create workqueue kmpathd");
>                dm_unregister_target(&multipath_target);
> diff --git a/drivers/message/i2o/driver.c b/drivers/message/i2o/driver.c
> index a0421ef..8a5b2d8 100644
> --- a/drivers/message/i2o/driver.c
> +++ b/drivers/message/i2o/driver.c
> @@ -84,7 +84,8 @@ int i2o_driver_register(struct i2o_driver *drv)
>        osm_debug("Register driver %s\n", drv->name);
>
>        if (drv->event) {
> -               drv->event_queue = create_workqueue(drv->name);
> +               drv->event_queue = alloc_workqueue(drv->name,
> +                                                  WQ_MEM_RECLAIM, 1);
>                if (!drv->event_queue) {
>                        osm_err("Could not initialize event queue for driver "
>                                "%s\n", drv->name);
> diff --git a/drivers/scsi/qla2xxx/qla_os.c b/drivers/scsi/qla2xxx/qla_os.c
> index 1e4bff6..a6dd94d 100644
> --- a/drivers/scsi/qla2xxx/qla_os.c
> +++ b/drivers/scsi/qla2xxx/qla_os.c
> @@ -351,7 +351,7 @@ static int qla25xx_setup_mode(struct scsi_qla_host *vha)
>                                "Can't create request queue\n");
>                        goto fail;
>                }
> -               ha->wq = create_workqueue("qla2xxx_wq");
> +               ha->wq = alloc_workqueue("qla2xxx_wq", WQ_MEM_RECLAIM, 1);
>                vha->req = ha->req_q_map[req];
>                options |= BIT_1;
>                for (ques = 1; ques < ha->max_rsp_queues; ques++) {
> diff --git a/fs/bio-integrity.c b/fs/bio-integrity.c
> index 4d0ff5e..b37653f 100644
> --- a/fs/bio-integrity.c
> +++ b/fs/bio-integrity.c
> @@ -782,7 +782,7 @@ void __init bio_integrity_init(void)
>  {
>        unsigned int i;
>
> -       kintegrityd_wq = create_workqueue("kintegrityd");
> +       kintegrityd_wq = alloc_workqueue("kintegrityd", WQ_MEM_RECLAIM, 1);
>        if (!kintegrityd_wq)
>                panic("Failed to create kintegrityd\n");
>
> diff --git a/fs/ext4/super.c b/fs/ext4/super.c
> index 2614774..5efccc6 100644
> --- a/fs/ext4/super.c
> +++ b/fs/ext4/super.c
> @@ -3038,7 +3038,8 @@ no_journal:
>                goto failed_mount_wq;
>        }
>
> -       EXT4_SB(sb)->dio_unwritten_wq = create_workqueue("ext4-dio-unwritten");
> +       EXT4_SB(sb)->dio_unwritten_wq = alloc_workqueue("ext4-dio-unwritten",
> +                                                       WQ_MEM_RECLAIM, 1);
>        if (!EXT4_SB(sb)->dio_unwritten_wq) {
>                printk(KERN_ERR "EXT4-fs: failed to create DIO workqueue\n");
>                goto failed_mount_wq;
> diff --git a/fs/reiserfs/journal.c b/fs/reiserfs/journal.c
> index 812e2c0..03275f1 100644
> --- a/fs/reiserfs/journal.c
> +++ b/fs/reiserfs/journal.c
> @@ -2950,7 +2950,7 @@ int journal_init(struct super_block *sb, const char *j_dev_name,
>        reiserfs_mounted_fs_count++;
>        if (reiserfs_mounted_fs_count <= 1) {
>                reiserfs_write_unlock(sb);
> -               commit_wq = create_workqueue("reiserfs");
> +               commit_wq = alloc_workqueue("reiserfs", WQ_MEM_RECLAIM, 1);
>                reiserfs_write_lock(sb);
>        }
>
> diff --git a/include/linux/workqueue.h b/include/linux/workqueue.h
> index 25e02c9..77cb9d9 100644
> --- a/include/linux/workqueue.h
> +++ b/include/linux/workqueue.h
> @@ -247,6 +247,8 @@ enum {
>        WQ_HIGHPRI              = 1 << 4, /* high priority */
>        WQ_CPU_INTENSIVE        = 1 << 5, /* cpu instensive workqueue */
>
> +       WQ_MEM_RECLAIM          = WQ_RESCUER | WQ_HIGHPRI,
> +
>        WQ_DYING                = 1 << 6, /* internal: workqueue is dying */
>
>        WQ_MAX_ACTIVE           = 512,    /* I like 512, better ideas? */
>

Re: -mm: xfs lockdep warning

[Tejun Heo]

Hello,

On 10/07/2010 03:21 PM, Torsten Kaiser wrote:
>> Can you test whether the following patch fix the issue?
> 
> Yes. After adding this patch to 2.6.36-rc6 it survived my testcase. It
> first seems to get stuck again, but after less then a minute the
> system recovered and all compiles finished successfully.
> 
> I suspect the combination of CFLAGs, the gcc version and maybe even
> changes in KOffice from 2.2.1 to 2.2.2 are import for my testcase. It
> looks like at the start of the complies something is really eating an
> enormous amount of memory and I did not hit such a case with earlier
> kernels.

It's great to hear that the patch helped but would you be able to
repeat the same test w/ 2.6.35 and see whether there's any noticeable
difference?  Some level of stuttering is expected if the system is hit
with sudden huge spike of memory pressure but let's see if it has
regressed somehow.

Thanks.

-- 
tejun

Re: -mm: xfs lockdep warning

[Torsten Kaiser]

[-- Attachment #1: Type: text/plain, Size: 3419 bytes --]

On Thu, Oct 7, 2010 at 4:05 PM, Tejun Heo <tj@kernel.org> wrote:
> Hello,
>
> On 10/07/2010 03:21 PM, Torsten Kaiser wrote:
>>> Can you test whether the following patch fix the issue?
>>
>> Yes. After adding this patch to 2.6.36-rc6 it survived my testcase. It
>> first seems to get stuck again, but after less then a minute the
>> system recovered and all compiles finished successfully.
>>
>> I suspect the combination of CFLAGs, the gcc version and maybe even
>> changes in KOffice from 2.2.1 to 2.2.2 are import for my testcase. It
>> looks like at the start of the complies something is really eating an
>> enormous amount of memory and I did not hit such a case with earlier
>> kernels.
>
> It's great to hear that the patch helped but would you be able to

:-(
The second try with the patched 2.6.36-rc6 got stuck again.

> repeat the same test w/ 2.6.35 and see whether there's any noticeable
> difference?

I just tested vanilla 2.6.35, userspace and the testcase as with the
2.6.36-rc6 kernel.
Behavior was the same as with the first try of the patched 2.6.36-rc6:
The testcase started to build up CPU load and memory pressure, then
the mouse got stuck. After ~1 min the system recovered and KDE became
usable again. (Compile just finish OK)

vmstat 60 from 2.6.35:
 2  0      0 2365848   1060 742072    0    0   205     7 6467 12252 26  5 62  8
17  0      0 837936   1060 1079520    0    0  1025    24 2210 13483 69 20  8  2
18  0      0 694296   1060 1127728    0    0   231    14  626 1883 74 26  0  0
20  0      0 228540    124 998440    0    0    48    12  604 1820 72 28  0  0
23  1     24 195456     52 657216    0    0   339    11  563 1669 84 16  0  0
 0 42 937144  14004      0 164352    0 15617   290 15633  827 1254 24  9  1 66
21  1 861184 776104      0 149908  998 3852  2790  3902 1137 2454 45 15  0 40
21  0 798544 1131964      0 190004  212    0   530    15  599 1469 87 13  0  0
12  1 779488 1536772      0 225312  195    0   398    12  534 1087 88 12  0  0
11  0 620620 2201408      0 273976 1338    0  1638    40  851 2281 75 19  3  3
12  1 588196 1572564      0 344232  521    0  1326   152  787 1888 86 14  0  0
 8  0 510876 2286000      0 311464  127    0   442   398  758 1558 88 12  0  0

> Some level of stuttering is expected if the system is hit
> with sudden huge spike of memory pressure but let's see if it has
> regressed somehow.

Yes, if a swapstorm occurs, I can live with a (short) "lockup". For
example compiling openoffice on tmpfs is a case where I have seen
similar short periods of the mouse getting stuck. But with 2.6.36-rc5
it was the first time that the system did not seem to recover, so I
think it should count as a regression compared to 2.6.35.

As the second try with the patched 2.6.36-rc6 got stuck again, could
it be that your patch is incomplete?
>From drivers/md/dm-crypt.c:
        cc->io_queue = create_singlethread_workqueue("kcryptd_io");
        cc->crypt_queue = create_singlethread_workqueue("kcryptd");

Do these workqueues also need WQ_MEM_RECLAIM?

I have attached the SysRq-M output from the patched -rc6. The symptoms
were the same as with the unpatched kernel. The system got stuck, the
hung_task_timeout triggered for a numer of programs in sync_page() or
do_lookup() and one kworker/1:1:496 was probable the nail in the
coffin.

Do you want me to do more tests with 2.6.35 or the patched .36-rc6? Or
any other patch I should try?

Torsten

[-- Attachment #2: hang-with-patch.txt --]
[-- Type: text/plain, Size: 33978 bytes --]

[  646.760693] SysRq : Show Memory
[  657.523845] SysRq : Changing Loglevel
[  657.527640] Loglevel set to 6
[  659.770834] SysRq : Show Memory
[  659.774109] Mem-Info:
[  659.776450] Node 0 DMA per-cpu:
[  659.779709] CPU    0: hi:    0, btch:   1 usd:   0
[  659.780824] CPU    1: hi:    0, btch:   1 usd:   0
[  659.780824] CPU    2: hi:    0, btch:   1 usd:   0
[  659.780824] CPU    3: hi:    0, btch:   1 usd:   0
[  659.780824] Node 0 DMA32 per-cpu:
[  659.780824] CPU    0: hi:  186, btch:  31 usd:   0
[  659.780824] CPU    1: hi:  186, btch:  31 usd:   0
[  659.780824] CPU    2: hi:  186, btch:  31 usd:   0
[  659.780824] CPU    3: hi:  186, btch:  31 usd:   0
[  659.780824] Node 1 DMA32 per-cpu:
[  659.780824] CPU    0: hi:  186, btch:  31 usd:   0
[  659.780824] CPU    1: hi:  186, btch:  31 usd:   0
[  659.780824] CPU    2: hi:  186, btch:  31 usd:   0
[  659.840014] CPU    3: hi:  186, btch:  31 usd:   0
[  659.840014] Node 1 Normal per-cpu:
[  659.840014] CPU    0: hi:  186, btch:  31 usd:   0
[  659.840014] CPU    1: hi:  186, btch:  31 usd:   0
[  659.840014] CPU    2: hi:  186, btch:  31 usd:   0
[  659.840014] CPU    3: hi:  186, btch:  31 usd:   0
[  659.840014] active_anon:677975 inactive_anon:261811 isolated_anon:1496
[  659.840014]  active_file:411 inactive_file:459 isolated_file:0
[  659.840014]  unevictable:1352 dirty:0 writeback:170816 unstable:0
[  659.840014]  free:3585 slab_reclaimable:8839 slab_unreclaimable:37060
[  659.840014]  mapped:1587 shmem:23114 pagetables:10122 bounce:0
[  659.840014] Node 0 DMA free:8036kB min:28kB low:32kB high:40kB active_anon:3736kB inactive_anon:3984kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15768kB mlocked:0kB dirty:0kB writeback:560kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:156kB kernel_stack:0kB pagetables:16kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1181 all_unreclaimable? no
[  659.840014] lowmem_reserve[]: 0 2004 2004 2004
[  659.840014] Node 0 DMA32 free:3176kB min:4036kB low:5044kB high:6052kB active_anon:1363688kB inactive_anon:454876kB active_file:1608kB inactive_file:1820kB unevictable:580kB isolated(anon):1280kB isolated(file):0kB present:2052320kB mlocked:580kB dirty:0kB writeback:402768kB mapped:4068kB shmem:11732kB slab_reclaimable:17924kB slab_unreclaimable:77816kB kernel_stack:1984kB pagetables:24012kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:137356 all_unreclaimable? no
[  659.840014] lowmem_reserve[]: 0 0 0 0
[  659.840014] Node 1 DMA32 free:2340kB min:3036kB low:3792kB high:4552kB active_anon:1136304kB inactive_anon:379920kB active_file:0kB inactive_file:0kB unevictable:1276kB isolated(anon):3296kB isolated(file):0kB present:1544000kB mlocked:1276kB dirty:0kB writeback:221672kB mapped:0kB shmem:76796kB slab_reclaimable:10652kB slab_unreclaimable:30568kB kernel_stack:296kB pagetables:5036kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:120927 all_unreclaimable? no
[  659.840014] lowmem_reserve[]: 0 0 505 505
[  659.840014] Node 1 Normal free:788kB min:1016kB low:1268kB high:1524kB active_anon:208172kB inactive_anon:208512kB active_file:36kB inactive_file:16kB unevictable:3552kB isolated(anon):1408kB isolated(file):0kB present:517120kB mlocked:3552kB dirty:0kB writeback:58264kB mapped:2280kB shmem:3928kB slab_reclaimable:6780kB slab_unreclaimable:39700kB kernel_stack:1128kB pagetables:11424kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:89156 all_unreclaimable? no
[  659.840014] lowmem_reserve[]: 0 0 0 0
[  659.840014] Node 0 DMA: 43*4kB 1*8kB 3*16kB 2*32kB 1*64kB 2*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 8036kB
[  659.840014] Node 0 DMA32: 276*4kB 7*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 3176kB
[  659.840014] Node 1 DMA32: 49*4kB 2*8kB 7*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2340kB
[  659.840014] Node 1 Normal: 21*4kB 10*8kB 7*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 788kB
[  659.840014] 206580 total pagecache pages
[  659.840014] 181890 pages in swap cache
[  659.840014] Swap cache stats: add 251105, delete 69215, find 11614/11665
[  659.840014] Free swap  = 9000388kB
[  659.840014] Total swap = 10000316kB
[  659.840014] 1048575 pages RAM
[  659.840014] 33808 pages reserved
[  659.840014] 62524 pages shared
[  659.840014] 996022 pages non-shared
[  692.249734] SysRq : Show Memory
[  692.252998] Mem-Info:
[  692.255347] Node 0 DMA per-cpu:
[  692.258615] CPU    0: hi:    0, btch:   1 usd:   0
[  692.259723] CPU    1: hi:    0, btch:   1 usd:   0
[  692.259723] CPU    2: hi:    0, btch:   1 usd:   0
[  692.259723] CPU    3: hi:    0, btch:   1 usd:   0
[  692.259723] Node 0 DMA32 per-cpu:
[  692.259723] CPU    0: hi:  186, btch:  31 usd:   0
[  692.259723] CPU    1: hi:  186, btch:  31 usd:   0
[  692.259723] CPU    2: hi:  186, btch:  31 usd:   0
[  692.259723] CPU    3: hi:  186, btch:  31 usd:   0
[  692.259723] Node 1 DMA32 per-cpu:
[  692.259723] CPU    0: hi:  186, btch:  31 usd:   0
[  692.259723] CPU    1: hi:  186, btch:  31 usd:   0
[  692.259723] CPU    2: hi:  186, btch:  31 usd:   0
[  692.259723] CPU    3: hi:  186, btch:  31 usd:   0
[  692.259723] Node 1 Normal per-cpu:
[  692.259723] CPU    0: hi:  186, btch:  31 usd:   0
[  692.259723] CPU    1: hi:  186, btch:  31 usd:   0
[  692.259723] CPU    2: hi:  186, btch:  31 usd:   0
[  692.259723] CPU    3: hi:  186, btch:  31 usd:   0
[  692.259723] active_anon:677453 inactive_anon:261670 isolated_anon:1473
[  692.259723]  active_file:72 inactive_file:42 isolated_file:0
[  692.259723]  unevictable:1352 dirty:0 writeback:184172 unstable:0
[  692.259723]  free:3710 slab_reclaimable:8763 slab_unreclaimable:38996
[  692.259723]  mapped:832 shmem:20558 pagetables:10122 bounce:0
[  692.259723] Node 0 DMA free:8040kB min:28kB low:32kB high:40kB active_anon:3736kB inactive_anon:3996kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15768kB mlocked:0kB dirty:0kB writeback:1476kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:156kB kernel_stack:0kB pagetables:16kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:743 all_unreclaimable? no
[  692.259723] lowmem_reserve[]: 0 2004 2004 2004
[  692.259723] Node 0 DMA32 free:3544kB min:4036kB low:5044kB high:6052kB active_anon:1362472kB inactive_anon:454652kB active_file:252kB inactive_file:156kB unevictable:580kB isolated(anon):768kB isolated(file):0kB present:2052320kB mlocked:580kB dirty:0kB writeback:416996kB mapped:1048kB shmem:5908kB slab_reclaimable:17636kB slab_unreclaimable:84436kB kernel_stack:1984kB pagetables:24012kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:85247 all_unreclaimable? no
[  692.259723] lowmem_reserve[]: 0 0 0 0
[  692.259723] Node 1 DMA32 free:2352kB min:3036kB low:3792kB high:4552kB active_anon:1135940kB inactive_anon:379184kB active_file:0kB inactive_file:0kB unevictable:1276kB isolated(anon):4100kB isolated(file):0kB present:1544000kB mlocked:1276kB dirty:0kB writeback:234936kB mapped:0kB shmem:73664kB slab_reclaimable:10652kB slab_unreclaimable:30632kB kernel_stack:296kB pagetables:5036kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:74277 all_unreclaimable? no
[  692.259723] lowmem_reserve[]: 0 0 505 505
[  692.259723] Node 1 Normal free:904kB min:1016kB low:1268kB high:1524kB active_anon:207664kB inactive_anon:208812kB active_file:36kB inactive_file:12kB unevictable:3552kB isolated(anon):1024kB isolated(file):0kB present:517120kB mlocked:3552kB dirty:0kB writeback:83280kB mapped:2280kB shmem:2660kB slab_reclaimable:6764kB slab_unreclaimable:40760kB kernel_stack:1128kB pagetables:11424kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[  692.259723] lowmem_reserve[]: 0 0 0 0
[  692.259723] Node 0 DMA: 44*4kB 1*8kB 3*16kB 2*32kB 1*64kB 2*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 8040kB
[  692.259723] Node 0 DMA32: 338*4kB 10*8kB 6*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 3544kB
[  692.259723] Node 1 DMA32: 46*4kB 1*8kB 3*16kB 2*32kB 2*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2352kB
[  692.259723] Node 1 Normal: 88*4kB 3*8kB 3*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 904kB
[  692.259723] 216617 total pagecache pages
[  692.259723] 195285 pages in swap cache
[  692.259723] Swap cache stats: add 265249, delete 69964, find 11614/11666
[  692.259723] Free swap  = 8943908kB
[  692.259723] Total swap = 10000316kB
[  692.259723] 1048575 pages RAM
[  692.259723] 33808 pages reserved
[  692.259723] 50180 pages shared
[  692.259723] 996672 pages non-shared
[  721.740042] INFO: task kworker/1:1:496 blocked for more than 120 seconds.
[  721.747127] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  721.755350]  ffff88007ffe1bb0 0000000000000046 ffff88007ffe1b10 ffffffff81082de0
[  721.763158]  0000000000012780 ffff88007ff88b60 ffff88007ff88db8 ffff88007ffe1fd8
[  721.770975]  ffff88007ff88dc0 ffff88007ffe0010 0000000000012780 ffff88007ffe1fd8
[  721.778783] Call Trace:
[  721.781363]  [<ffffffff81082de0>] ? mempool_alloc_slab+0x10/0x20
[  721.787657]  [<ffffffff8156661d>] schedule_timeout+0x19d/0x200
[  721.793777]  [<ffffffff8102f559>] ? enqueue_task_fair+0x159/0x180
[  721.800170]  [<ffffffff81565ace>] wait_for_common+0xfe/0x170
[  721.806108]  [<ffffffff81035da0>] ? default_wake_function+0x0/0x10
[  721.812588]  [<ffffffff81565be8>] wait_for_completion+0x18/0x20
[  721.818785]  [<ffffffff8105335d>] kthread_create+0x8d/0x110
[  721.824628]  [<ffffffff8104f2c0>] ? worker_thread+0x0/0x370
[  721.830475]  [<ffffffff81031a3d>] ? finish_task_switch+0x3d/0xc0
[  721.836762]  [<ffffffff810b914c>] ? kmem_cache_alloc+0x8c/0xa0
[  721.842825]  [<ffffffff8104f04f>] create_worker+0xff/0x170
[  721.848479]  [<ffffffff8104f14d>] manage_workers+0x8d/0x200
[  721.854239]  [<ffffffff8104f5ba>] worker_thread+0x2fa/0x370
[  721.859987]  [<ffffffff8104f2c0>] ? worker_thread+0x0/0x370
[  721.865748]  [<ffffffff810530d6>] kthread+0x96/0xa0
[  721.870758]  [<ffffffff81003154>] kernel_thread_helper+0x4/0x10
[  721.876870]  [<ffffffff81053040>] ? kthread+0x0/0xa0
[  721.881998]  [<ffffffff81003150>] ? kernel_thread_helper+0x0/0x10
[  721.888281] INFO: task X:2546 blocked for more than 120 seconds.
[  721.894501] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  721.902611]  ffff880006b35d78 0000000000003082 ffff88011fc04c20 0000000000000001
[  721.910415]  0000000000012780 ffff88000781b330 ffff88000781b588 ffff880006b35fd8
[  721.918205]  ffff88000781b590 ffff880006b34010 0000000000012780 ffff880006b35fd8
[  721.926090] Call Trace:
[  721.928556]  [<ffffffff81080bb0>] ? sync_page+0x0/0x50
[  721.933884]  [<ffffffff81566372>] io_schedule+0x42/0x60
[  721.939146]  [<ffffffff81080bed>] sync_page+0x3d/0x50
[  721.944275]  [<ffffffff81566732>] __wait_on_bit_lock+0x52/0xb0
[  721.950153]  [<ffffffff81080b92>] __lock_page+0x62/0x70
[  721.955411]  [<ffffffff81053590>] ? wake_bit_function+0x0/0x40
[  721.961313]  [<ffffffff8109ad61>] handle_mm_fault+0xa01/0xa20
[  721.967094]  [<ffffffff810244e0>] do_page_fault+0x100/0x270
[  721.972728]  [<ffffffff815682df>] page_fault+0x1f/0x30
[  721.977917] INFO: task virtuoso-t:3276 blocked for more than 120 seconds.
[  721.984792] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  721.992707]  ffff88006702dd78 0000000000000086 ffff88011fc04c20 0000000000000001
[  722.000237]  0000000000012780 ffff880073e738e0 ffff880073e73b38 ffff88006702dfd8
[  722.007744]  ffff880073e73b40 ffff88006702c010 0000000000012780 ffff88006702dfd8
[  722.015333] Call Trace:
[  722.017799]  [<ffffffff81080bb0>] ? sync_page+0x0/0x50
[  722.023012]  [<ffffffff81566372>] io_schedule+0x42/0x60
[  722.028255]  [<ffffffff81080bed>] sync_page+0x3d/0x50
[  722.033709]  [<ffffffff81566732>] __wait_on_bit_lock+0x52/0xb0
[  722.039764]  [<ffffffff81080b92>] __lock_page+0x62/0x70
[  722.045234]  [<ffffffff81053590>] ? wake_bit_function+0x0/0x40
[  722.051231]  [<ffffffff8109ad61>] handle_mm_fault+0xa01/0xa20
[  722.057185]  [<ffffffff810244e0>] do_page_fault+0x100/0x270
[  722.062957]  [<ffffffff815682df>] page_fault+0x1f/0x30
[  722.068272] INFO: task as:8337 blocked for more than 120 seconds.
[  722.074568] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  722.082652]  ffff88003dd1fd78 0000000000000086 ffff88011fc04c20 0000000000000001
[  722.090459]  0000000000012780 ffff880048e56660 ffff880048e568b8 ffff88003dd1ffd8
[  722.098154]  ffff880048e568c0 ffff88003dd1e010 0000000000012780 ffff88003dd1ffd8
[  722.105875] Call Trace:
[  722.108427]  [<ffffffff81080bb0>] ? sync_page+0x0/0x50
[  722.113850]  [<ffffffff81566372>] io_schedule+0x42/0x60
[  722.119317]  [<ffffffff81080bed>] sync_page+0x3d/0x50
[  722.124578]  [<ffffffff81566732>] __wait_on_bit_lock+0x52/0xb0
[  722.130706]  [<ffffffff81080b92>] __lock_page+0x62/0x70
[  722.136174]  [<ffffffff81053590>] ? wake_bit_function+0x0/0x40
[  722.142275]  [<ffffffff8109ad61>] handle_mm_fault+0xa01/0xa20
[  722.148289]  [<ffffffff810a01bc>] ? do_brk+0x25c/0x340
[  722.153698]  [<ffffffff810244e0>] do_page_fault+0x100/0x270
[  722.159545]  [<ffffffff815682df>] page_fault+0x1f/0x30
[  722.164958] INFO: task cc1plus:9239 blocked for more than 120 seconds.
[  722.171820] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  722.180031]  ffff8800c4e1fd78 0000000000000082 ffff88011fc04c20 0000000000000001
[  722.187813]  0000000000012780 ffff8800cdee5550 ffff8800cdee57a8 ffff8800c4e1ffd8
[  722.195609]  ffff8800cdee57b0 ffff8800c4e1e010 0000000000012780 ffff8800c4e1ffd8
[  722.203377] Call Trace:
[  722.205922]  [<ffffffff81080bb0>] ? sync_page+0x0/0x50
[  722.211298]  [<ffffffff81566372>] io_schedule+0x42/0x60
[  722.216756]  [<ffffffff81080bed>] sync_page+0x3d/0x50
[  722.222027]  [<ffffffff81566732>] __wait_on_bit_lock+0x52/0xb0
[  722.228063]  [<ffffffff81080b92>] __lock_page+0x62/0x70
[  722.233475]  [<ffffffff81053590>] ? wake_bit_function+0x0/0x40
[  722.239540]  [<ffffffff8109ad61>] handle_mm_fault+0xa01/0xa20
[  722.245532]  [<ffffffff810244e0>] do_page_fault+0x100/0x270
[  722.251303]  [<ffffffff815682df>] page_fault+0x1f/0x30
[  722.256652] INFO: task cc1plus:12681 blocked for more than 120 seconds.
[  722.263531] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  722.271722]  ffff8800c7fdfd78 0000000000000086 ffff88011fc04c20 0000000000000001
[  722.279487]  0000000000012780 ffff8800cdee0000 ffff8800cdee0258 ffff8800c7fdffd8
[  722.287207]  ffff8800cdee0260 ffff8800c7fde010 0000000000012780 ffff8800c7fdffd8
[  722.294929] Call Trace:
[  722.297448]  [<ffffffff81080bb0>] ? sync_page+0x0/0x50
[  722.302790]  [<ffffffff81566372>] io_schedule+0x42/0x60
[  722.308215]  [<ffffffff81080bed>] sync_page+0x3d/0x50
[  722.313505]  [<ffffffff81566732>] __wait_on_bit_lock+0x52/0xb0
[  722.319533]  [<ffffffff81080b92>] __lock_page+0x62/0x70
[  722.324961]  [<ffffffff81053590>] ? wake_bit_function+0x0/0x40
[  722.331045]  [<ffffffff8109ad61>] handle_mm_fault+0xa01/0xa20
[  722.336988]  [<ffffffff810244e0>] do_page_fault+0x100/0x270
[  722.342830]  [<ffffffff815682df>] page_fault+0x1f/0x30
[  722.348180] INFO: task cc1plus:13371 blocked for more than 120 seconds.
[  722.355065] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  722.363467]  ffff88010742bd78 0000000000000086 ffff88011fc04c20 0000000000000001
[  722.370997]  0000000000012780 ffff8800cdee05b0 ffff8800cdee0808 ffff88010742bfd8
[  722.378512]  ffff8800cdee0810 ffff88010742a010 0000000000012780 ffff88010742bfd8
[  722.386076] Call Trace:
[  722.388534]  [<ffffffff81080bb0>] ? sync_page+0x0/0x50
[  722.393720]  [<ffffffff81566372>] io_schedule+0x42/0x60
[  722.398965]  [<ffffffff81080bed>] sync_page+0x3d/0x50
[  722.404086]  [<ffffffff81566732>] __wait_on_bit_lock+0x52/0xb0
[  722.409934]  [<ffffffff81080b92>] __lock_page+0x62/0x70
[  722.415205]  [<ffffffff81053590>] ? wake_bit_function+0x0/0x40
[  722.421125]  [<ffffffff8109ad61>] handle_mm_fault+0xa01/0xa20
[  722.426869]  [<ffffffff810244e0>] do_page_fault+0x100/0x270
[  722.432486]  [<ffffffff815682df>] page_fault+0x1f/0x30
[  722.437640] INFO: task make:13846 blocked for more than 120 seconds.
[  722.444054] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  722.451935]  ffff880104533b88 0000000000000086 ffff8800cb19e760 0000000000000202
[  722.459417]  0000000000012780 ffff880080c64fa0 ffff880080c651f8 ffff880104533fd8
[  722.466936]  ffff880080c65200 ffff880104532010 0000000000012780 ffff880104533fd8
[  722.474774] Call Trace:
[  722.477240]  [<ffffffff81080bb0>] ? sync_page+0x0/0x50
[  722.482442]  [<ffffffff81080bb0>] ? sync_page+0x0/0x50
[  722.487615]  [<ffffffff81566372>] io_schedule+0x42/0x60
[  722.492919]  [<ffffffff81080bed>] sync_page+0x3d/0x50
[  722.498007]  [<ffffffff81566867>] __wait_on_bit+0x57/0x80
[  722.503467]  [<ffffffff81080d8e>] wait_on_page_bit+0x6e/0x80
[  722.509168]  [<ffffffff81053590>] ? wake_bit_function+0x0/0x40
[  722.515080]  [<ffffffff81092454>] shmem_getpage+0x5b4/0x860
[  722.520723]  [<ffffffff81092d46>] shmem_file_aio_read+0x1e6/0x380
[  722.526909]  [<ffffffff810bfc52>] do_sync_read+0xd2/0x110
[  722.532415]  [<ffffffff811d49aa>] ? security_file_permission+0x6a/0x70
[  722.539040]  [<ffffffff810bff95>] vfs_read+0xc5/0x180
[  722.544209]  [<ffffffff810c063c>] sys_read+0x4c/0x80
[  722.549255]  [<ffffffff8100242b>] system_call_fastpath+0x16/0x1b
[  722.555373] INFO: task cc1plus:13856 blocked for more than 120 seconds.
[  722.562112] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  722.570090]  ffff8800dca63bc8 0000000000000086 ffff8800dca63b68 ffffffff8102f559
[  722.577673]  0000000000012780 ffff88010e79ec10 ffff88010e79ee68 ffff8800dca63fd8
[  722.585582]  ffff88010e79ee70 ffff8800dca62010 0000000000012780 ffff8800dca63fd8
[  722.593122] Call Trace:
[  722.595595]  [<ffffffff8102f559>] ? enqueue_task_fair+0x159/0x180
[  722.601747]  [<ffffffff81566e91>] __mutex_lock_slowpath+0xe1/0x160
[  722.608054]  [<ffffffff810d3252>] ? __d_lookup+0xa2/0x140
[  722.613595]  [<ffffffff8156692e>] mutex_lock+0x1e/0x40
[  722.618845]  [<ffffffff810cb1ba>] do_lookup+0xfa/0x170
[  722.624140]  [<ffffffff810cbcb5>] link_path_walk+0x4d5/0x9c0
[  722.629913]  [<ffffffff810cc2b0>] path_walk+0x60/0xe0
[  722.635095]  [<ffffffff810cc383>] do_path_lookup+0x53/0x60
[  722.640720]  [<ffffffff810ccfc3>] user_path_at+0x53/0xa0
[  722.646154]  [<ffffffff810a374d>] ? page_add_new_anon_rmap+0x7d/0xc0
[  722.652664]  [<ffffffff8109ab32>] ? handle_mm_fault+0x7d2/0xa20
[  722.658702]  [<ffffffff810c4187>] vfs_fstatat+0x37/0x70
[  722.664051]  [<ffffffff810c4296>] vfs_stat+0x16/0x20
[  722.669121]  [<ffffffff810c42bf>] sys_newstat+0x1f/0x50
[  722.674494]  [<ffffffff8102450c>] ? do_page_fault+0x12c/0x270
[  722.680361]  [<ffffffff815682df>] ? page_fault+0x1f/0x30
[  722.685786]  [<ffffffff8100242b>] system_call_fastpath+0x16/0x1b
[  722.691940] INFO: task cc1plus:13859 blocked for more than 120 seconds.
[  722.698644] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  722.706623]  ffff8800c4dc1bc8 0000000000000086 ffff8800c4dc1b68 ffffffff00000000
[  722.714199]  0000000000012780 ffff88010e79be90 ffff88010e79c0e8 ffff8800c4dc1fd8
[  722.721837]  ffff88010e79c0f0 ffff8800c4dc0010 0000000000012780 ffff8800c4dc1fd8
[  722.729431] Call Trace:
[  722.731929]  [<ffffffff81566e91>] __mutex_lock_slowpath+0xe1/0x160
[  722.738191]  [<ffffffff810d3252>] ? __d_lookup+0xa2/0x140
[  722.743690]  [<ffffffff8156692e>] mutex_lock+0x1e/0x40
[  722.748907]  [<ffffffff810cb1ba>] do_lookup+0xfa/0x170
[  722.754202]  [<ffffffff810cbcb5>] link_path_walk+0x4d5/0x9c0
[  722.759980]  [<ffffffff810cc2b0>] path_walk+0x60/0xe0
[  722.765173]  [<ffffffff810cc383>] do_path_lookup+0x53/0x60
[  722.770800]  [<ffffffff810ccfc3>] user_path_at+0x53/0xa0
[  722.776223]  [<ffffffff810a374d>] ? page_add_new_anon_rmap+0x7d/0xc0
[  722.782743]  [<ffffffff8109ab32>] ? handle_mm_fault+0x7d2/0xa20
[  722.788797]  [<ffffffff810c4187>] vfs_fstatat+0x37/0x70
[  722.794111]  [<ffffffff810c4296>] vfs_stat+0x16/0x20
[  722.799138]  [<ffffffff810c42bf>] sys_newstat+0x1f/0x50
[  722.804494]  [<ffffffff8102450c>] ? do_page_fault+0x12c/0x270
[  722.810378]  [<ffffffff815682df>] ? page_fault+0x1f/0x30
[  722.815795]  [<ffffffff8100242b>] system_call_fastpath+0x16/0x1b
[  736.990877] SysRq : Show Memory
[  736.994135] Mem-Info:
[  736.996476] Node 0 DMA per-cpu:
[  736.999744] CPU    0: hi:    0, btch:   1 usd:   0
[  737.000866] CPU    1: hi:    0, btch:   1 usd:   0
[  737.000866] CPU    2: hi:    0, btch:   1 usd:   0
[  737.000866] CPU    3: hi:    0, btch:   1 usd:   0
[  737.000866] Node 0 DMA32 per-cpu:
[  737.000866] CPU    0: hi:  186, btch:  31 usd:   0
[  737.000866] CPU    1: hi:  186, btch:  31 usd:   0
[  737.000866] CPU    2: hi:  186, btch:  31 usd:   0
[  737.000866] CPU    3: hi:  186, btch:  31 usd:   0
[  737.000866] Node 1 DMA32 per-cpu:
[  737.000866] CPU    0: hi:  186, btch:  31 usd:   0
[  737.000866] CPU    1: hi:  186, btch:  31 usd:   0
[  737.000866] CPU    2: hi:  186, btch:  31 usd:   0
[  737.000866] CPU    3: hi:  186, btch:  31 usd:   0
[  737.000866] Node 1 Normal per-cpu:
[  737.000866] CPU    0: hi:  186, btch:  31 usd:   0
[  737.000866] CPU    1: hi:  186, btch:  31 usd:   0
[  737.000866] CPU    2: hi:  186, btch:  31 usd:   0
[  737.000866] CPU    3: hi:  186, btch:  31 usd:   0
[  737.000866] active_anon:677421 inactive_anon:261648 isolated_anon:1537
[  737.000866]  active_file:55 inactive_file:17 isolated_file:0
[  737.000866]  unevictable:1352 dirty:0 writeback:185484 unstable:0
[  737.000866]  free:3559 slab_reclaimable:8759 slab_unreclaimable:39192
[  737.000866]  mapped:790 shmem:20039 pagetables:10122 bounce:0
[  737.100049] Node 0 DMA free:8040kB min:28kB low:32kB high:40kB active_anon:3736kB inactive_anon:4012kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15768kB mlocked:0kB dirty:0kB writeback:1476kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:156kB kernel_stack:0kB pagetables:16kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[  737.100049] lowmem_reserve[]: 0 2004 2004 2004
[  737.100049] Node 0 DMA32 free:3080kB min:4036kB low:5044kB high:6052kB active_anon:1362344kB inactive_anon:454652kB active_file:204kB inactive_file:40kB unevictable:580kB isolated(anon):768kB isolated(file):0kB present:2052320kB mlocked:580kB dirty:0kB writeback:417620kB mapped:884kB shmem:5900kB slab_reclaimable:17620kB slab_unreclaimable:85076kB kernel_stack:1984kB pagetables:24012kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:413 all_unreclaimable? no
[  737.100049] lowmem_reserve[]: 0 0 0 0
[  737.100049] Node 1 DMA32 free:2348kB min:3036kB low:3792kB high:4552kB active_anon:1135940kB inactive_anon:379172kB active_file:0kB inactive_file:0kB unevictable:1276kB isolated(anon):4100kB isolated(file):0kB present:1544000kB mlocked:1276kB dirty:0kB writeback:237876kB mapped:0kB shmem:71704kB slab_reclaimable:10652kB slab_unreclaimable:30636kB kernel_stack:296kB pagetables:5036kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:716 all_unreclaimable? no
[  737.100049] lowmem_reserve[]: 0 0 505 505
[  737.100049] Node 1 Normal free:768kB min:1016kB low:1268kB high:1524kB active_anon:207664kB inactive_anon:208556kB active_file:16kB inactive_file:28kB unevictable:3552kB isolated(anon):1280kB isolated(file):0kB present:517120kB mlocked:3552kB dirty:0kB writeback:84964kB mapped:2276kB shmem:2552kB slab_reclaimable:6764kB slab_unreclaimable:40900kB kernel_stack:1128kB pagetables:11424kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:391 all_unreclaimable? no
[  737.100049] lowmem_reserve[]: 0 0 0 0
[  737.100049] Node 0 DMA: 44*4kB 1*8kB 3*16kB 2*32kB 1*64kB 2*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 8040kB
[  737.100049] Node 0 DMA32: 266*4kB 0*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 3080kB
[  737.100049] Node 1 DMA32: 45*4kB 1*8kB 3*16kB 2*32kB 2*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2348kB
[  737.100049] Node 1 Normal: 54*4kB 3*8kB 3*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 768kB
[  737.100049] 217413 total pagecache pages
[  737.100049] 196596 pages in swap cache
[  737.100049] Swap cache stats: add 266560, delete 69964, find 11614/11666
[  737.100049] Free swap  = 8938664kB
[  737.100049] Total swap = 10000316kB
[  737.100049] 1048575 pages RAM
[  737.100049] 33808 pages reserved
[  737.100049] 50100 pages shared
[  737.100049] 996764 pages non-shared
[  747.384761] SysRq : Show Memory
[  747.388025] Mem-Info:
[  747.390366] Node 0 DMA per-cpu:
[  747.393633] CPU    0: hi:    0, btch:   1 usd:   0
[  747.394750] CPU    1: hi:    0, btch:   1 usd:   0
[  747.394750] CPU    2: hi:    0, btch:   1 usd:   0
[  747.394750] CPU    3: hi:    0, btch:   1 usd:   0
[  747.394750] Node 0 DMA32 per-cpu:
[  747.394750] CPU    0: hi:  186, btch:  31 usd:   0
[  747.394750] CPU    1: hi:  186, btch:  31 usd:   0
[  747.394750] CPU    2: hi:  186, btch:  31 usd:   0
[  747.394750] CPU    3: hi:  186, btch:  31 usd:   0
[  747.394750] Node 1 DMA32 per-cpu:
[  747.394750] CPU    0: hi:  186, btch:  31 usd:   0
[  747.394750] CPU    1: hi:  186, btch:  31 usd:   0
[  747.394750] CPU    2: hi:  186, btch:  31 usd:   0
[  747.394750] CPU    3: hi:  186, btch:  31 usd:   0
[  747.394750] Node 1 Normal per-cpu:
[  747.394750] CPU    0: hi:  186, btch:  31 usd:   0
[  747.394750] CPU    1: hi:  186, btch:  31 usd:   0
[  747.394750] CPU    2: hi:  186, btch:  31 usd:   0
[  747.394750] CPU    3: hi:  186, btch:  31 usd:   0
[  747.394750] active_anon:677421 inactive_anon:261648 isolated_anon:1537
[  747.394750]  active_file:11 inactive_file:11 isolated_file:0
[  747.394750]  unevictable:1352 dirty:0 writeback:185484 unstable:0
[  747.394750]  free:3613 slab_reclaimable:8755 slab_unreclaimable:39192
[  747.394750]  mapped:740 shmem:20039 pagetables:10122 bounce:0
[  747.394750] Node 0 DMA free:8040kB min:28kB low:32kB high:40kB active_anon:3736kB inactive_anon:4012kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15768kB mlocked:0kB dirty:0kB writeback:1476kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:156kB kernel_stack:0kB pagetables:16kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[  747.394750] lowmem_reserve[]: 0 2004 2004 2004
[  747.394750] Node 0 DMA32 free:3296kB min:4036kB low:5044kB high:6052kB active_anon:1362344kB inactive_anon:454724kB active_file:28kB inactive_file:16kB unevictable:580kB isolated(anon):768kB isolated(file):0kB present:2052320kB mlocked:580kB dirty:0kB writeback:417620kB mapped:684kB shmem:5900kB slab_reclaimable:17604kB slab_unreclaimable:85076kB kernel_stack:1984kB pagetables:24012kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[  747.394750] lowmem_reserve[]: 0 0 0 0
[  747.394750] Node 1 DMA32 free:2348kB min:3036kB low:3792kB high:4552kB active_anon:1135940kB inactive_anon:379284kB active_file:0kB inactive_file:0kB unevictable:1276kB isolated(anon):4100kB isolated(file):0kB present:1544000kB mlocked:1276kB dirty:0kB writeback:237876kB mapped:0kB shmem:71704kB slab_reclaimable:10652kB slab_unreclaimable:30636kB kernel_stack:296kB pagetables:5036kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[  747.394750] lowmem_reserve[]: 0 0 505 505
[  747.394750] Node 1 Normal free:768kB min:1016kB low:1268kB high:1524kB active_anon:207664kB inactive_anon:208572kB active_file:16kB inactive_file:28kB unevictable:3552kB isolated(anon):1280kB isolated(file):0kB present:517120kB mlocked:3552kB dirty:0kB writeback:84964kB mapped:2276kB shmem:2552kB slab_reclaimable:6764kB slab_unreclaimable:40900kB kernel_stack:1128kB pagetables:11424kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[  747.394750] lowmem_reserve[]: 0 0 0 0
[  747.394750] Node 0 DMA: 44*4kB 1*8kB 3*16kB 2*32kB 1*64kB 2*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 8040kB
[  747.394750] Node 0 DMA32: 302*4kB 5*8kB 2*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 3296kB
[  747.394750] Node 1 DMA32: 45*4kB 1*8kB 3*16kB 2*32kB 2*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2348kB
[  747.394750] Node 1 Normal: 54*4kB 3*8kB 3*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 768kB
[  747.730058] 217363 total pagecache pages
[  747.730058] 196596 pages in swap cache
[  747.730058] Swap cache stats: add 266560, delete 69964, find 11614/11666
[  747.730058] Free swap  = 8938664kB
[  747.730058] Total swap = 10000316kB
[  747.730058] 1048575 pages RAM
[  747.730058] 33808 pages reserved
[  747.730058] 48587 pages shared
[  747.730058] 996797 pages non-shared
[  765.630495] SysRq : Show Memory
[  765.633743] Mem-Info:
[  765.636069] Node 0 DMA per-cpu:
[  765.639295] CPU    0: hi:    0, btch:   1 usd:   0
[  765.640488] CPU    1: hi:    0, btch:   1 usd:   0
[  765.640488] CPU    2: hi:    0, btch:   1 usd:   0
[  765.640488] CPU    3: hi:    0, btch:   1 usd:   0
[  765.640488] Node 0 DMA32 per-cpu:
[  765.640488] CPU    0: hi:  186, btch:  31 usd:   0
[  765.640488] CPU    1: hi:  186, btch:  31 usd:   0
[  765.640488] CPU    2: hi:  186, btch:  31 usd:   0
[  765.640488] CPU    3: hi:  186, btch:  31 usd:   0
[  765.640488] Node 1 DMA32 per-cpu:
[  765.640488] CPU    0: hi:  186, btch:  31 usd:   0
[  765.640488] CPU    1: hi:  186, btch:  31 usd:   0
[  765.640488] CPU    2: hi:  186, btch:  31 usd:   0
[  765.640488] CPU    3: hi:  186, btch:  31 usd:   0
[  765.640488] Node 1 Normal per-cpu:
[  765.640488] CPU    0: hi:  186, btch:  31 usd:   0
[  765.640488] CPU    1: hi:  186, btch:  31 usd:   0
[  765.640488] CPU    2: hi:  186, btch:  31 usd:   0
[  765.640488] CPU    3: hi:  186, btch:  31 usd:   0
[  765.640488] active_anon:677421 inactive_anon:261658 isolated_anon:1537
[  765.640488]  active_file:3 inactive_file:11 isolated_file:0
[  765.640488]  unevictable:1352 dirty:0 writeback:185934 unstable:0
[  765.640488]  free:3549 slab_reclaimable:8755 slab_unreclaimable:39259
[  765.640488]  mapped:732 shmem:19874 pagetables:10122 bounce:0
[  765.640488] Node 0 DMA free:8040kB min:28kB low:32kB high:40kB active_anon:3736kB inactive_anon:3988kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15768kB mlocked:0kB dirty:0kB writeback:1476kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:156kB kernel_stack:0kB pagetables:16kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[  765.640488] lowmem_reserve[]: 0 2004 2004 2004
[  765.640488] Node 0 DMA32 free:3036kB min:4036kB low:5044kB high:6052kB active_anon:1362344kB inactive_anon:454840kB active_file:12kB inactive_file:32kB unevictable:580kB isolated(anon):768kB isolated(file):0kB present:2052320kB mlocked:580kB dirty:0kB writeback:417620kB mapped:684kB shmem:5900kB slab_reclaimable:17604kB slab_unreclaimable:85332kB kernel_stack:1984kB pagetables:24012kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[  765.640488] lowmem_reserve[]: 0 0 0 0
[  765.845104] Node 1 DMA32 free:2348kB min:3036kB low:3792kB high:4552kB active_anon:1135940kB inactive_anon:379248kB active_file:0kB inactive_file:0kB unevictable:1276kB isolated(anon):4100kB isolated(file):0kB present:1544000kB mlocked:1276kB dirty:0kB writeback:239616kB mapped:0kB shmem:71100kB slab_reclaimable:10652kB slab_unreclaimable:30636kB kernel_stack:296kB pagetables:5036kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[  765.845104] lowmem_reserve[]: 0 0 505 505
[  765.845104] Node 1 Normal free:788kB min:1016kB low:1268kB high:1524kB active_anon:207664kB inactive_anon:208556kB active_file:0kB inactive_file:12kB unevictable:3552kB isolated(anon):1280kB isolated(file):0kB present:517120kB mlocked:3552kB dirty:0kB writeback:85024kB mapped:2244kB shmem:2496kB slab_reclaimable:6764kB slab_unreclaimable:40912kB kernel_stack:1128kB pagetables:11424kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[  765.845104] lowmem_reserve[]: 0 0 0 0
[  765.845104] Node 0 DMA: 44*4kB 1*8kB 3*16kB 2*32kB 1*64kB 2*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 8040kB
[  765.845104] Node 0 DMA32: 255*4kB 0*8kB 0*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 3036kB
[  765.845104] Node 1 DMA32: 45*4kB 1*8kB 3*16kB 2*32kB 2*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2348kB
[  765.845104] Node 1 Normal: 55*4kB 5*8kB 3*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 788kB
[  765.845104] 217638 total pagecache pages
[  765.845104] 197044 pages in swap cache
[  765.845104] Swap cache stats: add 267008, delete 69964, find 11614/11666
[  765.845104] Free swap  = 8936872kB
[  765.845104] Total swap = 10000316kB
[  765.845104] 1048575 pages RAM
[  765.845104] 33808 pages reserved
[  765.845104] 48325 pages shared
[  765.845104] 996865 pages non-shared

Re: -mm: xfs lockdep warning

[Tejun Heo]

Hello,

On 10/07/2010 05:15 PM, Torsten Kaiser wrote:
> :-(
> The second try with the patched 2.6.36-rc6 got stuck again.

:-(

>> repeat the same test w/ 2.6.35 and see whether there's any noticeable
>> difference?
> 
> I just tested vanilla 2.6.35, userspace and the testcase as with the
> 2.6.36-rc6 kernel.
> Behavior was the same as with the first try of the patched 2.6.36-rc6:
> The testcase started to build up CPU load and memory pressure, then
> the mouse got stuck. After ~1 min the system recovered and KDE became
> usable again. (Compile just finish OK)
> 
> vmstat 60 from 2.6.35:
>  2  0      0 2365848   1060 742072    0    0   205     7 6467 12252 26  5 62  8
> 17  0      0 837936   1060 1079520    0    0  1025    24 2210 13483 69 20  8  2
> 18  0      0 694296   1060 1127728    0    0   231    14  626 1883 74 26  0  0
> 20  0      0 228540    124 998440    0    0    48    12  604 1820 72 28  0  0
> 23  1     24 195456     52 657216    0    0   339    11  563 1669 84 16  0  0
>  0 42 937144  14004      0 164352    0 15617   290 15633  827 1254 24  9  1 66
> 21  1 861184 776104      0 149908  998 3852  2790  3902 1137 2454 45 15  0 40
> 21  0 798544 1131964      0 190004  212    0   530    15  599 1469 87 13  0  0
> 12  1 779488 1536772      0 225312  195    0   398    12  534 1087 88 12  0  0
> 11  0 620620 2201408      0 273976 1338    0  1638    40  851 2281 75 19  3  3
> 12  1 588196 1572564      0 344232  521    0  1326   152  787 1888 86 14  0  0
>  8  0 510876 2286000      0 311464  127    0   442   398  758 1558 88 12  0  0
> 
>> Some level of stuttering is expected if the system is hit
>> with sudden huge spike of memory pressure but let's see if it has
>> regressed somehow.
> 
> Yes, if a swapstorm occurs, I can live with a (short) "lockup". For
> example compiling openoffice on tmpfs is a case where I have seen
> similar short periods of the mouse getting stuck. But with 2.6.36-rc5
> it was the first time that the system did not seem to recover, so I
> think it should count as a regression compared to 2.6.35.
> 
> As the second try with the patched 2.6.36-rc6 got stuck again, could
> it be that your patch is incomplete?
>>From drivers/md/dm-crypt.c:
>         cc->io_queue = create_singlethread_workqueue("kcryptd_io");
>         cc->crypt_queue = create_singlethread_workqueue("kcryptd");
> 
> Do these workqueues also need WQ_MEM_RECLAIM?

No, singlethread wq's are currently mapped to unbound wq's and
automatically HIGHPRI.

> I have attached the SysRq-M output from the patched -rc6. The symptoms
> were the same as with the unpatched kernel. The system got stuck, the
> hung_task_timeout triggered for a numer of programs in sync_page() or
> do_lookup() and one kworker/1:1:496 was probable the nail in the
> coffin.
>
> Do you want me to do more tests with 2.6.35 or the patched .36-rc6? Or
> any other patch I should try?

496 being stuck should be okay.  They are allowed to be stuck there
and that's exactly why we have the rescuers.  Can you please capture
the output of sysrq-t w/ the patch applied and hung trigger tripped?
I'd really like to know what the rescuers are doing.

Thanks.

-- 
tejun

Re: -mm: xfs lockdep warning

[Tejun Heo]

(adding back lkml)

I can now more or less reliably reproduce the hang.  I'll write when I
know more.

Thanks.

-- 
tejun