* Hang in XFS reclaim on 3.7.0-rc3 @ 2012-10-29 20:03 ` Torsten Kaiser 0 siblings, 0 replies; 31+ messages in thread From: Torsten Kaiser @ 2012-10-29 20:03 UTC (permalink / raw) To: xfs; +Cc: Linux Kernel After experiencing a hang of all IO yesterday ( http://marc.info/?l=linux-kernel&m=135142236520624&w=2 ), I turned on LOCKDEP after upgrading to -rc3. I then tried to replicate the load that hung yesterday and got the following lockdep report, implicating XFS instead of by stacking swap onto dm-crypt and md. Oct 29 20:27:11 thoregon kernel: [ 2675.571958] usb 7-2: USB disconnect, device number 2 Oct 29 20:30:01 thoregon kernel: [ 2844.971913] Oct 29 20:30:01 thoregon kernel: [ 2844.971920] ================================= Oct 29 20:30:01 thoregon kernel: [ 2844.971921] [ INFO: inconsistent lock state ] Oct 29 20:30:01 thoregon kernel: [ 2844.971924] 3.7.0-rc3 #1 Not tainted Oct 29 20:30:01 thoregon kernel: [ 2844.971925] --------------------------------- Oct 29 20:30:01 thoregon kernel: [ 2844.971927] inconsistent {RECLAIM_FS-ON-W} -> {IN-RECLAIM_FS-W} usage. Oct 29 20:30:01 thoregon kernel: [ 2844.971929] kswapd0/725 [HC0[0]:SC0[0]:HE1:SE1] takes: Oct 29 20:30:01 thoregon kernel: [ 2844.971931] (&(&ip->i_lock)->mr_lock){++++?.}, at: [<ffffffff811e7ef4>] xfs_ilock+0x84/0xb0 Oct 29 20:30:01 thoregon kernel: [ 2844.971941] {RECLAIM_FS-ON-W} state was registered at: Oct 29 20:30:01 thoregon kernel: [ 2844.971942] [<ffffffff8108137e>] mark_held_locks+0x7e/0x130 Oct 29 20:30:01 thoregon kernel: [ 2844.971947] [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0 Oct 29 20:30:01 thoregon kernel: [ 2844.971949] [<ffffffff810e9dd5>] kmem_cache_alloc+0x35/0xe0 Oct 29 20:30:01 thoregon kernel: [ 2844.971952] [<ffffffff810dba31>] vm_map_ram+0x271/0x770 Oct 29 20:30:01 thoregon kernel: [ 2844.971955] [<ffffffff811e10a6>] _xfs_buf_map_pages+0x46/0xe0 Oct 29 20:30:01 thoregon kernel: [ 2844.971959] [<ffffffff811e1fba>] xfs_buf_get_map+0x8a/0x130 Oct 29 20:30:01 thoregon kernel: [ 2844.971961] [<ffffffff81233849>] xfs_trans_get_buf_map+0xa9/0xd0 Oct 29 20:30:01 thoregon kernel: [ 2844.971964] [<ffffffff8121e339>] xfs_ifree_cluster+0x129/0x670 Oct 29 20:30:01 thoregon kernel: [ 2844.971967] [<ffffffff8121f959>] xfs_ifree+0xe9/0xf0 Oct 29 20:30:01 thoregon kernel: [ 2844.971969] [<ffffffff811f4abf>] xfs_inactive+0x2af/0x480 Oct 29 20:30:01 thoregon kernel: [ 2844.971972] [<ffffffff811efb90>] xfs_fs_evict_inode+0x70/0x80 Oct 29 20:30:01 thoregon kernel: [ 2844.971974] [<ffffffff8110cb8f>] evict+0xaf/0x1b0 Oct 29 20:30:01 thoregon kernel: [ 2844.971977] [<ffffffff8110cd95>] iput+0x105/0x210 Oct 29 20:30:01 thoregon kernel: [ 2844.971979] [<ffffffff811070d0>] dentry_iput+0xa0/0xe0 Oct 29 20:30:01 thoregon kernel: [ 2844.971981] [<ffffffff81108310>] dput+0x150/0x280 Oct 29 20:30:01 thoregon kernel: [ 2844.971983] [<ffffffff811020fb>] sys_renameat+0x21b/0x290 Oct 29 20:30:01 thoregon kernel: [ 2844.971986] [<ffffffff81102186>] sys_rename+0x16/0x20 Oct 29 20:30:01 thoregon kernel: [ 2844.971988] [<ffffffff816b2292>] system_call_fastpath+0x16/0x1b Oct 29 20:30:01 thoregon kernel: [ 2844.971992] irq event stamp: 155377 Oct 29 20:30:01 thoregon kernel: [ 2844.971993] hardirqs last enabled at (155377): [<ffffffff816ae1ed>] mutex_trylock+0xfd/0x170 Oct 29 20:30:01 thoregon kernel: [ 2844.971997] hardirqs last disabled at (155376): [<ffffffff816ae12e>] mutex_trylock+0x3e/0x170 Oct 29 20:30:01 thoregon kernel: [ 2844.971999] softirqs last enabled at (155368): [<ffffffff81042fb1>] __do_softirq+0x111/0x170 Oct 29 20:30:01 thoregon kernel: [ 2844.972002] softirqs last disabled at (155353): [<ffffffff816b33bc>] call_softirq+0x1c/0x30 Oct 29 20:30:01 thoregon kernel: [ 2844.972004] Oct 29 20:30:01 thoregon kernel: [ 2844.972004] other info that might help us debug this: Oct 29 20:30:01 thoregon kernel: [ 2844.972006] Possible unsafe locking scenario: Oct 29 20:30:01 thoregon kernel: [ 2844.972006] Oct 29 20:30:01 thoregon kernel: [ 2844.972007] CPU0 Oct 29 20:30:01 thoregon kernel: [ 2844.972007] ---- Oct 29 20:30:01 thoregon kernel: [ 2844.972008] lock(&(&ip->i_lock)->mr_lock); Oct 29 20:30:01 thoregon kernel: [ 2844.972009] <Interrupt> Oct 29 20:30:01 thoregon kernel: [ 2844.972010] lock(&(&ip->i_lock)->mr_lock); Oct 29 20:30:01 thoregon kernel: [ 2844.972012] Oct 29 20:30:01 thoregon kernel: [ 2844.972012] *** DEADLOCK *** Oct 29 20:30:01 thoregon kernel: [ 2844.972012] Oct 29 20:30:01 thoregon kernel: [ 2844.972013] 3 locks held by kswapd0/725: Oct 29 20:30:01 thoregon kernel: [ 2844.972014] #0: (shrinker_rwsem){++++..}, at: [<ffffffff810bbd22>] shrink_slab+0x32/0x1f0 Oct 29 20:30:01 thoregon kernel: [ 2844.972020] #1: (&type->s_umount_key#20){++++.+}, at: [<ffffffff810f5a8e>] grab_super_passive+0x3e/0x90 Oct 29 20:30:01 thoregon kernel: [ 2844.972024] #2: (&pag->pag_ici_reclaim_lock){+.+...}, at: [<ffffffff811f263c>] xfs_reclaim_inodes_ag+0xbc/0x4f0 Oct 29 20:30:01 thoregon kernel: [ 2844.972027] Oct 29 20:30:01 thoregon kernel: [ 2844.972027] stack backtrace: Oct 29 20:30:01 thoregon kernel: [ 2844.972029] Pid: 725, comm: kswapd0 Not tainted 3.7.0-rc3 #1 Oct 29 20:30:01 thoregon kernel: [ 2844.972031] Call Trace: Oct 29 20:30:01 thoregon kernel: [ 2844.972035] [<ffffffff816a782c>] print_usage_bug+0x1f5/0x206 Oct 29 20:30:01 thoregon kernel: [ 2844.972039] [<ffffffff8100ed8a>] ? save_stack_trace+0x2a/0x50 Oct 29 20:30:01 thoregon kernel: [ 2844.972042] [<ffffffff8107e9fd>] mark_lock+0x28d/0x2f0 Oct 29 20:30:01 thoregon kernel: [ 2844.972044] [<ffffffff8107de30>] ? print_irq_inversion_bug.part.37+0x1f0/0x1f0 Oct 29 20:30:01 thoregon kernel: [ 2844.972047] [<ffffffff8107efdf>] __lock_acquire+0x57f/0x1c00 Oct 29 20:30:01 thoregon kernel: [ 2844.972049] [<ffffffff8107c899>] ? __lock_is_held+0x59/0x70 Oct 29 20:30:01 thoregon kernel: [ 2844.972051] [<ffffffff81080b55>] lock_acquire+0x55/0x70 Oct 29 20:30:01 thoregon kernel: [ 2844.972053] [<ffffffff811e7ef4>] ? xfs_ilock+0x84/0xb0 Oct 29 20:30:01 thoregon kernel: [ 2844.972056] [<ffffffff8106126a>] down_write_nested+0x4a/0x70 Oct 29 20:30:01 thoregon kernel: [ 2844.972058] [<ffffffff811e7ef4>] ? xfs_ilock+0x84/0xb0 Oct 29 20:30:01 thoregon kernel: [ 2844.972060] [<ffffffff811e7ef4>] xfs_ilock+0x84/0xb0 Oct 29 20:30:01 thoregon kernel: [ 2844.972063] [<ffffffff811f1a76>] xfs_reclaim_inode+0x136/0x340 Oct 29 20:30:01 thoregon kernel: [ 2844.972065] [<ffffffff811f283f>] xfs_reclaim_inodes_ag+0x2bf/0x4f0 Oct 29 20:30:01 thoregon kernel: [ 2844.972066] [<ffffffff811f2660>] ? xfs_reclaim_inodes_ag+0xe0/0x4f0 Oct 29 20:30:01 thoregon kernel: [ 2844.972069] [<ffffffff811f2bae>] xfs_reclaim_inodes_nr+0x2e/0x40 Oct 29 20:30:01 thoregon kernel: [ 2844.972071] [<ffffffff811ef480>] xfs_fs_free_cached_objects+0x10/0x20 Oct 29 20:30:01 thoregon kernel: [ 2844.972073] [<ffffffff810f5bf3>] prune_super+0x113/0x1a0 Oct 29 20:30:01 thoregon kernel: [ 2844.972075] [<ffffffff810bbe0e>] shrink_slab+0x11e/0x1f0 Oct 29 20:30:01 thoregon kernel: [ 2844.972077] [<ffffffff810be400>] kswapd+0x690/0xa10 Oct 29 20:30:01 thoregon kernel: [ 2844.972080] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60 Oct 29 20:30:01 thoregon kernel: [ 2844.972082] [<ffffffff810bdd70>] ? shrink_lruvec+0x540/0x540 Oct 29 20:30:01 thoregon kernel: [ 2844.972084] [<ffffffff8105c246>] kthread+0xd6/0xe0 Oct 29 20:30:01 thoregon kernel: [ 2844.972087] [<ffffffff816b148b>] ? _raw_spin_unlock_irq+0x2b/0x50 Oct 29 20:30:01 thoregon kernel: [ 2844.972089] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 Oct 29 20:30:01 thoregon kernel: [ 2844.972091] [<ffffffff816b21ec>] ret_from_fork+0x7c/0xb0 Oct 29 20:30:01 thoregon kernel: [ 2844.972093] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 Oct 29 20:30:01 thoregon cron[24374]: (root) CMD (test -x /usr/sbin/run-crons && /usr/sbin/run-crons) As kswapd got stuck yesterday, I think LOCKDEP found a real problem. If you need more information, please ask. I will try to provide it. Thanks, Torsten ^ permalink raw reply [flat|nested] 31+ messages in thread
* Hang in XFS reclaim on 3.7.0-rc3 @ 2012-10-29 20:03 ` Torsten Kaiser 0 siblings, 0 replies; 31+ messages in thread From: Torsten Kaiser @ 2012-10-29 20:03 UTC (permalink / raw) To: xfs; +Cc: Linux Kernel After experiencing a hang of all IO yesterday ( http://marc.info/?l=linux-kernel&m=135142236520624&w=2 ), I turned on LOCKDEP after upgrading to -rc3. I then tried to replicate the load that hung yesterday and got the following lockdep report, implicating XFS instead of by stacking swap onto dm-crypt and md. Oct 29 20:27:11 thoregon kernel: [ 2675.571958] usb 7-2: USB disconnect, device number 2 Oct 29 20:30:01 thoregon kernel: [ 2844.971913] Oct 29 20:30:01 thoregon kernel: [ 2844.971920] ================================= Oct 29 20:30:01 thoregon kernel: [ 2844.971921] [ INFO: inconsistent lock state ] Oct 29 20:30:01 thoregon kernel: [ 2844.971924] 3.7.0-rc3 #1 Not tainted Oct 29 20:30:01 thoregon kernel: [ 2844.971925] --------------------------------- Oct 29 20:30:01 thoregon kernel: [ 2844.971927] inconsistent {RECLAIM_FS-ON-W} -> {IN-RECLAIM_FS-W} usage. Oct 29 20:30:01 thoregon kernel: [ 2844.971929] kswapd0/725 [HC0[0]:SC0[0]:HE1:SE1] takes: Oct 29 20:30:01 thoregon kernel: [ 2844.971931] (&(&ip->i_lock)->mr_lock){++++?.}, at: [<ffffffff811e7ef4>] xfs_ilock+0x84/0xb0 Oct 29 20:30:01 thoregon kernel: [ 2844.971941] {RECLAIM_FS-ON-W} state was registered at: Oct 29 20:30:01 thoregon kernel: [ 2844.971942] [<ffffffff8108137e>] mark_held_locks+0x7e/0x130 Oct 29 20:30:01 thoregon kernel: [ 2844.971947] [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0 Oct 29 20:30:01 thoregon kernel: [ 2844.971949] [<ffffffff810e9dd5>] kmem_cache_alloc+0x35/0xe0 Oct 29 20:30:01 thoregon kernel: [ 2844.971952] [<ffffffff810dba31>] vm_map_ram+0x271/0x770 Oct 29 20:30:01 thoregon kernel: [ 2844.971955] [<ffffffff811e10a6>] _xfs_buf_map_pages+0x46/0xe0 Oct 29 20:30:01 thoregon kernel: [ 2844.971959] [<ffffffff811e1fba>] xfs_buf_get_map+0x8a/0x130 Oct 29 20:30:01 thoregon kernel: [ 2844.971961] [<ffffffff81233849>] xfs_trans_get_buf_map+0xa9/0xd0 Oct 29 20:30:01 thoregon kernel: [ 2844.971964] [<ffffffff8121e339>] xfs_ifree_cluster+0x129/0x670 Oct 29 20:30:01 thoregon kernel: [ 2844.971967] [<ffffffff8121f959>] xfs_ifree+0xe9/0xf0 Oct 29 20:30:01 thoregon kernel: [ 2844.971969] [<ffffffff811f4abf>] xfs_inactive+0x2af/0x480 Oct 29 20:30:01 thoregon kernel: [ 2844.971972] [<ffffffff811efb90>] xfs_fs_evict_inode+0x70/0x80 Oct 29 20:30:01 thoregon kernel: [ 2844.971974] [<ffffffff8110cb8f>] evict+0xaf/0x1b0 Oct 29 20:30:01 thoregon kernel: [ 2844.971977] [<ffffffff8110cd95>] iput+0x105/0x210 Oct 29 20:30:01 thoregon kernel: [ 2844.971979] [<ffffffff811070d0>] dentry_iput+0xa0/0xe0 Oct 29 20:30:01 thoregon kernel: [ 2844.971981] [<ffffffff81108310>] dput+0x150/0x280 Oct 29 20:30:01 thoregon kernel: [ 2844.971983] [<ffffffff811020fb>] sys_renameat+0x21b/0x290 Oct 29 20:30:01 thoregon kernel: [ 2844.971986] [<ffffffff81102186>] sys_rename+0x16/0x20 Oct 29 20:30:01 thoregon kernel: [ 2844.971988] [<ffffffff816b2292>] system_call_fastpath+0x16/0x1b Oct 29 20:30:01 thoregon kernel: [ 2844.971992] irq event stamp: 155377 Oct 29 20:30:01 thoregon kernel: [ 2844.971993] hardirqs last enabled at (155377): [<ffffffff816ae1ed>] mutex_trylock+0xfd/0x170 Oct 29 20:30:01 thoregon kernel: [ 2844.971997] hardirqs last disabled at (155376): [<ffffffff816ae12e>] mutex_trylock+0x3e/0x170 Oct 29 20:30:01 thoregon kernel: [ 2844.971999] softirqs last enabled at (155368): [<ffffffff81042fb1>] __do_softirq+0x111/0x170 Oct 29 20:30:01 thoregon kernel: [ 2844.972002] softirqs last disabled at (155353): [<ffffffff816b33bc>] call_softirq+0x1c/0x30 Oct 29 20:30:01 thoregon kernel: [ 2844.972004] Oct 29 20:30:01 thoregon kernel: [ 2844.972004] other info that might help us debug this: Oct 29 20:30:01 thoregon kernel: [ 2844.972006] Possible unsafe locking scenario: Oct 29 20:30:01 thoregon kernel: [ 2844.972006] Oct 29 20:30:01 thoregon kernel: [ 2844.972007] CPU0 Oct 29 20:30:01 thoregon kernel: [ 2844.972007] ---- Oct 29 20:30:01 thoregon kernel: [ 2844.972008] lock(&(&ip->i_lock)->mr_lock); Oct 29 20:30:01 thoregon kernel: [ 2844.972009] <Interrupt> Oct 29 20:30:01 thoregon kernel: [ 2844.972010] lock(&(&ip->i_lock)->mr_lock); Oct 29 20:30:01 thoregon kernel: [ 2844.972012] Oct 29 20:30:01 thoregon kernel: [ 2844.972012] *** DEADLOCK *** Oct 29 20:30:01 thoregon kernel: [ 2844.972012] Oct 29 20:30:01 thoregon kernel: [ 2844.972013] 3 locks held by kswapd0/725: Oct 29 20:30:01 thoregon kernel: [ 2844.972014] #0: (shrinker_rwsem){++++..}, at: [<ffffffff810bbd22>] shrink_slab+0x32/0x1f0 Oct 29 20:30:01 thoregon kernel: [ 2844.972020] #1: (&type->s_umount_key#20){++++.+}, at: [<ffffffff810f5a8e>] grab_super_passive+0x3e/0x90 Oct 29 20:30:01 thoregon kernel: [ 2844.972024] #2: (&pag->pag_ici_reclaim_lock){+.+...}, at: [<ffffffff811f263c>] xfs_reclaim_inodes_ag+0xbc/0x4f0 Oct 29 20:30:01 thoregon kernel: [ 2844.972027] Oct 29 20:30:01 thoregon kernel: [ 2844.972027] stack backtrace: Oct 29 20:30:01 thoregon kernel: [ 2844.972029] Pid: 725, comm: kswapd0 Not tainted 3.7.0-rc3 #1 Oct 29 20:30:01 thoregon kernel: [ 2844.972031] Call Trace: Oct 29 20:30:01 thoregon kernel: [ 2844.972035] [<ffffffff816a782c>] print_usage_bug+0x1f5/0x206 Oct 29 20:30:01 thoregon kernel: [ 2844.972039] [<ffffffff8100ed8a>] ? save_stack_trace+0x2a/0x50 Oct 29 20:30:01 thoregon kernel: [ 2844.972042] [<ffffffff8107e9fd>] mark_lock+0x28d/0x2f0 Oct 29 20:30:01 thoregon kernel: [ 2844.972044] [<ffffffff8107de30>] ? print_irq_inversion_bug.part.37+0x1f0/0x1f0 Oct 29 20:30:01 thoregon kernel: [ 2844.972047] [<ffffffff8107efdf>] __lock_acquire+0x57f/0x1c00 Oct 29 20:30:01 thoregon kernel: [ 2844.972049] [<ffffffff8107c899>] ? __lock_is_held+0x59/0x70 Oct 29 20:30:01 thoregon kernel: [ 2844.972051] [<ffffffff81080b55>] lock_acquire+0x55/0x70 Oct 29 20:30:01 thoregon kernel: [ 2844.972053] [<ffffffff811e7ef4>] ? xfs_ilock+0x84/0xb0 Oct 29 20:30:01 thoregon kernel: [ 2844.972056] [<ffffffff8106126a>] down_write_nested+0x4a/0x70 Oct 29 20:30:01 thoregon kernel: [ 2844.972058] [<ffffffff811e7ef4>] ? xfs_ilock+0x84/0xb0 Oct 29 20:30:01 thoregon kernel: [ 2844.972060] [<ffffffff811e7ef4>] xfs_ilock+0x84/0xb0 Oct 29 20:30:01 thoregon kernel: [ 2844.972063] [<ffffffff811f1a76>] xfs_reclaim_inode+0x136/0x340 Oct 29 20:30:01 thoregon kernel: [ 2844.972065] [<ffffffff811f283f>] xfs_reclaim_inodes_ag+0x2bf/0x4f0 Oct 29 20:30:01 thoregon kernel: [ 2844.972066] [<ffffffff811f2660>] ? xfs_reclaim_inodes_ag+0xe0/0x4f0 Oct 29 20:30:01 thoregon kernel: [ 2844.972069] [<ffffffff811f2bae>] xfs_reclaim_inodes_nr+0x2e/0x40 Oct 29 20:30:01 thoregon kernel: [ 2844.972071] [<ffffffff811ef480>] xfs_fs_free_cached_objects+0x10/0x20 Oct 29 20:30:01 thoregon kernel: [ 2844.972073] [<ffffffff810f5bf3>] prune_super+0x113/0x1a0 Oct 29 20:30:01 thoregon kernel: [ 2844.972075] [<ffffffff810bbe0e>] shrink_slab+0x11e/0x1f0 Oct 29 20:30:01 thoregon kernel: [ 2844.972077] [<ffffffff810be400>] kswapd+0x690/0xa10 Oct 29 20:30:01 thoregon kernel: [ 2844.972080] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60 Oct 29 20:30:01 thoregon kernel: [ 2844.972082] [<ffffffff810bdd70>] ? shrink_lruvec+0x540/0x540 Oct 29 20:30:01 thoregon kernel: [ 2844.972084] [<ffffffff8105c246>] kthread+0xd6/0xe0 Oct 29 20:30:01 thoregon kernel: [ 2844.972087] [<ffffffff816b148b>] ? _raw_spin_unlock_irq+0x2b/0x50 Oct 29 20:30:01 thoregon kernel: [ 2844.972089] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 Oct 29 20:30:01 thoregon kernel: [ 2844.972091] [<ffffffff816b21ec>] ret_from_fork+0x7c/0xb0 Oct 29 20:30:01 thoregon kernel: [ 2844.972093] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 Oct 29 20:30:01 thoregon cron[24374]: (root) CMD (test -x /usr/sbin/run-crons && /usr/sbin/run-crons) As kswapd got stuck yesterday, I think LOCKDEP found a real problem. If you need more information, please ask. I will try to provide it. Thanks, Torsten _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Hang in XFS reclaim on 3.7.0-rc3 2012-10-29 20:03 ` Torsten Kaiser @ 2012-10-29 22:26 ` Dave Chinner -1 siblings, 0 replies; 31+ messages in thread From: Dave Chinner @ 2012-10-29 22:26 UTC (permalink / raw) To: Torsten Kaiser; +Cc: xfs, Linux Kernel On Mon, Oct 29, 2012 at 09:03:15PM +0100, Torsten Kaiser wrote: > After experiencing a hang of all IO yesterday ( > http://marc.info/?l=linux-kernel&m=135142236520624&w=2 ), I turned on > LOCKDEP after upgrading to -rc3. > > I then tried to replicate the load that hung yesterday and got the > following lockdep report, implicating XFS instead of by stacking swap > onto dm-crypt and md. > > [ 2844.971913] > [ 2844.971920] ================================= > [ 2844.971921] [ INFO: inconsistent lock state ] > [ 2844.971924] 3.7.0-rc3 #1 Not tainted > [ 2844.971925] --------------------------------- > [ 2844.971927] inconsistent {RECLAIM_FS-ON-W} -> {IN-RECLAIM_FS-W} usage. > [ 2844.971929] kswapd0/725 [HC0[0]:SC0[0]:HE1:SE1] takes: > [ 2844.971931] (&(&ip->i_lock)->mr_lock){++++?.}, at: [<ffffffff811e7ef4>] xfs_ilock+0x84/0xb0 > [ 2844.971941] {RECLAIM_FS-ON-W} state was registered at: > [ 2844.971942] [<ffffffff8108137e>] mark_held_locks+0x7e/0x130 > [ 2844.971947] [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0 > [ 2844.971949] [<ffffffff810e9dd5>] kmem_cache_alloc+0x35/0xe0 > [ 2844.971952] [<ffffffff810dba31>] vm_map_ram+0x271/0x770 > [ 2844.971955] [<ffffffff811e10a6>] _xfs_buf_map_pages+0x46/0xe0 > [ 2844.971959] [<ffffffff811e1fba>] xfs_buf_get_map+0x8a/0x130 > [ 2844.971961] [<ffffffff81233849>] xfs_trans_get_buf_map+0xa9/0xd0 > [ 2844.971964] [<ffffffff8121e339>] xfs_ifree_cluster+0x129/0x670 > [ 2844.971967] [<ffffffff8121f959>] xfs_ifree+0xe9/0xf0 > [ 2844.971969] [<ffffffff811f4abf>] xfs_inactive+0x2af/0x480 > [ 2844.971972] [<ffffffff811efb90>] xfs_fs_evict_inode+0x70/0x80 > [ 2844.971974] [<ffffffff8110cb8f>] evict+0xaf/0x1b0 > [ 2844.971977] [<ffffffff8110cd95>] iput+0x105/0x210 > [ 2844.971979] [<ffffffff811070d0>] dentry_iput+0xa0/0xe0 > [ 2844.971981] [<ffffffff81108310>] dput+0x150/0x280 > [ 2844.971983] [<ffffffff811020fb>] sys_renameat+0x21b/0x290 > [ 2844.971986] [<ffffffff81102186>] sys_rename+0x16/0x20 > [ 2844.971988] [<ffffffff816b2292>] system_call_fastpath+0x16/0x1b We shouldn't be mapping pages there. See if the patch below fixes it. Fundamentally, though, the lockdep warning has come about because vm_map_ram is doing a GFP_KERNEL allocation when we need it to be doing GFP_NOFS - we are within a transaction here, so memory reclaim is not allowed to recurse back into the filesystem. mm-folk: can we please get this vmalloc/gfp_flags passing API fixed once and for all? This is the fourth time in the last month or so that I've seen XFS bug reports with silent hangs and associated lockdep output that implicate GFP_KERNEL allocations from vm_map_ram in GFP_NOFS conditions as the potential cause.... Cheers, Dave. -- Dave Chinner david@fromorbit.com xfs: don't vmap inode cluster buffers during free From: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <dchinner@redhat.com> --- fs/xfs/xfs_inode.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c index c4add46..82f6e5d 100644 --- a/fs/xfs/xfs_inode.c +++ b/fs/xfs/xfs_inode.c @@ -1781,7 +1781,8 @@ xfs_ifree_cluster( * to mark all the active inodes on the buffer stale. */ bp = xfs_trans_get_buf(tp, mp->m_ddev_targp, blkno, - mp->m_bsize * blks_per_cluster, 0); + mp->m_bsize * blks_per_cluster, + XBF_UNMAPPED); if (!bp) return ENOMEM; ^ permalink raw reply related [flat|nested] 31+ messages in thread
* Re: Hang in XFS reclaim on 3.7.0-rc3 @ 2012-10-29 22:26 ` Dave Chinner 0 siblings, 0 replies; 31+ messages in thread From: Dave Chinner @ 2012-10-29 22:26 UTC (permalink / raw) To: Torsten Kaiser; +Cc: Linux Kernel, xfs On Mon, Oct 29, 2012 at 09:03:15PM +0100, Torsten Kaiser wrote: > After experiencing a hang of all IO yesterday ( > http://marc.info/?l=linux-kernel&m=135142236520624&w=2 ), I turned on > LOCKDEP after upgrading to -rc3. > > I then tried to replicate the load that hung yesterday and got the > following lockdep report, implicating XFS instead of by stacking swap > onto dm-crypt and md. > > [ 2844.971913] > [ 2844.971920] ================================= > [ 2844.971921] [ INFO: inconsistent lock state ] > [ 2844.971924] 3.7.0-rc3 #1 Not tainted > [ 2844.971925] --------------------------------- > [ 2844.971927] inconsistent {RECLAIM_FS-ON-W} -> {IN-RECLAIM_FS-W} usage. > [ 2844.971929] kswapd0/725 [HC0[0]:SC0[0]:HE1:SE1] takes: > [ 2844.971931] (&(&ip->i_lock)->mr_lock){++++?.}, at: [<ffffffff811e7ef4>] xfs_ilock+0x84/0xb0 > [ 2844.971941] {RECLAIM_FS-ON-W} state was registered at: > [ 2844.971942] [<ffffffff8108137e>] mark_held_locks+0x7e/0x130 > [ 2844.971947] [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0 > [ 2844.971949] [<ffffffff810e9dd5>] kmem_cache_alloc+0x35/0xe0 > [ 2844.971952] [<ffffffff810dba31>] vm_map_ram+0x271/0x770 > [ 2844.971955] [<ffffffff811e10a6>] _xfs_buf_map_pages+0x46/0xe0 > [ 2844.971959] [<ffffffff811e1fba>] xfs_buf_get_map+0x8a/0x130 > [ 2844.971961] [<ffffffff81233849>] xfs_trans_get_buf_map+0xa9/0xd0 > [ 2844.971964] [<ffffffff8121e339>] xfs_ifree_cluster+0x129/0x670 > [ 2844.971967] [<ffffffff8121f959>] xfs_ifree+0xe9/0xf0 > [ 2844.971969] [<ffffffff811f4abf>] xfs_inactive+0x2af/0x480 > [ 2844.971972] [<ffffffff811efb90>] xfs_fs_evict_inode+0x70/0x80 > [ 2844.971974] [<ffffffff8110cb8f>] evict+0xaf/0x1b0 > [ 2844.971977] [<ffffffff8110cd95>] iput+0x105/0x210 > [ 2844.971979] [<ffffffff811070d0>] dentry_iput+0xa0/0xe0 > [ 2844.971981] [<ffffffff81108310>] dput+0x150/0x280 > [ 2844.971983] [<ffffffff811020fb>] sys_renameat+0x21b/0x290 > [ 2844.971986] [<ffffffff81102186>] sys_rename+0x16/0x20 > [ 2844.971988] [<ffffffff816b2292>] system_call_fastpath+0x16/0x1b We shouldn't be mapping pages there. See if the patch below fixes it. Fundamentally, though, the lockdep warning has come about because vm_map_ram is doing a GFP_KERNEL allocation when we need it to be doing GFP_NOFS - we are within a transaction here, so memory reclaim is not allowed to recurse back into the filesystem. mm-folk: can we please get this vmalloc/gfp_flags passing API fixed once and for all? This is the fourth time in the last month or so that I've seen XFS bug reports with silent hangs and associated lockdep output that implicate GFP_KERNEL allocations from vm_map_ram in GFP_NOFS conditions as the potential cause.... Cheers, Dave. -- Dave Chinner david@fromorbit.com xfs: don't vmap inode cluster buffers during free From: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <dchinner@redhat.com> --- fs/xfs/xfs_inode.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c index c4add46..82f6e5d 100644 --- a/fs/xfs/xfs_inode.c +++ b/fs/xfs/xfs_inode.c @@ -1781,7 +1781,8 @@ xfs_ifree_cluster( * to mark all the active inodes on the buffer stale. */ bp = xfs_trans_get_buf(tp, mp->m_ddev_targp, blkno, - mp->m_bsize * blks_per_cluster, 0); + mp->m_bsize * blks_per_cluster, + XBF_UNMAPPED); if (!bp) return ENOMEM; _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply related [flat|nested] 31+ messages in thread
* Re: Hang in XFS reclaim on 3.7.0-rc3 2012-10-29 22:26 ` Dave Chinner (?) @ 2012-10-29 22:41 ` Dave Chinner -1 siblings, 0 replies; 31+ messages in thread From: Dave Chinner @ 2012-10-29 22:41 UTC (permalink / raw) To: Torsten Kaiser; +Cc: xfs, Linux Kernel, linux-mm [add the linux-mm cc I forgot to add before sending] On Tue, Oct 30, 2012 at 09:26:13AM +1100, Dave Chinner wrote: > On Mon, Oct 29, 2012 at 09:03:15PM +0100, Torsten Kaiser wrote: > > After experiencing a hang of all IO yesterday ( > > http://marc.info/?l=linux-kernel&m=135142236520624&w=2 ), I turned on > > LOCKDEP after upgrading to -rc3. > > > > I then tried to replicate the load that hung yesterday and got the > > following lockdep report, implicating XFS instead of by stacking swap > > onto dm-crypt and md. > > > > [ 2844.971913] > > [ 2844.971920] ================================= > > [ 2844.971921] [ INFO: inconsistent lock state ] > > [ 2844.971924] 3.7.0-rc3 #1 Not tainted > > [ 2844.971925] --------------------------------- > > [ 2844.971927] inconsistent {RECLAIM_FS-ON-W} -> {IN-RECLAIM_FS-W} usage. > > [ 2844.971929] kswapd0/725 [HC0[0]:SC0[0]:HE1:SE1] takes: > > [ 2844.971931] (&(&ip->i_lock)->mr_lock){++++?.}, at: [<ffffffff811e7ef4>] xfs_ilock+0x84/0xb0 > > [ 2844.971941] {RECLAIM_FS-ON-W} state was registered at: > > [ 2844.971942] [<ffffffff8108137e>] mark_held_locks+0x7e/0x130 > > [ 2844.971947] [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0 > > [ 2844.971949] [<ffffffff810e9dd5>] kmem_cache_alloc+0x35/0xe0 > > [ 2844.971952] [<ffffffff810dba31>] vm_map_ram+0x271/0x770 > > [ 2844.971955] [<ffffffff811e10a6>] _xfs_buf_map_pages+0x46/0xe0 > > [ 2844.971959] [<ffffffff811e1fba>] xfs_buf_get_map+0x8a/0x130 > > [ 2844.971961] [<ffffffff81233849>] xfs_trans_get_buf_map+0xa9/0xd0 > > [ 2844.971964] [<ffffffff8121e339>] xfs_ifree_cluster+0x129/0x670 > > [ 2844.971967] [<ffffffff8121f959>] xfs_ifree+0xe9/0xf0 > > [ 2844.971969] [<ffffffff811f4abf>] xfs_inactive+0x2af/0x480 > > [ 2844.971972] [<ffffffff811efb90>] xfs_fs_evict_inode+0x70/0x80 > > [ 2844.971974] [<ffffffff8110cb8f>] evict+0xaf/0x1b0 > > [ 2844.971977] [<ffffffff8110cd95>] iput+0x105/0x210 > > [ 2844.971979] [<ffffffff811070d0>] dentry_iput+0xa0/0xe0 > > [ 2844.971981] [<ffffffff81108310>] dput+0x150/0x280 > > [ 2844.971983] [<ffffffff811020fb>] sys_renameat+0x21b/0x290 > > [ 2844.971986] [<ffffffff81102186>] sys_rename+0x16/0x20 > > [ 2844.971988] [<ffffffff816b2292>] system_call_fastpath+0x16/0x1b > > We shouldn't be mapping pages there. See if the patch below fixes > it. > > Fundamentally, though, the lockdep warning has come about because > vm_map_ram is doing a GFP_KERNEL allocation when we need it to be > doing GFP_NOFS - we are within a transaction here, so memory reclaim > is not allowed to recurse back into the filesystem. > > mm-folk: can we please get this vmalloc/gfp_flags passing API > fixed once and for all? This is the fourth time in the last month or > so that I've seen XFS bug reports with silent hangs and associated > lockdep output that implicate GFP_KERNEL allocations from vm_map_ram > in GFP_NOFS conditions as the potential cause.... > > Cheers, > > Dave. > -- > Dave Chinner > david@fromorbit.com > > xfs: don't vmap inode cluster buffers during free > > From: Dave Chinner <dchinner@redhat.com> > > Signed-off-by: Dave Chinner <dchinner@redhat.com> > --- > fs/xfs/xfs_inode.c | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) > > diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c > index c4add46..82f6e5d 100644 > --- a/fs/xfs/xfs_inode.c > +++ b/fs/xfs/xfs_inode.c > @@ -1781,7 +1781,8 @@ xfs_ifree_cluster( > * to mark all the active inodes on the buffer stale. > */ > bp = xfs_trans_get_buf(tp, mp->m_ddev_targp, blkno, > - mp->m_bsize * blks_per_cluster, 0); > + mp->m_bsize * blks_per_cluster, > + XBF_UNMAPPED); > > if (!bp) > return ENOMEM; > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ > > -- > This message has been scanned for viruses and > dangerous content by MailScanner, and is > believed to be clean. > > -- Dave Chinner david@fromorbit.com ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Hang in XFS reclaim on 3.7.0-rc3 @ 2012-10-29 22:41 ` Dave Chinner 0 siblings, 0 replies; 31+ messages in thread From: Dave Chinner @ 2012-10-29 22:41 UTC (permalink / raw) To: Torsten Kaiser; +Cc: xfs, Linux Kernel, linux-mm [add the linux-mm cc I forgot to add before sending] On Tue, Oct 30, 2012 at 09:26:13AM +1100, Dave Chinner wrote: > On Mon, Oct 29, 2012 at 09:03:15PM +0100, Torsten Kaiser wrote: > > After experiencing a hang of all IO yesterday ( > > http://marc.info/?l=linux-kernel&m=135142236520624&w=2 ), I turned on > > LOCKDEP after upgrading to -rc3. > > > > I then tried to replicate the load that hung yesterday and got the > > following lockdep report, implicating XFS instead of by stacking swap > > onto dm-crypt and md. > > > > [ 2844.971913] > > [ 2844.971920] ================================= > > [ 2844.971921] [ INFO: inconsistent lock state ] > > [ 2844.971924] 3.7.0-rc3 #1 Not tainted > > [ 2844.971925] --------------------------------- > > [ 2844.971927] inconsistent {RECLAIM_FS-ON-W} -> {IN-RECLAIM_FS-W} usage. > > [ 2844.971929] kswapd0/725 [HC0[0]:SC0[0]:HE1:SE1] takes: > > [ 2844.971931] (&(&ip->i_lock)->mr_lock){++++?.}, at: [<ffffffff811e7ef4>] xfs_ilock+0x84/0xb0 > > [ 2844.971941] {RECLAIM_FS-ON-W} state was registered at: > > [ 2844.971942] [<ffffffff8108137e>] mark_held_locks+0x7e/0x130 > > [ 2844.971947] [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0 > > [ 2844.971949] [<ffffffff810e9dd5>] kmem_cache_alloc+0x35/0xe0 > > [ 2844.971952] [<ffffffff810dba31>] vm_map_ram+0x271/0x770 > > [ 2844.971955] [<ffffffff811e10a6>] _xfs_buf_map_pages+0x46/0xe0 > > [ 2844.971959] [<ffffffff811e1fba>] xfs_buf_get_map+0x8a/0x130 > > [ 2844.971961] [<ffffffff81233849>] xfs_trans_get_buf_map+0xa9/0xd0 > > [ 2844.971964] [<ffffffff8121e339>] xfs_ifree_cluster+0x129/0x670 > > [ 2844.971967] [<ffffffff8121f959>] xfs_ifree+0xe9/0xf0 > > [ 2844.971969] [<ffffffff811f4abf>] xfs_inactive+0x2af/0x480 > > [ 2844.971972] [<ffffffff811efb90>] xfs_fs_evict_inode+0x70/0x80 > > [ 2844.971974] [<ffffffff8110cb8f>] evict+0xaf/0x1b0 > > [ 2844.971977] [<ffffffff8110cd95>] iput+0x105/0x210 > > [ 2844.971979] [<ffffffff811070d0>] dentry_iput+0xa0/0xe0 > > [ 2844.971981] [<ffffffff81108310>] dput+0x150/0x280 > > [ 2844.971983] [<ffffffff811020fb>] sys_renameat+0x21b/0x290 > > [ 2844.971986] [<ffffffff81102186>] sys_rename+0x16/0x20 > > [ 2844.971988] [<ffffffff816b2292>] system_call_fastpath+0x16/0x1b > > We shouldn't be mapping pages there. See if the patch below fixes > it. > > Fundamentally, though, the lockdep warning has come about because > vm_map_ram is doing a GFP_KERNEL allocation when we need it to be > doing GFP_NOFS - we are within a transaction here, so memory reclaim > is not allowed to recurse back into the filesystem. > > mm-folk: can we please get this vmalloc/gfp_flags passing API > fixed once and for all? This is the fourth time in the last month or > so that I've seen XFS bug reports with silent hangs and associated > lockdep output that implicate GFP_KERNEL allocations from vm_map_ram > in GFP_NOFS conditions as the potential cause.... > > Cheers, > > Dave. > -- > Dave Chinner > david@fromorbit.com > > xfs: don't vmap inode cluster buffers during free > > From: Dave Chinner <dchinner@redhat.com> > > Signed-off-by: Dave Chinner <dchinner@redhat.com> > --- > fs/xfs/xfs_inode.c | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) > > diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c > index c4add46..82f6e5d 100644 > --- a/fs/xfs/xfs_inode.c > +++ b/fs/xfs/xfs_inode.c > @@ -1781,7 +1781,8 @@ xfs_ifree_cluster( > * to mark all the active inodes on the buffer stale. > */ > bp = xfs_trans_get_buf(tp, mp->m_ddev_targp, blkno, > - mp->m_bsize * blks_per_cluster, 0); > + mp->m_bsize * blks_per_cluster, > + XBF_UNMAPPED); > > if (!bp) > return ENOMEM; > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ > > -- > This message has been scanned for viruses and > dangerous content by MailScanner, and is > believed to be clean. > > -- Dave Chinner david@fromorbit.com -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Hang in XFS reclaim on 3.7.0-rc3 @ 2012-10-29 22:41 ` Dave Chinner 0 siblings, 0 replies; 31+ messages in thread From: Dave Chinner @ 2012-10-29 22:41 UTC (permalink / raw) To: Torsten Kaiser; +Cc: linux-mm, Linux Kernel, xfs [add the linux-mm cc I forgot to add before sending] On Tue, Oct 30, 2012 at 09:26:13AM +1100, Dave Chinner wrote: > On Mon, Oct 29, 2012 at 09:03:15PM +0100, Torsten Kaiser wrote: > > After experiencing a hang of all IO yesterday ( > > http://marc.info/?l=linux-kernel&m=135142236520624&w=2 ), I turned on > > LOCKDEP after upgrading to -rc3. > > > > I then tried to replicate the load that hung yesterday and got the > > following lockdep report, implicating XFS instead of by stacking swap > > onto dm-crypt and md. > > > > [ 2844.971913] > > [ 2844.971920] ================================= > > [ 2844.971921] [ INFO: inconsistent lock state ] > > [ 2844.971924] 3.7.0-rc3 #1 Not tainted > > [ 2844.971925] --------------------------------- > > [ 2844.971927] inconsistent {RECLAIM_FS-ON-W} -> {IN-RECLAIM_FS-W} usage. > > [ 2844.971929] kswapd0/725 [HC0[0]:SC0[0]:HE1:SE1] takes: > > [ 2844.971931] (&(&ip->i_lock)->mr_lock){++++?.}, at: [<ffffffff811e7ef4>] xfs_ilock+0x84/0xb0 > > [ 2844.971941] {RECLAIM_FS-ON-W} state was registered at: > > [ 2844.971942] [<ffffffff8108137e>] mark_held_locks+0x7e/0x130 > > [ 2844.971947] [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0 > > [ 2844.971949] [<ffffffff810e9dd5>] kmem_cache_alloc+0x35/0xe0 > > [ 2844.971952] [<ffffffff810dba31>] vm_map_ram+0x271/0x770 > > [ 2844.971955] [<ffffffff811e10a6>] _xfs_buf_map_pages+0x46/0xe0 > > [ 2844.971959] [<ffffffff811e1fba>] xfs_buf_get_map+0x8a/0x130 > > [ 2844.971961] [<ffffffff81233849>] xfs_trans_get_buf_map+0xa9/0xd0 > > [ 2844.971964] [<ffffffff8121e339>] xfs_ifree_cluster+0x129/0x670 > > [ 2844.971967] [<ffffffff8121f959>] xfs_ifree+0xe9/0xf0 > > [ 2844.971969] [<ffffffff811f4abf>] xfs_inactive+0x2af/0x480 > > [ 2844.971972] [<ffffffff811efb90>] xfs_fs_evict_inode+0x70/0x80 > > [ 2844.971974] [<ffffffff8110cb8f>] evict+0xaf/0x1b0 > > [ 2844.971977] [<ffffffff8110cd95>] iput+0x105/0x210 > > [ 2844.971979] [<ffffffff811070d0>] dentry_iput+0xa0/0xe0 > > [ 2844.971981] [<ffffffff81108310>] dput+0x150/0x280 > > [ 2844.971983] [<ffffffff811020fb>] sys_renameat+0x21b/0x290 > > [ 2844.971986] [<ffffffff81102186>] sys_rename+0x16/0x20 > > [ 2844.971988] [<ffffffff816b2292>] system_call_fastpath+0x16/0x1b > > We shouldn't be mapping pages there. See if the patch below fixes > it. > > Fundamentally, though, the lockdep warning has come about because > vm_map_ram is doing a GFP_KERNEL allocation when we need it to be > doing GFP_NOFS - we are within a transaction here, so memory reclaim > is not allowed to recurse back into the filesystem. > > mm-folk: can we please get this vmalloc/gfp_flags passing API > fixed once and for all? This is the fourth time in the last month or > so that I've seen XFS bug reports with silent hangs and associated > lockdep output that implicate GFP_KERNEL allocations from vm_map_ram > in GFP_NOFS conditions as the potential cause.... > > Cheers, > > Dave. > -- > Dave Chinner > david@fromorbit.com > > xfs: don't vmap inode cluster buffers during free > > From: Dave Chinner <dchinner@redhat.com> > > Signed-off-by: Dave Chinner <dchinner@redhat.com> > --- > fs/xfs/xfs_inode.c | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) > > diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c > index c4add46..82f6e5d 100644 > --- a/fs/xfs/xfs_inode.c > +++ b/fs/xfs/xfs_inode.c > @@ -1781,7 +1781,8 @@ xfs_ifree_cluster( > * to mark all the active inodes on the buffer stale. > */ > bp = xfs_trans_get_buf(tp, mp->m_ddev_targp, blkno, > - mp->m_bsize * blks_per_cluster, 0); > + mp->m_bsize * blks_per_cluster, > + XBF_UNMAPPED); > > if (!bp) > return ENOMEM; > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ > > -- > This message has been scanned for viruses and > dangerous content by MailScanner, and is > believed to be clean. > > -- Dave Chinner david@fromorbit.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Hang in XFS reclaim on 3.7.0-rc3 2012-10-29 22:26 ` Dave Chinner @ 2012-10-30 20:37 ` Torsten Kaiser -1 siblings, 0 replies; 31+ messages in thread From: Torsten Kaiser @ 2012-10-30 20:37 UTC (permalink / raw) To: Dave Chinner; +Cc: xfs, Linux Kernel On Mon, Oct 29, 2012 at 11:26 PM, Dave Chinner <david@fromorbit.com> wrote: > On Mon, Oct 29, 2012 at 09:03:15PM +0100, Torsten Kaiser wrote: >> After experiencing a hang of all IO yesterday ( >> http://marc.info/?l=linux-kernel&m=135142236520624&w=2 ), I turned on >> LOCKDEP after upgrading to -rc3. >> >> I then tried to replicate the load that hung yesterday and got the >> following lockdep report, implicating XFS instead of by stacking swap >> onto dm-crypt and md. >> >> [ 2844.971913] >> [ 2844.971920] ================================= >> [ 2844.971921] [ INFO: inconsistent lock state ] >> [ 2844.971924] 3.7.0-rc3 #1 Not tainted >> [ 2844.971925] --------------------------------- >> [ 2844.971927] inconsistent {RECLAIM_FS-ON-W} -> {IN-RECLAIM_FS-W} usage. >> [ 2844.971929] kswapd0/725 [HC0[0]:SC0[0]:HE1:SE1] takes: >> [ 2844.971931] (&(&ip->i_lock)->mr_lock){++++?.}, at: [<ffffffff811e7ef4>] xfs_ilock+0x84/0xb0 >> [ 2844.971941] {RECLAIM_FS-ON-W} state was registered at: >> [ 2844.971942] [<ffffffff8108137e>] mark_held_locks+0x7e/0x130 >> [ 2844.971947] [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0 >> [ 2844.971949] [<ffffffff810e9dd5>] kmem_cache_alloc+0x35/0xe0 >> [ 2844.971952] [<ffffffff810dba31>] vm_map_ram+0x271/0x770 >> [ 2844.971955] [<ffffffff811e10a6>] _xfs_buf_map_pages+0x46/0xe0 >> [ 2844.971959] [<ffffffff811e1fba>] xfs_buf_get_map+0x8a/0x130 >> [ 2844.971961] [<ffffffff81233849>] xfs_trans_get_buf_map+0xa9/0xd0 >> [ 2844.971964] [<ffffffff8121e339>] xfs_ifree_cluster+0x129/0x670 >> [ 2844.971967] [<ffffffff8121f959>] xfs_ifree+0xe9/0xf0 >> [ 2844.971969] [<ffffffff811f4abf>] xfs_inactive+0x2af/0x480 >> [ 2844.971972] [<ffffffff811efb90>] xfs_fs_evict_inode+0x70/0x80 >> [ 2844.971974] [<ffffffff8110cb8f>] evict+0xaf/0x1b0 >> [ 2844.971977] [<ffffffff8110cd95>] iput+0x105/0x210 >> [ 2844.971979] [<ffffffff811070d0>] dentry_iput+0xa0/0xe0 >> [ 2844.971981] [<ffffffff81108310>] dput+0x150/0x280 >> [ 2844.971983] [<ffffffff811020fb>] sys_renameat+0x21b/0x290 >> [ 2844.971986] [<ffffffff81102186>] sys_rename+0x16/0x20 >> [ 2844.971988] [<ffffffff816b2292>] system_call_fastpath+0x16/0x1b > > We shouldn't be mapping pages there. See if the patch below fixes > it. Applying your fix and rerunning my test workload did not trigger this or any other LOCKDEP reports. While I'm not 100% sure about my test case always hitting this, your description makes me quite confident, that it really fixed this issue. I will keep LOCKDEP enabled on that system, and if there really is another splat, I will report back here. But I rather doubt that this will be needed. Thanks for the very quick fix! Torsten > Fundamentally, though, the lockdep warning has come about because > vm_map_ram is doing a GFP_KERNEL allocation when we need it to be > doing GFP_NOFS - we are within a transaction here, so memory reclaim > is not allowed to recurse back into the filesystem. > > mm-folk: can we please get this vmalloc/gfp_flags passing API > fixed once and for all? This is the fourth time in the last month or > so that I've seen XFS bug reports with silent hangs and associated > lockdep output that implicate GFP_KERNEL allocations from vm_map_ram > in GFP_NOFS conditions as the potential cause.... > > Cheers, > > Dave. > -- > Dave Chinner > david@fromorbit.com > > xfs: don't vmap inode cluster buffers during free > > From: Dave Chinner <dchinner@redhat.com> > > Signed-off-by: Dave Chinner <dchinner@redhat.com> > --- > fs/xfs/xfs_inode.c | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) > > diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c > index c4add46..82f6e5d 100644 > --- a/fs/xfs/xfs_inode.c > +++ b/fs/xfs/xfs_inode.c > @@ -1781,7 +1781,8 @@ xfs_ifree_cluster( > * to mark all the active inodes on the buffer stale. > */ > bp = xfs_trans_get_buf(tp, mp->m_ddev_targp, blkno, > - mp->m_bsize * blks_per_cluster, 0); > + mp->m_bsize * blks_per_cluster, > + XBF_UNMAPPED); > > if (!bp) > return ENOMEM; ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Hang in XFS reclaim on 3.7.0-rc3 @ 2012-10-30 20:37 ` Torsten Kaiser 0 siblings, 0 replies; 31+ messages in thread From: Torsten Kaiser @ 2012-10-30 20:37 UTC (permalink / raw) To: Dave Chinner; +Cc: Linux Kernel, xfs On Mon, Oct 29, 2012 at 11:26 PM, Dave Chinner <david@fromorbit.com> wrote: > On Mon, Oct 29, 2012 at 09:03:15PM +0100, Torsten Kaiser wrote: >> After experiencing a hang of all IO yesterday ( >> http://marc.info/?l=linux-kernel&m=135142236520624&w=2 ), I turned on >> LOCKDEP after upgrading to -rc3. >> >> I then tried to replicate the load that hung yesterday and got the >> following lockdep report, implicating XFS instead of by stacking swap >> onto dm-crypt and md. >> >> [ 2844.971913] >> [ 2844.971920] ================================= >> [ 2844.971921] [ INFO: inconsistent lock state ] >> [ 2844.971924] 3.7.0-rc3 #1 Not tainted >> [ 2844.971925] --------------------------------- >> [ 2844.971927] inconsistent {RECLAIM_FS-ON-W} -> {IN-RECLAIM_FS-W} usage. >> [ 2844.971929] kswapd0/725 [HC0[0]:SC0[0]:HE1:SE1] takes: >> [ 2844.971931] (&(&ip->i_lock)->mr_lock){++++?.}, at: [<ffffffff811e7ef4>] xfs_ilock+0x84/0xb0 >> [ 2844.971941] {RECLAIM_FS-ON-W} state was registered at: >> [ 2844.971942] [<ffffffff8108137e>] mark_held_locks+0x7e/0x130 >> [ 2844.971947] [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0 >> [ 2844.971949] [<ffffffff810e9dd5>] kmem_cache_alloc+0x35/0xe0 >> [ 2844.971952] [<ffffffff810dba31>] vm_map_ram+0x271/0x770 >> [ 2844.971955] [<ffffffff811e10a6>] _xfs_buf_map_pages+0x46/0xe0 >> [ 2844.971959] [<ffffffff811e1fba>] xfs_buf_get_map+0x8a/0x130 >> [ 2844.971961] [<ffffffff81233849>] xfs_trans_get_buf_map+0xa9/0xd0 >> [ 2844.971964] [<ffffffff8121e339>] xfs_ifree_cluster+0x129/0x670 >> [ 2844.971967] [<ffffffff8121f959>] xfs_ifree+0xe9/0xf0 >> [ 2844.971969] [<ffffffff811f4abf>] xfs_inactive+0x2af/0x480 >> [ 2844.971972] [<ffffffff811efb90>] xfs_fs_evict_inode+0x70/0x80 >> [ 2844.971974] [<ffffffff8110cb8f>] evict+0xaf/0x1b0 >> [ 2844.971977] [<ffffffff8110cd95>] iput+0x105/0x210 >> [ 2844.971979] [<ffffffff811070d0>] dentry_iput+0xa0/0xe0 >> [ 2844.971981] [<ffffffff81108310>] dput+0x150/0x280 >> [ 2844.971983] [<ffffffff811020fb>] sys_renameat+0x21b/0x290 >> [ 2844.971986] [<ffffffff81102186>] sys_rename+0x16/0x20 >> [ 2844.971988] [<ffffffff816b2292>] system_call_fastpath+0x16/0x1b > > We shouldn't be mapping pages there. See if the patch below fixes > it. Applying your fix and rerunning my test workload did not trigger this or any other LOCKDEP reports. While I'm not 100% sure about my test case always hitting this, your description makes me quite confident, that it really fixed this issue. I will keep LOCKDEP enabled on that system, and if there really is another splat, I will report back here. But I rather doubt that this will be needed. Thanks for the very quick fix! Torsten > Fundamentally, though, the lockdep warning has come about because > vm_map_ram is doing a GFP_KERNEL allocation when we need it to be > doing GFP_NOFS - we are within a transaction here, so memory reclaim > is not allowed to recurse back into the filesystem. > > mm-folk: can we please get this vmalloc/gfp_flags passing API > fixed once and for all? This is the fourth time in the last month or > so that I've seen XFS bug reports with silent hangs and associated > lockdep output that implicate GFP_KERNEL allocations from vm_map_ram > in GFP_NOFS conditions as the potential cause.... > > Cheers, > > Dave. > -- > Dave Chinner > david@fromorbit.com > > xfs: don't vmap inode cluster buffers during free > > From: Dave Chinner <dchinner@redhat.com> > > Signed-off-by: Dave Chinner <dchinner@redhat.com> > --- > fs/xfs/xfs_inode.c | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) > > diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c > index c4add46..82f6e5d 100644 > --- a/fs/xfs/xfs_inode.c > +++ b/fs/xfs/xfs_inode.c > @@ -1781,7 +1781,8 @@ xfs_ifree_cluster( > * to mark all the active inodes on the buffer stale. > */ > bp = xfs_trans_get_buf(tp, mp->m_ddev_targp, blkno, > - mp->m_bsize * blks_per_cluster, 0); > + mp->m_bsize * blks_per_cluster, > + XBF_UNMAPPED); > > if (!bp) > return ENOMEM; _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Hang in XFS reclaim on 3.7.0-rc3 2012-10-30 20:37 ` Torsten Kaiser @ 2012-10-30 20:46 ` Christoph Hellwig -1 siblings, 0 replies; 31+ messages in thread From: Christoph Hellwig @ 2012-10-30 20:46 UTC (permalink / raw) To: Torsten Kaiser; +Cc: Dave Chinner, Linux Kernel, xfs On Tue, Oct 30, 2012 at 09:37:06PM +0100, Torsten Kaiser wrote: > On Mon, Oct 29, 2012 at 11:26 PM, Dave Chinner <david@fromorbit.com> wrote: For some reason I only managed to get the two mails from Torsten into my xfs list folder, but the one from Dave is missing. I did see Dave's mail in my linux-mm folder, though. > > From: Dave Chinner <dchinner@redhat.com> > > > > Signed-off-by: Dave Chinner <dchinner@redhat.com> Looks good, Reviewed-by: Christoph Hellwig <hch@lst.de> And I agree that vmap needs to be fixed to pass through the gfp flags ASAP. ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Hang in XFS reclaim on 3.7.0-rc3 @ 2012-10-30 20:46 ` Christoph Hellwig 0 siblings, 0 replies; 31+ messages in thread From: Christoph Hellwig @ 2012-10-30 20:46 UTC (permalink / raw) To: Torsten Kaiser; +Cc: Linux Kernel, xfs On Tue, Oct 30, 2012 at 09:37:06PM +0100, Torsten Kaiser wrote: > On Mon, Oct 29, 2012 at 11:26 PM, Dave Chinner <david@fromorbit.com> wrote: For some reason I only managed to get the two mails from Torsten into my xfs list folder, but the one from Dave is missing. I did see Dave's mail in my linux-mm folder, though. > > From: Dave Chinner <dchinner@redhat.com> > > > > Signed-off-by: Dave Chinner <dchinner@redhat.com> Looks good, Reviewed-by: Christoph Hellwig <hch@lst.de> And I agree that vmap needs to be fixed to pass through the gfp flags ASAP. _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Hang in XFS reclaim on 3.7.0-rc3 2012-10-30 20:37 ` Torsten Kaiser @ 2012-11-18 10:24 ` Torsten Kaiser -1 siblings, 0 replies; 31+ messages in thread From: Torsten Kaiser @ 2012-11-18 10:24 UTC (permalink / raw) To: Dave Chinner; +Cc: xfs, Linux Kernel On Tue, Oct 30, 2012 at 9:37 PM, Torsten Kaiser <just.for.lkml@googlemail.com> wrote: > I will keep LOCKDEP enabled on that system, and if there really is > another splat, I will report back here. But I rather doubt that this > will be needed. After the patch, I did not see this problem again, but today I found another LOCKDEP report that also looks XFS related. I found it twice in the logs, and as both were slightly different, I will attach both versions. Nov 6 21:57:09 thoregon kernel: [ 9941.104345] Nov 6 21:57:09 thoregon kernel: [ 9941.104350] ================================= Nov 6 21:57:09 thoregon kernel: [ 9941.104351] [ INFO: inconsistent lock state ] Nov 6 21:57:09 thoregon kernel: [ 9941.104353] 3.7.0-rc4 #1 Not tainted Nov 6 21:57:09 thoregon kernel: [ 9941.104354] --------------------------------- Nov 6 21:57:09 thoregon kernel: [ 9941.104355] inconsistent {RECLAIM_FS-ON-W} -> {IN-RECLAIM_FS-W} usage. Nov 6 21:57:09 thoregon kernel: [ 9941.104357] kswapd0/725 [HC0[0]:SC0[0]:HE1:SE1] takes: Nov 6 21:57:09 thoregon kernel: [ 9941.104359] (&(&ip->i_lock)->mr_lock){++++?.}, at: [<ffffffff811e8164>] xfs_ilock+0x84/0xb0 Nov 6 21:57:09 thoregon kernel: [ 9941.104366] {RECLAIM_FS-ON-W} state was registered at: Nov 6 21:57:09 thoregon kernel: [ 9941.104367] [<ffffffff8108137e>] mark_held_locks+0x7e/0x130 Nov 6 21:57:09 thoregon kernel: [ 9941.104371] [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0 Nov 6 21:57:09 thoregon kernel: [ 9941.104373] [<ffffffff810b5a55>] __alloc_pages_nodemask+0x75/0x800 Nov 6 21:57:09 thoregon kernel: [ 9941.104375] [<ffffffff810b6262>] __get_free_pages+0x12/0x40 Nov 6 21:57:09 thoregon kernel: [ 9941.104377] [<ffffffff8102d7f0>] pte_alloc_one_kernel+0x10/0x20 Nov 6 21:57:09 thoregon kernel: [ 9941.104380] [<ffffffff810cc3e6>] __pte_alloc_kernel+0x16/0x90 Nov 6 21:57:09 thoregon kernel: [ 9941.104382] [<ffffffff810d9f37>] vmap_page_range_noflush+0x287/0x320 Nov 6 21:57:09 thoregon kernel: [ 9941.104385] [<ffffffff810dbe54>] vm_map_ram+0x694/0x770 Nov 6 21:57:09 thoregon kernel: [ 9941.104386] [<ffffffff811e1316>] _xfs_buf_map_pages+0x46/0xe0 Nov 6 21:57:09 thoregon kernel: [ 9941.104389] [<ffffffff811e222a>] xfs_buf_get_map+0x8a/0x130 Nov 6 21:57:09 thoregon kernel: [ 9941.104391] [<ffffffff81233ab9>] xfs_trans_get_buf_map+0xa9/0xd0 Nov 6 21:57:09 thoregon kernel: [ 9941.104393] [<ffffffff8121e5a9>] xfs_ifree_cluster+0x129/0x670 Nov 6 21:57:09 thoregon kernel: [ 9941.104396] [<ffffffff8121fbc9>] xfs_ifree+0xe9/0xf0 Nov 6 21:57:09 thoregon kernel: [ 9941.104398] [<ffffffff811f4d2f>] xfs_inactive+0x2af/0x480 Nov 6 21:57:09 thoregon kernel: [ 9941.104400] [<ffffffff811efe00>] xfs_fs_evict_inode+0x70/0x80 Nov 6 21:57:09 thoregon kernel: [ 9941.104402] [<ffffffff8110cb8f>] evict+0xaf/0x1b0 Nov 6 21:57:09 thoregon kernel: [ 9941.104405] [<ffffffff8110cd95>] iput+0x105/0x210 Nov 6 21:57:09 thoregon kernel: [ 9941.104406] [<ffffffff81107ba0>] d_delete+0x150/0x190 Nov 6 21:57:09 thoregon kernel: [ 9941.104408] [<ffffffff810ff8a7>] vfs_rmdir+0x107/0x120 Nov 6 21:57:09 thoregon kernel: [ 9941.104411] [<ffffffff810ff9a4>] do_rmdir+0xe4/0x130 Nov 6 21:57:09 thoregon kernel: [ 9941.104413] [<ffffffff81101c01>] sys_rmdir+0x11/0x20 Nov 6 21:57:09 thoregon kernel: [ 9941.104415] [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b Nov 6 21:57:09 thoregon kernel: [ 9941.104417] irq event stamp: 18505 Nov 6 21:57:09 thoregon kernel: [ 9941.104418] hardirqs last enabled at (18505): [<ffffffff816aec5d>] mutex_trylock+0xfd/0x170 Nov 6 21:57:09 thoregon kernel: [ 9941.104421] hardirqs last disabled at (18504): [<ffffffff816aeb9e>] mutex_trylock+0x3e/0x170 Nov 6 21:57:09 thoregon kernel: [ 9941.104423] softirqs last enabled at (18492): [<ffffffff81042fb1>] __do_softirq+0x111/0x170 Nov 6 21:57:09 thoregon kernel: [ 9941.104426] softirqs last disabled at (18477): [<ffffffff816b3e3c>] call_softirq+0x1c/0x30 Nov 6 21:57:09 thoregon kernel: [ 9941.104428] Nov 6 21:57:09 thoregon kernel: [ 9941.104428] other info that might help us debug this: Nov 6 21:57:09 thoregon kernel: [ 9941.104429] Possible unsafe locking scenario: Nov 6 21:57:09 thoregon kernel: [ 9941.104429] Nov 6 21:57:09 thoregon kernel: [ 9941.104430] CPU0 Nov 6 21:57:09 thoregon kernel: [ 9941.104431] ---- Nov 6 21:57:09 thoregon kernel: [ 9941.104432] lock(&(&ip->i_lock)->mr_lock); Nov 6 21:57:09 thoregon kernel: [ 9941.104433] <Interrupt> Nov 6 21:57:09 thoregon kernel: [ 9941.104434] lock(&(&ip->i_lock)->mr_lock); Nov 6 21:57:09 thoregon kernel: [ 9941.104435] Nov 6 21:57:09 thoregon kernel: [ 9941.104435] *** DEADLOCK *** Nov 6 21:57:09 thoregon kernel: [ 9941.104435] Nov 6 21:57:09 thoregon kernel: [ 9941.104437] 3 locks held by kswapd0/725: Nov 6 21:57:09 thoregon kernel: [ 9941.104438] #0: (shrinker_rwsem){++++..}, at: [<ffffffff810bbd22>] shrink_slab+0x32/0x1f0 Nov 6 21:57:09 thoregon kernel: [ 9941.104442] #1: (&type->s_umount_key#20){++++.+}, at: [<ffffffff810f5a8e>] grab_super_passive+0x3e/0x90 Nov 6 21:57:09 thoregon kernel: [ 9941.104446] #2: (&pag->pag_ici_reclaim_lock){+.+...}, at: [<ffffffff811f28ac>] xfs_reclaim_inodes_ag+0xbc/0x4f0 Nov 6 21:57:09 thoregon kernel: [ 9941.104449] Nov 6 21:57:09 thoregon kernel: [ 9941.104449] stack backtrace: Nov 6 21:57:09 thoregon kernel: [ 9941.104451] Pid: 725, comm: kswapd0 Not tainted 3.7.0-rc4 #1 Nov 6 21:57:09 thoregon kernel: [ 9941.104452] Call Trace: Nov 6 21:57:09 thoregon kernel: [ 9941.104456] [<ffffffff816a829c>] print_usage_bug+0x1f5/0x206 Nov 6 21:57:09 thoregon kernel: [ 9941.104460] [<ffffffff8100ed8a>] ? save_stack_trace+0x2a/0x50 Nov 6 21:57:09 thoregon kernel: [ 9941.104462] [<ffffffff8107e9fd>] mark_lock+0x28d/0x2f0 Nov 6 21:57:09 thoregon kernel: [ 9941.104464] [<ffffffff8107de30>] ? print_irq_inversion_bug.part.37+0x1f0/0x1f0 Nov 6 21:57:09 thoregon kernel: [ 9941.104466] [<ffffffff8107efdf>] __lock_acquire+0x57f/0x1c00 Nov 6 21:57:09 thoregon kernel: [ 9941.104468] [<ffffffff8107c899>] ? __lock_is_held+0x59/0x70 Nov 6 21:57:09 thoregon kernel: [ 9941.104470] [<ffffffff81080b55>] lock_acquire+0x55/0x70 Nov 6 21:57:09 thoregon kernel: [ 9941.104472] [<ffffffff811e8164>] ? xfs_ilock+0x84/0xb0 Nov 6 21:57:09 thoregon kernel: [ 9941.104476] [<ffffffff8106126a>] down_write_nested+0x4a/0x70 Nov 6 21:57:09 thoregon kernel: [ 9941.104477] [<ffffffff811e8164>] ? xfs_ilock+0x84/0xb0 Nov 6 21:57:09 thoregon kernel: [ 9941.104479] [<ffffffff811e8164>] xfs_ilock+0x84/0xb0 Nov 6 21:57:09 thoregon kernel: [ 9941.104481] [<ffffffff811f1ce6>] xfs_reclaim_inode+0x136/0x340 Nov 6 21:57:09 thoregon kernel: [ 9941.104483] [<ffffffff811f2aaf>] xfs_reclaim_inodes_ag+0x2bf/0x4f0 Nov 6 21:57:09 thoregon kernel: [ 9941.104485] [<ffffffff811f28d0>] ? xfs_reclaim_inodes_ag+0xe0/0x4f0 Nov 6 21:57:09 thoregon kernel: [ 9941.104487] [<ffffffff811f2e1e>] xfs_reclaim_inodes_nr+0x2e/0x40 Nov 6 21:57:09 thoregon kernel: [ 9941.104489] [<ffffffff811ef6f0>] xfs_fs_free_cached_objects+0x10/0x20 Nov 6 21:57:09 thoregon kernel: [ 9941.104491] [<ffffffff810f5bf3>] prune_super+0x113/0x1a0 Nov 6 21:57:09 thoregon kernel: [ 9941.104493] [<ffffffff810bbe0e>] shrink_slab+0x11e/0x1f0 Nov 6 21:57:09 thoregon kernel: [ 9941.104496] [<ffffffff810be400>] kswapd+0x690/0xa10 Nov 6 21:57:09 thoregon kernel: [ 9941.104498] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60 Nov 6 21:57:09 thoregon kernel: [ 9941.104500] [<ffffffff810bdd70>] ? shrink_lruvec+0x540/0x540 Nov 6 21:57:09 thoregon kernel: [ 9941.104502] [<ffffffff8105c246>] kthread+0xd6/0xe0 Nov 6 21:57:09 thoregon kernel: [ 9941.104504] [<ffffffff816b1efb>] ? _raw_spin_unlock_irq+0x2b/0x50 Nov 6 21:57:09 thoregon kernel: [ 9941.104506] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 Nov 6 21:57:09 thoregon kernel: [ 9941.104508] [<ffffffff816b2c6c>] ret_from_fork+0x7c/0xb0 Nov 6 21:57:09 thoregon kernel: [ 9941.104510] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 Nov 17 14:07:38 thoregon kernel: [66571.610863] Nov 17 14:07:38 thoregon kernel: [66571.610869] ========================================================= Nov 17 14:07:38 thoregon kernel: [66571.610870] [ INFO: possible irq lock inversion dependency detected ] Nov 17 14:07:38 thoregon kernel: [66571.610873] 3.7.0-rc5 #1 Not tainted Nov 17 14:07:38 thoregon kernel: [66571.610874] --------------------------------------------------------- Nov 17 14:07:38 thoregon kernel: [66571.610875] cc1/21330 just changed the state of lock: Nov 17 14:07:38 thoregon kernel: [66571.610877] (sb_internal){.+.+.?}, at: [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50 Nov 17 14:07:38 thoregon kernel: [66571.610885] but this lock took another, RECLAIM_FS-unsafe lock in the past: Nov 17 14:07:38 thoregon kernel: [66571.610886] (&(&ip->i_lock)->mr_lock/1){+.+.+.} Nov 17 14:07:38 thoregon kernel: [66571.610886] Nov 17 14:07:39 thoregon kernel: [66571.610886] and interrupts could create inverse lock ordering between them. Nov 17 14:07:39 thoregon kernel: [66571.610886] Nov 17 14:07:39 thoregon kernel: [66571.610890] Nov 17 14:07:39 thoregon kernel: [66571.610890] other info that might help us debug this: Nov 17 14:07:39 thoregon kernel: [66571.610891] Possible interrupt unsafe locking scenario: Nov 17 14:07:39 thoregon kernel: [66571.610891] Nov 17 14:07:39 thoregon kernel: [66571.610892] CPU0 CPU1 Nov 17 14:07:39 thoregon kernel: [66571.610893] ---- ---- Nov 17 14:07:39 thoregon kernel: [66571.610894] lock(&(&ip->i_lock)->mr_lock/1); Nov 17 14:07:39 thoregon kernel: [66571.610896] local_irq_disable(); Nov 17 14:07:39 thoregon kernel: [66571.610897] lock(sb_internal); Nov 17 14:07:39 thoregon kernel: [66571.610898] lock(&(&ip->i_lock)->mr_lock/1); Nov 17 14:07:39 thoregon kernel: [66571.610900] <Interrupt> Nov 17 14:07:39 thoregon kernel: [66571.610901] lock(sb_internal); Nov 17 14:07:39 thoregon kernel: [66571.610902] Nov 17 14:07:39 thoregon kernel: [66571.610902] *** DEADLOCK *** Nov 17 14:07:39 thoregon kernel: [66571.610902] Nov 17 14:07:39 thoregon kernel: [66571.610904] 3 locks held by cc1/21330: Nov 17 14:07:39 thoregon kernel: [66571.610905] #0: (&mm->mmap_sem){++++++}, at: [<ffffffff81029d8b>] __do_page_fault+0xfb/0x480 Nov 17 14:07:39 thoregon kernel: [66571.610910] #1: (shrinker_rwsem){++++..}, at: [<ffffffff810bbd02>] shrink_slab+0x32/0x1f0 Nov 17 14:07:39 thoregon kernel: [66571.610915] #2: (&type->s_umount_key#20){++++.+}, at: [<ffffffff810f5a7e>] grab_super_passive+0x3e/0x90 Nov 17 14:07:39 thoregon kernel: [66571.610921] Nov 17 14:07:39 thoregon kernel: [66571.610921] the shortest dependencies between 2nd lock and 1st lock: Nov 17 14:07:39 thoregon kernel: [66571.610927] -> (&(&ip->i_lock)->mr_lock/1){+.+.+.} ops: 169649 { Nov 17 14:07:39 thoregon kernel: [66571.610931] HARDIRQ-ON-W at: Nov 17 14:07:39 thoregon kernel: [66571.610932] [<ffffffff8107f091>] __lock_acquire+0x631/0x1c00 Nov 17 14:07:39 thoregon kernel: [66571.610935] [<ffffffff81080b55>] lock_acquire+0x55/0x70 Nov 17 14:07:39 thoregon kernel: [66571.610937] [<ffffffff8106126a>] down_write_nested+0x4a/0x70 Nov 17 14:07:39 thoregon kernel: [66571.610941] [<ffffffff811e8164>] xfs_ilock+0x84/0xb0 Nov 17 14:07:39 thoregon kernel: [66571.610944] [<ffffffff811f51a4>] xfs_create+0x1d4/0x5a0 Nov 17 14:07:39 thoregon kernel: [66571.610946] [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 Nov 17 14:07:39 thoregon kernel: [66571.610948] [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 Nov 17 14:07:39 thoregon kernel: [66571.610950] [<ffffffff81100322>] vfs_create+0x72/0xc0 Nov 17 14:07:39 thoregon kernel: [66571.610953] [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 Nov 17 14:07:39 thoregon kernel: [66571.610955] [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 Nov 17 14:07:39 thoregon kernel: [66571.610957] [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 Nov 17 14:07:39 thoregon kernel: [66571.610959] [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 Nov 17 14:07:39 thoregon kernel: [66571.610962] [<ffffffff810f222c>] sys_open+0x1c/0x20 Nov 17 14:07:39 thoregon kernel: [66571.610964] [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b Nov 17 14:07:39 thoregon kernel: [66571.610967] SOFTIRQ-ON-W at: Nov 17 14:07:39 thoregon kernel: [66571.610968] [<ffffffff8107f0c7>] __lock_acquire+0x667/0x1c00 Nov 17 14:07:39 thoregon kernel: [66571.610970] [<ffffffff81080b55>] lock_acquire+0x55/0x70 Nov 17 14:07:39 thoregon kernel: [66571.610972] [<ffffffff8106126a>] down_write_nested+0x4a/0x70 Nov 17 14:07:39 thoregon kernel: [66571.610974] [<ffffffff811e8164>] xfs_ilock+0x84/0xb0 Nov 17 14:07:39 thoregon kernel: [66571.610976] [<ffffffff811f51a4>] xfs_create+0x1d4/0x5a0 Nov 17 14:07:39 thoregon kernel: [66571.610977] [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 Nov 17 14:07:39 thoregon kernel: [66571.610979] [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 Nov 17 14:07:39 thoregon kernel: [66571.610981] [<ffffffff81100322>] vfs_create+0x72/0xc0 Nov 17 14:07:39 thoregon kernel: [66571.610983] [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 Nov 17 14:07:39 thoregon kernel: [66571.610985] [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 Nov 17 14:07:39 thoregon kernel: [66571.610987] [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 Nov 17 14:07:39 thoregon kernel: [66571.610989] [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 Nov 17 14:07:39 thoregon kernel: [66571.610991] [<ffffffff810f222c>] sys_open+0x1c/0x20 Nov 17 14:07:39 thoregon kernel: [66571.610993] [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b Nov 17 14:07:39 thoregon kernel: [66571.610995] RECLAIM_FS-ON-W at: Nov 17 14:07:39 thoregon kernel: [66571.610996] [<ffffffff8108137e>] mark_held_locks+0x7e/0x130 Nov 17 14:07:39 thoregon kernel: [66571.610998] [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0 Nov 17 14:07:39 thoregon kernel: [66571.610999] [<ffffffff810e9dc5>] kmem_cache_alloc+0x35/0xe0 Nov 17 14:07:39 thoregon kernel: [66571.611002] [<ffffffff810dba21>] vm_map_ram+0x271/0x770 Nov 17 14:07:39 thoregon kernel: [66571.611004] [<ffffffff811e12b6>] _xfs_buf_map_pages+0x46/0xe0 Nov 17 14:07:39 thoregon kernel: [66571.611008] [<ffffffff811e21ca>] xfs_buf_get_map+0x8a/0x130 Nov 17 14:07:39 thoregon kernel: [66571.611009] [<ffffffff81233989>] xfs_trans_get_buf_map+0xa9/0xd0 Nov 17 14:07:39 thoregon kernel: [66571.611011] [<ffffffff8121bc2d>] xfs_ialloc_inode_init+0xcd/0x1d0 Nov 17 14:07:39 thoregon kernel: [66571.611015] [<ffffffff8121c16f>] xfs_ialloc_ag_alloc+0x18f/0x500 Nov 17 14:07:39 thoregon kernel: [66571.611017] [<ffffffff8121d955>] xfs_dialloc+0x185/0x2a0 Nov 17 14:07:39 thoregon kernel: [66571.611019] [<ffffffff8121f068>] xfs_ialloc+0x58/0x650 Nov 17 14:07:39 thoregon kernel: [66571.611021] [<ffffffff811f3995>] xfs_dir_ialloc+0x65/0x270 Nov 17 14:07:39 thoregon kernel: [66571.611023] [<ffffffff811f537c>] xfs_create+0x3ac/0x5a0 Nov 17 14:07:39 thoregon kernel: [66571.611024] [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 Nov 17 14:07:39 thoregon kernel: [66571.611026] [<ffffffff811ecb61>] xfs_vn_mkdir+0x11/0x20 Nov 17 14:07:39 thoregon kernel: [66571.611028] [<ffffffff8110016f>] vfs_mkdir+0x7f/0xd0 Nov 17 14:07:39 thoregon kernel: [66571.611030] [<ffffffff81101b83>] sys_mkdirat+0x43/0x80 Nov 17 14:07:39 thoregon kernel: [66571.611032] [<ffffffff81101bd4>] sys_mkdir+0x14/0x20 Nov 17 14:07:39 thoregon kernel: [66571.611034] [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b Nov 17 14:07:39 thoregon kernel: [66571.611036] INITIAL USE at: Nov 17 14:07:39 thoregon kernel: [66571.611037] [<ffffffff8107ed49>] __lock_acquire+0x2e9/0x1c00 Nov 17 14:07:39 thoregon kernel: [66571.611038] [<ffffffff81080b55>] lock_acquire+0x55/0x70 Nov 17 14:07:39 thoregon kernel: [66571.611040] [<ffffffff8106126a>] down_write_nested+0x4a/0x70 Nov 17 14:07:39 thoregon kernel: [66571.611042] [<ffffffff811e8164>] xfs_ilock+0x84/0xb0 Nov 17 14:07:39 thoregon kernel: [66571.611044] [<ffffffff811f51a4>] xfs_create+0x1d4/0x5a0 Nov 17 14:07:39 thoregon kernel: [66571.611046] [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 Nov 17 14:07:39 thoregon kernel: [66571.611047] [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 Nov 17 14:07:39 thoregon kernel: [66571.611049] [<ffffffff81100322>] vfs_create+0x72/0xc0 Nov 17 14:07:39 thoregon kernel: [66571.611051] [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 Nov 17 14:07:39 thoregon kernel: [66571.611053] [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 Nov 17 14:07:39 thoregon kernel: [66571.611055] [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 Nov 17 14:07:39 thoregon kernel: [66571.611057] [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 Nov 17 14:07:39 thoregon kernel: [66571.611059] [<ffffffff810f222c>] sys_open+0x1c/0x20 Nov 17 14:07:39 thoregon kernel: [66571.611061] [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b Nov 17 14:07:39 thoregon kernel: [66571.611063] } Nov 17 14:07:39 thoregon kernel: [66571.611064] ... key at: [<ffffffff825b4b81>] __key.41357+0x1/0x8 Nov 17 14:07:39 thoregon kernel: [66571.611066] ... acquired at: Nov 17 14:07:39 thoregon kernel: [66571.611067] [<ffffffff81080b55>] lock_acquire+0x55/0x70 Nov 17 14:07:39 thoregon kernel: [66571.611069] [<ffffffff8106126a>] down_write_nested+0x4a/0x70 Nov 17 14:07:39 thoregon kernel: [66571.611071] [<ffffffff811e8164>] xfs_ilock+0x84/0xb0 Nov 17 14:07:39 thoregon kernel: [66571.611073] [<ffffffff811f51a4>] xfs_create+0x1d4/0x5a0 Nov 17 14:07:39 thoregon kernel: [66571.611074] [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 Nov 17 14:07:39 thoregon kernel: [66571.611076] [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 Nov 17 14:07:39 thoregon kernel: [66571.611078] [<ffffffff81100322>] vfs_create+0x72/0xc0 Nov 17 14:07:39 thoregon kernel: [66571.611080] [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 Nov 17 14:07:39 thoregon kernel: [66571.611082] [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 Nov 17 14:07:39 thoregon kernel: [66571.611084] [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 Nov 17 14:07:39 thoregon kernel: [66571.611086] [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 Nov 17 14:07:39 thoregon kernel: [66571.611088] [<ffffffff810f222c>] sys_open+0x1c/0x20 Nov 17 14:07:39 thoregon kernel: [66571.611090] [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b Nov 17 14:07:39 thoregon kernel: [66571.611091] Nov 17 14:07:39 thoregon kernel: [66571.611092] -> (sb_internal){.+.+.?} ops: 1341531 { Nov 17 14:07:39 thoregon kernel: [66571.611095] HARDIRQ-ON-R at: Nov 17 14:07:39 thoregon kernel: [66571.611096] [<ffffffff8107ef6a>] __lock_acquire+0x50a/0x1c00 Nov 17 14:07:39 thoregon kernel: [66571.611098] [<ffffffff81080b55>] lock_acquire+0x55/0x70 Nov 17 14:07:39 thoregon kernel: [66571.611100] [<ffffffff810f451b>] __sb_start_write+0xab/0x190 Nov 17 14:07:39 thoregon kernel: [66571.611102] [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50 Nov 17 14:07:39 thoregon kernel: [66571.611104] [<ffffffff811f5157>] xfs_create+0x187/0x5a0 Nov 17 14:07:39 thoregon kernel: [66571.611105] [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 Nov 17 14:07:39 thoregon kernel: [66571.611107] [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 Nov 17 14:07:39 thoregon kernel: [66571.611109] [<ffffffff81100322>] vfs_create+0x72/0xc0 Nov 17 14:07:39 thoregon kernel: [66571.611111] [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 Nov 17 14:07:39 thoregon kernel: [66571.611113] [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 Nov 17 14:07:39 thoregon kernel: [66571.611115] [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 Nov 17 14:07:39 thoregon kernel: [66571.611117] [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 Nov 17 14:07:39 thoregon kernel: [66571.611119] [<ffffffff810f222c>] sys_open+0x1c/0x20 Nov 17 14:07:39 thoregon kernel: [66571.611121] [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b Nov 17 14:07:39 thoregon kernel: [66571.611123] SOFTIRQ-ON-R at: Nov 17 14:07:39 thoregon kernel: [66571.611124] [<ffffffff8107f0c7>] __lock_acquire+0x667/0x1c00 Nov 17 14:07:39 thoregon kernel: [66571.611126] [<ffffffff81080b55>] lock_acquire+0x55/0x70 Nov 17 14:07:39 thoregon kernel: [66571.611128] [<ffffffff810f451b>] __sb_start_write+0xab/0x190 Nov 17 14:07:39 thoregon kernel: [66571.611130] [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50 Nov 17 14:07:39 thoregon kernel: [66571.611132] [<ffffffff811f5157>] xfs_create+0x187/0x5a0 Nov 17 14:07:39 thoregon kernel: [66571.611133] [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 Nov 17 14:07:39 thoregon kernel: [66571.611135] [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 Nov 17 14:07:39 thoregon kernel: [66571.611137] [<ffffffff81100322>] vfs_create+0x72/0xc0 Nov 17 14:07:39 thoregon kernel: [66571.611139] [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 Nov 17 14:07:39 thoregon kernel: [66571.611141] [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 Nov 17 14:07:39 thoregon kernel: [66571.611143] [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 Nov 17 14:07:39 thoregon kernel: [66571.611145] [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 Nov 17 14:07:39 thoregon kernel: [66571.611147] [<ffffffff810f222c>] sys_open+0x1c/0x20 Nov 17 14:07:39 thoregon kernel: [66571.611149] [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b Nov 17 14:07:39 thoregon kernel: [66571.611151] IN-RECLAIM_FS-R at: Nov 17 14:07:39 thoregon kernel: [66571.611152] [<ffffffff8107efdf>] __lock_acquire+0x57f/0x1c00 Nov 17 14:07:39 thoregon kernel: [66571.611154] [<ffffffff81080b55>] lock_acquire+0x55/0x70 Nov 17 14:07:39 thoregon kernel: [66571.611156] [<ffffffff810f451b>] __sb_start_write+0xab/0x190 Nov 17 14:07:39 thoregon kernel: [66571.611158] [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50 Nov 17 14:07:39 thoregon kernel: [66571.611159] [<ffffffff811f3e64>] xfs_free_eofblocks+0x104/0x250 Nov 17 14:07:39 thoregon kernel: [66571.611161] [<ffffffff811f4b39>] xfs_inactive+0xa9/0x480 Nov 17 14:07:39 thoregon kernel: [66571.611163] [<ffffffff811efe10>] xfs_fs_evict_inode+0x70/0x80 Nov 17 14:07:39 thoregon kernel: [66571.611165] [<ffffffff8110cb7f>] evict+0xaf/0x1b0 Nov 17 14:07:39 thoregon kernel: [66571.611168] [<ffffffff8110d209>] dispose_list+0x39/0x50 Nov 17 14:07:39 thoregon kernel: [66571.611170] [<ffffffff8110dc63>] prune_icache_sb+0x183/0x340 Nov 17 14:07:39 thoregon kernel: [66571.611172] [<ffffffff810f5bc3>] prune_super+0xf3/0x1a0 Nov 17 14:07:39 thoregon kernel: [66571.611173] [<ffffffff810bbdee>] shrink_slab+0x11e/0x1f0 Nov 17 14:07:39 thoregon kernel: [66571.611175] [<ffffffff810be98f>] try_to_free_pages+0x21f/0x4e0 Nov 17 14:07:39 thoregon kernel: [66571.611177] [<ffffffff810b5ec6>] __alloc_pages_nodemask+0x506/0x800 Nov 17 14:07:39 thoregon kernel: [66571.611179] [<ffffffff810ce56e>] handle_pte_fault+0x5ae/0x7a0 Nov 17 14:07:39 thoregon kernel: [66571.611182] [<ffffffff810cf769>] handle_mm_fault+0x1f9/0x2a0 Nov 17 14:07:39 thoregon kernel: [66571.611184] [<ffffffff81029dfc>] __do_page_fault+0x16c/0x480 Nov 17 14:07:39 thoregon kernel: [66571.611186] [<ffffffff8102a139>] do_page_fault+0x9/0x10 Nov 17 14:07:39 thoregon kernel: [66571.611188] [<ffffffff816b287f>] page_fault+0x1f/0x30 Nov 17 14:07:39 thoregon kernel: [66571.611190] RECLAIM_FS-ON-R at: Nov 17 14:07:39 thoregon kernel: [66571.611191] [<ffffffff8108137e>] mark_held_locks+0x7e/0x130 Nov 17 14:07:39 thoregon kernel: [66571.611193] [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0 Nov 17 14:07:39 thoregon kernel: [66571.611195] [<ffffffff810e9dc5>] kmem_cache_alloc+0x35/0xe0 Nov 17 14:07:39 thoregon kernel: [66571.611197] [<ffffffff811f6d4f>] kmem_zone_alloc+0x5f/0xe0 Nov 17 14:07:39 thoregon kernel: [66571.611198] [<ffffffff811f6de8>] kmem_zone_zalloc+0x18/0x50 Nov 17 14:07:39 thoregon kernel: [66571.611200] [<ffffffff8122b0b2>] _xfs_trans_alloc+0x32/0x90 Nov 17 14:07:39 thoregon kernel: [66571.611202] [<ffffffff8122b148>] xfs_trans_alloc+0x38/0x50 Nov 17 14:07:39 thoregon kernel: [66571.611204] [<ffffffff811f5157>] xfs_create+0x187/0x5a0 Nov 17 14:07:39 thoregon kernel: [66571.611205] [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 Nov 17 14:07:39 thoregon kernel: [66571.611207] [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 Nov 17 14:07:39 thoregon kernel: [66571.611209] [<ffffffff81100322>] vfs_create+0x72/0xc0 Nov 17 14:07:39 thoregon kernel: [66571.611211] [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 Nov 17 14:07:39 thoregon kernel: [66571.611213] [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 Nov 17 14:07:39 thoregon kernel: [66571.611215] [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 Nov 17 14:07:39 thoregon kernel: [66571.611217] [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 Nov 17 14:07:39 thoregon kernel: [66571.611219] [<ffffffff810f222c>] sys_open+0x1c/0x20 Nov 17 14:07:39 thoregon kernel: [66571.611221] [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b Nov 17 14:07:39 thoregon kernel: [66571.611223] INITIAL USE at: Nov 17 14:07:39 thoregon kernel: [66571.611224] [<ffffffff8107ed49>] __lock_acquire+0x2e9/0x1c00 Nov 17 14:07:39 thoregon kernel: [66571.611225] [<ffffffff81080b55>] lock_acquire+0x55/0x70 Nov 17 14:07:39 thoregon kernel: [66571.611227] [<ffffffff810f451b>] __sb_start_write+0xab/0x190 Nov 17 14:07:39 thoregon kernel: [66571.611229] [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50 Nov 17 14:07:39 thoregon kernel: [66571.611231] [<ffffffff811f5157>] xfs_create+0x187/0x5a0 Nov 17 14:07:39 thoregon kernel: [66571.611232] [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 Nov 17 14:07:39 thoregon kernel: [66571.611234] [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 Nov 17 14:07:39 thoregon kernel: [66571.611236] [<ffffffff81100322>] vfs_create+0x72/0xc0 Nov 17 14:07:39 thoregon kernel: [66571.611238] [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 Nov 17 14:07:39 thoregon kernel: [66571.611240] [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 Nov 17 14:07:39 thoregon kernel: [66571.611242] [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 Nov 17 14:07:39 thoregon kernel: [66571.611244] [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 Nov 17 14:07:39 thoregon kernel: [66571.611246] [<ffffffff810f222c>] sys_open+0x1c/0x20 Nov 17 14:07:39 thoregon kernel: [66571.611248] [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b Nov 17 14:07:39 thoregon kernel: [66571.611250] } Nov 17 14:07:39 thoregon kernel: [66571.611251] ... key at: [<ffffffff81c34e40>] xfs_fs_type+0x60/0x80 Nov 17 14:07:39 thoregon kernel: [66571.611254] ... acquired at: Nov 17 14:07:39 thoregon kernel: [66571.611254] [<ffffffff8107df3b>] check_usage_forwards+0x10b/0x140 Nov 17 14:07:39 thoregon kernel: [66571.611256] [<ffffffff8107e900>] mark_lock+0x190/0x2f0 Nov 17 14:07:39 thoregon kernel: [66571.611258] [<ffffffff8107efdf>] __lock_acquire+0x57f/0x1c00 Nov 17 14:07:39 thoregon kernel: [66571.611260] [<ffffffff81080b55>] lock_acquire+0x55/0x70 Nov 17 14:07:39 thoregon kernel: [66571.611261] [<ffffffff810f451b>] __sb_start_write+0xab/0x190 Nov 17 14:07:39 thoregon kernel: [66571.611263] [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50 Nov 17 14:07:39 thoregon kernel: [66571.611265] [<ffffffff811f3e64>] xfs_free_eofblocks+0x104/0x250 Nov 17 14:07:39 thoregon kernel: [66571.611266] [<ffffffff811f4b39>] xfs_inactive+0xa9/0x480 Nov 17 14:07:39 thoregon kernel: [66571.611268] [<ffffffff811efe10>] xfs_fs_evict_inode+0x70/0x80 Nov 17 14:07:39 thoregon kernel: [66571.611270] [<ffffffff8110cb7f>] evict+0xaf/0x1b0 Nov 17 14:07:39 thoregon kernel: [66571.611271] [<ffffffff8110d209>] dispose_list+0x39/0x50 Nov 17 14:07:39 thoregon kernel: [66571.611273] [<ffffffff8110dc63>] prune_icache_sb+0x183/0x340 Nov 17 14:07:39 thoregon kernel: [66571.611275] [<ffffffff810f5bc3>] prune_super+0xf3/0x1a0 Nov 17 14:07:39 thoregon kernel: [66571.611277] [<ffffffff810bbdee>] shrink_slab+0x11e/0x1f0 Nov 17 14:07:39 thoregon kernel: [66571.611278] [<ffffffff810be98f>] try_to_free_pages+0x21f/0x4e0 Nov 17 14:07:39 thoregon kernel: [66571.611280] [<ffffffff810b5ec6>] __alloc_pages_nodemask+0x506/0x800 Nov 17 14:07:39 thoregon kernel: [66571.611282] [<ffffffff810ce56e>] handle_pte_fault+0x5ae/0x7a0 Nov 17 14:07:39 thoregon kernel: [66571.611284] [<ffffffff810cf769>] handle_mm_fault+0x1f9/0x2a0 Nov 17 14:07:39 thoregon kernel: [66571.611286] [<ffffffff81029dfc>] __do_page_fault+0x16c/0x480 Nov 17 14:07:39 thoregon kernel: [66571.611287] [<ffffffff8102a139>] do_page_fault+0x9/0x10 Nov 17 14:07:39 thoregon kernel: [66571.611289] [<ffffffff816b287f>] page_fault+0x1f/0x30 Nov 17 14:07:39 thoregon kernel: [66571.611291] Nov 17 14:07:39 thoregon kernel: [66571.611292] Nov 17 14:07:39 thoregon kernel: [66571.611292] stack backtrace: Nov 17 14:07:39 thoregon kernel: [66571.611294] Pid: 21330, comm: cc1 Not tainted 3.7.0-rc5 #1 Nov 17 14:07:39 thoregon kernel: [66571.611295] Call Trace: Nov 17 14:07:39 thoregon kernel: [66571.611298] [<ffffffff8107de28>] print_irq_inversion_bug.part.37+0x1e8/0x1f0 Nov 17 14:07:39 thoregon kernel: [66571.611300] [<ffffffff8107df3b>] check_usage_forwards+0x10b/0x140 Nov 17 14:07:39 thoregon kernel: [66571.611303] [<ffffffff8107e900>] mark_lock+0x190/0x2f0 Nov 17 14:07:39 thoregon kernel: [66571.611306] [<ffffffff8150406e>] ? dm_request+0x2e/0x2a0 Nov 17 14:07:39 thoregon kernel: [66571.611308] [<ffffffff8107de30>] ? print_irq_inversion_bug.part.37+0x1f0/0x1f0 Nov 17 14:07:39 thoregon kernel: [66571.611310] [<ffffffff8107efdf>] __lock_acquire+0x57f/0x1c00 Nov 17 14:07:39 thoregon kernel: [66571.611313] [<ffffffff812202f4>] ? xfs_iext_bno_to_ext+0x84/0x160 Nov 17 14:07:39 thoregon kernel: [66571.611316] [<ffffffff8120a023>] ? xfs_bmbt_get_all+0x13/0x20 Nov 17 14:07:39 thoregon kernel: [66571.611318] [<ffffffff81200fb4>] ? xfs_bmap_search_multi_extents+0xa4/0x110 Nov 17 14:07:39 thoregon kernel: [66571.611320] [<ffffffff81080b55>] lock_acquire+0x55/0x70 Nov 17 14:07:39 thoregon kernel: [66571.611322] [<ffffffff8122b138>] ? xfs_trans_alloc+0x28/0x50 Nov 17 14:07:39 thoregon kernel: [66571.611324] [<ffffffff810f451b>] __sb_start_write+0xab/0x190 Nov 17 14:07:39 thoregon kernel: [66571.611326] [<ffffffff8122b138>] ? xfs_trans_alloc+0x28/0x50 Nov 17 14:07:39 thoregon kernel: [66571.611328] [<ffffffff8122b138>] ? xfs_trans_alloc+0x28/0x50 Nov 17 14:07:39 thoregon kernel: [66571.611330] [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50 Nov 17 14:07:39 thoregon kernel: [66571.611332] [<ffffffff811f3e64>] xfs_free_eofblocks+0x104/0x250 Nov 17 14:07:39 thoregon kernel: [66571.611334] [<ffffffff816b204b>] ? _raw_spin_unlock_irq+0x2b/0x50 Nov 17 14:07:39 thoregon kernel: [66571.611336] [<ffffffff811f4b39>] xfs_inactive+0xa9/0x480 Nov 17 14:07:39 thoregon kernel: [66571.611337] [<ffffffff816b204b>] ? _raw_spin_unlock_irq+0x2b/0x50 Nov 17 14:07:39 thoregon kernel: [66571.611340] [<ffffffff811efe10>] xfs_fs_evict_inode+0x70/0x80 Nov 17 14:07:39 thoregon kernel: [66571.611342] [<ffffffff8110cb7f>] evict+0xaf/0x1b0 Nov 17 14:07:39 thoregon kernel: [66571.611344] [<ffffffff8110d209>] dispose_list+0x39/0x50 Nov 17 14:07:39 thoregon kernel: [66571.611346] [<ffffffff8110dc63>] prune_icache_sb+0x183/0x340 Nov 17 14:07:39 thoregon kernel: [66571.611347] [<ffffffff810f5bc3>] prune_super+0xf3/0x1a0 Nov 17 14:07:39 thoregon kernel: [66571.611349] [<ffffffff810bbdee>] shrink_slab+0x11e/0x1f0 Nov 17 14:07:39 thoregon kernel: [66571.611352] [<ffffffff810be98f>] try_to_free_pages+0x21f/0x4e0 Nov 17 14:07:39 thoregon kernel: [66571.611354] [<ffffffff810b5ec6>] __alloc_pages_nodemask+0x506/0x800 Nov 17 14:07:39 thoregon kernel: [66571.611356] [<ffffffff810b9e40>] ? lru_deactivate_fn+0x1c0/0x1c0 Nov 17 14:07:39 thoregon kernel: [66571.611358] [<ffffffff810ce56e>] handle_pte_fault+0x5ae/0x7a0 Nov 17 14:07:39 thoregon kernel: [66571.611360] [<ffffffff810cf769>] handle_mm_fault+0x1f9/0x2a0 Nov 17 14:07:39 thoregon kernel: [66571.611363] [<ffffffff81029dfc>] __do_page_fault+0x16c/0x480 Nov 17 14:07:39 thoregon kernel: [66571.611366] [<ffffffff8129c7ad>] ? trace_hardirqs_off_thunk+0x3a/0x3c Nov 17 14:07:39 thoregon kernel: [66571.611368] [<ffffffff8102a139>] do_page_fault+0x9/0x10 Nov 17 14:07:39 thoregon kernel: [66571.611370] [<ffffffff816b287f>] page_fault+0x1f/0x30 ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Hang in XFS reclaim on 3.7.0-rc3 @ 2012-11-18 10:24 ` Torsten Kaiser 0 siblings, 0 replies; 31+ messages in thread From: Torsten Kaiser @ 2012-11-18 10:24 UTC (permalink / raw) To: Dave Chinner; +Cc: Linux Kernel, xfs On Tue, Oct 30, 2012 at 9:37 PM, Torsten Kaiser <just.for.lkml@googlemail.com> wrote: > I will keep LOCKDEP enabled on that system, and if there really is > another splat, I will report back here. But I rather doubt that this > will be needed. After the patch, I did not see this problem again, but today I found another LOCKDEP report that also looks XFS related. I found it twice in the logs, and as both were slightly different, I will attach both versions. Nov 6 21:57:09 thoregon kernel: [ 9941.104345] Nov 6 21:57:09 thoregon kernel: [ 9941.104350] ================================= Nov 6 21:57:09 thoregon kernel: [ 9941.104351] [ INFO: inconsistent lock state ] Nov 6 21:57:09 thoregon kernel: [ 9941.104353] 3.7.0-rc4 #1 Not tainted Nov 6 21:57:09 thoregon kernel: [ 9941.104354] --------------------------------- Nov 6 21:57:09 thoregon kernel: [ 9941.104355] inconsistent {RECLAIM_FS-ON-W} -> {IN-RECLAIM_FS-W} usage. Nov 6 21:57:09 thoregon kernel: [ 9941.104357] kswapd0/725 [HC0[0]:SC0[0]:HE1:SE1] takes: Nov 6 21:57:09 thoregon kernel: [ 9941.104359] (&(&ip->i_lock)->mr_lock){++++?.}, at: [<ffffffff811e8164>] xfs_ilock+0x84/0xb0 Nov 6 21:57:09 thoregon kernel: [ 9941.104366] {RECLAIM_FS-ON-W} state was registered at: Nov 6 21:57:09 thoregon kernel: [ 9941.104367] [<ffffffff8108137e>] mark_held_locks+0x7e/0x130 Nov 6 21:57:09 thoregon kernel: [ 9941.104371] [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0 Nov 6 21:57:09 thoregon kernel: [ 9941.104373] [<ffffffff810b5a55>] __alloc_pages_nodemask+0x75/0x800 Nov 6 21:57:09 thoregon kernel: [ 9941.104375] [<ffffffff810b6262>] __get_free_pages+0x12/0x40 Nov 6 21:57:09 thoregon kernel: [ 9941.104377] [<ffffffff8102d7f0>] pte_alloc_one_kernel+0x10/0x20 Nov 6 21:57:09 thoregon kernel: [ 9941.104380] [<ffffffff810cc3e6>] __pte_alloc_kernel+0x16/0x90 Nov 6 21:57:09 thoregon kernel: [ 9941.104382] [<ffffffff810d9f37>] vmap_page_range_noflush+0x287/0x320 Nov 6 21:57:09 thoregon kernel: [ 9941.104385] [<ffffffff810dbe54>] vm_map_ram+0x694/0x770 Nov 6 21:57:09 thoregon kernel: [ 9941.104386] [<ffffffff811e1316>] _xfs_buf_map_pages+0x46/0xe0 Nov 6 21:57:09 thoregon kernel: [ 9941.104389] [<ffffffff811e222a>] xfs_buf_get_map+0x8a/0x130 Nov 6 21:57:09 thoregon kernel: [ 9941.104391] [<ffffffff81233ab9>] xfs_trans_get_buf_map+0xa9/0xd0 Nov 6 21:57:09 thoregon kernel: [ 9941.104393] [<ffffffff8121e5a9>] xfs_ifree_cluster+0x129/0x670 Nov 6 21:57:09 thoregon kernel: [ 9941.104396] [<ffffffff8121fbc9>] xfs_ifree+0xe9/0xf0 Nov 6 21:57:09 thoregon kernel: [ 9941.104398] [<ffffffff811f4d2f>] xfs_inactive+0x2af/0x480 Nov 6 21:57:09 thoregon kernel: [ 9941.104400] [<ffffffff811efe00>] xfs_fs_evict_inode+0x70/0x80 Nov 6 21:57:09 thoregon kernel: [ 9941.104402] [<ffffffff8110cb8f>] evict+0xaf/0x1b0 Nov 6 21:57:09 thoregon kernel: [ 9941.104405] [<ffffffff8110cd95>] iput+0x105/0x210 Nov 6 21:57:09 thoregon kernel: [ 9941.104406] [<ffffffff81107ba0>] d_delete+0x150/0x190 Nov 6 21:57:09 thoregon kernel: [ 9941.104408] [<ffffffff810ff8a7>] vfs_rmdir+0x107/0x120 Nov 6 21:57:09 thoregon kernel: [ 9941.104411] [<ffffffff810ff9a4>] do_rmdir+0xe4/0x130 Nov 6 21:57:09 thoregon kernel: [ 9941.104413] [<ffffffff81101c01>] sys_rmdir+0x11/0x20 Nov 6 21:57:09 thoregon kernel: [ 9941.104415] [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b Nov 6 21:57:09 thoregon kernel: [ 9941.104417] irq event stamp: 18505 Nov 6 21:57:09 thoregon kernel: [ 9941.104418] hardirqs last enabled at (18505): [<ffffffff816aec5d>] mutex_trylock+0xfd/0x170 Nov 6 21:57:09 thoregon kernel: [ 9941.104421] hardirqs last disabled at (18504): [<ffffffff816aeb9e>] mutex_trylock+0x3e/0x170 Nov 6 21:57:09 thoregon kernel: [ 9941.104423] softirqs last enabled at (18492): [<ffffffff81042fb1>] __do_softirq+0x111/0x170 Nov 6 21:57:09 thoregon kernel: [ 9941.104426] softirqs last disabled at (18477): [<ffffffff816b3e3c>] call_softirq+0x1c/0x30 Nov 6 21:57:09 thoregon kernel: [ 9941.104428] Nov 6 21:57:09 thoregon kernel: [ 9941.104428] other info that might help us debug this: Nov 6 21:57:09 thoregon kernel: [ 9941.104429] Possible unsafe locking scenario: Nov 6 21:57:09 thoregon kernel: [ 9941.104429] Nov 6 21:57:09 thoregon kernel: [ 9941.104430] CPU0 Nov 6 21:57:09 thoregon kernel: [ 9941.104431] ---- Nov 6 21:57:09 thoregon kernel: [ 9941.104432] lock(&(&ip->i_lock)->mr_lock); Nov 6 21:57:09 thoregon kernel: [ 9941.104433] <Interrupt> Nov 6 21:57:09 thoregon kernel: [ 9941.104434] lock(&(&ip->i_lock)->mr_lock); Nov 6 21:57:09 thoregon kernel: [ 9941.104435] Nov 6 21:57:09 thoregon kernel: [ 9941.104435] *** DEADLOCK *** Nov 6 21:57:09 thoregon kernel: [ 9941.104435] Nov 6 21:57:09 thoregon kernel: [ 9941.104437] 3 locks held by kswapd0/725: Nov 6 21:57:09 thoregon kernel: [ 9941.104438] #0: (shrinker_rwsem){++++..}, at: [<ffffffff810bbd22>] shrink_slab+0x32/0x1f0 Nov 6 21:57:09 thoregon kernel: [ 9941.104442] #1: (&type->s_umount_key#20){++++.+}, at: [<ffffffff810f5a8e>] grab_super_passive+0x3e/0x90 Nov 6 21:57:09 thoregon kernel: [ 9941.104446] #2: (&pag->pag_ici_reclaim_lock){+.+...}, at: [<ffffffff811f28ac>] xfs_reclaim_inodes_ag+0xbc/0x4f0 Nov 6 21:57:09 thoregon kernel: [ 9941.104449] Nov 6 21:57:09 thoregon kernel: [ 9941.104449] stack backtrace: Nov 6 21:57:09 thoregon kernel: [ 9941.104451] Pid: 725, comm: kswapd0 Not tainted 3.7.0-rc4 #1 Nov 6 21:57:09 thoregon kernel: [ 9941.104452] Call Trace: Nov 6 21:57:09 thoregon kernel: [ 9941.104456] [<ffffffff816a829c>] print_usage_bug+0x1f5/0x206 Nov 6 21:57:09 thoregon kernel: [ 9941.104460] [<ffffffff8100ed8a>] ? save_stack_trace+0x2a/0x50 Nov 6 21:57:09 thoregon kernel: [ 9941.104462] [<ffffffff8107e9fd>] mark_lock+0x28d/0x2f0 Nov 6 21:57:09 thoregon kernel: [ 9941.104464] [<ffffffff8107de30>] ? print_irq_inversion_bug.part.37+0x1f0/0x1f0 Nov 6 21:57:09 thoregon kernel: [ 9941.104466] [<ffffffff8107efdf>] __lock_acquire+0x57f/0x1c00 Nov 6 21:57:09 thoregon kernel: [ 9941.104468] [<ffffffff8107c899>] ? __lock_is_held+0x59/0x70 Nov 6 21:57:09 thoregon kernel: [ 9941.104470] [<ffffffff81080b55>] lock_acquire+0x55/0x70 Nov 6 21:57:09 thoregon kernel: [ 9941.104472] [<ffffffff811e8164>] ? xfs_ilock+0x84/0xb0 Nov 6 21:57:09 thoregon kernel: [ 9941.104476] [<ffffffff8106126a>] down_write_nested+0x4a/0x70 Nov 6 21:57:09 thoregon kernel: [ 9941.104477] [<ffffffff811e8164>] ? xfs_ilock+0x84/0xb0 Nov 6 21:57:09 thoregon kernel: [ 9941.104479] [<ffffffff811e8164>] xfs_ilock+0x84/0xb0 Nov 6 21:57:09 thoregon kernel: [ 9941.104481] [<ffffffff811f1ce6>] xfs_reclaim_inode+0x136/0x340 Nov 6 21:57:09 thoregon kernel: [ 9941.104483] [<ffffffff811f2aaf>] xfs_reclaim_inodes_ag+0x2bf/0x4f0 Nov 6 21:57:09 thoregon kernel: [ 9941.104485] [<ffffffff811f28d0>] ? xfs_reclaim_inodes_ag+0xe0/0x4f0 Nov 6 21:57:09 thoregon kernel: [ 9941.104487] [<ffffffff811f2e1e>] xfs_reclaim_inodes_nr+0x2e/0x40 Nov 6 21:57:09 thoregon kernel: [ 9941.104489] [<ffffffff811ef6f0>] xfs_fs_free_cached_objects+0x10/0x20 Nov 6 21:57:09 thoregon kernel: [ 9941.104491] [<ffffffff810f5bf3>] prune_super+0x113/0x1a0 Nov 6 21:57:09 thoregon kernel: [ 9941.104493] [<ffffffff810bbe0e>] shrink_slab+0x11e/0x1f0 Nov 6 21:57:09 thoregon kernel: [ 9941.104496] [<ffffffff810be400>] kswapd+0x690/0xa10 Nov 6 21:57:09 thoregon kernel: [ 9941.104498] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60 Nov 6 21:57:09 thoregon kernel: [ 9941.104500] [<ffffffff810bdd70>] ? shrink_lruvec+0x540/0x540 Nov 6 21:57:09 thoregon kernel: [ 9941.104502] [<ffffffff8105c246>] kthread+0xd6/0xe0 Nov 6 21:57:09 thoregon kernel: [ 9941.104504] [<ffffffff816b1efb>] ? _raw_spin_unlock_irq+0x2b/0x50 Nov 6 21:57:09 thoregon kernel: [ 9941.104506] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 Nov 6 21:57:09 thoregon kernel: [ 9941.104508] [<ffffffff816b2c6c>] ret_from_fork+0x7c/0xb0 Nov 6 21:57:09 thoregon kernel: [ 9941.104510] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 Nov 17 14:07:38 thoregon kernel: [66571.610863] Nov 17 14:07:38 thoregon kernel: [66571.610869] ========================================================= Nov 17 14:07:38 thoregon kernel: [66571.610870] [ INFO: possible irq lock inversion dependency detected ] Nov 17 14:07:38 thoregon kernel: [66571.610873] 3.7.0-rc5 #1 Not tainted Nov 17 14:07:38 thoregon kernel: [66571.610874] --------------------------------------------------------- Nov 17 14:07:38 thoregon kernel: [66571.610875] cc1/21330 just changed the state of lock: Nov 17 14:07:38 thoregon kernel: [66571.610877] (sb_internal){.+.+.?}, at: [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50 Nov 17 14:07:38 thoregon kernel: [66571.610885] but this lock took another, RECLAIM_FS-unsafe lock in the past: Nov 17 14:07:38 thoregon kernel: [66571.610886] (&(&ip->i_lock)->mr_lock/1){+.+.+.} Nov 17 14:07:38 thoregon kernel: [66571.610886] Nov 17 14:07:39 thoregon kernel: [66571.610886] and interrupts could create inverse lock ordering between them. Nov 17 14:07:39 thoregon kernel: [66571.610886] Nov 17 14:07:39 thoregon kernel: [66571.610890] Nov 17 14:07:39 thoregon kernel: [66571.610890] other info that might help us debug this: Nov 17 14:07:39 thoregon kernel: [66571.610891] Possible interrupt unsafe locking scenario: Nov 17 14:07:39 thoregon kernel: [66571.610891] Nov 17 14:07:39 thoregon kernel: [66571.610892] CPU0 CPU1 Nov 17 14:07:39 thoregon kernel: [66571.610893] ---- ---- Nov 17 14:07:39 thoregon kernel: [66571.610894] lock(&(&ip->i_lock)->mr_lock/1); Nov 17 14:07:39 thoregon kernel: [66571.610896] local_irq_disable(); Nov 17 14:07:39 thoregon kernel: [66571.610897] lock(sb_internal); Nov 17 14:07:39 thoregon kernel: [66571.610898] lock(&(&ip->i_lock)->mr_lock/1); Nov 17 14:07:39 thoregon kernel: [66571.610900] <Interrupt> Nov 17 14:07:39 thoregon kernel: [66571.610901] lock(sb_internal); Nov 17 14:07:39 thoregon kernel: [66571.610902] Nov 17 14:07:39 thoregon kernel: [66571.610902] *** DEADLOCK *** Nov 17 14:07:39 thoregon kernel: [66571.610902] Nov 17 14:07:39 thoregon kernel: [66571.610904] 3 locks held by cc1/21330: Nov 17 14:07:39 thoregon kernel: [66571.610905] #0: (&mm->mmap_sem){++++++}, at: [<ffffffff81029d8b>] __do_page_fault+0xfb/0x480 Nov 17 14:07:39 thoregon kernel: [66571.610910] #1: (shrinker_rwsem){++++..}, at: [<ffffffff810bbd02>] shrink_slab+0x32/0x1f0 Nov 17 14:07:39 thoregon kernel: [66571.610915] #2: (&type->s_umount_key#20){++++.+}, at: [<ffffffff810f5a7e>] grab_super_passive+0x3e/0x90 Nov 17 14:07:39 thoregon kernel: [66571.610921] Nov 17 14:07:39 thoregon kernel: [66571.610921] the shortest dependencies between 2nd lock and 1st lock: Nov 17 14:07:39 thoregon kernel: [66571.610927] -> (&(&ip->i_lock)->mr_lock/1){+.+.+.} ops: 169649 { Nov 17 14:07:39 thoregon kernel: [66571.610931] HARDIRQ-ON-W at: Nov 17 14:07:39 thoregon kernel: [66571.610932] [<ffffffff8107f091>] __lock_acquire+0x631/0x1c00 Nov 17 14:07:39 thoregon kernel: [66571.610935] [<ffffffff81080b55>] lock_acquire+0x55/0x70 Nov 17 14:07:39 thoregon kernel: [66571.610937] [<ffffffff8106126a>] down_write_nested+0x4a/0x70 Nov 17 14:07:39 thoregon kernel: [66571.610941] [<ffffffff811e8164>] xfs_ilock+0x84/0xb0 Nov 17 14:07:39 thoregon kernel: [66571.610944] [<ffffffff811f51a4>] xfs_create+0x1d4/0x5a0 Nov 17 14:07:39 thoregon kernel: [66571.610946] [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 Nov 17 14:07:39 thoregon kernel: [66571.610948] [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 Nov 17 14:07:39 thoregon kernel: [66571.610950] [<ffffffff81100322>] vfs_create+0x72/0xc0 Nov 17 14:07:39 thoregon kernel: [66571.610953] [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 Nov 17 14:07:39 thoregon kernel: [66571.610955] [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 Nov 17 14:07:39 thoregon kernel: [66571.610957] [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 Nov 17 14:07:39 thoregon kernel: [66571.610959] [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 Nov 17 14:07:39 thoregon kernel: [66571.610962] [<ffffffff810f222c>] sys_open+0x1c/0x20 Nov 17 14:07:39 thoregon kernel: [66571.610964] [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b Nov 17 14:07:39 thoregon kernel: [66571.610967] SOFTIRQ-ON-W at: Nov 17 14:07:39 thoregon kernel: [66571.610968] [<ffffffff8107f0c7>] __lock_acquire+0x667/0x1c00 Nov 17 14:07:39 thoregon kernel: [66571.610970] [<ffffffff81080b55>] lock_acquire+0x55/0x70 Nov 17 14:07:39 thoregon kernel: [66571.610972] [<ffffffff8106126a>] down_write_nested+0x4a/0x70 Nov 17 14:07:39 thoregon kernel: [66571.610974] [<ffffffff811e8164>] xfs_ilock+0x84/0xb0 Nov 17 14:07:39 thoregon kernel: [66571.610976] [<ffffffff811f51a4>] xfs_create+0x1d4/0x5a0 Nov 17 14:07:39 thoregon kernel: [66571.610977] [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 Nov 17 14:07:39 thoregon kernel: [66571.610979] [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 Nov 17 14:07:39 thoregon kernel: [66571.610981] [<ffffffff81100322>] vfs_create+0x72/0xc0 Nov 17 14:07:39 thoregon kernel: [66571.610983] [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 Nov 17 14:07:39 thoregon kernel: [66571.610985] [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 Nov 17 14:07:39 thoregon kernel: [66571.610987] [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 Nov 17 14:07:39 thoregon kernel: [66571.610989] [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 Nov 17 14:07:39 thoregon kernel: [66571.610991] [<ffffffff810f222c>] sys_open+0x1c/0x20 Nov 17 14:07:39 thoregon kernel: [66571.610993] [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b Nov 17 14:07:39 thoregon kernel: [66571.610995] RECLAIM_FS-ON-W at: Nov 17 14:07:39 thoregon kernel: [66571.610996] [<ffffffff8108137e>] mark_held_locks+0x7e/0x130 Nov 17 14:07:39 thoregon kernel: [66571.610998] [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0 Nov 17 14:07:39 thoregon kernel: [66571.610999] [<ffffffff810e9dc5>] kmem_cache_alloc+0x35/0xe0 Nov 17 14:07:39 thoregon kernel: [66571.611002] [<ffffffff810dba21>] vm_map_ram+0x271/0x770 Nov 17 14:07:39 thoregon kernel: [66571.611004] [<ffffffff811e12b6>] _xfs_buf_map_pages+0x46/0xe0 Nov 17 14:07:39 thoregon kernel: [66571.611008] [<ffffffff811e21ca>] xfs_buf_get_map+0x8a/0x130 Nov 17 14:07:39 thoregon kernel: [66571.611009] [<ffffffff81233989>] xfs_trans_get_buf_map+0xa9/0xd0 Nov 17 14:07:39 thoregon kernel: [66571.611011] [<ffffffff8121bc2d>] xfs_ialloc_inode_init+0xcd/0x1d0 Nov 17 14:07:39 thoregon kernel: [66571.611015] [<ffffffff8121c16f>] xfs_ialloc_ag_alloc+0x18f/0x500 Nov 17 14:07:39 thoregon kernel: [66571.611017] [<ffffffff8121d955>] xfs_dialloc+0x185/0x2a0 Nov 17 14:07:39 thoregon kernel: [66571.611019] [<ffffffff8121f068>] xfs_ialloc+0x58/0x650 Nov 17 14:07:39 thoregon kernel: [66571.611021] [<ffffffff811f3995>] xfs_dir_ialloc+0x65/0x270 Nov 17 14:07:39 thoregon kernel: [66571.611023] [<ffffffff811f537c>] xfs_create+0x3ac/0x5a0 Nov 17 14:07:39 thoregon kernel: [66571.611024] [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 Nov 17 14:07:39 thoregon kernel: [66571.611026] [<ffffffff811ecb61>] xfs_vn_mkdir+0x11/0x20 Nov 17 14:07:39 thoregon kernel: [66571.611028] [<ffffffff8110016f>] vfs_mkdir+0x7f/0xd0 Nov 17 14:07:39 thoregon kernel: [66571.611030] [<ffffffff81101b83>] sys_mkdirat+0x43/0x80 Nov 17 14:07:39 thoregon kernel: [66571.611032] [<ffffffff81101bd4>] sys_mkdir+0x14/0x20 Nov 17 14:07:39 thoregon kernel: [66571.611034] [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b Nov 17 14:07:39 thoregon kernel: [66571.611036] INITIAL USE at: Nov 17 14:07:39 thoregon kernel: [66571.611037] [<ffffffff8107ed49>] __lock_acquire+0x2e9/0x1c00 Nov 17 14:07:39 thoregon kernel: [66571.611038] [<ffffffff81080b55>] lock_acquire+0x55/0x70 Nov 17 14:07:39 thoregon kernel: [66571.611040] [<ffffffff8106126a>] down_write_nested+0x4a/0x70 Nov 17 14:07:39 thoregon kernel: [66571.611042] [<ffffffff811e8164>] xfs_ilock+0x84/0xb0 Nov 17 14:07:39 thoregon kernel: [66571.611044] [<ffffffff811f51a4>] xfs_create+0x1d4/0x5a0 Nov 17 14:07:39 thoregon kernel: [66571.611046] [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 Nov 17 14:07:39 thoregon kernel: [66571.611047] [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 Nov 17 14:07:39 thoregon kernel: [66571.611049] [<ffffffff81100322>] vfs_create+0x72/0xc0 Nov 17 14:07:39 thoregon kernel: [66571.611051] [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 Nov 17 14:07:39 thoregon kernel: [66571.611053] [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 Nov 17 14:07:39 thoregon kernel: [66571.611055] [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 Nov 17 14:07:39 thoregon kernel: [66571.611057] [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 Nov 17 14:07:39 thoregon kernel: [66571.611059] [<ffffffff810f222c>] sys_open+0x1c/0x20 Nov 17 14:07:39 thoregon kernel: [66571.611061] [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b Nov 17 14:07:39 thoregon kernel: [66571.611063] } Nov 17 14:07:39 thoregon kernel: [66571.611064] ... key at: [<ffffffff825b4b81>] __key.41357+0x1/0x8 Nov 17 14:07:39 thoregon kernel: [66571.611066] ... acquired at: Nov 17 14:07:39 thoregon kernel: [66571.611067] [<ffffffff81080b55>] lock_acquire+0x55/0x70 Nov 17 14:07:39 thoregon kernel: [66571.611069] [<ffffffff8106126a>] down_write_nested+0x4a/0x70 Nov 17 14:07:39 thoregon kernel: [66571.611071] [<ffffffff811e8164>] xfs_ilock+0x84/0xb0 Nov 17 14:07:39 thoregon kernel: [66571.611073] [<ffffffff811f51a4>] xfs_create+0x1d4/0x5a0 Nov 17 14:07:39 thoregon kernel: [66571.611074] [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 Nov 17 14:07:39 thoregon kernel: [66571.611076] [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 Nov 17 14:07:39 thoregon kernel: [66571.611078] [<ffffffff81100322>] vfs_create+0x72/0xc0 Nov 17 14:07:39 thoregon kernel: [66571.611080] [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 Nov 17 14:07:39 thoregon kernel: [66571.611082] [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 Nov 17 14:07:39 thoregon kernel: [66571.611084] [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 Nov 17 14:07:39 thoregon kernel: [66571.611086] [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 Nov 17 14:07:39 thoregon kernel: [66571.611088] [<ffffffff810f222c>] sys_open+0x1c/0x20 Nov 17 14:07:39 thoregon kernel: [66571.611090] [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b Nov 17 14:07:39 thoregon kernel: [66571.611091] Nov 17 14:07:39 thoregon kernel: [66571.611092] -> (sb_internal){.+.+.?} ops: 1341531 { Nov 17 14:07:39 thoregon kernel: [66571.611095] HARDIRQ-ON-R at: Nov 17 14:07:39 thoregon kernel: [66571.611096] [<ffffffff8107ef6a>] __lock_acquire+0x50a/0x1c00 Nov 17 14:07:39 thoregon kernel: [66571.611098] [<ffffffff81080b55>] lock_acquire+0x55/0x70 Nov 17 14:07:39 thoregon kernel: [66571.611100] [<ffffffff810f451b>] __sb_start_write+0xab/0x190 Nov 17 14:07:39 thoregon kernel: [66571.611102] [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50 Nov 17 14:07:39 thoregon kernel: [66571.611104] [<ffffffff811f5157>] xfs_create+0x187/0x5a0 Nov 17 14:07:39 thoregon kernel: [66571.611105] [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 Nov 17 14:07:39 thoregon kernel: [66571.611107] [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 Nov 17 14:07:39 thoregon kernel: [66571.611109] [<ffffffff81100322>] vfs_create+0x72/0xc0 Nov 17 14:07:39 thoregon kernel: [66571.611111] [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 Nov 17 14:07:39 thoregon kernel: [66571.611113] [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 Nov 17 14:07:39 thoregon kernel: [66571.611115] [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 Nov 17 14:07:39 thoregon kernel: [66571.611117] [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 Nov 17 14:07:39 thoregon kernel: [66571.611119] [<ffffffff810f222c>] sys_open+0x1c/0x20 Nov 17 14:07:39 thoregon kernel: [66571.611121] [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b Nov 17 14:07:39 thoregon kernel: [66571.611123] SOFTIRQ-ON-R at: Nov 17 14:07:39 thoregon kernel: [66571.611124] [<ffffffff8107f0c7>] __lock_acquire+0x667/0x1c00 Nov 17 14:07:39 thoregon kernel: [66571.611126] [<ffffffff81080b55>] lock_acquire+0x55/0x70 Nov 17 14:07:39 thoregon kernel: [66571.611128] [<ffffffff810f451b>] __sb_start_write+0xab/0x190 Nov 17 14:07:39 thoregon kernel: [66571.611130] [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50 Nov 17 14:07:39 thoregon kernel: [66571.611132] [<ffffffff811f5157>] xfs_create+0x187/0x5a0 Nov 17 14:07:39 thoregon kernel: [66571.611133] [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 Nov 17 14:07:39 thoregon kernel: [66571.611135] [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 Nov 17 14:07:39 thoregon kernel: [66571.611137] [<ffffffff81100322>] vfs_create+0x72/0xc0 Nov 17 14:07:39 thoregon kernel: [66571.611139] [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 Nov 17 14:07:39 thoregon kernel: [66571.611141] [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 Nov 17 14:07:39 thoregon kernel: [66571.611143] [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 Nov 17 14:07:39 thoregon kernel: [66571.611145] [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 Nov 17 14:07:39 thoregon kernel: [66571.611147] [<ffffffff810f222c>] sys_open+0x1c/0x20 Nov 17 14:07:39 thoregon kernel: [66571.611149] [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b Nov 17 14:07:39 thoregon kernel: [66571.611151] IN-RECLAIM_FS-R at: Nov 17 14:07:39 thoregon kernel: [66571.611152] [<ffffffff8107efdf>] __lock_acquire+0x57f/0x1c00 Nov 17 14:07:39 thoregon kernel: [66571.611154] [<ffffffff81080b55>] lock_acquire+0x55/0x70 Nov 17 14:07:39 thoregon kernel: [66571.611156] [<ffffffff810f451b>] __sb_start_write+0xab/0x190 Nov 17 14:07:39 thoregon kernel: [66571.611158] [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50 Nov 17 14:07:39 thoregon kernel: [66571.611159] [<ffffffff811f3e64>] xfs_free_eofblocks+0x104/0x250 Nov 17 14:07:39 thoregon kernel: [66571.611161] [<ffffffff811f4b39>] xfs_inactive+0xa9/0x480 Nov 17 14:07:39 thoregon kernel: [66571.611163] [<ffffffff811efe10>] xfs_fs_evict_inode+0x70/0x80 Nov 17 14:07:39 thoregon kernel: [66571.611165] [<ffffffff8110cb7f>] evict+0xaf/0x1b0 Nov 17 14:07:39 thoregon kernel: [66571.611168] [<ffffffff8110d209>] dispose_list+0x39/0x50 Nov 17 14:07:39 thoregon kernel: [66571.611170] [<ffffffff8110dc63>] prune_icache_sb+0x183/0x340 Nov 17 14:07:39 thoregon kernel: [66571.611172] [<ffffffff810f5bc3>] prune_super+0xf3/0x1a0 Nov 17 14:07:39 thoregon kernel: [66571.611173] [<ffffffff810bbdee>] shrink_slab+0x11e/0x1f0 Nov 17 14:07:39 thoregon kernel: [66571.611175] [<ffffffff810be98f>] try_to_free_pages+0x21f/0x4e0 Nov 17 14:07:39 thoregon kernel: [66571.611177] [<ffffffff810b5ec6>] __alloc_pages_nodemask+0x506/0x800 Nov 17 14:07:39 thoregon kernel: [66571.611179] [<ffffffff810ce56e>] handle_pte_fault+0x5ae/0x7a0 Nov 17 14:07:39 thoregon kernel: [66571.611182] [<ffffffff810cf769>] handle_mm_fault+0x1f9/0x2a0 Nov 17 14:07:39 thoregon kernel: [66571.611184] [<ffffffff81029dfc>] __do_page_fault+0x16c/0x480 Nov 17 14:07:39 thoregon kernel: [66571.611186] [<ffffffff8102a139>] do_page_fault+0x9/0x10 Nov 17 14:07:39 thoregon kernel: [66571.611188] [<ffffffff816b287f>] page_fault+0x1f/0x30 Nov 17 14:07:39 thoregon kernel: [66571.611190] RECLAIM_FS-ON-R at: Nov 17 14:07:39 thoregon kernel: [66571.611191] [<ffffffff8108137e>] mark_held_locks+0x7e/0x130 Nov 17 14:07:39 thoregon kernel: [66571.611193] [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0 Nov 17 14:07:39 thoregon kernel: [66571.611195] [<ffffffff810e9dc5>] kmem_cache_alloc+0x35/0xe0 Nov 17 14:07:39 thoregon kernel: [66571.611197] [<ffffffff811f6d4f>] kmem_zone_alloc+0x5f/0xe0 Nov 17 14:07:39 thoregon kernel: [66571.611198] [<ffffffff811f6de8>] kmem_zone_zalloc+0x18/0x50 Nov 17 14:07:39 thoregon kernel: [66571.611200] [<ffffffff8122b0b2>] _xfs_trans_alloc+0x32/0x90 Nov 17 14:07:39 thoregon kernel: [66571.611202] [<ffffffff8122b148>] xfs_trans_alloc+0x38/0x50 Nov 17 14:07:39 thoregon kernel: [66571.611204] [<ffffffff811f5157>] xfs_create+0x187/0x5a0 Nov 17 14:07:39 thoregon kernel: [66571.611205] [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 Nov 17 14:07:39 thoregon kernel: [66571.611207] [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 Nov 17 14:07:39 thoregon kernel: [66571.611209] [<ffffffff81100322>] vfs_create+0x72/0xc0 Nov 17 14:07:39 thoregon kernel: [66571.611211] [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 Nov 17 14:07:39 thoregon kernel: [66571.611213] [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 Nov 17 14:07:39 thoregon kernel: [66571.611215] [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 Nov 17 14:07:39 thoregon kernel: [66571.611217] [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 Nov 17 14:07:39 thoregon kernel: [66571.611219] [<ffffffff810f222c>] sys_open+0x1c/0x20 Nov 17 14:07:39 thoregon kernel: [66571.611221] [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b Nov 17 14:07:39 thoregon kernel: [66571.611223] INITIAL USE at: Nov 17 14:07:39 thoregon kernel: [66571.611224] [<ffffffff8107ed49>] __lock_acquire+0x2e9/0x1c00 Nov 17 14:07:39 thoregon kernel: [66571.611225] [<ffffffff81080b55>] lock_acquire+0x55/0x70 Nov 17 14:07:39 thoregon kernel: [66571.611227] [<ffffffff810f451b>] __sb_start_write+0xab/0x190 Nov 17 14:07:39 thoregon kernel: [66571.611229] [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50 Nov 17 14:07:39 thoregon kernel: [66571.611231] [<ffffffff811f5157>] xfs_create+0x187/0x5a0 Nov 17 14:07:39 thoregon kernel: [66571.611232] [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 Nov 17 14:07:39 thoregon kernel: [66571.611234] [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 Nov 17 14:07:39 thoregon kernel: [66571.611236] [<ffffffff81100322>] vfs_create+0x72/0xc0 Nov 17 14:07:39 thoregon kernel: [66571.611238] [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 Nov 17 14:07:39 thoregon kernel: [66571.611240] [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 Nov 17 14:07:39 thoregon kernel: [66571.611242] [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 Nov 17 14:07:39 thoregon kernel: [66571.611244] [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 Nov 17 14:07:39 thoregon kernel: [66571.611246] [<ffffffff810f222c>] sys_open+0x1c/0x20 Nov 17 14:07:39 thoregon kernel: [66571.611248] [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b Nov 17 14:07:39 thoregon kernel: [66571.611250] } Nov 17 14:07:39 thoregon kernel: [66571.611251] ... key at: [<ffffffff81c34e40>] xfs_fs_type+0x60/0x80 Nov 17 14:07:39 thoregon kernel: [66571.611254] ... acquired at: Nov 17 14:07:39 thoregon kernel: [66571.611254] [<ffffffff8107df3b>] check_usage_forwards+0x10b/0x140 Nov 17 14:07:39 thoregon kernel: [66571.611256] [<ffffffff8107e900>] mark_lock+0x190/0x2f0 Nov 17 14:07:39 thoregon kernel: [66571.611258] [<ffffffff8107efdf>] __lock_acquire+0x57f/0x1c00 Nov 17 14:07:39 thoregon kernel: [66571.611260] [<ffffffff81080b55>] lock_acquire+0x55/0x70 Nov 17 14:07:39 thoregon kernel: [66571.611261] [<ffffffff810f451b>] __sb_start_write+0xab/0x190 Nov 17 14:07:39 thoregon kernel: [66571.611263] [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50 Nov 17 14:07:39 thoregon kernel: [66571.611265] [<ffffffff811f3e64>] xfs_free_eofblocks+0x104/0x250 Nov 17 14:07:39 thoregon kernel: [66571.611266] [<ffffffff811f4b39>] xfs_inactive+0xa9/0x480 Nov 17 14:07:39 thoregon kernel: [66571.611268] [<ffffffff811efe10>] xfs_fs_evict_inode+0x70/0x80 Nov 17 14:07:39 thoregon kernel: [66571.611270] [<ffffffff8110cb7f>] evict+0xaf/0x1b0 Nov 17 14:07:39 thoregon kernel: [66571.611271] [<ffffffff8110d209>] dispose_list+0x39/0x50 Nov 17 14:07:39 thoregon kernel: [66571.611273] [<ffffffff8110dc63>] prune_icache_sb+0x183/0x340 Nov 17 14:07:39 thoregon kernel: [66571.611275] [<ffffffff810f5bc3>] prune_super+0xf3/0x1a0 Nov 17 14:07:39 thoregon kernel: [66571.611277] [<ffffffff810bbdee>] shrink_slab+0x11e/0x1f0 Nov 17 14:07:39 thoregon kernel: [66571.611278] [<ffffffff810be98f>] try_to_free_pages+0x21f/0x4e0 Nov 17 14:07:39 thoregon kernel: [66571.611280] [<ffffffff810b5ec6>] __alloc_pages_nodemask+0x506/0x800 Nov 17 14:07:39 thoregon kernel: [66571.611282] [<ffffffff810ce56e>] handle_pte_fault+0x5ae/0x7a0 Nov 17 14:07:39 thoregon kernel: [66571.611284] [<ffffffff810cf769>] handle_mm_fault+0x1f9/0x2a0 Nov 17 14:07:39 thoregon kernel: [66571.611286] [<ffffffff81029dfc>] __do_page_fault+0x16c/0x480 Nov 17 14:07:39 thoregon kernel: [66571.611287] [<ffffffff8102a139>] do_page_fault+0x9/0x10 Nov 17 14:07:39 thoregon kernel: [66571.611289] [<ffffffff816b287f>] page_fault+0x1f/0x30 Nov 17 14:07:39 thoregon kernel: [66571.611291] Nov 17 14:07:39 thoregon kernel: [66571.611292] Nov 17 14:07:39 thoregon kernel: [66571.611292] stack backtrace: Nov 17 14:07:39 thoregon kernel: [66571.611294] Pid: 21330, comm: cc1 Not tainted 3.7.0-rc5 #1 Nov 17 14:07:39 thoregon kernel: [66571.611295] Call Trace: Nov 17 14:07:39 thoregon kernel: [66571.611298] [<ffffffff8107de28>] print_irq_inversion_bug.part.37+0x1e8/0x1f0 Nov 17 14:07:39 thoregon kernel: [66571.611300] [<ffffffff8107df3b>] check_usage_forwards+0x10b/0x140 Nov 17 14:07:39 thoregon kernel: [66571.611303] [<ffffffff8107e900>] mark_lock+0x190/0x2f0 Nov 17 14:07:39 thoregon kernel: [66571.611306] [<ffffffff8150406e>] ? dm_request+0x2e/0x2a0 Nov 17 14:07:39 thoregon kernel: [66571.611308] [<ffffffff8107de30>] ? print_irq_inversion_bug.part.37+0x1f0/0x1f0 Nov 17 14:07:39 thoregon kernel: [66571.611310] [<ffffffff8107efdf>] __lock_acquire+0x57f/0x1c00 Nov 17 14:07:39 thoregon kernel: [66571.611313] [<ffffffff812202f4>] ? xfs_iext_bno_to_ext+0x84/0x160 Nov 17 14:07:39 thoregon kernel: [66571.611316] [<ffffffff8120a023>] ? xfs_bmbt_get_all+0x13/0x20 Nov 17 14:07:39 thoregon kernel: [66571.611318] [<ffffffff81200fb4>] ? xfs_bmap_search_multi_extents+0xa4/0x110 Nov 17 14:07:39 thoregon kernel: [66571.611320] [<ffffffff81080b55>] lock_acquire+0x55/0x70 Nov 17 14:07:39 thoregon kernel: [66571.611322] [<ffffffff8122b138>] ? xfs_trans_alloc+0x28/0x50 Nov 17 14:07:39 thoregon kernel: [66571.611324] [<ffffffff810f451b>] __sb_start_write+0xab/0x190 Nov 17 14:07:39 thoregon kernel: [66571.611326] [<ffffffff8122b138>] ? xfs_trans_alloc+0x28/0x50 Nov 17 14:07:39 thoregon kernel: [66571.611328] [<ffffffff8122b138>] ? xfs_trans_alloc+0x28/0x50 Nov 17 14:07:39 thoregon kernel: [66571.611330] [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50 Nov 17 14:07:39 thoregon kernel: [66571.611332] [<ffffffff811f3e64>] xfs_free_eofblocks+0x104/0x250 Nov 17 14:07:39 thoregon kernel: [66571.611334] [<ffffffff816b204b>] ? _raw_spin_unlock_irq+0x2b/0x50 Nov 17 14:07:39 thoregon kernel: [66571.611336] [<ffffffff811f4b39>] xfs_inactive+0xa9/0x480 Nov 17 14:07:39 thoregon kernel: [66571.611337] [<ffffffff816b204b>] ? _raw_spin_unlock_irq+0x2b/0x50 Nov 17 14:07:39 thoregon kernel: [66571.611340] [<ffffffff811efe10>] xfs_fs_evict_inode+0x70/0x80 Nov 17 14:07:39 thoregon kernel: [66571.611342] [<ffffffff8110cb7f>] evict+0xaf/0x1b0 Nov 17 14:07:39 thoregon kernel: [66571.611344] [<ffffffff8110d209>] dispose_list+0x39/0x50 Nov 17 14:07:39 thoregon kernel: [66571.611346] [<ffffffff8110dc63>] prune_icache_sb+0x183/0x340 Nov 17 14:07:39 thoregon kernel: [66571.611347] [<ffffffff810f5bc3>] prune_super+0xf3/0x1a0 Nov 17 14:07:39 thoregon kernel: [66571.611349] [<ffffffff810bbdee>] shrink_slab+0x11e/0x1f0 Nov 17 14:07:39 thoregon kernel: [66571.611352] [<ffffffff810be98f>] try_to_free_pages+0x21f/0x4e0 Nov 17 14:07:39 thoregon kernel: [66571.611354] [<ffffffff810b5ec6>] __alloc_pages_nodemask+0x506/0x800 Nov 17 14:07:39 thoregon kernel: [66571.611356] [<ffffffff810b9e40>] ? lru_deactivate_fn+0x1c0/0x1c0 Nov 17 14:07:39 thoregon kernel: [66571.611358] [<ffffffff810ce56e>] handle_pte_fault+0x5ae/0x7a0 Nov 17 14:07:39 thoregon kernel: [66571.611360] [<ffffffff810cf769>] handle_mm_fault+0x1f9/0x2a0 Nov 17 14:07:39 thoregon kernel: [66571.611363] [<ffffffff81029dfc>] __do_page_fault+0x16c/0x480 Nov 17 14:07:39 thoregon kernel: [66571.611366] [<ffffffff8129c7ad>] ? trace_hardirqs_off_thunk+0x3a/0x3c Nov 17 14:07:39 thoregon kernel: [66571.611368] [<ffffffff8102a139>] do_page_fault+0x9/0x10 Nov 17 14:07:39 thoregon kernel: [66571.611370] [<ffffffff816b287f>] page_fault+0x1f/0x30 _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Hang in XFS reclaim on 3.7.0-rc3 2012-11-18 10:24 ` Torsten Kaiser @ 2012-11-18 15:29 ` Torsten Kaiser -1 siblings, 0 replies; 31+ messages in thread From: Torsten Kaiser @ 2012-11-18 15:29 UTC (permalink / raw) To: Dave Chinner; +Cc: xfs, Linux Kernel On Sun, Nov 18, 2012 at 11:24 AM, Torsten Kaiser <just.for.lkml@googlemail.com> wrote: > On Tue, Oct 30, 2012 at 9:37 PM, Torsten Kaiser > <just.for.lkml@googlemail.com> wrote: >> I will keep LOCKDEP enabled on that system, and if there really is >> another splat, I will report back here. But I rather doubt that this >> will be needed. > > After the patch, I did not see this problem again, but today I found > another LOCKDEP report that also looks XFS related. > I found it twice in the logs, and as both were slightly different, I > will attach both versions. > Nov 6 21:57:09 thoregon kernel: [ 9941.104353] 3.7.0-rc4 #1 Not tainted > Nov 6 21:57:09 thoregon kernel: [ 9941.104355] inconsistent > {RECLAIM_FS-ON-W} -> {IN-RECLAIM_FS-W} usage. > Nov 6 21:57:09 thoregon kernel: [ 9941.104430] CPU0 > Nov 6 21:57:09 thoregon kernel: [ 9941.104431] ---- > Nov 6 21:57:09 thoregon kernel: [ 9941.104432] lock(&(&ip->i_lock)->mr_lock); > Nov 6 21:57:09 thoregon kernel: [ 9941.104433] <Interrupt> > Nov 6 21:57:09 thoregon kernel: [ 9941.104434] > lock(&(&ip->i_lock)->mr_lock); > Nov 6 21:57:09 thoregon kernel: [ 9941.104435] > Nov 6 21:57:09 thoregon kernel: [ 9941.104435] *** DEADLOCK *** Sorry! Copied the wrong report. Your fix only landed in -rc5, so my vanilla -rc4 did (also) report the old problem again. And I copy&pasted that report instead of the second appearance of the new problem. Here is the correct second report of the sb_internal vs ip->i_lock->mr_lock problem: [110926.972477] [110926.972482] ========================================================= [110926.972484] [ INFO: possible irq lock inversion dependency detected ] [110926.972486] 3.7.0-rc4 #1 Not tainted [110926.972487] --------------------------------------------------------- [110926.972489] kswapd0/725 just changed the state of lock: [110926.972490] (sb_internal){.+.+.?}, at: [<ffffffff8122b268>] xfs_trans_alloc+0x28/0x50 [110926.972499] but this lock took another, RECLAIM_FS-unsafe lock in the past: [110926.972500] (&(&ip->i_lock)->mr_lock/1){+.+.+.} [110926.972500] [110926.972500] and interrupts could create inverse lock ordering between them. [110926.972500] [110926.972503] [110926.972503] other info that might help us debug this: [110926.972504] Possible interrupt unsafe locking scenario: [110926.972504] [110926.972505] CPU0 CPU1 [110926.972506] ---- ---- [110926.972507] lock(&(&ip->i_lock)->mr_lock/1); [110926.972509] local_irq_disable(); [110926.972509] lock(sb_internal); [110926.972511] lock(&(&ip->i_lock)->mr_lock/1); [110926.972512] <Interrupt> [110926.972513] lock(sb_internal); [110926.972514] [110926.972514] *** DEADLOCK *** [110926.972514] [110926.972516] 2 locks held by kswapd0/725: [110926.972517] #0: (shrinker_rwsem){++++..}, at: [<ffffffff810bbd22>] shrink_slab+0x32/0x1f0 [110926.972522] #1: (&type->s_umount_key#20){++++.+}, at: [<ffffffff810f5a8e>] grab_super_passive+0x3e/0x90 [110926.972527] [110926.972527] the shortest dependencies between 2nd lock and 1st lock: [110926.972533] -> (&(&ip->i_lock)->mr_lock/1){+.+.+.} ops: 58117 { [110926.972536] HARDIRQ-ON-W at: [110926.972537] [<ffffffff8107f091>] __lock_acquire+0x631/0x1c00 [110926.972540] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [110926.972542] [<ffffffff8106126a>] down_write_nested+0x4a/0x70 [110926.972545] [<ffffffff811e8164>] xfs_ilock+0x84/0xb0 [110926.972548] [<ffffffff811f5194>] xfs_create+0x1d4/0x5a0 [110926.972550] [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0 [110926.972552] [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10 [110926.972554] [<ffffffff81100332>] vfs_create+0x72/0xc0 [110926.972556] [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80 [110926.972558] [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490 [110926.972560] [<ffffffff8110184d>] do_filp_open+0x3d/0xa0 [110926.972562] [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0 [110926.972565] [<ffffffff810f223c>] sys_open+0x1c/0x20 [110926.972567] [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b [110926.972570] SOFTIRQ-ON-W at: [110926.972571] [<ffffffff8107f0c7>] __lock_acquire+0x667/0x1c00 [110926.972573] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [110926.972574] [<ffffffff8106126a>] down_write_nested+0x4a/0x70 [110926.972576] [<ffffffff811e8164>] xfs_ilock+0x84/0xb0 [110926.972578] [<ffffffff811f5194>] xfs_create+0x1d4/0x5a0 [110926.972580] [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0 [110926.972581] [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10 [110926.972583] [<ffffffff81100332>] vfs_create+0x72/0xc0 [110926.972585] [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80 [110926.972587] [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490 [110926.972589] [<ffffffff8110184d>] do_filp_open+0x3d/0xa0 [110926.972591] [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0 [110926.972593] [<ffffffff810f223c>] sys_open+0x1c/0x20 [110926.972595] [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b [110926.972597] RECLAIM_FS-ON-W at: [110926.972598] [<ffffffff8108137e>] mark_held_locks+0x7e/0x130 [110926.972600] [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0 [110926.972601] [<ffffffff810e9dd5>] kmem_cache_alloc+0x35/0xe0 [110926.972603] [<ffffffff810dba31>] vm_map_ram+0x271/0x770 [110926.972606] [<ffffffff811e1316>] _xfs_buf_map_pages+0x46/0xe0 [110926.972609] [<ffffffff811e222a>] xfs_buf_get_map+0x8a/0x130 [110926.972610] [<ffffffff81233ab9>] xfs_trans_get_buf_map+0xa9/0xd0 [110926.972613] [<ffffffff8121bced>] xfs_ialloc_inode_init+0xcd/0x1d0 [110926.972616] [<ffffffff8121c25e>] xfs_ialloc_ag_alloc+0x1be/0x560 [110926.972618] [<ffffffff8121da65>] xfs_dialloc+0x185/0x2a0 [110926.972619] [<ffffffff8121f198>] xfs_ialloc+0x58/0x650 [110926.972621] [<ffffffff811f3985>] xfs_dir_ialloc+0x65/0x270 [110926.972623] [<ffffffff811f536c>] xfs_create+0x3ac/0x5a0 [110926.972624] [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0 [110926.972626] [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10 [110926.972628] [<ffffffff81100332>] vfs_create+0x72/0xc0 [110926.972630] [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80 [110926.972632] [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490 [110926.972634] [<ffffffff8110184d>] do_filp_open+0x3d/0xa0 [110926.972636] [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0 [110926.972638] [<ffffffff810f223c>] sys_open+0x1c/0x20 [110926.972640] [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b [110926.972642] INITIAL USE at: [110926.972642] [<ffffffff8107ed49>] __lock_acquire+0x2e9/0x1c00 [110926.972644] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [110926.972646] [<ffffffff8106126a>] down_write_nested+0x4a/0x70 [110926.972648] [<ffffffff811e8164>] xfs_ilock+0x84/0xb0 [110926.972650] [<ffffffff811f5194>] xfs_create+0x1d4/0x5a0 [110926.972651] [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0 [110926.972653] [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10 [110926.972655] [<ffffffff81100332>] vfs_create+0x72/0xc0 [110926.972657] [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80 [110926.972659] [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490 [110926.972661] [<ffffffff8110184d>] do_filp_open+0x3d/0xa0 [110926.972663] [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0 [110926.972664] [<ffffffff810f223c>] sys_open+0x1c/0x20 [110926.972666] [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b [110926.972668] } [110926.972669] ... key at: [<ffffffff825b4b81>] __key.41355+0x1/0x8 [110926.972672] ... acquired at: [110926.972672] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [110926.972674] [<ffffffff8106126a>] down_write_nested+0x4a/0x70 [110926.972676] [<ffffffff811e8164>] xfs_ilock+0x84/0xb0 [110926.972678] [<ffffffff811f5194>] xfs_create+0x1d4/0x5a0 [110926.972679] [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0 [110926.972681] [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10 [110926.972683] [<ffffffff81100332>] vfs_create+0x72/0xc0 [110926.972684] [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80 [110926.972686] [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490 [110926.972688] [<ffffffff8110184d>] do_filp_open+0x3d/0xa0 [110926.972690] [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0 [110926.972692] [<ffffffff810f223c>] sys_open+0x1c/0x20 [110926.972694] [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b [110926.972696] [110926.972696] -> (sb_internal){.+.+.?} ops: 1710064 { [110926.972699] HARDIRQ-ON-R at: [110926.972700] [<ffffffff8107ef6a>] __lock_acquire+0x50a/0x1c00 [110926.972702] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [110926.972704] [<ffffffff810f452b>] __sb_start_write+0xab/0x190 [110926.972705] [<ffffffff8122b268>] xfs_trans_alloc+0x28/0x50 [110926.972707] [<ffffffff811f5147>] xfs_create+0x187/0x5a0 [110926.972709] [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0 [110926.972711] [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10 [110926.972712] [<ffffffff81100332>] vfs_create+0x72/0xc0 [110926.972714] [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80 [110926.972716] [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490 [110926.972718] [<ffffffff8110184d>] do_filp_open+0x3d/0xa0 [110926.972720] [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0 [110926.972722] [<ffffffff810f223c>] sys_open+0x1c/0x20 [110926.972724] [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b [110926.972726] SOFTIRQ-ON-R at: [110926.972727] [<ffffffff8107f0c7>] __lock_acquire+0x667/0x1c00 [110926.972728] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [110926.972730] [<ffffffff810f452b>] __sb_start_write+0xab/0x190 [110926.972732] [<ffffffff8122b268>] xfs_trans_alloc+0x28/0x50 [110926.972734] [<ffffffff811f5147>] xfs_create+0x187/0x5a0 [110926.972735] [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0 [110926.972737] [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10 [110926.972739] [<ffffffff81100332>] vfs_create+0x72/0xc0 [110926.972741] [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80 [110926.972743] [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490 [110926.972745] [<ffffffff8110184d>] do_filp_open+0x3d/0xa0 [110926.972747] [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0 [110926.972749] [<ffffffff810f223c>] sys_open+0x1c/0x20 [110926.972750] [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b [110926.972752] IN-RECLAIM_FS-R at: [110926.972753] [<ffffffff8107efdf>] __lock_acquire+0x57f/0x1c00 [110926.972755] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [110926.972757] [<ffffffff810f452b>] __sb_start_write+0xab/0x190 [110926.972758] [<ffffffff8122b268>] xfs_trans_alloc+0x28/0x50 [110926.972760] [<ffffffff811f3e54>] xfs_free_eofblocks+0x104/0x250 [110926.972762] [<ffffffff811f4b29>] xfs_inactive+0xa9/0x480 [110926.972763] [<ffffffff811efe00>] xfs_fs_evict_inode+0x70/0x80 [110926.972765] [<ffffffff8110cb8f>] evict+0xaf/0x1b0 [110926.972768] [<ffffffff8110d219>] dispose_list+0x39/0x50 [110926.972770] [<ffffffff8110dc73>] prune_icache_sb+0x183/0x340 [110926.972772] [<ffffffff810f5bd3>] prune_super+0xf3/0x1a0 [110926.972773] [<ffffffff810bbe0e>] shrink_slab+0x11e/0x1f0 [110926.972775] [<ffffffff810be400>] kswapd+0x690/0xa10 [110926.972777] [<ffffffff8105c246>] kthread+0xd6/0xe0 [110926.972779] [<ffffffff816b2c6c>] ret_from_fork+0x7c/0xb0 [110926.972781] RECLAIM_FS-ON-R at: [110926.972782] [<ffffffff8108137e>] mark_held_locks+0x7e/0x130 [110926.972784] [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0 [110926.972785] [<ffffffff810e9dd5>] kmem_cache_alloc+0x35/0xe0 [110926.972787] [<ffffffff811f6d3f>] kmem_zone_alloc+0x5f/0xe0 [110926.972789] [<ffffffff811f6dd8>] kmem_zone_zalloc+0x18/0x50 [110926.972790] [<ffffffff8122b1e2>] _xfs_trans_alloc+0x32/0x90 [110926.972792] [<ffffffff8122b278>] xfs_trans_alloc+0x38/0x50 [110926.972794] [<ffffffff811f5147>] xfs_create+0x187/0x5a0 [110926.972796] [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0 [110926.972797] [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10 [110926.972799] [<ffffffff81100332>] vfs_create+0x72/0xc0 [110926.972801] [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80 [110926.972803] [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490 [110926.972805] [<ffffffff8110184d>] do_filp_open+0x3d/0xa0 [110926.972807] [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0 [110926.972809] [<ffffffff810f223c>] sys_open+0x1c/0x20 [110926.972811] [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b [110926.972813] INITIAL USE at: [110926.972814] [<ffffffff8107ed49>] __lock_acquire+0x2e9/0x1c00 [110926.972815] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [110926.972817] [<ffffffff810f452b>] __sb_start_write+0xab/0x190 [110926.972819] [<ffffffff8122b268>] xfs_trans_alloc+0x28/0x50 [110926.972820] [<ffffffff811f5147>] xfs_create+0x187/0x5a0 [110926.972822] [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0 [110926.972824] [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10 [110926.972826] [<ffffffff81100332>] vfs_create+0x72/0xc0 [110926.972827] [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80 [110926.972829] [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490 [110926.972831] [<ffffffff8110184d>] do_filp_open+0x3d/0xa0 [110926.972833] [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0 [110926.972835] [<ffffffff810f223c>] sys_open+0x1c/0x20 [110926.972837] [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b [110926.972839] } [110926.972840] ... key at: [<ffffffff81c34e40>] xfs_fs_type+0x60/0x80 [110926.972842] ... acquired at: [110926.972843] [<ffffffff8107df3b>] check_usage_forwards+0x10b/0x140 [110926.972845] [<ffffffff8107e900>] mark_lock+0x190/0x2f0 [110926.972846] [<ffffffff8107efdf>] __lock_acquire+0x57f/0x1c00 [110926.972848] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [110926.972850] [<ffffffff810f452b>] __sb_start_write+0xab/0x190 [110926.972851] [<ffffffff8122b268>] xfs_trans_alloc+0x28/0x50 [110926.972853] [<ffffffff811f3e54>] xfs_free_eofblocks+0x104/0x250 [110926.972855] [<ffffffff811f4b29>] xfs_inactive+0xa9/0x480 [110926.972856] [<ffffffff811efe00>] xfs_fs_evict_inode+0x70/0x80 [110926.972858] [<ffffffff8110cb8f>] evict+0xaf/0x1b0 [110926.972860] [<ffffffff8110d219>] dispose_list+0x39/0x50 [110926.972861] [<ffffffff8110dc73>] prune_icache_sb+0x183/0x340 [110926.972863] [<ffffffff810f5bd3>] prune_super+0xf3/0x1a0 [110926.972865] [<ffffffff810bbe0e>] shrink_slab+0x11e/0x1f0 [110926.972866] [<ffffffff810be400>] kswapd+0x690/0xa10 [110926.972868] [<ffffffff8105c246>] kthread+0xd6/0xe0 [110926.972870] [<ffffffff816b2c6c>] ret_from_fork+0x7c/0xb0 [110926.972871] [110926.972872] [110926.972872] stack backtrace: [110926.972874] Pid: 725, comm: kswapd0 Not tainted 3.7.0-rc4 #1 [110926.972875] Call Trace: [110926.972878] [<ffffffff8107de28>] print_irq_inversion_bug.part.37+0x1e8/0x1f0 [110926.972880] [<ffffffff8107df3b>] check_usage_forwards+0x10b/0x140 [110926.972883] [<ffffffff8107e900>] mark_lock+0x190/0x2f0 [110926.972885] [<ffffffff8107de30>] ? print_irq_inversion_bug.part.37+0x1f0/0x1f0 [110926.972887] [<ffffffff8107efdf>] __lock_acquire+0x57f/0x1c00 [110926.972889] [<ffffffff81220424>] ? xfs_iext_bno_to_ext+0x84/0x160 [110926.972892] [<ffffffff8120a0e3>] ? xfs_bmbt_get_all+0x13/0x20 [110926.972895] [<ffffffff81201104>] ? xfs_bmap_search_multi_extents+0xa4/0x110 [110926.972897] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [110926.972899] [<ffffffff8122b268>] ? xfs_trans_alloc+0x28/0x50 [110926.972901] [<ffffffff810f452b>] __sb_start_write+0xab/0x190 [110926.972903] [<ffffffff8122b268>] ? xfs_trans_alloc+0x28/0x50 [110926.972905] [<ffffffff8122b268>] ? xfs_trans_alloc+0x28/0x50 [110926.972907] [<ffffffff8122b268>] xfs_trans_alloc+0x28/0x50 [110926.972908] [<ffffffff811f3e54>] xfs_free_eofblocks+0x104/0x250 [110926.972910] [<ffffffff816b1efb>] ? _raw_spin_unlock_irq+0x2b/0x50 [110926.972912] [<ffffffff811f4b29>] xfs_inactive+0xa9/0x480 [110926.972914] [<ffffffff816b1efb>] ? _raw_spin_unlock_irq+0x2b/0x50 [110926.972916] [<ffffffff811efe00>] xfs_fs_evict_inode+0x70/0x80 [110926.972918] [<ffffffff8110cb8f>] evict+0xaf/0x1b0 [110926.972920] [<ffffffff8110d219>] dispose_list+0x39/0x50 [110926.972922] [<ffffffff8110dc73>] prune_icache_sb+0x183/0x340 [110926.972924] [<ffffffff810f5bd3>] prune_super+0xf3/0x1a0 [110926.972926] [<ffffffff810bbe0e>] shrink_slab+0x11e/0x1f0 [110926.972928] [<ffffffff810be400>] kswapd+0x690/0xa10 [110926.972930] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60 [110926.972932] [<ffffffff810bdd70>] ? shrink_lruvec+0x540/0x540 [110926.972934] [<ffffffff8105c246>] kthread+0xd6/0xe0 [110926.972936] [<ffffffff816b1efb>] ? _raw_spin_unlock_irq+0x2b/0x50 [110926.972938] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 [110926.972940] [<ffffffff816b2c6c>] ret_from_fork+0x7c/0xb0 [110926.972942] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 > Nov 17 14:07:38 thoregon kernel: [66571.610863] > Nov 17 14:07:38 thoregon kernel: [66571.610869] > ========================================================= > Nov 17 14:07:38 thoregon kernel: [66571.610870] [ INFO: possible irq > lock inversion dependency detected ] > Nov 17 14:07:38 thoregon kernel: [66571.610873] 3.7.0-rc5 #1 Not tainted > Nov 17 14:07:38 thoregon kernel: [66571.610874] > --------------------------------------------------------- > Nov 17 14:07:38 thoregon kernel: [66571.610875] cc1/21330 just changed > the state of lock: > Nov 17 14:07:38 thoregon kernel: [66571.610877] > (sb_internal){.+.+.?}, at: [<ffffffff8122b138>] > xfs_trans_alloc+0x28/0x50 > Nov 17 14:07:38 thoregon kernel: [66571.610885] but this lock took > another, RECLAIM_FS-unsafe lock in the past: > Nov 17 14:07:38 thoregon kernel: [66571.610886] > (&(&ip->i_lock)->mr_lock/1){+.+.+.} > Nov 17 14:07:38 thoregon kernel: [66571.610886] > Nov 17 14:07:39 thoregon kernel: [66571.610886] and interrupts could > create inverse lock ordering between them. > Nov 17 14:07:39 thoregon kernel: [66571.610886] > Nov 17 14:07:39 thoregon kernel: [66571.610890] > Nov 17 14:07:39 thoregon kernel: [66571.610890] other info that might > help us debug this: > Nov 17 14:07:39 thoregon kernel: [66571.610891] Possible interrupt > unsafe locking scenario: > Nov 17 14:07:39 thoregon kernel: [66571.610891] > Nov 17 14:07:39 thoregon kernel: [66571.610892] CPU0 > CPU1 > Nov 17 14:07:39 thoregon kernel: [66571.610893] ---- > ---- > Nov 17 14:07:39 thoregon kernel: [66571.610894] > lock(&(&ip->i_lock)->mr_lock/1); > Nov 17 14:07:39 thoregon kernel: [66571.610896] > local_irq_disable(); > Nov 17 14:07:39 thoregon kernel: [66571.610897] > lock(sb_internal); > Nov 17 14:07:39 thoregon kernel: [66571.610898] > lock(&(&ip->i_lock)->mr_lock/1); > Nov 17 14:07:39 thoregon kernel: [66571.610900] <Interrupt> > Nov 17 14:07:39 thoregon kernel: [66571.610901] lock(sb_internal); > Nov 17 14:07:39 thoregon kernel: [66571.610902] > Nov 17 14:07:39 thoregon kernel: [66571.610902] *** DEADLOCK *** > Nov 17 14:07:39 thoregon kernel: [66571.610902] > Nov 17 14:07:39 thoregon kernel: [66571.610904] 3 locks held by cc1/21330: > Nov 17 14:07:39 thoregon kernel: [66571.610905] #0: > (&mm->mmap_sem){++++++}, at: [<ffffffff81029d8b>] > __do_page_fault+0xfb/0x480 > Nov 17 14:07:39 thoregon kernel: [66571.610910] #1: > (shrinker_rwsem){++++..}, at: [<ffffffff810bbd02>] > shrink_slab+0x32/0x1f0 > Nov 17 14:07:39 thoregon kernel: [66571.610915] #2: > (&type->s_umount_key#20){++++.+}, at: [<ffffffff810f5a7e>] > grab_super_passive+0x3e/0x90 > Nov 17 14:07:39 thoregon kernel: [66571.610921] > Nov 17 14:07:39 thoregon kernel: [66571.610921] the shortest > dependencies between 2nd lock and 1st lock: > Nov 17 14:07:39 thoregon kernel: [66571.610927] -> > (&(&ip->i_lock)->mr_lock/1){+.+.+.} ops: 169649 { > Nov 17 14:07:39 thoregon kernel: [66571.610931] HARDIRQ-ON-W at: > Nov 17 14:07:39 thoregon kernel: [66571.610932] > [<ffffffff8107f091>] __lock_acquire+0x631/0x1c00 > Nov 17 14:07:39 thoregon kernel: [66571.610935] > [<ffffffff81080b55>] lock_acquire+0x55/0x70 > Nov 17 14:07:39 thoregon kernel: [66571.610937] > [<ffffffff8106126a>] down_write_nested+0x4a/0x70 > Nov 17 14:07:39 thoregon kernel: [66571.610941] > [<ffffffff811e8164>] xfs_ilock+0x84/0xb0 > Nov 17 14:07:39 thoregon kernel: [66571.610944] > [<ffffffff811f51a4>] xfs_create+0x1d4/0x5a0 > Nov 17 14:07:39 thoregon kernel: [66571.610946] > [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 > Nov 17 14:07:39 thoregon kernel: [66571.610948] > [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 > Nov 17 14:07:39 thoregon kernel: [66571.610950] > [<ffffffff81100322>] vfs_create+0x72/0xc0 > Nov 17 14:07:39 thoregon kernel: [66571.610953] > [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 > Nov 17 14:07:39 thoregon kernel: [66571.610955] > [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 > Nov 17 14:07:39 thoregon kernel: [66571.610957] > [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 > Nov 17 14:07:39 thoregon kernel: [66571.610959] > [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 > Nov 17 14:07:39 thoregon kernel: [66571.610962] > [<ffffffff810f222c>] sys_open+0x1c/0x20 > Nov 17 14:07:39 thoregon kernel: [66571.610964] > [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b > Nov 17 14:07:39 thoregon kernel: [66571.610967] SOFTIRQ-ON-W at: > Nov 17 14:07:39 thoregon kernel: [66571.610968] > [<ffffffff8107f0c7>] __lock_acquire+0x667/0x1c00 > Nov 17 14:07:39 thoregon kernel: [66571.610970] > [<ffffffff81080b55>] lock_acquire+0x55/0x70 > Nov 17 14:07:39 thoregon kernel: [66571.610972] > [<ffffffff8106126a>] down_write_nested+0x4a/0x70 > Nov 17 14:07:39 thoregon kernel: [66571.610974] > [<ffffffff811e8164>] xfs_ilock+0x84/0xb0 > Nov 17 14:07:39 thoregon kernel: [66571.610976] > [<ffffffff811f51a4>] xfs_create+0x1d4/0x5a0 > Nov 17 14:07:39 thoregon kernel: [66571.610977] > [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 > Nov 17 14:07:39 thoregon kernel: [66571.610979] > [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 > Nov 17 14:07:39 thoregon kernel: [66571.610981] > [<ffffffff81100322>] vfs_create+0x72/0xc0 > Nov 17 14:07:39 thoregon kernel: [66571.610983] > [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 > Nov 17 14:07:39 thoregon kernel: [66571.610985] > [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 > Nov 17 14:07:39 thoregon kernel: [66571.610987] > [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 > Nov 17 14:07:39 thoregon kernel: [66571.610989] > [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 > Nov 17 14:07:39 thoregon kernel: [66571.610991] > [<ffffffff810f222c>] sys_open+0x1c/0x20 > Nov 17 14:07:39 thoregon kernel: [66571.610993] > [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b > Nov 17 14:07:39 thoregon kernel: [66571.610995] RECLAIM_FS-ON-W at: > Nov 17 14:07:39 thoregon kernel: [66571.610996] > [<ffffffff8108137e>] mark_held_locks+0x7e/0x130 > Nov 17 14:07:39 thoregon kernel: [66571.610998] > [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0 > Nov 17 14:07:39 thoregon kernel: [66571.610999] > [<ffffffff810e9dc5>] kmem_cache_alloc+0x35/0xe0 > Nov 17 14:07:39 thoregon kernel: [66571.611002] > [<ffffffff810dba21>] vm_map_ram+0x271/0x770 > Nov 17 14:07:39 thoregon kernel: [66571.611004] > [<ffffffff811e12b6>] _xfs_buf_map_pages+0x46/0xe0 > Nov 17 14:07:39 thoregon kernel: [66571.611008] > [<ffffffff811e21ca>] xfs_buf_get_map+0x8a/0x130 > Nov 17 14:07:39 thoregon kernel: [66571.611009] > [<ffffffff81233989>] xfs_trans_get_buf_map+0xa9/0xd0 > Nov 17 14:07:39 thoregon kernel: [66571.611011] > [<ffffffff8121bc2d>] xfs_ialloc_inode_init+0xcd/0x1d0 > Nov 17 14:07:39 thoregon kernel: [66571.611015] > [<ffffffff8121c16f>] xfs_ialloc_ag_alloc+0x18f/0x500 > Nov 17 14:07:39 thoregon kernel: [66571.611017] > [<ffffffff8121d955>] xfs_dialloc+0x185/0x2a0 > Nov 17 14:07:39 thoregon kernel: [66571.611019] > [<ffffffff8121f068>] xfs_ialloc+0x58/0x650 > Nov 17 14:07:39 thoregon kernel: [66571.611021] > [<ffffffff811f3995>] xfs_dir_ialloc+0x65/0x270 > Nov 17 14:07:39 thoregon kernel: [66571.611023] > [<ffffffff811f537c>] xfs_create+0x3ac/0x5a0 > Nov 17 14:07:39 thoregon kernel: [66571.611024] > [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 > Nov 17 14:07:39 thoregon kernel: [66571.611026] > [<ffffffff811ecb61>] xfs_vn_mkdir+0x11/0x20 > Nov 17 14:07:39 thoregon kernel: [66571.611028] > [<ffffffff8110016f>] vfs_mkdir+0x7f/0xd0 > Nov 17 14:07:39 thoregon kernel: [66571.611030] > [<ffffffff81101b83>] sys_mkdirat+0x43/0x80 > Nov 17 14:07:39 thoregon kernel: [66571.611032] > [<ffffffff81101bd4>] sys_mkdir+0x14/0x20 > Nov 17 14:07:39 thoregon kernel: [66571.611034] > [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b > Nov 17 14:07:39 thoregon kernel: [66571.611036] INITIAL USE at: > Nov 17 14:07:39 thoregon kernel: [66571.611037] > [<ffffffff8107ed49>] __lock_acquire+0x2e9/0x1c00 > Nov 17 14:07:39 thoregon kernel: [66571.611038] > [<ffffffff81080b55>] lock_acquire+0x55/0x70 > Nov 17 14:07:39 thoregon kernel: [66571.611040] > [<ffffffff8106126a>] down_write_nested+0x4a/0x70 > Nov 17 14:07:39 thoregon kernel: [66571.611042] > [<ffffffff811e8164>] xfs_ilock+0x84/0xb0 > Nov 17 14:07:39 thoregon kernel: [66571.611044] > [<ffffffff811f51a4>] xfs_create+0x1d4/0x5a0 > Nov 17 14:07:39 thoregon kernel: [66571.611046] > [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 > Nov 17 14:07:39 thoregon kernel: [66571.611047] > [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 > Nov 17 14:07:39 thoregon kernel: [66571.611049] > [<ffffffff81100322>] vfs_create+0x72/0xc0 > Nov 17 14:07:39 thoregon kernel: [66571.611051] > [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 > Nov 17 14:07:39 thoregon kernel: [66571.611053] > [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 > Nov 17 14:07:39 thoregon kernel: [66571.611055] > [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 > Nov 17 14:07:39 thoregon kernel: [66571.611057] > [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 > Nov 17 14:07:39 thoregon kernel: [66571.611059] > [<ffffffff810f222c>] sys_open+0x1c/0x20 > Nov 17 14:07:39 thoregon kernel: [66571.611061] > [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b > Nov 17 14:07:39 thoregon kernel: [66571.611063] } > Nov 17 14:07:39 thoregon kernel: [66571.611064] ... key at: > [<ffffffff825b4b81>] __key.41357+0x1/0x8 > Nov 17 14:07:39 thoregon kernel: [66571.611066] ... acquired at: > Nov 17 14:07:39 thoregon kernel: [66571.611067] > [<ffffffff81080b55>] lock_acquire+0x55/0x70 > Nov 17 14:07:39 thoregon kernel: [66571.611069] > [<ffffffff8106126a>] down_write_nested+0x4a/0x70 > Nov 17 14:07:39 thoregon kernel: [66571.611071] > [<ffffffff811e8164>] xfs_ilock+0x84/0xb0 > Nov 17 14:07:39 thoregon kernel: [66571.611073] > [<ffffffff811f51a4>] xfs_create+0x1d4/0x5a0 > Nov 17 14:07:39 thoregon kernel: [66571.611074] > [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 > Nov 17 14:07:39 thoregon kernel: [66571.611076] > [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 > Nov 17 14:07:39 thoregon kernel: [66571.611078] > [<ffffffff81100322>] vfs_create+0x72/0xc0 > Nov 17 14:07:39 thoregon kernel: [66571.611080] > [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 > Nov 17 14:07:39 thoregon kernel: [66571.611082] > [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 > Nov 17 14:07:39 thoregon kernel: [66571.611084] > [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 > Nov 17 14:07:39 thoregon kernel: [66571.611086] > [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 > Nov 17 14:07:39 thoregon kernel: [66571.611088] > [<ffffffff810f222c>] sys_open+0x1c/0x20 > Nov 17 14:07:39 thoregon kernel: [66571.611090] > [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b > Nov 17 14:07:39 thoregon kernel: [66571.611091] > Nov 17 14:07:39 thoregon kernel: [66571.611092] -> > (sb_internal){.+.+.?} ops: 1341531 { > Nov 17 14:07:39 thoregon kernel: [66571.611095] HARDIRQ-ON-R at: > Nov 17 14:07:39 thoregon kernel: [66571.611096] > [<ffffffff8107ef6a>] __lock_acquire+0x50a/0x1c00 > Nov 17 14:07:39 thoregon kernel: [66571.611098] > [<ffffffff81080b55>] lock_acquire+0x55/0x70 > Nov 17 14:07:39 thoregon kernel: [66571.611100] > [<ffffffff810f451b>] __sb_start_write+0xab/0x190 > Nov 17 14:07:39 thoregon kernel: [66571.611102] > [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50 > Nov 17 14:07:39 thoregon kernel: [66571.611104] > [<ffffffff811f5157>] xfs_create+0x187/0x5a0 > Nov 17 14:07:39 thoregon kernel: [66571.611105] > [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 > Nov 17 14:07:39 thoregon kernel: [66571.611107] > [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 > Nov 17 14:07:39 thoregon kernel: [66571.611109] > [<ffffffff81100322>] vfs_create+0x72/0xc0 > Nov 17 14:07:39 thoregon kernel: [66571.611111] > [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 > Nov 17 14:07:39 thoregon kernel: [66571.611113] > [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 > Nov 17 14:07:39 thoregon kernel: [66571.611115] > [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 > Nov 17 14:07:39 thoregon kernel: [66571.611117] > [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 > Nov 17 14:07:39 thoregon kernel: [66571.611119] > [<ffffffff810f222c>] sys_open+0x1c/0x20 > Nov 17 14:07:39 thoregon kernel: [66571.611121] > [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b > Nov 17 14:07:39 thoregon kernel: [66571.611123] SOFTIRQ-ON-R at: > Nov 17 14:07:39 thoregon kernel: [66571.611124] > [<ffffffff8107f0c7>] __lock_acquire+0x667/0x1c00 > Nov 17 14:07:39 thoregon kernel: [66571.611126] > [<ffffffff81080b55>] lock_acquire+0x55/0x70 > Nov 17 14:07:39 thoregon kernel: [66571.611128] > [<ffffffff810f451b>] __sb_start_write+0xab/0x190 > Nov 17 14:07:39 thoregon kernel: [66571.611130] > [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50 > Nov 17 14:07:39 thoregon kernel: [66571.611132] > [<ffffffff811f5157>] xfs_create+0x187/0x5a0 > Nov 17 14:07:39 thoregon kernel: [66571.611133] > [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 > Nov 17 14:07:39 thoregon kernel: [66571.611135] > [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 > Nov 17 14:07:39 thoregon kernel: [66571.611137] > [<ffffffff81100322>] vfs_create+0x72/0xc0 > Nov 17 14:07:39 thoregon kernel: [66571.611139] > [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 > Nov 17 14:07:39 thoregon kernel: [66571.611141] > [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 > Nov 17 14:07:39 thoregon kernel: [66571.611143] > [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 > Nov 17 14:07:39 thoregon kernel: [66571.611145] > [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 > Nov 17 14:07:39 thoregon kernel: [66571.611147] > [<ffffffff810f222c>] sys_open+0x1c/0x20 > Nov 17 14:07:39 thoregon kernel: [66571.611149] > [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b > Nov 17 14:07:39 thoregon kernel: [66571.611151] IN-RECLAIM_FS-R at: > Nov 17 14:07:39 thoregon kernel: [66571.611152] > [<ffffffff8107efdf>] __lock_acquire+0x57f/0x1c00 > Nov 17 14:07:39 thoregon kernel: [66571.611154] > [<ffffffff81080b55>] lock_acquire+0x55/0x70 > Nov 17 14:07:39 thoregon kernel: [66571.611156] > [<ffffffff810f451b>] __sb_start_write+0xab/0x190 > Nov 17 14:07:39 thoregon kernel: [66571.611158] > [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50 > Nov 17 14:07:39 thoregon kernel: [66571.611159] > [<ffffffff811f3e64>] xfs_free_eofblocks+0x104/0x250 > Nov 17 14:07:39 thoregon kernel: [66571.611161] > [<ffffffff811f4b39>] xfs_inactive+0xa9/0x480 > Nov 17 14:07:39 thoregon kernel: [66571.611163] > [<ffffffff811efe10>] xfs_fs_evict_inode+0x70/0x80 > Nov 17 14:07:39 thoregon kernel: [66571.611165] > [<ffffffff8110cb7f>] evict+0xaf/0x1b0 > Nov 17 14:07:39 thoregon kernel: [66571.611168] > [<ffffffff8110d209>] dispose_list+0x39/0x50 > Nov 17 14:07:39 thoregon kernel: [66571.611170] > [<ffffffff8110dc63>] prune_icache_sb+0x183/0x340 > Nov 17 14:07:39 thoregon kernel: [66571.611172] > [<ffffffff810f5bc3>] prune_super+0xf3/0x1a0 > Nov 17 14:07:39 thoregon kernel: [66571.611173] > [<ffffffff810bbdee>] shrink_slab+0x11e/0x1f0 > Nov 17 14:07:39 thoregon kernel: [66571.611175] > [<ffffffff810be98f>] try_to_free_pages+0x21f/0x4e0 > Nov 17 14:07:39 thoregon kernel: [66571.611177] > [<ffffffff810b5ec6>] __alloc_pages_nodemask+0x506/0x800 > Nov 17 14:07:39 thoregon kernel: [66571.611179] > [<ffffffff810ce56e>] handle_pte_fault+0x5ae/0x7a0 > Nov 17 14:07:39 thoregon kernel: [66571.611182] > [<ffffffff810cf769>] handle_mm_fault+0x1f9/0x2a0 > Nov 17 14:07:39 thoregon kernel: [66571.611184] > [<ffffffff81029dfc>] __do_page_fault+0x16c/0x480 > Nov 17 14:07:39 thoregon kernel: [66571.611186] > [<ffffffff8102a139>] do_page_fault+0x9/0x10 > Nov 17 14:07:39 thoregon kernel: [66571.611188] > [<ffffffff816b287f>] page_fault+0x1f/0x30 > Nov 17 14:07:39 thoregon kernel: [66571.611190] RECLAIM_FS-ON-R at: > Nov 17 14:07:39 thoregon kernel: [66571.611191] > [<ffffffff8108137e>] mark_held_locks+0x7e/0x130 > Nov 17 14:07:39 thoregon kernel: [66571.611193] > [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0 > Nov 17 14:07:39 thoregon kernel: [66571.611195] > [<ffffffff810e9dc5>] kmem_cache_alloc+0x35/0xe0 > Nov 17 14:07:39 thoregon kernel: [66571.611197] > [<ffffffff811f6d4f>] kmem_zone_alloc+0x5f/0xe0 > Nov 17 14:07:39 thoregon kernel: [66571.611198] > [<ffffffff811f6de8>] kmem_zone_zalloc+0x18/0x50 > Nov 17 14:07:39 thoregon kernel: [66571.611200] > [<ffffffff8122b0b2>] _xfs_trans_alloc+0x32/0x90 > Nov 17 14:07:39 thoregon kernel: [66571.611202] > [<ffffffff8122b148>] xfs_trans_alloc+0x38/0x50 > Nov 17 14:07:39 thoregon kernel: [66571.611204] > [<ffffffff811f5157>] xfs_create+0x187/0x5a0 > Nov 17 14:07:39 thoregon kernel: [66571.611205] > [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 > Nov 17 14:07:39 thoregon kernel: [66571.611207] > [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 > Nov 17 14:07:39 thoregon kernel: [66571.611209] > [<ffffffff81100322>] vfs_create+0x72/0xc0 > Nov 17 14:07:39 thoregon kernel: [66571.611211] > [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 > Nov 17 14:07:39 thoregon kernel: [66571.611213] > [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 > Nov 17 14:07:39 thoregon kernel: [66571.611215] > [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 > Nov 17 14:07:39 thoregon kernel: [66571.611217] > [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 > Nov 17 14:07:39 thoregon kernel: [66571.611219] > [<ffffffff810f222c>] sys_open+0x1c/0x20 > Nov 17 14:07:39 thoregon kernel: [66571.611221] > [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b > Nov 17 14:07:39 thoregon kernel: [66571.611223] INITIAL USE at: > Nov 17 14:07:39 thoregon kernel: [66571.611224] > [<ffffffff8107ed49>] __lock_acquire+0x2e9/0x1c00 > Nov 17 14:07:39 thoregon kernel: [66571.611225] > [<ffffffff81080b55>] lock_acquire+0x55/0x70 > Nov 17 14:07:39 thoregon kernel: [66571.611227] > [<ffffffff810f451b>] __sb_start_write+0xab/0x190 > Nov 17 14:07:39 thoregon kernel: [66571.611229] > [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50 > Nov 17 14:07:39 thoregon kernel: [66571.611231] > [<ffffffff811f5157>] xfs_create+0x187/0x5a0 > Nov 17 14:07:39 thoregon kernel: [66571.611232] > [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 > Nov 17 14:07:39 thoregon kernel: [66571.611234] > [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 > Nov 17 14:07:39 thoregon kernel: [66571.611236] > [<ffffffff81100322>] vfs_create+0x72/0xc0 > Nov 17 14:07:39 thoregon kernel: [66571.611238] > [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 > Nov 17 14:07:39 thoregon kernel: [66571.611240] > [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 > Nov 17 14:07:39 thoregon kernel: [66571.611242] > [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 > Nov 17 14:07:39 thoregon kernel: [66571.611244] > [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 > Nov 17 14:07:39 thoregon kernel: [66571.611246] > [<ffffffff810f222c>] sys_open+0x1c/0x20 > Nov 17 14:07:39 thoregon kernel: [66571.611248] > [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b > Nov 17 14:07:39 thoregon kernel: [66571.611250] } > Nov 17 14:07:39 thoregon kernel: [66571.611251] ... key at: > [<ffffffff81c34e40>] xfs_fs_type+0x60/0x80 > Nov 17 14:07:39 thoregon kernel: [66571.611254] ... acquired at: > Nov 17 14:07:39 thoregon kernel: [66571.611254] > [<ffffffff8107df3b>] check_usage_forwards+0x10b/0x140 > Nov 17 14:07:39 thoregon kernel: [66571.611256] > [<ffffffff8107e900>] mark_lock+0x190/0x2f0 > Nov 17 14:07:39 thoregon kernel: [66571.611258] > [<ffffffff8107efdf>] __lock_acquire+0x57f/0x1c00 > Nov 17 14:07:39 thoregon kernel: [66571.611260] > [<ffffffff81080b55>] lock_acquire+0x55/0x70 > Nov 17 14:07:39 thoregon kernel: [66571.611261] > [<ffffffff810f451b>] __sb_start_write+0xab/0x190 > Nov 17 14:07:39 thoregon kernel: [66571.611263] > [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50 > Nov 17 14:07:39 thoregon kernel: [66571.611265] > [<ffffffff811f3e64>] xfs_free_eofblocks+0x104/0x250 > Nov 17 14:07:39 thoregon kernel: [66571.611266] > [<ffffffff811f4b39>] xfs_inactive+0xa9/0x480 > Nov 17 14:07:39 thoregon kernel: [66571.611268] > [<ffffffff811efe10>] xfs_fs_evict_inode+0x70/0x80 > Nov 17 14:07:39 thoregon kernel: [66571.611270] > [<ffffffff8110cb7f>] evict+0xaf/0x1b0 > Nov 17 14:07:39 thoregon kernel: [66571.611271] > [<ffffffff8110d209>] dispose_list+0x39/0x50 > Nov 17 14:07:39 thoregon kernel: [66571.611273] > [<ffffffff8110dc63>] prune_icache_sb+0x183/0x340 > Nov 17 14:07:39 thoregon kernel: [66571.611275] > [<ffffffff810f5bc3>] prune_super+0xf3/0x1a0 > Nov 17 14:07:39 thoregon kernel: [66571.611277] > [<ffffffff810bbdee>] shrink_slab+0x11e/0x1f0 > Nov 17 14:07:39 thoregon kernel: [66571.611278] > [<ffffffff810be98f>] try_to_free_pages+0x21f/0x4e0 > Nov 17 14:07:39 thoregon kernel: [66571.611280] > [<ffffffff810b5ec6>] __alloc_pages_nodemask+0x506/0x800 > Nov 17 14:07:39 thoregon kernel: [66571.611282] > [<ffffffff810ce56e>] handle_pte_fault+0x5ae/0x7a0 > Nov 17 14:07:39 thoregon kernel: [66571.611284] > [<ffffffff810cf769>] handle_mm_fault+0x1f9/0x2a0 > Nov 17 14:07:39 thoregon kernel: [66571.611286] > [<ffffffff81029dfc>] __do_page_fault+0x16c/0x480 > Nov 17 14:07:39 thoregon kernel: [66571.611287] > [<ffffffff8102a139>] do_page_fault+0x9/0x10 > Nov 17 14:07:39 thoregon kernel: [66571.611289] > [<ffffffff816b287f>] page_fault+0x1f/0x30 > Nov 17 14:07:39 thoregon kernel: [66571.611291] > Nov 17 14:07:39 thoregon kernel: [66571.611292] > Nov 17 14:07:39 thoregon kernel: [66571.611292] stack backtrace: > Nov 17 14:07:39 thoregon kernel: [66571.611294] Pid: 21330, comm: cc1 > Not tainted 3.7.0-rc5 #1 > Nov 17 14:07:39 thoregon kernel: [66571.611295] Call Trace: > Nov 17 14:07:39 thoregon kernel: [66571.611298] [<ffffffff8107de28>] > print_irq_inversion_bug.part.37+0x1e8/0x1f0 > Nov 17 14:07:39 thoregon kernel: [66571.611300] [<ffffffff8107df3b>] > check_usage_forwards+0x10b/0x140 > Nov 17 14:07:39 thoregon kernel: [66571.611303] [<ffffffff8107e900>] > mark_lock+0x190/0x2f0 > Nov 17 14:07:39 thoregon kernel: [66571.611306] [<ffffffff8150406e>] > ? dm_request+0x2e/0x2a0 > Nov 17 14:07:39 thoregon kernel: [66571.611308] [<ffffffff8107de30>] > ? print_irq_inversion_bug.part.37+0x1f0/0x1f0 > Nov 17 14:07:39 thoregon kernel: [66571.611310] [<ffffffff8107efdf>] > __lock_acquire+0x57f/0x1c00 > Nov 17 14:07:39 thoregon kernel: [66571.611313] [<ffffffff812202f4>] > ? xfs_iext_bno_to_ext+0x84/0x160 > Nov 17 14:07:39 thoregon kernel: [66571.611316] [<ffffffff8120a023>] > ? xfs_bmbt_get_all+0x13/0x20 > Nov 17 14:07:39 thoregon kernel: [66571.611318] [<ffffffff81200fb4>] > ? xfs_bmap_search_multi_extents+0xa4/0x110 > Nov 17 14:07:39 thoregon kernel: [66571.611320] [<ffffffff81080b55>] > lock_acquire+0x55/0x70 > Nov 17 14:07:39 thoregon kernel: [66571.611322] [<ffffffff8122b138>] > ? xfs_trans_alloc+0x28/0x50 > Nov 17 14:07:39 thoregon kernel: [66571.611324] [<ffffffff810f451b>] > __sb_start_write+0xab/0x190 > Nov 17 14:07:39 thoregon kernel: [66571.611326] [<ffffffff8122b138>] > ? xfs_trans_alloc+0x28/0x50 > Nov 17 14:07:39 thoregon kernel: [66571.611328] [<ffffffff8122b138>] > ? xfs_trans_alloc+0x28/0x50 > Nov 17 14:07:39 thoregon kernel: [66571.611330] [<ffffffff8122b138>] > xfs_trans_alloc+0x28/0x50 > Nov 17 14:07:39 thoregon kernel: [66571.611332] [<ffffffff811f3e64>] > xfs_free_eofblocks+0x104/0x250 > Nov 17 14:07:39 thoregon kernel: [66571.611334] [<ffffffff816b204b>] > ? _raw_spin_unlock_irq+0x2b/0x50 > Nov 17 14:07:39 thoregon kernel: [66571.611336] [<ffffffff811f4b39>] > xfs_inactive+0xa9/0x480 > Nov 17 14:07:39 thoregon kernel: [66571.611337] [<ffffffff816b204b>] > ? _raw_spin_unlock_irq+0x2b/0x50 > Nov 17 14:07:39 thoregon kernel: [66571.611340] [<ffffffff811efe10>] > xfs_fs_evict_inode+0x70/0x80 > Nov 17 14:07:39 thoregon kernel: [66571.611342] [<ffffffff8110cb7f>] > evict+0xaf/0x1b0 > Nov 17 14:07:39 thoregon kernel: [66571.611344] [<ffffffff8110d209>] > dispose_list+0x39/0x50 > Nov 17 14:07:39 thoregon kernel: [66571.611346] [<ffffffff8110dc63>] > prune_icache_sb+0x183/0x340 > Nov 17 14:07:39 thoregon kernel: [66571.611347] [<ffffffff810f5bc3>] > prune_super+0xf3/0x1a0 > Nov 17 14:07:39 thoregon kernel: [66571.611349] [<ffffffff810bbdee>] > shrink_slab+0x11e/0x1f0 > Nov 17 14:07:39 thoregon kernel: [66571.611352] [<ffffffff810be98f>] > try_to_free_pages+0x21f/0x4e0 > Nov 17 14:07:39 thoregon kernel: [66571.611354] [<ffffffff810b5ec6>] > __alloc_pages_nodemask+0x506/0x800 > Nov 17 14:07:39 thoregon kernel: [66571.611356] [<ffffffff810b9e40>] > ? lru_deactivate_fn+0x1c0/0x1c0 > Nov 17 14:07:39 thoregon kernel: [66571.611358] [<ffffffff810ce56e>] > handle_pte_fault+0x5ae/0x7a0 > Nov 17 14:07:39 thoregon kernel: [66571.611360] [<ffffffff810cf769>] > handle_mm_fault+0x1f9/0x2a0 > Nov 17 14:07:39 thoregon kernel: [66571.611363] [<ffffffff81029dfc>] > __do_page_fault+0x16c/0x480 > Nov 17 14:07:39 thoregon kernel: [66571.611366] [<ffffffff8129c7ad>] > ? trace_hardirqs_off_thunk+0x3a/0x3c > Nov 17 14:07:39 thoregon kernel: [66571.611368] [<ffffffff8102a139>] > do_page_fault+0x9/0x10 > Nov 17 14:07:39 thoregon kernel: [66571.611370] [<ffffffff816b287f>] > page_fault+0x1f/0x30 ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Hang in XFS reclaim on 3.7.0-rc3 @ 2012-11-18 15:29 ` Torsten Kaiser 0 siblings, 0 replies; 31+ messages in thread From: Torsten Kaiser @ 2012-11-18 15:29 UTC (permalink / raw) To: Dave Chinner; +Cc: Linux Kernel, xfs On Sun, Nov 18, 2012 at 11:24 AM, Torsten Kaiser <just.for.lkml@googlemail.com> wrote: > On Tue, Oct 30, 2012 at 9:37 PM, Torsten Kaiser > <just.for.lkml@googlemail.com> wrote: >> I will keep LOCKDEP enabled on that system, and if there really is >> another splat, I will report back here. But I rather doubt that this >> will be needed. > > After the patch, I did not see this problem again, but today I found > another LOCKDEP report that also looks XFS related. > I found it twice in the logs, and as both were slightly different, I > will attach both versions. > Nov 6 21:57:09 thoregon kernel: [ 9941.104353] 3.7.0-rc4 #1 Not tainted > Nov 6 21:57:09 thoregon kernel: [ 9941.104355] inconsistent > {RECLAIM_FS-ON-W} -> {IN-RECLAIM_FS-W} usage. > Nov 6 21:57:09 thoregon kernel: [ 9941.104430] CPU0 > Nov 6 21:57:09 thoregon kernel: [ 9941.104431] ---- > Nov 6 21:57:09 thoregon kernel: [ 9941.104432] lock(&(&ip->i_lock)->mr_lock); > Nov 6 21:57:09 thoregon kernel: [ 9941.104433] <Interrupt> > Nov 6 21:57:09 thoregon kernel: [ 9941.104434] > lock(&(&ip->i_lock)->mr_lock); > Nov 6 21:57:09 thoregon kernel: [ 9941.104435] > Nov 6 21:57:09 thoregon kernel: [ 9941.104435] *** DEADLOCK *** Sorry! Copied the wrong report. Your fix only landed in -rc5, so my vanilla -rc4 did (also) report the old problem again. And I copy&pasted that report instead of the second appearance of the new problem. Here is the correct second report of the sb_internal vs ip->i_lock->mr_lock problem: [110926.972477] [110926.972482] ========================================================= [110926.972484] [ INFO: possible irq lock inversion dependency detected ] [110926.972486] 3.7.0-rc4 #1 Not tainted [110926.972487] --------------------------------------------------------- [110926.972489] kswapd0/725 just changed the state of lock: [110926.972490] (sb_internal){.+.+.?}, at: [<ffffffff8122b268>] xfs_trans_alloc+0x28/0x50 [110926.972499] but this lock took another, RECLAIM_FS-unsafe lock in the past: [110926.972500] (&(&ip->i_lock)->mr_lock/1){+.+.+.} [110926.972500] [110926.972500] and interrupts could create inverse lock ordering between them. [110926.972500] [110926.972503] [110926.972503] other info that might help us debug this: [110926.972504] Possible interrupt unsafe locking scenario: [110926.972504] [110926.972505] CPU0 CPU1 [110926.972506] ---- ---- [110926.972507] lock(&(&ip->i_lock)->mr_lock/1); [110926.972509] local_irq_disable(); [110926.972509] lock(sb_internal); [110926.972511] lock(&(&ip->i_lock)->mr_lock/1); [110926.972512] <Interrupt> [110926.972513] lock(sb_internal); [110926.972514] [110926.972514] *** DEADLOCK *** [110926.972514] [110926.972516] 2 locks held by kswapd0/725: [110926.972517] #0: (shrinker_rwsem){++++..}, at: [<ffffffff810bbd22>] shrink_slab+0x32/0x1f0 [110926.972522] #1: (&type->s_umount_key#20){++++.+}, at: [<ffffffff810f5a8e>] grab_super_passive+0x3e/0x90 [110926.972527] [110926.972527] the shortest dependencies between 2nd lock and 1st lock: [110926.972533] -> (&(&ip->i_lock)->mr_lock/1){+.+.+.} ops: 58117 { [110926.972536] HARDIRQ-ON-W at: [110926.972537] [<ffffffff8107f091>] __lock_acquire+0x631/0x1c00 [110926.972540] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [110926.972542] [<ffffffff8106126a>] down_write_nested+0x4a/0x70 [110926.972545] [<ffffffff811e8164>] xfs_ilock+0x84/0xb0 [110926.972548] [<ffffffff811f5194>] xfs_create+0x1d4/0x5a0 [110926.972550] [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0 [110926.972552] [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10 [110926.972554] [<ffffffff81100332>] vfs_create+0x72/0xc0 [110926.972556] [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80 [110926.972558] [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490 [110926.972560] [<ffffffff8110184d>] do_filp_open+0x3d/0xa0 [110926.972562] [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0 [110926.972565] [<ffffffff810f223c>] sys_open+0x1c/0x20 [110926.972567] [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b [110926.972570] SOFTIRQ-ON-W at: [110926.972571] [<ffffffff8107f0c7>] __lock_acquire+0x667/0x1c00 [110926.972573] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [110926.972574] [<ffffffff8106126a>] down_write_nested+0x4a/0x70 [110926.972576] [<ffffffff811e8164>] xfs_ilock+0x84/0xb0 [110926.972578] [<ffffffff811f5194>] xfs_create+0x1d4/0x5a0 [110926.972580] [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0 [110926.972581] [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10 [110926.972583] [<ffffffff81100332>] vfs_create+0x72/0xc0 [110926.972585] [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80 [110926.972587] [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490 [110926.972589] [<ffffffff8110184d>] do_filp_open+0x3d/0xa0 [110926.972591] [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0 [110926.972593] [<ffffffff810f223c>] sys_open+0x1c/0x20 [110926.972595] [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b [110926.972597] RECLAIM_FS-ON-W at: [110926.972598] [<ffffffff8108137e>] mark_held_locks+0x7e/0x130 [110926.972600] [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0 [110926.972601] [<ffffffff810e9dd5>] kmem_cache_alloc+0x35/0xe0 [110926.972603] [<ffffffff810dba31>] vm_map_ram+0x271/0x770 [110926.972606] [<ffffffff811e1316>] _xfs_buf_map_pages+0x46/0xe0 [110926.972609] [<ffffffff811e222a>] xfs_buf_get_map+0x8a/0x130 [110926.972610] [<ffffffff81233ab9>] xfs_trans_get_buf_map+0xa9/0xd0 [110926.972613] [<ffffffff8121bced>] xfs_ialloc_inode_init+0xcd/0x1d0 [110926.972616] [<ffffffff8121c25e>] xfs_ialloc_ag_alloc+0x1be/0x560 [110926.972618] [<ffffffff8121da65>] xfs_dialloc+0x185/0x2a0 [110926.972619] [<ffffffff8121f198>] xfs_ialloc+0x58/0x650 [110926.972621] [<ffffffff811f3985>] xfs_dir_ialloc+0x65/0x270 [110926.972623] [<ffffffff811f536c>] xfs_create+0x3ac/0x5a0 [110926.972624] [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0 [110926.972626] [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10 [110926.972628] [<ffffffff81100332>] vfs_create+0x72/0xc0 [110926.972630] [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80 [110926.972632] [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490 [110926.972634] [<ffffffff8110184d>] do_filp_open+0x3d/0xa0 [110926.972636] [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0 [110926.972638] [<ffffffff810f223c>] sys_open+0x1c/0x20 [110926.972640] [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b [110926.972642] INITIAL USE at: [110926.972642] [<ffffffff8107ed49>] __lock_acquire+0x2e9/0x1c00 [110926.972644] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [110926.972646] [<ffffffff8106126a>] down_write_nested+0x4a/0x70 [110926.972648] [<ffffffff811e8164>] xfs_ilock+0x84/0xb0 [110926.972650] [<ffffffff811f5194>] xfs_create+0x1d4/0x5a0 [110926.972651] [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0 [110926.972653] [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10 [110926.972655] [<ffffffff81100332>] vfs_create+0x72/0xc0 [110926.972657] [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80 [110926.972659] [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490 [110926.972661] [<ffffffff8110184d>] do_filp_open+0x3d/0xa0 [110926.972663] [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0 [110926.972664] [<ffffffff810f223c>] sys_open+0x1c/0x20 [110926.972666] [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b [110926.972668] } [110926.972669] ... key at: [<ffffffff825b4b81>] __key.41355+0x1/0x8 [110926.972672] ... acquired at: [110926.972672] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [110926.972674] [<ffffffff8106126a>] down_write_nested+0x4a/0x70 [110926.972676] [<ffffffff811e8164>] xfs_ilock+0x84/0xb0 [110926.972678] [<ffffffff811f5194>] xfs_create+0x1d4/0x5a0 [110926.972679] [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0 [110926.972681] [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10 [110926.972683] [<ffffffff81100332>] vfs_create+0x72/0xc0 [110926.972684] [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80 [110926.972686] [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490 [110926.972688] [<ffffffff8110184d>] do_filp_open+0x3d/0xa0 [110926.972690] [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0 [110926.972692] [<ffffffff810f223c>] sys_open+0x1c/0x20 [110926.972694] [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b [110926.972696] [110926.972696] -> (sb_internal){.+.+.?} ops: 1710064 { [110926.972699] HARDIRQ-ON-R at: [110926.972700] [<ffffffff8107ef6a>] __lock_acquire+0x50a/0x1c00 [110926.972702] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [110926.972704] [<ffffffff810f452b>] __sb_start_write+0xab/0x190 [110926.972705] [<ffffffff8122b268>] xfs_trans_alloc+0x28/0x50 [110926.972707] [<ffffffff811f5147>] xfs_create+0x187/0x5a0 [110926.972709] [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0 [110926.972711] [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10 [110926.972712] [<ffffffff81100332>] vfs_create+0x72/0xc0 [110926.972714] [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80 [110926.972716] [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490 [110926.972718] [<ffffffff8110184d>] do_filp_open+0x3d/0xa0 [110926.972720] [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0 [110926.972722] [<ffffffff810f223c>] sys_open+0x1c/0x20 [110926.972724] [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b [110926.972726] SOFTIRQ-ON-R at: [110926.972727] [<ffffffff8107f0c7>] __lock_acquire+0x667/0x1c00 [110926.972728] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [110926.972730] [<ffffffff810f452b>] __sb_start_write+0xab/0x190 [110926.972732] [<ffffffff8122b268>] xfs_trans_alloc+0x28/0x50 [110926.972734] [<ffffffff811f5147>] xfs_create+0x187/0x5a0 [110926.972735] [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0 [110926.972737] [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10 [110926.972739] [<ffffffff81100332>] vfs_create+0x72/0xc0 [110926.972741] [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80 [110926.972743] [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490 [110926.972745] [<ffffffff8110184d>] do_filp_open+0x3d/0xa0 [110926.972747] [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0 [110926.972749] [<ffffffff810f223c>] sys_open+0x1c/0x20 [110926.972750] [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b [110926.972752] IN-RECLAIM_FS-R at: [110926.972753] [<ffffffff8107efdf>] __lock_acquire+0x57f/0x1c00 [110926.972755] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [110926.972757] [<ffffffff810f452b>] __sb_start_write+0xab/0x190 [110926.972758] [<ffffffff8122b268>] xfs_trans_alloc+0x28/0x50 [110926.972760] [<ffffffff811f3e54>] xfs_free_eofblocks+0x104/0x250 [110926.972762] [<ffffffff811f4b29>] xfs_inactive+0xa9/0x480 [110926.972763] [<ffffffff811efe00>] xfs_fs_evict_inode+0x70/0x80 [110926.972765] [<ffffffff8110cb8f>] evict+0xaf/0x1b0 [110926.972768] [<ffffffff8110d219>] dispose_list+0x39/0x50 [110926.972770] [<ffffffff8110dc73>] prune_icache_sb+0x183/0x340 [110926.972772] [<ffffffff810f5bd3>] prune_super+0xf3/0x1a0 [110926.972773] [<ffffffff810bbe0e>] shrink_slab+0x11e/0x1f0 [110926.972775] [<ffffffff810be400>] kswapd+0x690/0xa10 [110926.972777] [<ffffffff8105c246>] kthread+0xd6/0xe0 [110926.972779] [<ffffffff816b2c6c>] ret_from_fork+0x7c/0xb0 [110926.972781] RECLAIM_FS-ON-R at: [110926.972782] [<ffffffff8108137e>] mark_held_locks+0x7e/0x130 [110926.972784] [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0 [110926.972785] [<ffffffff810e9dd5>] kmem_cache_alloc+0x35/0xe0 [110926.972787] [<ffffffff811f6d3f>] kmem_zone_alloc+0x5f/0xe0 [110926.972789] [<ffffffff811f6dd8>] kmem_zone_zalloc+0x18/0x50 [110926.972790] [<ffffffff8122b1e2>] _xfs_trans_alloc+0x32/0x90 [110926.972792] [<ffffffff8122b278>] xfs_trans_alloc+0x38/0x50 [110926.972794] [<ffffffff811f5147>] xfs_create+0x187/0x5a0 [110926.972796] [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0 [110926.972797] [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10 [110926.972799] [<ffffffff81100332>] vfs_create+0x72/0xc0 [110926.972801] [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80 [110926.972803] [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490 [110926.972805] [<ffffffff8110184d>] do_filp_open+0x3d/0xa0 [110926.972807] [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0 [110926.972809] [<ffffffff810f223c>] sys_open+0x1c/0x20 [110926.972811] [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b [110926.972813] INITIAL USE at: [110926.972814] [<ffffffff8107ed49>] __lock_acquire+0x2e9/0x1c00 [110926.972815] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [110926.972817] [<ffffffff810f452b>] __sb_start_write+0xab/0x190 [110926.972819] [<ffffffff8122b268>] xfs_trans_alloc+0x28/0x50 [110926.972820] [<ffffffff811f5147>] xfs_create+0x187/0x5a0 [110926.972822] [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0 [110926.972824] [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10 [110926.972826] [<ffffffff81100332>] vfs_create+0x72/0xc0 [110926.972827] [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80 [110926.972829] [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490 [110926.972831] [<ffffffff8110184d>] do_filp_open+0x3d/0xa0 [110926.972833] [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0 [110926.972835] [<ffffffff810f223c>] sys_open+0x1c/0x20 [110926.972837] [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b [110926.972839] } [110926.972840] ... key at: [<ffffffff81c34e40>] xfs_fs_type+0x60/0x80 [110926.972842] ... acquired at: [110926.972843] [<ffffffff8107df3b>] check_usage_forwards+0x10b/0x140 [110926.972845] [<ffffffff8107e900>] mark_lock+0x190/0x2f0 [110926.972846] [<ffffffff8107efdf>] __lock_acquire+0x57f/0x1c00 [110926.972848] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [110926.972850] [<ffffffff810f452b>] __sb_start_write+0xab/0x190 [110926.972851] [<ffffffff8122b268>] xfs_trans_alloc+0x28/0x50 [110926.972853] [<ffffffff811f3e54>] xfs_free_eofblocks+0x104/0x250 [110926.972855] [<ffffffff811f4b29>] xfs_inactive+0xa9/0x480 [110926.972856] [<ffffffff811efe00>] xfs_fs_evict_inode+0x70/0x80 [110926.972858] [<ffffffff8110cb8f>] evict+0xaf/0x1b0 [110926.972860] [<ffffffff8110d219>] dispose_list+0x39/0x50 [110926.972861] [<ffffffff8110dc73>] prune_icache_sb+0x183/0x340 [110926.972863] [<ffffffff810f5bd3>] prune_super+0xf3/0x1a0 [110926.972865] [<ffffffff810bbe0e>] shrink_slab+0x11e/0x1f0 [110926.972866] [<ffffffff810be400>] kswapd+0x690/0xa10 [110926.972868] [<ffffffff8105c246>] kthread+0xd6/0xe0 [110926.972870] [<ffffffff816b2c6c>] ret_from_fork+0x7c/0xb0 [110926.972871] [110926.972872] [110926.972872] stack backtrace: [110926.972874] Pid: 725, comm: kswapd0 Not tainted 3.7.0-rc4 #1 [110926.972875] Call Trace: [110926.972878] [<ffffffff8107de28>] print_irq_inversion_bug.part.37+0x1e8/0x1f0 [110926.972880] [<ffffffff8107df3b>] check_usage_forwards+0x10b/0x140 [110926.972883] [<ffffffff8107e900>] mark_lock+0x190/0x2f0 [110926.972885] [<ffffffff8107de30>] ? print_irq_inversion_bug.part.37+0x1f0/0x1f0 [110926.972887] [<ffffffff8107efdf>] __lock_acquire+0x57f/0x1c00 [110926.972889] [<ffffffff81220424>] ? xfs_iext_bno_to_ext+0x84/0x160 [110926.972892] [<ffffffff8120a0e3>] ? xfs_bmbt_get_all+0x13/0x20 [110926.972895] [<ffffffff81201104>] ? xfs_bmap_search_multi_extents+0xa4/0x110 [110926.972897] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [110926.972899] [<ffffffff8122b268>] ? xfs_trans_alloc+0x28/0x50 [110926.972901] [<ffffffff810f452b>] __sb_start_write+0xab/0x190 [110926.972903] [<ffffffff8122b268>] ? xfs_trans_alloc+0x28/0x50 [110926.972905] [<ffffffff8122b268>] ? xfs_trans_alloc+0x28/0x50 [110926.972907] [<ffffffff8122b268>] xfs_trans_alloc+0x28/0x50 [110926.972908] [<ffffffff811f3e54>] xfs_free_eofblocks+0x104/0x250 [110926.972910] [<ffffffff816b1efb>] ? _raw_spin_unlock_irq+0x2b/0x50 [110926.972912] [<ffffffff811f4b29>] xfs_inactive+0xa9/0x480 [110926.972914] [<ffffffff816b1efb>] ? _raw_spin_unlock_irq+0x2b/0x50 [110926.972916] [<ffffffff811efe00>] xfs_fs_evict_inode+0x70/0x80 [110926.972918] [<ffffffff8110cb8f>] evict+0xaf/0x1b0 [110926.972920] [<ffffffff8110d219>] dispose_list+0x39/0x50 [110926.972922] [<ffffffff8110dc73>] prune_icache_sb+0x183/0x340 [110926.972924] [<ffffffff810f5bd3>] prune_super+0xf3/0x1a0 [110926.972926] [<ffffffff810bbe0e>] shrink_slab+0x11e/0x1f0 [110926.972928] [<ffffffff810be400>] kswapd+0x690/0xa10 [110926.972930] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60 [110926.972932] [<ffffffff810bdd70>] ? shrink_lruvec+0x540/0x540 [110926.972934] [<ffffffff8105c246>] kthread+0xd6/0xe0 [110926.972936] [<ffffffff816b1efb>] ? _raw_spin_unlock_irq+0x2b/0x50 [110926.972938] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 [110926.972940] [<ffffffff816b2c6c>] ret_from_fork+0x7c/0xb0 [110926.972942] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 > Nov 17 14:07:38 thoregon kernel: [66571.610863] > Nov 17 14:07:38 thoregon kernel: [66571.610869] > ========================================================= > Nov 17 14:07:38 thoregon kernel: [66571.610870] [ INFO: possible irq > lock inversion dependency detected ] > Nov 17 14:07:38 thoregon kernel: [66571.610873] 3.7.0-rc5 #1 Not tainted > Nov 17 14:07:38 thoregon kernel: [66571.610874] > --------------------------------------------------------- > Nov 17 14:07:38 thoregon kernel: [66571.610875] cc1/21330 just changed > the state of lock: > Nov 17 14:07:38 thoregon kernel: [66571.610877] > (sb_internal){.+.+.?}, at: [<ffffffff8122b138>] > xfs_trans_alloc+0x28/0x50 > Nov 17 14:07:38 thoregon kernel: [66571.610885] but this lock took > another, RECLAIM_FS-unsafe lock in the past: > Nov 17 14:07:38 thoregon kernel: [66571.610886] > (&(&ip->i_lock)->mr_lock/1){+.+.+.} > Nov 17 14:07:38 thoregon kernel: [66571.610886] > Nov 17 14:07:39 thoregon kernel: [66571.610886] and interrupts could > create inverse lock ordering between them. > Nov 17 14:07:39 thoregon kernel: [66571.610886] > Nov 17 14:07:39 thoregon kernel: [66571.610890] > Nov 17 14:07:39 thoregon kernel: [66571.610890] other info that might > help us debug this: > Nov 17 14:07:39 thoregon kernel: [66571.610891] Possible interrupt > unsafe locking scenario: > Nov 17 14:07:39 thoregon kernel: [66571.610891] > Nov 17 14:07:39 thoregon kernel: [66571.610892] CPU0 > CPU1 > Nov 17 14:07:39 thoregon kernel: [66571.610893] ---- > ---- > Nov 17 14:07:39 thoregon kernel: [66571.610894] > lock(&(&ip->i_lock)->mr_lock/1); > Nov 17 14:07:39 thoregon kernel: [66571.610896] > local_irq_disable(); > Nov 17 14:07:39 thoregon kernel: [66571.610897] > lock(sb_internal); > Nov 17 14:07:39 thoregon kernel: [66571.610898] > lock(&(&ip->i_lock)->mr_lock/1); > Nov 17 14:07:39 thoregon kernel: [66571.610900] <Interrupt> > Nov 17 14:07:39 thoregon kernel: [66571.610901] lock(sb_internal); > Nov 17 14:07:39 thoregon kernel: [66571.610902] > Nov 17 14:07:39 thoregon kernel: [66571.610902] *** DEADLOCK *** > Nov 17 14:07:39 thoregon kernel: [66571.610902] > Nov 17 14:07:39 thoregon kernel: [66571.610904] 3 locks held by cc1/21330: > Nov 17 14:07:39 thoregon kernel: [66571.610905] #0: > (&mm->mmap_sem){++++++}, at: [<ffffffff81029d8b>] > __do_page_fault+0xfb/0x480 > Nov 17 14:07:39 thoregon kernel: [66571.610910] #1: > (shrinker_rwsem){++++..}, at: [<ffffffff810bbd02>] > shrink_slab+0x32/0x1f0 > Nov 17 14:07:39 thoregon kernel: [66571.610915] #2: > (&type->s_umount_key#20){++++.+}, at: [<ffffffff810f5a7e>] > grab_super_passive+0x3e/0x90 > Nov 17 14:07:39 thoregon kernel: [66571.610921] > Nov 17 14:07:39 thoregon kernel: [66571.610921] the shortest > dependencies between 2nd lock and 1st lock: > Nov 17 14:07:39 thoregon kernel: [66571.610927] -> > (&(&ip->i_lock)->mr_lock/1){+.+.+.} ops: 169649 { > Nov 17 14:07:39 thoregon kernel: [66571.610931] HARDIRQ-ON-W at: > Nov 17 14:07:39 thoregon kernel: [66571.610932] > [<ffffffff8107f091>] __lock_acquire+0x631/0x1c00 > Nov 17 14:07:39 thoregon kernel: [66571.610935] > [<ffffffff81080b55>] lock_acquire+0x55/0x70 > Nov 17 14:07:39 thoregon kernel: [66571.610937] > [<ffffffff8106126a>] down_write_nested+0x4a/0x70 > Nov 17 14:07:39 thoregon kernel: [66571.610941] > [<ffffffff811e8164>] xfs_ilock+0x84/0xb0 > Nov 17 14:07:39 thoregon kernel: [66571.610944] > [<ffffffff811f51a4>] xfs_create+0x1d4/0x5a0 > Nov 17 14:07:39 thoregon kernel: [66571.610946] > [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 > Nov 17 14:07:39 thoregon kernel: [66571.610948] > [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 > Nov 17 14:07:39 thoregon kernel: [66571.610950] > [<ffffffff81100322>] vfs_create+0x72/0xc0 > Nov 17 14:07:39 thoregon kernel: [66571.610953] > [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 > Nov 17 14:07:39 thoregon kernel: [66571.610955] > [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 > Nov 17 14:07:39 thoregon kernel: [66571.610957] > [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 > Nov 17 14:07:39 thoregon kernel: [66571.610959] > [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 > Nov 17 14:07:39 thoregon kernel: [66571.610962] > [<ffffffff810f222c>] sys_open+0x1c/0x20 > Nov 17 14:07:39 thoregon kernel: [66571.610964] > [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b > Nov 17 14:07:39 thoregon kernel: [66571.610967] SOFTIRQ-ON-W at: > Nov 17 14:07:39 thoregon kernel: [66571.610968] > [<ffffffff8107f0c7>] __lock_acquire+0x667/0x1c00 > Nov 17 14:07:39 thoregon kernel: [66571.610970] > [<ffffffff81080b55>] lock_acquire+0x55/0x70 > Nov 17 14:07:39 thoregon kernel: [66571.610972] > [<ffffffff8106126a>] down_write_nested+0x4a/0x70 > Nov 17 14:07:39 thoregon kernel: [66571.610974] > [<ffffffff811e8164>] xfs_ilock+0x84/0xb0 > Nov 17 14:07:39 thoregon kernel: [66571.610976] > [<ffffffff811f51a4>] xfs_create+0x1d4/0x5a0 > Nov 17 14:07:39 thoregon kernel: [66571.610977] > [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 > Nov 17 14:07:39 thoregon kernel: [66571.610979] > [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 > Nov 17 14:07:39 thoregon kernel: [66571.610981] > [<ffffffff81100322>] vfs_create+0x72/0xc0 > Nov 17 14:07:39 thoregon kernel: [66571.610983] > [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 > Nov 17 14:07:39 thoregon kernel: [66571.610985] > [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 > Nov 17 14:07:39 thoregon kernel: [66571.610987] > [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 > Nov 17 14:07:39 thoregon kernel: [66571.610989] > [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 > Nov 17 14:07:39 thoregon kernel: [66571.610991] > [<ffffffff810f222c>] sys_open+0x1c/0x20 > Nov 17 14:07:39 thoregon kernel: [66571.610993] > [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b > Nov 17 14:07:39 thoregon kernel: [66571.610995] RECLAIM_FS-ON-W at: > Nov 17 14:07:39 thoregon kernel: [66571.610996] > [<ffffffff8108137e>] mark_held_locks+0x7e/0x130 > Nov 17 14:07:39 thoregon kernel: [66571.610998] > [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0 > Nov 17 14:07:39 thoregon kernel: [66571.610999] > [<ffffffff810e9dc5>] kmem_cache_alloc+0x35/0xe0 > Nov 17 14:07:39 thoregon kernel: [66571.611002] > [<ffffffff810dba21>] vm_map_ram+0x271/0x770 > Nov 17 14:07:39 thoregon kernel: [66571.611004] > [<ffffffff811e12b6>] _xfs_buf_map_pages+0x46/0xe0 > Nov 17 14:07:39 thoregon kernel: [66571.611008] > [<ffffffff811e21ca>] xfs_buf_get_map+0x8a/0x130 > Nov 17 14:07:39 thoregon kernel: [66571.611009] > [<ffffffff81233989>] xfs_trans_get_buf_map+0xa9/0xd0 > Nov 17 14:07:39 thoregon kernel: [66571.611011] > [<ffffffff8121bc2d>] xfs_ialloc_inode_init+0xcd/0x1d0 > Nov 17 14:07:39 thoregon kernel: [66571.611015] > [<ffffffff8121c16f>] xfs_ialloc_ag_alloc+0x18f/0x500 > Nov 17 14:07:39 thoregon kernel: [66571.611017] > [<ffffffff8121d955>] xfs_dialloc+0x185/0x2a0 > Nov 17 14:07:39 thoregon kernel: [66571.611019] > [<ffffffff8121f068>] xfs_ialloc+0x58/0x650 > Nov 17 14:07:39 thoregon kernel: [66571.611021] > [<ffffffff811f3995>] xfs_dir_ialloc+0x65/0x270 > Nov 17 14:07:39 thoregon kernel: [66571.611023] > [<ffffffff811f537c>] xfs_create+0x3ac/0x5a0 > Nov 17 14:07:39 thoregon kernel: [66571.611024] > [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 > Nov 17 14:07:39 thoregon kernel: [66571.611026] > [<ffffffff811ecb61>] xfs_vn_mkdir+0x11/0x20 > Nov 17 14:07:39 thoregon kernel: [66571.611028] > [<ffffffff8110016f>] vfs_mkdir+0x7f/0xd0 > Nov 17 14:07:39 thoregon kernel: [66571.611030] > [<ffffffff81101b83>] sys_mkdirat+0x43/0x80 > Nov 17 14:07:39 thoregon kernel: [66571.611032] > [<ffffffff81101bd4>] sys_mkdir+0x14/0x20 > Nov 17 14:07:39 thoregon kernel: [66571.611034] > [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b > Nov 17 14:07:39 thoregon kernel: [66571.611036] INITIAL USE at: > Nov 17 14:07:39 thoregon kernel: [66571.611037] > [<ffffffff8107ed49>] __lock_acquire+0x2e9/0x1c00 > Nov 17 14:07:39 thoregon kernel: [66571.611038] > [<ffffffff81080b55>] lock_acquire+0x55/0x70 > Nov 17 14:07:39 thoregon kernel: [66571.611040] > [<ffffffff8106126a>] down_write_nested+0x4a/0x70 > Nov 17 14:07:39 thoregon kernel: [66571.611042] > [<ffffffff811e8164>] xfs_ilock+0x84/0xb0 > Nov 17 14:07:39 thoregon kernel: [66571.611044] > [<ffffffff811f51a4>] xfs_create+0x1d4/0x5a0 > Nov 17 14:07:39 thoregon kernel: [66571.611046] > [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 > Nov 17 14:07:39 thoregon kernel: [66571.611047] > [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 > Nov 17 14:07:39 thoregon kernel: [66571.611049] > [<ffffffff81100322>] vfs_create+0x72/0xc0 > Nov 17 14:07:39 thoregon kernel: [66571.611051] > [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 > Nov 17 14:07:39 thoregon kernel: [66571.611053] > [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 > Nov 17 14:07:39 thoregon kernel: [66571.611055] > [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 > Nov 17 14:07:39 thoregon kernel: [66571.611057] > [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 > Nov 17 14:07:39 thoregon kernel: [66571.611059] > [<ffffffff810f222c>] sys_open+0x1c/0x20 > Nov 17 14:07:39 thoregon kernel: [66571.611061] > [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b > Nov 17 14:07:39 thoregon kernel: [66571.611063] } > Nov 17 14:07:39 thoregon kernel: [66571.611064] ... key at: > [<ffffffff825b4b81>] __key.41357+0x1/0x8 > Nov 17 14:07:39 thoregon kernel: [66571.611066] ... acquired at: > Nov 17 14:07:39 thoregon kernel: [66571.611067] > [<ffffffff81080b55>] lock_acquire+0x55/0x70 > Nov 17 14:07:39 thoregon kernel: [66571.611069] > [<ffffffff8106126a>] down_write_nested+0x4a/0x70 > Nov 17 14:07:39 thoregon kernel: [66571.611071] > [<ffffffff811e8164>] xfs_ilock+0x84/0xb0 > Nov 17 14:07:39 thoregon kernel: [66571.611073] > [<ffffffff811f51a4>] xfs_create+0x1d4/0x5a0 > Nov 17 14:07:39 thoregon kernel: [66571.611074] > [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 > Nov 17 14:07:39 thoregon kernel: [66571.611076] > [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 > Nov 17 14:07:39 thoregon kernel: [66571.611078] > [<ffffffff81100322>] vfs_create+0x72/0xc0 > Nov 17 14:07:39 thoregon kernel: [66571.611080] > [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 > Nov 17 14:07:39 thoregon kernel: [66571.611082] > [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 > Nov 17 14:07:39 thoregon kernel: [66571.611084] > [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 > Nov 17 14:07:39 thoregon kernel: [66571.611086] > [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 > Nov 17 14:07:39 thoregon kernel: [66571.611088] > [<ffffffff810f222c>] sys_open+0x1c/0x20 > Nov 17 14:07:39 thoregon kernel: [66571.611090] > [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b > Nov 17 14:07:39 thoregon kernel: [66571.611091] > Nov 17 14:07:39 thoregon kernel: [66571.611092] -> > (sb_internal){.+.+.?} ops: 1341531 { > Nov 17 14:07:39 thoregon kernel: [66571.611095] HARDIRQ-ON-R at: > Nov 17 14:07:39 thoregon kernel: [66571.611096] > [<ffffffff8107ef6a>] __lock_acquire+0x50a/0x1c00 > Nov 17 14:07:39 thoregon kernel: [66571.611098] > [<ffffffff81080b55>] lock_acquire+0x55/0x70 > Nov 17 14:07:39 thoregon kernel: [66571.611100] > [<ffffffff810f451b>] __sb_start_write+0xab/0x190 > Nov 17 14:07:39 thoregon kernel: [66571.611102] > [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50 > Nov 17 14:07:39 thoregon kernel: [66571.611104] > [<ffffffff811f5157>] xfs_create+0x187/0x5a0 > Nov 17 14:07:39 thoregon kernel: [66571.611105] > [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 > Nov 17 14:07:39 thoregon kernel: [66571.611107] > [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 > Nov 17 14:07:39 thoregon kernel: [66571.611109] > [<ffffffff81100322>] vfs_create+0x72/0xc0 > Nov 17 14:07:39 thoregon kernel: [66571.611111] > [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 > Nov 17 14:07:39 thoregon kernel: [66571.611113] > [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 > Nov 17 14:07:39 thoregon kernel: [66571.611115] > [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 > Nov 17 14:07:39 thoregon kernel: [66571.611117] > [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 > Nov 17 14:07:39 thoregon kernel: [66571.611119] > [<ffffffff810f222c>] sys_open+0x1c/0x20 > Nov 17 14:07:39 thoregon kernel: [66571.611121] > [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b > Nov 17 14:07:39 thoregon kernel: [66571.611123] SOFTIRQ-ON-R at: > Nov 17 14:07:39 thoregon kernel: [66571.611124] > [<ffffffff8107f0c7>] __lock_acquire+0x667/0x1c00 > Nov 17 14:07:39 thoregon kernel: [66571.611126] > [<ffffffff81080b55>] lock_acquire+0x55/0x70 > Nov 17 14:07:39 thoregon kernel: [66571.611128] > [<ffffffff810f451b>] __sb_start_write+0xab/0x190 > Nov 17 14:07:39 thoregon kernel: [66571.611130] > [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50 > Nov 17 14:07:39 thoregon kernel: [66571.611132] > [<ffffffff811f5157>] xfs_create+0x187/0x5a0 > Nov 17 14:07:39 thoregon kernel: [66571.611133] > [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 > Nov 17 14:07:39 thoregon kernel: [66571.611135] > [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 > Nov 17 14:07:39 thoregon kernel: [66571.611137] > [<ffffffff81100322>] vfs_create+0x72/0xc0 > Nov 17 14:07:39 thoregon kernel: [66571.611139] > [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 > Nov 17 14:07:39 thoregon kernel: [66571.611141] > [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 > Nov 17 14:07:39 thoregon kernel: [66571.611143] > [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 > Nov 17 14:07:39 thoregon kernel: [66571.611145] > [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 > Nov 17 14:07:39 thoregon kernel: [66571.611147] > [<ffffffff810f222c>] sys_open+0x1c/0x20 > Nov 17 14:07:39 thoregon kernel: [66571.611149] > [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b > Nov 17 14:07:39 thoregon kernel: [66571.611151] IN-RECLAIM_FS-R at: > Nov 17 14:07:39 thoregon kernel: [66571.611152] > [<ffffffff8107efdf>] __lock_acquire+0x57f/0x1c00 > Nov 17 14:07:39 thoregon kernel: [66571.611154] > [<ffffffff81080b55>] lock_acquire+0x55/0x70 > Nov 17 14:07:39 thoregon kernel: [66571.611156] > [<ffffffff810f451b>] __sb_start_write+0xab/0x190 > Nov 17 14:07:39 thoregon kernel: [66571.611158] > [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50 > Nov 17 14:07:39 thoregon kernel: [66571.611159] > [<ffffffff811f3e64>] xfs_free_eofblocks+0x104/0x250 > Nov 17 14:07:39 thoregon kernel: [66571.611161] > [<ffffffff811f4b39>] xfs_inactive+0xa9/0x480 > Nov 17 14:07:39 thoregon kernel: [66571.611163] > [<ffffffff811efe10>] xfs_fs_evict_inode+0x70/0x80 > Nov 17 14:07:39 thoregon kernel: [66571.611165] > [<ffffffff8110cb7f>] evict+0xaf/0x1b0 > Nov 17 14:07:39 thoregon kernel: [66571.611168] > [<ffffffff8110d209>] dispose_list+0x39/0x50 > Nov 17 14:07:39 thoregon kernel: [66571.611170] > [<ffffffff8110dc63>] prune_icache_sb+0x183/0x340 > Nov 17 14:07:39 thoregon kernel: [66571.611172] > [<ffffffff810f5bc3>] prune_super+0xf3/0x1a0 > Nov 17 14:07:39 thoregon kernel: [66571.611173] > [<ffffffff810bbdee>] shrink_slab+0x11e/0x1f0 > Nov 17 14:07:39 thoregon kernel: [66571.611175] > [<ffffffff810be98f>] try_to_free_pages+0x21f/0x4e0 > Nov 17 14:07:39 thoregon kernel: [66571.611177] > [<ffffffff810b5ec6>] __alloc_pages_nodemask+0x506/0x800 > Nov 17 14:07:39 thoregon kernel: [66571.611179] > [<ffffffff810ce56e>] handle_pte_fault+0x5ae/0x7a0 > Nov 17 14:07:39 thoregon kernel: [66571.611182] > [<ffffffff810cf769>] handle_mm_fault+0x1f9/0x2a0 > Nov 17 14:07:39 thoregon kernel: [66571.611184] > [<ffffffff81029dfc>] __do_page_fault+0x16c/0x480 > Nov 17 14:07:39 thoregon kernel: [66571.611186] > [<ffffffff8102a139>] do_page_fault+0x9/0x10 > Nov 17 14:07:39 thoregon kernel: [66571.611188] > [<ffffffff816b287f>] page_fault+0x1f/0x30 > Nov 17 14:07:39 thoregon kernel: [66571.611190] RECLAIM_FS-ON-R at: > Nov 17 14:07:39 thoregon kernel: [66571.611191] > [<ffffffff8108137e>] mark_held_locks+0x7e/0x130 > Nov 17 14:07:39 thoregon kernel: [66571.611193] > [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0 > Nov 17 14:07:39 thoregon kernel: [66571.611195] > [<ffffffff810e9dc5>] kmem_cache_alloc+0x35/0xe0 > Nov 17 14:07:39 thoregon kernel: [66571.611197] > [<ffffffff811f6d4f>] kmem_zone_alloc+0x5f/0xe0 > Nov 17 14:07:39 thoregon kernel: [66571.611198] > [<ffffffff811f6de8>] kmem_zone_zalloc+0x18/0x50 > Nov 17 14:07:39 thoregon kernel: [66571.611200] > [<ffffffff8122b0b2>] _xfs_trans_alloc+0x32/0x90 > Nov 17 14:07:39 thoregon kernel: [66571.611202] > [<ffffffff8122b148>] xfs_trans_alloc+0x38/0x50 > Nov 17 14:07:39 thoregon kernel: [66571.611204] > [<ffffffff811f5157>] xfs_create+0x187/0x5a0 > Nov 17 14:07:39 thoregon kernel: [66571.611205] > [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 > Nov 17 14:07:39 thoregon kernel: [66571.611207] > [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 > Nov 17 14:07:39 thoregon kernel: [66571.611209] > [<ffffffff81100322>] vfs_create+0x72/0xc0 > Nov 17 14:07:39 thoregon kernel: [66571.611211] > [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 > Nov 17 14:07:39 thoregon kernel: [66571.611213] > [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 > Nov 17 14:07:39 thoregon kernel: [66571.611215] > [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 > Nov 17 14:07:39 thoregon kernel: [66571.611217] > [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 > Nov 17 14:07:39 thoregon kernel: [66571.611219] > [<ffffffff810f222c>] sys_open+0x1c/0x20 > Nov 17 14:07:39 thoregon kernel: [66571.611221] > [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b > Nov 17 14:07:39 thoregon kernel: [66571.611223] INITIAL USE at: > Nov 17 14:07:39 thoregon kernel: [66571.611224] > [<ffffffff8107ed49>] __lock_acquire+0x2e9/0x1c00 > Nov 17 14:07:39 thoregon kernel: [66571.611225] > [<ffffffff81080b55>] lock_acquire+0x55/0x70 > Nov 17 14:07:39 thoregon kernel: [66571.611227] > [<ffffffff810f451b>] __sb_start_write+0xab/0x190 > Nov 17 14:07:39 thoregon kernel: [66571.611229] > [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50 > Nov 17 14:07:39 thoregon kernel: [66571.611231] > [<ffffffff811f5157>] xfs_create+0x187/0x5a0 > Nov 17 14:07:39 thoregon kernel: [66571.611232] > [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 > Nov 17 14:07:39 thoregon kernel: [66571.611234] > [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 > Nov 17 14:07:39 thoregon kernel: [66571.611236] > [<ffffffff81100322>] vfs_create+0x72/0xc0 > Nov 17 14:07:39 thoregon kernel: [66571.611238] > [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 > Nov 17 14:07:39 thoregon kernel: [66571.611240] > [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 > Nov 17 14:07:39 thoregon kernel: [66571.611242] > [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 > Nov 17 14:07:39 thoregon kernel: [66571.611244] > [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 > Nov 17 14:07:39 thoregon kernel: [66571.611246] > [<ffffffff810f222c>] sys_open+0x1c/0x20 > Nov 17 14:07:39 thoregon kernel: [66571.611248] > [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b > Nov 17 14:07:39 thoregon kernel: [66571.611250] } > Nov 17 14:07:39 thoregon kernel: [66571.611251] ... key at: > [<ffffffff81c34e40>] xfs_fs_type+0x60/0x80 > Nov 17 14:07:39 thoregon kernel: [66571.611254] ... acquired at: > Nov 17 14:07:39 thoregon kernel: [66571.611254] > [<ffffffff8107df3b>] check_usage_forwards+0x10b/0x140 > Nov 17 14:07:39 thoregon kernel: [66571.611256] > [<ffffffff8107e900>] mark_lock+0x190/0x2f0 > Nov 17 14:07:39 thoregon kernel: [66571.611258] > [<ffffffff8107efdf>] __lock_acquire+0x57f/0x1c00 > Nov 17 14:07:39 thoregon kernel: [66571.611260] > [<ffffffff81080b55>] lock_acquire+0x55/0x70 > Nov 17 14:07:39 thoregon kernel: [66571.611261] > [<ffffffff810f451b>] __sb_start_write+0xab/0x190 > Nov 17 14:07:39 thoregon kernel: [66571.611263] > [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50 > Nov 17 14:07:39 thoregon kernel: [66571.611265] > [<ffffffff811f3e64>] xfs_free_eofblocks+0x104/0x250 > Nov 17 14:07:39 thoregon kernel: [66571.611266] > [<ffffffff811f4b39>] xfs_inactive+0xa9/0x480 > Nov 17 14:07:39 thoregon kernel: [66571.611268] > [<ffffffff811efe10>] xfs_fs_evict_inode+0x70/0x80 > Nov 17 14:07:39 thoregon kernel: [66571.611270] > [<ffffffff8110cb7f>] evict+0xaf/0x1b0 > Nov 17 14:07:39 thoregon kernel: [66571.611271] > [<ffffffff8110d209>] dispose_list+0x39/0x50 > Nov 17 14:07:39 thoregon kernel: [66571.611273] > [<ffffffff8110dc63>] prune_icache_sb+0x183/0x340 > Nov 17 14:07:39 thoregon kernel: [66571.611275] > [<ffffffff810f5bc3>] prune_super+0xf3/0x1a0 > Nov 17 14:07:39 thoregon kernel: [66571.611277] > [<ffffffff810bbdee>] shrink_slab+0x11e/0x1f0 > Nov 17 14:07:39 thoregon kernel: [66571.611278] > [<ffffffff810be98f>] try_to_free_pages+0x21f/0x4e0 > Nov 17 14:07:39 thoregon kernel: [66571.611280] > [<ffffffff810b5ec6>] __alloc_pages_nodemask+0x506/0x800 > Nov 17 14:07:39 thoregon kernel: [66571.611282] > [<ffffffff810ce56e>] handle_pte_fault+0x5ae/0x7a0 > Nov 17 14:07:39 thoregon kernel: [66571.611284] > [<ffffffff810cf769>] handle_mm_fault+0x1f9/0x2a0 > Nov 17 14:07:39 thoregon kernel: [66571.611286] > [<ffffffff81029dfc>] __do_page_fault+0x16c/0x480 > Nov 17 14:07:39 thoregon kernel: [66571.611287] > [<ffffffff8102a139>] do_page_fault+0x9/0x10 > Nov 17 14:07:39 thoregon kernel: [66571.611289] > [<ffffffff816b287f>] page_fault+0x1f/0x30 > Nov 17 14:07:39 thoregon kernel: [66571.611291] > Nov 17 14:07:39 thoregon kernel: [66571.611292] > Nov 17 14:07:39 thoregon kernel: [66571.611292] stack backtrace: > Nov 17 14:07:39 thoregon kernel: [66571.611294] Pid: 21330, comm: cc1 > Not tainted 3.7.0-rc5 #1 > Nov 17 14:07:39 thoregon kernel: [66571.611295] Call Trace: > Nov 17 14:07:39 thoregon kernel: [66571.611298] [<ffffffff8107de28>] > print_irq_inversion_bug.part.37+0x1e8/0x1f0 > Nov 17 14:07:39 thoregon kernel: [66571.611300] [<ffffffff8107df3b>] > check_usage_forwards+0x10b/0x140 > Nov 17 14:07:39 thoregon kernel: [66571.611303] [<ffffffff8107e900>] > mark_lock+0x190/0x2f0 > Nov 17 14:07:39 thoregon kernel: [66571.611306] [<ffffffff8150406e>] > ? dm_request+0x2e/0x2a0 > Nov 17 14:07:39 thoregon kernel: [66571.611308] [<ffffffff8107de30>] > ? print_irq_inversion_bug.part.37+0x1f0/0x1f0 > Nov 17 14:07:39 thoregon kernel: [66571.611310] [<ffffffff8107efdf>] > __lock_acquire+0x57f/0x1c00 > Nov 17 14:07:39 thoregon kernel: [66571.611313] [<ffffffff812202f4>] > ? xfs_iext_bno_to_ext+0x84/0x160 > Nov 17 14:07:39 thoregon kernel: [66571.611316] [<ffffffff8120a023>] > ? xfs_bmbt_get_all+0x13/0x20 > Nov 17 14:07:39 thoregon kernel: [66571.611318] [<ffffffff81200fb4>] > ? xfs_bmap_search_multi_extents+0xa4/0x110 > Nov 17 14:07:39 thoregon kernel: [66571.611320] [<ffffffff81080b55>] > lock_acquire+0x55/0x70 > Nov 17 14:07:39 thoregon kernel: [66571.611322] [<ffffffff8122b138>] > ? xfs_trans_alloc+0x28/0x50 > Nov 17 14:07:39 thoregon kernel: [66571.611324] [<ffffffff810f451b>] > __sb_start_write+0xab/0x190 > Nov 17 14:07:39 thoregon kernel: [66571.611326] [<ffffffff8122b138>] > ? xfs_trans_alloc+0x28/0x50 > Nov 17 14:07:39 thoregon kernel: [66571.611328] [<ffffffff8122b138>] > ? xfs_trans_alloc+0x28/0x50 > Nov 17 14:07:39 thoregon kernel: [66571.611330] [<ffffffff8122b138>] > xfs_trans_alloc+0x28/0x50 > Nov 17 14:07:39 thoregon kernel: [66571.611332] [<ffffffff811f3e64>] > xfs_free_eofblocks+0x104/0x250 > Nov 17 14:07:39 thoregon kernel: [66571.611334] [<ffffffff816b204b>] > ? _raw_spin_unlock_irq+0x2b/0x50 > Nov 17 14:07:39 thoregon kernel: [66571.611336] [<ffffffff811f4b39>] > xfs_inactive+0xa9/0x480 > Nov 17 14:07:39 thoregon kernel: [66571.611337] [<ffffffff816b204b>] > ? _raw_spin_unlock_irq+0x2b/0x50 > Nov 17 14:07:39 thoregon kernel: [66571.611340] [<ffffffff811efe10>] > xfs_fs_evict_inode+0x70/0x80 > Nov 17 14:07:39 thoregon kernel: [66571.611342] [<ffffffff8110cb7f>] > evict+0xaf/0x1b0 > Nov 17 14:07:39 thoregon kernel: [66571.611344] [<ffffffff8110d209>] > dispose_list+0x39/0x50 > Nov 17 14:07:39 thoregon kernel: [66571.611346] [<ffffffff8110dc63>] > prune_icache_sb+0x183/0x340 > Nov 17 14:07:39 thoregon kernel: [66571.611347] [<ffffffff810f5bc3>] > prune_super+0xf3/0x1a0 > Nov 17 14:07:39 thoregon kernel: [66571.611349] [<ffffffff810bbdee>] > shrink_slab+0x11e/0x1f0 > Nov 17 14:07:39 thoregon kernel: [66571.611352] [<ffffffff810be98f>] > try_to_free_pages+0x21f/0x4e0 > Nov 17 14:07:39 thoregon kernel: [66571.611354] [<ffffffff810b5ec6>] > __alloc_pages_nodemask+0x506/0x800 > Nov 17 14:07:39 thoregon kernel: [66571.611356] [<ffffffff810b9e40>] > ? lru_deactivate_fn+0x1c0/0x1c0 > Nov 17 14:07:39 thoregon kernel: [66571.611358] [<ffffffff810ce56e>] > handle_pte_fault+0x5ae/0x7a0 > Nov 17 14:07:39 thoregon kernel: [66571.611360] [<ffffffff810cf769>] > handle_mm_fault+0x1f9/0x2a0 > Nov 17 14:07:39 thoregon kernel: [66571.611363] [<ffffffff81029dfc>] > __do_page_fault+0x16c/0x480 > Nov 17 14:07:39 thoregon kernel: [66571.611366] [<ffffffff8129c7ad>] > ? trace_hardirqs_off_thunk+0x3a/0x3c > Nov 17 14:07:39 thoregon kernel: [66571.611368] [<ffffffff8102a139>] > do_page_fault+0x9/0x10 > Nov 17 14:07:39 thoregon kernel: [66571.611370] [<ffffffff816b287f>] > page_fault+0x1f/0x30 _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Hang in XFS reclaim on 3.7.0-rc3 2012-11-18 15:29 ` Torsten Kaiser @ 2012-11-18 23:51 ` Dave Chinner -1 siblings, 0 replies; 31+ messages in thread From: Dave Chinner @ 2012-11-18 23:51 UTC (permalink / raw) To: Torsten Kaiser; +Cc: xfs, Linux Kernel On Sun, Nov 18, 2012 at 04:29:22PM +0100, Torsten Kaiser wrote: > On Sun, Nov 18, 2012 at 11:24 AM, Torsten Kaiser > <just.for.lkml@googlemail.com> wrote: > > On Tue, Oct 30, 2012 at 9:37 PM, Torsten Kaiser > > <just.for.lkml@googlemail.com> wrote: > >> I will keep LOCKDEP enabled on that system, and if there really is > >> another splat, I will report back here. But I rather doubt that this > >> will be needed. > > > > After the patch, I did not see this problem again, but today I found > > another LOCKDEP report that also looks XFS related. > > I found it twice in the logs, and as both were slightly different, I > > will attach both versions. > > > Nov 6 21:57:09 thoregon kernel: [ 9941.104353] 3.7.0-rc4 #1 Not tainted > > Nov 6 21:57:09 thoregon kernel: [ 9941.104355] inconsistent > > {RECLAIM_FS-ON-W} -> {IN-RECLAIM_FS-W} usage. > > Nov 6 21:57:09 thoregon kernel: [ 9941.104430] CPU0 > > Nov 6 21:57:09 thoregon kernel: [ 9941.104431] ---- > > Nov 6 21:57:09 thoregon kernel: [ 9941.104432] lock(&(&ip->i_lock)->mr_lock); > > Nov 6 21:57:09 thoregon kernel: [ 9941.104433] <Interrupt> > > Nov 6 21:57:09 thoregon kernel: [ 9941.104434] > > lock(&(&ip->i_lock)->mr_lock); > > Nov 6 21:57:09 thoregon kernel: [ 9941.104435] > > Nov 6 21:57:09 thoregon kernel: [ 9941.104435] *** DEADLOCK *** > > Sorry! Copied the wrong report. Your fix only landed in -rc5, so my > vanilla -rc4 did (also) report the old problem again. > And I copy&pasted that report instead of the second appearance of the > new problem. Can you repost it with line wrapping turned off? The output simply becomes unreadable when it wraps.... Yeah, I know I can put it back together, but I've got better things to do with my time than stitch a couple of hundred lines of debug back into a readable format.... Cheers, Dave. -- Dave Chinner david@fromorbit.com ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Hang in XFS reclaim on 3.7.0-rc3 @ 2012-11-18 23:51 ` Dave Chinner 0 siblings, 0 replies; 31+ messages in thread From: Dave Chinner @ 2012-11-18 23:51 UTC (permalink / raw) To: Torsten Kaiser; +Cc: Linux Kernel, xfs On Sun, Nov 18, 2012 at 04:29:22PM +0100, Torsten Kaiser wrote: > On Sun, Nov 18, 2012 at 11:24 AM, Torsten Kaiser > <just.for.lkml@googlemail.com> wrote: > > On Tue, Oct 30, 2012 at 9:37 PM, Torsten Kaiser > > <just.for.lkml@googlemail.com> wrote: > >> I will keep LOCKDEP enabled on that system, and if there really is > >> another splat, I will report back here. But I rather doubt that this > >> will be needed. > > > > After the patch, I did not see this problem again, but today I found > > another LOCKDEP report that also looks XFS related. > > I found it twice in the logs, and as both were slightly different, I > > will attach both versions. > > > Nov 6 21:57:09 thoregon kernel: [ 9941.104353] 3.7.0-rc4 #1 Not tainted > > Nov 6 21:57:09 thoregon kernel: [ 9941.104355] inconsistent > > {RECLAIM_FS-ON-W} -> {IN-RECLAIM_FS-W} usage. > > Nov 6 21:57:09 thoregon kernel: [ 9941.104430] CPU0 > > Nov 6 21:57:09 thoregon kernel: [ 9941.104431] ---- > > Nov 6 21:57:09 thoregon kernel: [ 9941.104432] lock(&(&ip->i_lock)->mr_lock); > > Nov 6 21:57:09 thoregon kernel: [ 9941.104433] <Interrupt> > > Nov 6 21:57:09 thoregon kernel: [ 9941.104434] > > lock(&(&ip->i_lock)->mr_lock); > > Nov 6 21:57:09 thoregon kernel: [ 9941.104435] > > Nov 6 21:57:09 thoregon kernel: [ 9941.104435] *** DEADLOCK *** > > Sorry! Copied the wrong report. Your fix only landed in -rc5, so my > vanilla -rc4 did (also) report the old problem again. > And I copy&pasted that report instead of the second appearance of the > new problem. Can you repost it with line wrapping turned off? The output simply becomes unreadable when it wraps.... Yeah, I know I can put it back together, but I've got better things to do with my time than stitch a couple of hundred lines of debug back into a readable format.... Cheers, Dave. -- Dave Chinner david@fromorbit.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Hang in XFS reclaim on 3.7.0-rc3 2012-11-18 23:51 ` Dave Chinner @ 2012-11-19 6:50 ` Torsten Kaiser -1 siblings, 0 replies; 31+ messages in thread From: Torsten Kaiser @ 2012-11-19 6:50 UTC (permalink / raw) To: Dave Chinner; +Cc: xfs, Linux Kernel [-- Attachment #1: Type: text/plain, Size: 2121 bytes --] On Mon, Nov 19, 2012 at 12:51 AM, Dave Chinner <david@fromorbit.com> wrote: > On Sun, Nov 18, 2012 at 04:29:22PM +0100, Torsten Kaiser wrote: >> On Sun, Nov 18, 2012 at 11:24 AM, Torsten Kaiser >> <just.for.lkml@googlemail.com> wrote: >> > On Tue, Oct 30, 2012 at 9:37 PM, Torsten Kaiser >> > <just.for.lkml@googlemail.com> wrote: >> >> I will keep LOCKDEP enabled on that system, and if there really is >> >> another splat, I will report back here. But I rather doubt that this >> >> will be needed. >> > >> > After the patch, I did not see this problem again, but today I found >> > another LOCKDEP report that also looks XFS related. >> > I found it twice in the logs, and as both were slightly different, I >> > will attach both versions. >> >> > Nov 6 21:57:09 thoregon kernel: [ 9941.104353] 3.7.0-rc4 #1 Not tainted >> > Nov 6 21:57:09 thoregon kernel: [ 9941.104355] inconsistent >> > {RECLAIM_FS-ON-W} -> {IN-RECLAIM_FS-W} usage. >> > Nov 6 21:57:09 thoregon kernel: [ 9941.104430] CPU0 >> > Nov 6 21:57:09 thoregon kernel: [ 9941.104431] ---- >> > Nov 6 21:57:09 thoregon kernel: [ 9941.104432] lock(&(&ip->i_lock)->mr_lock); >> > Nov 6 21:57:09 thoregon kernel: [ 9941.104433] <Interrupt> >> > Nov 6 21:57:09 thoregon kernel: [ 9941.104434] >> > lock(&(&ip->i_lock)->mr_lock); >> > Nov 6 21:57:09 thoregon kernel: [ 9941.104435] >> > Nov 6 21:57:09 thoregon kernel: [ 9941.104435] *** DEADLOCK *** >> >> Sorry! Copied the wrong report. Your fix only landed in -rc5, so my >> vanilla -rc4 did (also) report the old problem again. >> And I copy&pasted that report instead of the second appearance of the >> new problem. > > Can you repost it with line wrapping turned off? The output simply > becomes unreadable when it wraps.... > > Yeah, I know I can put it back together, but I've got better things > to do with my time than stitch a couple of hundred lines of debug > back into a readable format.... Sorry about that, but I can't find any option to turn that off in Gmail. I have added the reports as attachment, I hope thats OK for you. Thanks for looking into this. Torsten [-- Attachment #2: lockdep-reports.txt --] [-- Type: text/plain, Size: 36869 bytes --] [110926.972477] [110926.972482] ========================================================= [110926.972484] [ INFO: possible irq lock inversion dependency detected ] [110926.972486] 3.7.0-rc4 #1 Not tainted [110926.972487] --------------------------------------------------------- [110926.972489] kswapd0/725 just changed the state of lock: [110926.972490] (sb_internal){.+.+.?}, at: [<ffffffff8122b268>] xfs_trans_alloc+0x28/0x50 [110926.972499] but this lock took another, RECLAIM_FS-unsafe lock in the past: [110926.972500] (&(&ip->i_lock)->mr_lock/1){+.+.+.} [110926.972500] [110926.972500] and interrupts could create inverse lock ordering between them. [110926.972500] [110926.972503] [110926.972503] other info that might help us debug this: [110926.972504] Possible interrupt unsafe locking scenario: [110926.972504] [110926.972505] CPU0 CPU1 [110926.972506] ---- ---- [110926.972507] lock(&(&ip->i_lock)->mr_lock/1); [110926.972509] local_irq_disable(); [110926.972509] lock(sb_internal); [110926.972511] lock(&(&ip->i_lock)->mr_lock/1); [110926.972512] <Interrupt> [110926.972513] lock(sb_internal); [110926.972514] [110926.972514] *** DEADLOCK *** [110926.972514] [110926.972516] 2 locks held by kswapd0/725: [110926.972517] #0: (shrinker_rwsem){++++..}, at: [<ffffffff810bbd22>] shrink_slab+0x32/0x1f0 [110926.972522] #1: (&type->s_umount_key#20){++++.+}, at: [<ffffffff810f5a8e>] grab_super_passive+0x3e/0x90 [110926.972527] [110926.972527] the shortest dependencies between 2nd lock and 1st lock: [110926.972533] -> (&(&ip->i_lock)->mr_lock/1){+.+.+.} ops: 58117 { [110926.972536] HARDIRQ-ON-W at: [110926.972537] [<ffffffff8107f091>] __lock_acquire+0x631/0x1c00 [110926.972540] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [110926.972542] [<ffffffff8106126a>] down_write_nested+0x4a/0x70 [110926.972545] [<ffffffff811e8164>] xfs_ilock+0x84/0xb0 [110926.972548] [<ffffffff811f5194>] xfs_create+0x1d4/0x5a0 [110926.972550] [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0 [110926.972552] [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10 [110926.972554] [<ffffffff81100332>] vfs_create+0x72/0xc0 [110926.972556] [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80 [110926.972558] [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490 [110926.972560] [<ffffffff8110184d>] do_filp_open+0x3d/0xa0 [110926.972562] [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0 [110926.972565] [<ffffffff810f223c>] sys_open+0x1c/0x20 [110926.972567] [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b [110926.972570] SOFTIRQ-ON-W at: [110926.972571] [<ffffffff8107f0c7>] __lock_acquire+0x667/0x1c00 [110926.972573] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [110926.972574] [<ffffffff8106126a>] down_write_nested+0x4a/0x70 [110926.972576] [<ffffffff811e8164>] xfs_ilock+0x84/0xb0 [110926.972578] [<ffffffff811f5194>] xfs_create+0x1d4/0x5a0 [110926.972580] [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0 [110926.972581] [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10 [110926.972583] [<ffffffff81100332>] vfs_create+0x72/0xc0 [110926.972585] [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80 [110926.972587] [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490 [110926.972589] [<ffffffff8110184d>] do_filp_open+0x3d/0xa0 [110926.972591] [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0 [110926.972593] [<ffffffff810f223c>] sys_open+0x1c/0x20 [110926.972595] [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b [110926.972597] RECLAIM_FS-ON-W at: [110926.972598] [<ffffffff8108137e>] mark_held_locks+0x7e/0x130 [110926.972600] [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0 [110926.972601] [<ffffffff810e9dd5>] kmem_cache_alloc+0x35/0xe0 [110926.972603] [<ffffffff810dba31>] vm_map_ram+0x271/0x770 [110926.972606] [<ffffffff811e1316>] _xfs_buf_map_pages+0x46/0xe0 [110926.972609] [<ffffffff811e222a>] xfs_buf_get_map+0x8a/0x130 [110926.972610] [<ffffffff81233ab9>] xfs_trans_get_buf_map+0xa9/0xd0 [110926.972613] [<ffffffff8121bced>] xfs_ialloc_inode_init+0xcd/0x1d0 [110926.972616] [<ffffffff8121c25e>] xfs_ialloc_ag_alloc+0x1be/0x560 [110926.972618] [<ffffffff8121da65>] xfs_dialloc+0x185/0x2a0 [110926.972619] [<ffffffff8121f198>] xfs_ialloc+0x58/0x650 [110926.972621] [<ffffffff811f3985>] xfs_dir_ialloc+0x65/0x270 [110926.972623] [<ffffffff811f536c>] xfs_create+0x3ac/0x5a0 [110926.972624] [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0 [110926.972626] [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10 [110926.972628] [<ffffffff81100332>] vfs_create+0x72/0xc0 [110926.972630] [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80 [110926.972632] [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490 [110926.972634] [<ffffffff8110184d>] do_filp_open+0x3d/0xa0 [110926.972636] [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0 [110926.972638] [<ffffffff810f223c>] sys_open+0x1c/0x20 [110926.972640] [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b [110926.972642] INITIAL USE at: [110926.972642] [<ffffffff8107ed49>] __lock_acquire+0x2e9/0x1c00 [110926.972644] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [110926.972646] [<ffffffff8106126a>] down_write_nested+0x4a/0x70 [110926.972648] [<ffffffff811e8164>] xfs_ilock+0x84/0xb0 [110926.972650] [<ffffffff811f5194>] xfs_create+0x1d4/0x5a0 [110926.972651] [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0 [110926.972653] [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10 [110926.972655] [<ffffffff81100332>] vfs_create+0x72/0xc0 [110926.972657] [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80 [110926.972659] [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490 [110926.972661] [<ffffffff8110184d>] do_filp_open+0x3d/0xa0 [110926.972663] [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0 [110926.972664] [<ffffffff810f223c>] sys_open+0x1c/0x20 [110926.972666] [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b [110926.972668] } [110926.972669] ... key at: [<ffffffff825b4b81>] __key.41355+0x1/0x8 [110926.972672] ... acquired at: [110926.972672] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [110926.972674] [<ffffffff8106126a>] down_write_nested+0x4a/0x70 [110926.972676] [<ffffffff811e8164>] xfs_ilock+0x84/0xb0 [110926.972678] [<ffffffff811f5194>] xfs_create+0x1d4/0x5a0 [110926.972679] [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0 [110926.972681] [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10 [110926.972683] [<ffffffff81100332>] vfs_create+0x72/0xc0 [110926.972684] [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80 [110926.972686] [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490 [110926.972688] [<ffffffff8110184d>] do_filp_open+0x3d/0xa0 [110926.972690] [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0 [110926.972692] [<ffffffff810f223c>] sys_open+0x1c/0x20 [110926.972694] [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b [110926.972696] [110926.972696] -> (sb_internal){.+.+.?} ops: 1710064 { [110926.972699] HARDIRQ-ON-R at: [110926.972700] [<ffffffff8107ef6a>] __lock_acquire+0x50a/0x1c00 [110926.972702] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [110926.972704] [<ffffffff810f452b>] __sb_start_write+0xab/0x190 [110926.972705] [<ffffffff8122b268>] xfs_trans_alloc+0x28/0x50 [110926.972707] [<ffffffff811f5147>] xfs_create+0x187/0x5a0 [110926.972709] [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0 [110926.972711] [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10 [110926.972712] [<ffffffff81100332>] vfs_create+0x72/0xc0 [110926.972714] [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80 [110926.972716] [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490 [110926.972718] [<ffffffff8110184d>] do_filp_open+0x3d/0xa0 [110926.972720] [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0 [110926.972722] [<ffffffff810f223c>] sys_open+0x1c/0x20 [110926.972724] [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b [110926.972726] SOFTIRQ-ON-R at: [110926.972727] [<ffffffff8107f0c7>] __lock_acquire+0x667/0x1c00 [110926.972728] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [110926.972730] [<ffffffff810f452b>] __sb_start_write+0xab/0x190 [110926.972732] [<ffffffff8122b268>] xfs_trans_alloc+0x28/0x50 [110926.972734] [<ffffffff811f5147>] xfs_create+0x187/0x5a0 [110926.972735] [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0 [110926.972737] [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10 [110926.972739] [<ffffffff81100332>] vfs_create+0x72/0xc0 [110926.972741] [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80 [110926.972743] [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490 [110926.972745] [<ffffffff8110184d>] do_filp_open+0x3d/0xa0 [110926.972747] [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0 [110926.972749] [<ffffffff810f223c>] sys_open+0x1c/0x20 [110926.972750] [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b [110926.972752] IN-RECLAIM_FS-R at: [110926.972753] [<ffffffff8107efdf>] __lock_acquire+0x57f/0x1c00 [110926.972755] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [110926.972757] [<ffffffff810f452b>] __sb_start_write+0xab/0x190 [110926.972758] [<ffffffff8122b268>] xfs_trans_alloc+0x28/0x50 [110926.972760] [<ffffffff811f3e54>] xfs_free_eofblocks+0x104/0x250 [110926.972762] [<ffffffff811f4b29>] xfs_inactive+0xa9/0x480 [110926.972763] [<ffffffff811efe00>] xfs_fs_evict_inode+0x70/0x80 [110926.972765] [<ffffffff8110cb8f>] evict+0xaf/0x1b0 [110926.972768] [<ffffffff8110d219>] dispose_list+0x39/0x50 [110926.972770] [<ffffffff8110dc73>] prune_icache_sb+0x183/0x340 [110926.972772] [<ffffffff810f5bd3>] prune_super+0xf3/0x1a0 [110926.972773] [<ffffffff810bbe0e>] shrink_slab+0x11e/0x1f0 [110926.972775] [<ffffffff810be400>] kswapd+0x690/0xa10 [110926.972777] [<ffffffff8105c246>] kthread+0xd6/0xe0 [110926.972779] [<ffffffff816b2c6c>] ret_from_fork+0x7c/0xb0 [110926.972781] RECLAIM_FS-ON-R at: [110926.972782] [<ffffffff8108137e>] mark_held_locks+0x7e/0x130 [110926.972784] [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0 [110926.972785] [<ffffffff810e9dd5>] kmem_cache_alloc+0x35/0xe0 [110926.972787] [<ffffffff811f6d3f>] kmem_zone_alloc+0x5f/0xe0 [110926.972789] [<ffffffff811f6dd8>] kmem_zone_zalloc+0x18/0x50 [110926.972790] [<ffffffff8122b1e2>] _xfs_trans_alloc+0x32/0x90 [110926.972792] [<ffffffff8122b278>] xfs_trans_alloc+0x38/0x50 [110926.972794] [<ffffffff811f5147>] xfs_create+0x187/0x5a0 [110926.972796] [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0 [110926.972797] [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10 [110926.972799] [<ffffffff81100332>] vfs_create+0x72/0xc0 [110926.972801] [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80 [110926.972803] [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490 [110926.972805] [<ffffffff8110184d>] do_filp_open+0x3d/0xa0 [110926.972807] [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0 [110926.972809] [<ffffffff810f223c>] sys_open+0x1c/0x20 [110926.972811] [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b [110926.972813] INITIAL USE at: [110926.972814] [<ffffffff8107ed49>] __lock_acquire+0x2e9/0x1c00 [110926.972815] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [110926.972817] [<ffffffff810f452b>] __sb_start_write+0xab/0x190 [110926.972819] [<ffffffff8122b268>] xfs_trans_alloc+0x28/0x50 [110926.972820] [<ffffffff811f5147>] xfs_create+0x187/0x5a0 [110926.972822] [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0 [110926.972824] [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10 [110926.972826] [<ffffffff81100332>] vfs_create+0x72/0xc0 [110926.972827] [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80 [110926.972829] [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490 [110926.972831] [<ffffffff8110184d>] do_filp_open+0x3d/0xa0 [110926.972833] [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0 [110926.972835] [<ffffffff810f223c>] sys_open+0x1c/0x20 [110926.972837] [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b [110926.972839] } [110926.972840] ... key at: [<ffffffff81c34e40>] xfs_fs_type+0x60/0x80 [110926.972842] ... acquired at: [110926.972843] [<ffffffff8107df3b>] check_usage_forwards+0x10b/0x140 [110926.972845] [<ffffffff8107e900>] mark_lock+0x190/0x2f0 [110926.972846] [<ffffffff8107efdf>] __lock_acquire+0x57f/0x1c00 [110926.972848] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [110926.972850] [<ffffffff810f452b>] __sb_start_write+0xab/0x190 [110926.972851] [<ffffffff8122b268>] xfs_trans_alloc+0x28/0x50 [110926.972853] [<ffffffff811f3e54>] xfs_free_eofblocks+0x104/0x250 [110926.972855] [<ffffffff811f4b29>] xfs_inactive+0xa9/0x480 [110926.972856] [<ffffffff811efe00>] xfs_fs_evict_inode+0x70/0x80 [110926.972858] [<ffffffff8110cb8f>] evict+0xaf/0x1b0 [110926.972860] [<ffffffff8110d219>] dispose_list+0x39/0x50 [110926.972861] [<ffffffff8110dc73>] prune_icache_sb+0x183/0x340 [110926.972863] [<ffffffff810f5bd3>] prune_super+0xf3/0x1a0 [110926.972865] [<ffffffff810bbe0e>] shrink_slab+0x11e/0x1f0 [110926.972866] [<ffffffff810be400>] kswapd+0x690/0xa10 [110926.972868] [<ffffffff8105c246>] kthread+0xd6/0xe0 [110926.972870] [<ffffffff816b2c6c>] ret_from_fork+0x7c/0xb0 [110926.972871] [110926.972872] [110926.972872] stack backtrace: [110926.972874] Pid: 725, comm: kswapd0 Not tainted 3.7.0-rc4 #1 [110926.972875] Call Trace: [110926.972878] [<ffffffff8107de28>] print_irq_inversion_bug.part.37+0x1e8/0x1f0 [110926.972880] [<ffffffff8107df3b>] check_usage_forwards+0x10b/0x140 [110926.972883] [<ffffffff8107e900>] mark_lock+0x190/0x2f0 [110926.972885] [<ffffffff8107de30>] ? print_irq_inversion_bug.part.37+0x1f0/0x1f0 [110926.972887] [<ffffffff8107efdf>] __lock_acquire+0x57f/0x1c00 [110926.972889] [<ffffffff81220424>] ? xfs_iext_bno_to_ext+0x84/0x160 [110926.972892] [<ffffffff8120a0e3>] ? xfs_bmbt_get_all+0x13/0x20 [110926.972895] [<ffffffff81201104>] ? xfs_bmap_search_multi_extents+0xa4/0x110 [110926.972897] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [110926.972899] [<ffffffff8122b268>] ? xfs_trans_alloc+0x28/0x50 [110926.972901] [<ffffffff810f452b>] __sb_start_write+0xab/0x190 [110926.972903] [<ffffffff8122b268>] ? xfs_trans_alloc+0x28/0x50 [110926.972905] [<ffffffff8122b268>] ? xfs_trans_alloc+0x28/0x50 [110926.972907] [<ffffffff8122b268>] xfs_trans_alloc+0x28/0x50 [110926.972908] [<ffffffff811f3e54>] xfs_free_eofblocks+0x104/0x250 [110926.972910] [<ffffffff816b1efb>] ? _raw_spin_unlock_irq+0x2b/0x50 [110926.972912] [<ffffffff811f4b29>] xfs_inactive+0xa9/0x480 [110926.972914] [<ffffffff816b1efb>] ? _raw_spin_unlock_irq+0x2b/0x50 [110926.972916] [<ffffffff811efe00>] xfs_fs_evict_inode+0x70/0x80 [110926.972918] [<ffffffff8110cb8f>] evict+0xaf/0x1b0 [110926.972920] [<ffffffff8110d219>] dispose_list+0x39/0x50 [110926.972922] [<ffffffff8110dc73>] prune_icache_sb+0x183/0x340 [110926.972924] [<ffffffff810f5bd3>] prune_super+0xf3/0x1a0 [110926.972926] [<ffffffff810bbe0e>] shrink_slab+0x11e/0x1f0 [110926.972928] [<ffffffff810be400>] kswapd+0x690/0xa10 [110926.972930] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60 [110926.972932] [<ffffffff810bdd70>] ? shrink_lruvec+0x540/0x540 [110926.972934] [<ffffffff8105c246>] kthread+0xd6/0xe0 [110926.972936] [<ffffffff816b1efb>] ? _raw_spin_unlock_irq+0x2b/0x50 [110926.972938] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 [110926.972940] [<ffffffff816b2c6c>] ret_from_fork+0x7c/0xb0 [110926.972942] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 [66571.610863] [66571.610869] ========================================================= [66571.610870] [ INFO: possible irq lock inversion dependency detected ] [66571.610873] 3.7.0-rc5 #1 Not tainted [66571.610874] --------------------------------------------------------- [66571.610875] cc1/21330 just changed the state of lock: [66571.610877] (sb_internal){.+.+.?}, at: [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50 [66571.610885] but this lock took another, RECLAIM_FS-unsafe lock in the past: [66571.610886] (&(&ip->i_lock)->mr_lock/1){+.+.+.} [66571.610886] [66571.610886] and interrupts could create inverse lock ordering between them. [66571.610886] [66571.610890] [66571.610890] other info that might help us debug this: [66571.610891] Possible interrupt unsafe locking scenario: [66571.610891] [66571.610892] CPU0 CPU1 [66571.610893] ---- ---- [66571.610894] lock(&(&ip->i_lock)->mr_lock/1); [66571.610896] local_irq_disable(); [66571.610897] lock(sb_internal); [66571.610898] lock(&(&ip->i_lock)->mr_lock/1); [66571.610900] <Interrupt> [66571.610901] lock(sb_internal); [66571.610902] [66571.610902] *** DEADLOCK *** [66571.610902] [66571.610904] 3 locks held by cc1/21330: [66571.610905] #0: (&mm->mmap_sem){++++++}, at: [<ffffffff81029d8b>] __do_page_fault+0xfb/0x480 [66571.610910] #1: (shrinker_rwsem){++++..}, at: [<ffffffff810bbd02>] shrink_slab+0x32/0x1f0 [66571.610915] #2: (&type->s_umount_key#20){++++.+}, at: [<ffffffff810f5a7e>] grab_super_passive+0x3e/0x90 [66571.610921] [66571.610921] the shortest dependencies between 2nd lock and 1st lock: [66571.610927] -> (&(&ip->i_lock)->mr_lock/1){+.+.+.} ops: 169649 { [66571.610931] HARDIRQ-ON-W at: [66571.610932] [<ffffffff8107f091>] __lock_acquire+0x631/0x1c00 [66571.610935] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [66571.610937] [<ffffffff8106126a>] down_write_nested+0x4a/0x70 [66571.610941] [<ffffffff811e8164>] xfs_ilock+0x84/0xb0 [66571.610944] [<ffffffff811f51a4>] xfs_create+0x1d4/0x5a0 [66571.610946] [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 [66571.610948] [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 [66571.610950] [<ffffffff81100322>] vfs_create+0x72/0xc0 [66571.610953] [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 [66571.610955] [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 [66571.610957] [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 [66571.610959] [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 [66571.610962] [<ffffffff810f222c>] sys_open+0x1c/0x20 [66571.610964] [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b [66571.610967] SOFTIRQ-ON-W at: [66571.610968] [<ffffffff8107f0c7>] __lock_acquire+0x667/0x1c00 [66571.610970] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [66571.610972] [<ffffffff8106126a>] down_write_nested+0x4a/0x70 [66571.610974] [<ffffffff811e8164>] xfs_ilock+0x84/0xb0 [66571.610976] [<ffffffff811f51a4>] xfs_create+0x1d4/0x5a0 [66571.610977] [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 [66571.610979] [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 [66571.610981] [<ffffffff81100322>] vfs_create+0x72/0xc0 [66571.610983] [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 [66571.610985] [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 [66571.610987] [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 [66571.610989] [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 [66571.610991] [<ffffffff810f222c>] sys_open+0x1c/0x20 [66571.610993] [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b [66571.610995] RECLAIM_FS-ON-W at: [66571.610996] [<ffffffff8108137e>] mark_held_locks+0x7e/0x130 [66571.610998] [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0 [66571.610999] [<ffffffff810e9dc5>] kmem_cache_alloc+0x35/0xe0 [66571.611002] [<ffffffff810dba21>] vm_map_ram+0x271/0x770 [66571.611004] [<ffffffff811e12b6>] _xfs_buf_map_pages+0x46/0xe0 [66571.611008] [<ffffffff811e21ca>] xfs_buf_get_map+0x8a/0x130 [66571.611009] [<ffffffff81233989>] xfs_trans_get_buf_map+0xa9/0xd0 [66571.611011] [<ffffffff8121bc2d>] xfs_ialloc_inode_init+0xcd/0x1d0 [66571.611015] [<ffffffff8121c16f>] xfs_ialloc_ag_alloc+0x18f/0x500 [66571.611017] [<ffffffff8121d955>] xfs_dialloc+0x185/0x2a0 [66571.611019] [<ffffffff8121f068>] xfs_ialloc+0x58/0x650 [66571.611021] [<ffffffff811f3995>] xfs_dir_ialloc+0x65/0x270 [66571.611023] [<ffffffff811f537c>] xfs_create+0x3ac/0x5a0 [66571.611024] [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 [66571.611026] [<ffffffff811ecb61>] xfs_vn_mkdir+0x11/0x20 [66571.611028] [<ffffffff8110016f>] vfs_mkdir+0x7f/0xd0 [66571.611030] [<ffffffff81101b83>] sys_mkdirat+0x43/0x80 [66571.611032] [<ffffffff81101bd4>] sys_mkdir+0x14/0x20 [66571.611034] [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b [66571.611036] INITIAL USE at: [66571.611037] [<ffffffff8107ed49>] __lock_acquire+0x2e9/0x1c00 [66571.611038] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [66571.611040] [<ffffffff8106126a>] down_write_nested+0x4a/0x70 [66571.611042] [<ffffffff811e8164>] xfs_ilock+0x84/0xb0 [66571.611044] [<ffffffff811f51a4>] xfs_create+0x1d4/0x5a0 [66571.611046] [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 [66571.611047] [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 [66571.611049] [<ffffffff81100322>] vfs_create+0x72/0xc0 [66571.611051] [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 [66571.611053] [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 [66571.611055] [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 [66571.611057] [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 [66571.611059] [<ffffffff810f222c>] sys_open+0x1c/0x20 [66571.611061] [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b [66571.611063] } [66571.611064] ... key at: [<ffffffff825b4b81>] __key.41357+0x1/0x8 [66571.611066] ... acquired at: [66571.611067] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [66571.611069] [<ffffffff8106126a>] down_write_nested+0x4a/0x70 [66571.611071] [<ffffffff811e8164>] xfs_ilock+0x84/0xb0 [66571.611073] [<ffffffff811f51a4>] xfs_create+0x1d4/0x5a0 [66571.611074] [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 [66571.611076] [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 [66571.611078] [<ffffffff81100322>] vfs_create+0x72/0xc0 [66571.611080] [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 [66571.611082] [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 [66571.611084] [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 [66571.611086] [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 [66571.611088] [<ffffffff810f222c>] sys_open+0x1c/0x20 [66571.611090] [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b [66571.611091] [66571.611092] -> (sb_internal){.+.+.?} ops: 1341531 { [66571.611095] HARDIRQ-ON-R at: [66571.611096] [<ffffffff8107ef6a>] __lock_acquire+0x50a/0x1c00 [66571.611098] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [66571.611100] [<ffffffff810f451b>] __sb_start_write+0xab/0x190 [66571.611102] [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50 [66571.611104] [<ffffffff811f5157>] xfs_create+0x187/0x5a0 [66571.611105] [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 [66571.611107] [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 [66571.611109] [<ffffffff81100322>] vfs_create+0x72/0xc0 [66571.611111] [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 [66571.611113] [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 [66571.611115] [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 [66571.611117] [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 [66571.611119] [<ffffffff810f222c>] sys_open+0x1c/0x20 [66571.611121] [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b [66571.611123] SOFTIRQ-ON-R at: [66571.611124] [<ffffffff8107f0c7>] __lock_acquire+0x667/0x1c00 [66571.611126] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [66571.611128] [<ffffffff810f451b>] __sb_start_write+0xab/0x190 [66571.611130] [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50 [66571.611132] [<ffffffff811f5157>] xfs_create+0x187/0x5a0 [66571.611133] [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 [66571.611135] [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 [66571.611137] [<ffffffff81100322>] vfs_create+0x72/0xc0 [66571.611139] [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 [66571.611141] [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 [66571.611143] [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 [66571.611145] [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 [66571.611147] [<ffffffff810f222c>] sys_open+0x1c/0x20 [66571.611149] [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b [66571.611151] IN-RECLAIM_FS-R at: [66571.611152] [<ffffffff8107efdf>] __lock_acquire+0x57f/0x1c00 [66571.611154] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [66571.611156] [<ffffffff810f451b>] __sb_start_write+0xab/0x190 [66571.611158] [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50 [66571.611159] [<ffffffff811f3e64>] xfs_free_eofblocks+0x104/0x250 [66571.611161] [<ffffffff811f4b39>] xfs_inactive+0xa9/0x480 [66571.611163] [<ffffffff811efe10>] xfs_fs_evict_inode+0x70/0x80 [66571.611165] [<ffffffff8110cb7f>] evict+0xaf/0x1b0 [66571.611168] [<ffffffff8110d209>] dispose_list+0x39/0x50 [66571.611170] [<ffffffff8110dc63>] prune_icache_sb+0x183/0x340 [66571.611172] [<ffffffff810f5bc3>] prune_super+0xf3/0x1a0 [66571.611173] [<ffffffff810bbdee>] shrink_slab+0x11e/0x1f0 [66571.611175] [<ffffffff810be98f>] try_to_free_pages+0x21f/0x4e0 [66571.611177] [<ffffffff810b5ec6>] __alloc_pages_nodemask+0x506/0x800 [66571.611179] [<ffffffff810ce56e>] handle_pte_fault+0x5ae/0x7a0 [66571.611182] [<ffffffff810cf769>] handle_mm_fault+0x1f9/0x2a0 [66571.611184] [<ffffffff81029dfc>] __do_page_fault+0x16c/0x480 [66571.611186] [<ffffffff8102a139>] do_page_fault+0x9/0x10 [66571.611188] [<ffffffff816b287f>] page_fault+0x1f/0x30 [66571.611190] RECLAIM_FS-ON-R at: [66571.611191] [<ffffffff8108137e>] mark_held_locks+0x7e/0x130 [66571.611193] [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0 [66571.611195] [<ffffffff810e9dc5>] kmem_cache_alloc+0x35/0xe0 [66571.611197] [<ffffffff811f6d4f>] kmem_zone_alloc+0x5f/0xe0 [66571.611198] [<ffffffff811f6de8>] kmem_zone_zalloc+0x18/0x50 [66571.611200] [<ffffffff8122b0b2>] _xfs_trans_alloc+0x32/0x90 [66571.611202] [<ffffffff8122b148>] xfs_trans_alloc+0x38/0x50 [66571.611204] [<ffffffff811f5157>] xfs_create+0x187/0x5a0 [66571.611205] [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 [66571.611207] [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 [66571.611209] [<ffffffff81100322>] vfs_create+0x72/0xc0 [66571.611211] [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 [66571.611213] [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 [66571.611215] [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 [66571.611217] [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 [66571.611219] [<ffffffff810f222c>] sys_open+0x1c/0x20 [66571.611221] [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b [66571.611223] INITIAL USE at: [66571.611224] [<ffffffff8107ed49>] __lock_acquire+0x2e9/0x1c00 [66571.611225] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [66571.611227] [<ffffffff810f451b>] __sb_start_write+0xab/0x190 [66571.611229] [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50 [66571.611231] [<ffffffff811f5157>] xfs_create+0x187/0x5a0 [66571.611232] [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 [66571.611234] [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 [66571.611236] [<ffffffff81100322>] vfs_create+0x72/0xc0 [66571.611238] [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 [66571.611240] [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 [66571.611242] [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 [66571.611244] [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 [66571.611246] [<ffffffff810f222c>] sys_open+0x1c/0x20 [66571.611248] [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b [66571.611250] } [66571.611251] ... key at: [<ffffffff81c34e40>] xfs_fs_type+0x60/0x80 [66571.611254] ... acquired at: [66571.611254] [<ffffffff8107df3b>] check_usage_forwards+0x10b/0x140 [66571.611256] [<ffffffff8107e900>] mark_lock+0x190/0x2f0 [66571.611258] [<ffffffff8107efdf>] __lock_acquire+0x57f/0x1c00 [66571.611260] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [66571.611261] [<ffffffff810f451b>] __sb_start_write+0xab/0x190 [66571.611263] [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50 [66571.611265] [<ffffffff811f3e64>] xfs_free_eofblocks+0x104/0x250 [66571.611266] [<ffffffff811f4b39>] xfs_inactive+0xa9/0x480 [66571.611268] [<ffffffff811efe10>] xfs_fs_evict_inode+0x70/0x80 [66571.611270] [<ffffffff8110cb7f>] evict+0xaf/0x1b0 [66571.611271] [<ffffffff8110d209>] dispose_list+0x39/0x50 [66571.611273] [<ffffffff8110dc63>] prune_icache_sb+0x183/0x340 [66571.611275] [<ffffffff810f5bc3>] prune_super+0xf3/0x1a0 [66571.611277] [<ffffffff810bbdee>] shrink_slab+0x11e/0x1f0 [66571.611278] [<ffffffff810be98f>] try_to_free_pages+0x21f/0x4e0 [66571.611280] [<ffffffff810b5ec6>] __alloc_pages_nodemask+0x506/0x800 [66571.611282] [<ffffffff810ce56e>] handle_pte_fault+0x5ae/0x7a0 [66571.611284] [<ffffffff810cf769>] handle_mm_fault+0x1f9/0x2a0 [66571.611286] [<ffffffff81029dfc>] __do_page_fault+0x16c/0x480 [66571.611287] [<ffffffff8102a139>] do_page_fault+0x9/0x10 [66571.611289] [<ffffffff816b287f>] page_fault+0x1f/0x30 [66571.611291] [66571.611292] [66571.611292] stack backtrace: [66571.611294] Pid: 21330, comm: cc1 Not tainted 3.7.0-rc5 #1 [66571.611295] Call Trace: [66571.611298] [<ffffffff8107de28>] print_irq_inversion_bug.part.37+0x1e8/0x1f0 [66571.611300] [<ffffffff8107df3b>] check_usage_forwards+0x10b/0x140 [66571.611303] [<ffffffff8107e900>] mark_lock+0x190/0x2f0 [66571.611306] [<ffffffff8150406e>] ? dm_request+0x2e/0x2a0 [66571.611308] [<ffffffff8107de30>] ? print_irq_inversion_bug.part.37+0x1f0/0x1f0 [66571.611310] [<ffffffff8107efdf>] __lock_acquire+0x57f/0x1c00 [66571.611313] [<ffffffff812202f4>] ? xfs_iext_bno_to_ext+0x84/0x160 [66571.611316] [<ffffffff8120a023>] ? xfs_bmbt_get_all+0x13/0x20 [66571.611318] [<ffffffff81200fb4>] ? xfs_bmap_search_multi_extents+0xa4/0x110 [66571.611320] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [66571.611322] [<ffffffff8122b138>] ? xfs_trans_alloc+0x28/0x50 [66571.611324] [<ffffffff810f451b>] __sb_start_write+0xab/0x190 [66571.611326] [<ffffffff8122b138>] ? xfs_trans_alloc+0x28/0x50 [66571.611328] [<ffffffff8122b138>] ? xfs_trans_alloc+0x28/0x50 [66571.611330] [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50 [66571.611332] [<ffffffff811f3e64>] xfs_free_eofblocks+0x104/0x250 [66571.611334] [<ffffffff816b204b>] ? _raw_spin_unlock_irq+0x2b/0x50 [66571.611336] [<ffffffff811f4b39>] xfs_inactive+0xa9/0x480 [66571.611337] [<ffffffff816b204b>] ? _raw_spin_unlock_irq+0x2b/0x50 [66571.611340] [<ffffffff811efe10>] xfs_fs_evict_inode+0x70/0x80 [66571.611342] [<ffffffff8110cb7f>] evict+0xaf/0x1b0 [66571.611344] [<ffffffff8110d209>] dispose_list+0x39/0x50 [66571.611346] [<ffffffff8110dc63>] prune_icache_sb+0x183/0x340 [66571.611347] [<ffffffff810f5bc3>] prune_super+0xf3/0x1a0 [66571.611349] [<ffffffff810bbdee>] shrink_slab+0x11e/0x1f0 [66571.611352] [<ffffffff810be98f>] try_to_free_pages+0x21f/0x4e0 [66571.611354] [<ffffffff810b5ec6>] __alloc_pages_nodemask+0x506/0x800 [66571.611356] [<ffffffff810b9e40>] ? lru_deactivate_fn+0x1c0/0x1c0 [66571.611358] [<ffffffff810ce56e>] handle_pte_fault+0x5ae/0x7a0 [66571.611360] [<ffffffff810cf769>] handle_mm_fault+0x1f9/0x2a0 [66571.611363] [<ffffffff81029dfc>] __do_page_fault+0x16c/0x480 [66571.611366] [<ffffffff8129c7ad>] ? trace_hardirqs_off_thunk+0x3a/0x3c [66571.611368] [<ffffffff8102a139>] do_page_fault+0x9/0x10 [66571.611370] [<ffffffff816b287f>] page_fault+0x1f/0x30 ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Hang in XFS reclaim on 3.7.0-rc3 @ 2012-11-19 6:50 ` Torsten Kaiser 0 siblings, 0 replies; 31+ messages in thread From: Torsten Kaiser @ 2012-11-19 6:50 UTC (permalink / raw) To: Dave Chinner; +Cc: Linux Kernel, xfs [-- Attachment #1: Type: text/plain, Size: 2121 bytes --] On Mon, Nov 19, 2012 at 12:51 AM, Dave Chinner <david@fromorbit.com> wrote: > On Sun, Nov 18, 2012 at 04:29:22PM +0100, Torsten Kaiser wrote: >> On Sun, Nov 18, 2012 at 11:24 AM, Torsten Kaiser >> <just.for.lkml@googlemail.com> wrote: >> > On Tue, Oct 30, 2012 at 9:37 PM, Torsten Kaiser >> > <just.for.lkml@googlemail.com> wrote: >> >> I will keep LOCKDEP enabled on that system, and if there really is >> >> another splat, I will report back here. But I rather doubt that this >> >> will be needed. >> > >> > After the patch, I did not see this problem again, but today I found >> > another LOCKDEP report that also looks XFS related. >> > I found it twice in the logs, and as both were slightly different, I >> > will attach both versions. >> >> > Nov 6 21:57:09 thoregon kernel: [ 9941.104353] 3.7.0-rc4 #1 Not tainted >> > Nov 6 21:57:09 thoregon kernel: [ 9941.104355] inconsistent >> > {RECLAIM_FS-ON-W} -> {IN-RECLAIM_FS-W} usage. >> > Nov 6 21:57:09 thoregon kernel: [ 9941.104430] CPU0 >> > Nov 6 21:57:09 thoregon kernel: [ 9941.104431] ---- >> > Nov 6 21:57:09 thoregon kernel: [ 9941.104432] lock(&(&ip->i_lock)->mr_lock); >> > Nov 6 21:57:09 thoregon kernel: [ 9941.104433] <Interrupt> >> > Nov 6 21:57:09 thoregon kernel: [ 9941.104434] >> > lock(&(&ip->i_lock)->mr_lock); >> > Nov 6 21:57:09 thoregon kernel: [ 9941.104435] >> > Nov 6 21:57:09 thoregon kernel: [ 9941.104435] *** DEADLOCK *** >> >> Sorry! Copied the wrong report. Your fix only landed in -rc5, so my >> vanilla -rc4 did (also) report the old problem again. >> And I copy&pasted that report instead of the second appearance of the >> new problem. > > Can you repost it with line wrapping turned off? The output simply > becomes unreadable when it wraps.... > > Yeah, I know I can put it back together, but I've got better things > to do with my time than stitch a couple of hundred lines of debug > back into a readable format.... Sorry about that, but I can't find any option to turn that off in Gmail. I have added the reports as attachment, I hope thats OK for you. Thanks for looking into this. Torsten [-- Attachment #2: lockdep-reports.txt --] [-- Type: text/plain, Size: 36869 bytes --] [110926.972477] [110926.972482] ========================================================= [110926.972484] [ INFO: possible irq lock inversion dependency detected ] [110926.972486] 3.7.0-rc4 #1 Not tainted [110926.972487] --------------------------------------------------------- [110926.972489] kswapd0/725 just changed the state of lock: [110926.972490] (sb_internal){.+.+.?}, at: [<ffffffff8122b268>] xfs_trans_alloc+0x28/0x50 [110926.972499] but this lock took another, RECLAIM_FS-unsafe lock in the past: [110926.972500] (&(&ip->i_lock)->mr_lock/1){+.+.+.} [110926.972500] [110926.972500] and interrupts could create inverse lock ordering between them. [110926.972500] [110926.972503] [110926.972503] other info that might help us debug this: [110926.972504] Possible interrupt unsafe locking scenario: [110926.972504] [110926.972505] CPU0 CPU1 [110926.972506] ---- ---- [110926.972507] lock(&(&ip->i_lock)->mr_lock/1); [110926.972509] local_irq_disable(); [110926.972509] lock(sb_internal); [110926.972511] lock(&(&ip->i_lock)->mr_lock/1); [110926.972512] <Interrupt> [110926.972513] lock(sb_internal); [110926.972514] [110926.972514] *** DEADLOCK *** [110926.972514] [110926.972516] 2 locks held by kswapd0/725: [110926.972517] #0: (shrinker_rwsem){++++..}, at: [<ffffffff810bbd22>] shrink_slab+0x32/0x1f0 [110926.972522] #1: (&type->s_umount_key#20){++++.+}, at: [<ffffffff810f5a8e>] grab_super_passive+0x3e/0x90 [110926.972527] [110926.972527] the shortest dependencies between 2nd lock and 1st lock: [110926.972533] -> (&(&ip->i_lock)->mr_lock/1){+.+.+.} ops: 58117 { [110926.972536] HARDIRQ-ON-W at: [110926.972537] [<ffffffff8107f091>] __lock_acquire+0x631/0x1c00 [110926.972540] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [110926.972542] [<ffffffff8106126a>] down_write_nested+0x4a/0x70 [110926.972545] [<ffffffff811e8164>] xfs_ilock+0x84/0xb0 [110926.972548] [<ffffffff811f5194>] xfs_create+0x1d4/0x5a0 [110926.972550] [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0 [110926.972552] [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10 [110926.972554] [<ffffffff81100332>] vfs_create+0x72/0xc0 [110926.972556] [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80 [110926.972558] [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490 [110926.972560] [<ffffffff8110184d>] do_filp_open+0x3d/0xa0 [110926.972562] [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0 [110926.972565] [<ffffffff810f223c>] sys_open+0x1c/0x20 [110926.972567] [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b [110926.972570] SOFTIRQ-ON-W at: [110926.972571] [<ffffffff8107f0c7>] __lock_acquire+0x667/0x1c00 [110926.972573] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [110926.972574] [<ffffffff8106126a>] down_write_nested+0x4a/0x70 [110926.972576] [<ffffffff811e8164>] xfs_ilock+0x84/0xb0 [110926.972578] [<ffffffff811f5194>] xfs_create+0x1d4/0x5a0 [110926.972580] [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0 [110926.972581] [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10 [110926.972583] [<ffffffff81100332>] vfs_create+0x72/0xc0 [110926.972585] [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80 [110926.972587] [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490 [110926.972589] [<ffffffff8110184d>] do_filp_open+0x3d/0xa0 [110926.972591] [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0 [110926.972593] [<ffffffff810f223c>] sys_open+0x1c/0x20 [110926.972595] [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b [110926.972597] RECLAIM_FS-ON-W at: [110926.972598] [<ffffffff8108137e>] mark_held_locks+0x7e/0x130 [110926.972600] [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0 [110926.972601] [<ffffffff810e9dd5>] kmem_cache_alloc+0x35/0xe0 [110926.972603] [<ffffffff810dba31>] vm_map_ram+0x271/0x770 [110926.972606] [<ffffffff811e1316>] _xfs_buf_map_pages+0x46/0xe0 [110926.972609] [<ffffffff811e222a>] xfs_buf_get_map+0x8a/0x130 [110926.972610] [<ffffffff81233ab9>] xfs_trans_get_buf_map+0xa9/0xd0 [110926.972613] [<ffffffff8121bced>] xfs_ialloc_inode_init+0xcd/0x1d0 [110926.972616] [<ffffffff8121c25e>] xfs_ialloc_ag_alloc+0x1be/0x560 [110926.972618] [<ffffffff8121da65>] xfs_dialloc+0x185/0x2a0 [110926.972619] [<ffffffff8121f198>] xfs_ialloc+0x58/0x650 [110926.972621] [<ffffffff811f3985>] xfs_dir_ialloc+0x65/0x270 [110926.972623] [<ffffffff811f536c>] xfs_create+0x3ac/0x5a0 [110926.972624] [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0 [110926.972626] [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10 [110926.972628] [<ffffffff81100332>] vfs_create+0x72/0xc0 [110926.972630] [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80 [110926.972632] [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490 [110926.972634] [<ffffffff8110184d>] do_filp_open+0x3d/0xa0 [110926.972636] [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0 [110926.972638] [<ffffffff810f223c>] sys_open+0x1c/0x20 [110926.972640] [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b [110926.972642] INITIAL USE at: [110926.972642] [<ffffffff8107ed49>] __lock_acquire+0x2e9/0x1c00 [110926.972644] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [110926.972646] [<ffffffff8106126a>] down_write_nested+0x4a/0x70 [110926.972648] [<ffffffff811e8164>] xfs_ilock+0x84/0xb0 [110926.972650] [<ffffffff811f5194>] xfs_create+0x1d4/0x5a0 [110926.972651] [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0 [110926.972653] [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10 [110926.972655] [<ffffffff81100332>] vfs_create+0x72/0xc0 [110926.972657] [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80 [110926.972659] [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490 [110926.972661] [<ffffffff8110184d>] do_filp_open+0x3d/0xa0 [110926.972663] [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0 [110926.972664] [<ffffffff810f223c>] sys_open+0x1c/0x20 [110926.972666] [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b [110926.972668] } [110926.972669] ... key at: [<ffffffff825b4b81>] __key.41355+0x1/0x8 [110926.972672] ... acquired at: [110926.972672] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [110926.972674] [<ffffffff8106126a>] down_write_nested+0x4a/0x70 [110926.972676] [<ffffffff811e8164>] xfs_ilock+0x84/0xb0 [110926.972678] [<ffffffff811f5194>] xfs_create+0x1d4/0x5a0 [110926.972679] [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0 [110926.972681] [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10 [110926.972683] [<ffffffff81100332>] vfs_create+0x72/0xc0 [110926.972684] [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80 [110926.972686] [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490 [110926.972688] [<ffffffff8110184d>] do_filp_open+0x3d/0xa0 [110926.972690] [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0 [110926.972692] [<ffffffff810f223c>] sys_open+0x1c/0x20 [110926.972694] [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b [110926.972696] [110926.972696] -> (sb_internal){.+.+.?} ops: 1710064 { [110926.972699] HARDIRQ-ON-R at: [110926.972700] [<ffffffff8107ef6a>] __lock_acquire+0x50a/0x1c00 [110926.972702] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [110926.972704] [<ffffffff810f452b>] __sb_start_write+0xab/0x190 [110926.972705] [<ffffffff8122b268>] xfs_trans_alloc+0x28/0x50 [110926.972707] [<ffffffff811f5147>] xfs_create+0x187/0x5a0 [110926.972709] [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0 [110926.972711] [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10 [110926.972712] [<ffffffff81100332>] vfs_create+0x72/0xc0 [110926.972714] [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80 [110926.972716] [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490 [110926.972718] [<ffffffff8110184d>] do_filp_open+0x3d/0xa0 [110926.972720] [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0 [110926.972722] [<ffffffff810f223c>] sys_open+0x1c/0x20 [110926.972724] [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b [110926.972726] SOFTIRQ-ON-R at: [110926.972727] [<ffffffff8107f0c7>] __lock_acquire+0x667/0x1c00 [110926.972728] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [110926.972730] [<ffffffff810f452b>] __sb_start_write+0xab/0x190 [110926.972732] [<ffffffff8122b268>] xfs_trans_alloc+0x28/0x50 [110926.972734] [<ffffffff811f5147>] xfs_create+0x187/0x5a0 [110926.972735] [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0 [110926.972737] [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10 [110926.972739] [<ffffffff81100332>] vfs_create+0x72/0xc0 [110926.972741] [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80 [110926.972743] [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490 [110926.972745] [<ffffffff8110184d>] do_filp_open+0x3d/0xa0 [110926.972747] [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0 [110926.972749] [<ffffffff810f223c>] sys_open+0x1c/0x20 [110926.972750] [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b [110926.972752] IN-RECLAIM_FS-R at: [110926.972753] [<ffffffff8107efdf>] __lock_acquire+0x57f/0x1c00 [110926.972755] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [110926.972757] [<ffffffff810f452b>] __sb_start_write+0xab/0x190 [110926.972758] [<ffffffff8122b268>] xfs_trans_alloc+0x28/0x50 [110926.972760] [<ffffffff811f3e54>] xfs_free_eofblocks+0x104/0x250 [110926.972762] [<ffffffff811f4b29>] xfs_inactive+0xa9/0x480 [110926.972763] [<ffffffff811efe00>] xfs_fs_evict_inode+0x70/0x80 [110926.972765] [<ffffffff8110cb8f>] evict+0xaf/0x1b0 [110926.972768] [<ffffffff8110d219>] dispose_list+0x39/0x50 [110926.972770] [<ffffffff8110dc73>] prune_icache_sb+0x183/0x340 [110926.972772] [<ffffffff810f5bd3>] prune_super+0xf3/0x1a0 [110926.972773] [<ffffffff810bbe0e>] shrink_slab+0x11e/0x1f0 [110926.972775] [<ffffffff810be400>] kswapd+0x690/0xa10 [110926.972777] [<ffffffff8105c246>] kthread+0xd6/0xe0 [110926.972779] [<ffffffff816b2c6c>] ret_from_fork+0x7c/0xb0 [110926.972781] RECLAIM_FS-ON-R at: [110926.972782] [<ffffffff8108137e>] mark_held_locks+0x7e/0x130 [110926.972784] [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0 [110926.972785] [<ffffffff810e9dd5>] kmem_cache_alloc+0x35/0xe0 [110926.972787] [<ffffffff811f6d3f>] kmem_zone_alloc+0x5f/0xe0 [110926.972789] [<ffffffff811f6dd8>] kmem_zone_zalloc+0x18/0x50 [110926.972790] [<ffffffff8122b1e2>] _xfs_trans_alloc+0x32/0x90 [110926.972792] [<ffffffff8122b278>] xfs_trans_alloc+0x38/0x50 [110926.972794] [<ffffffff811f5147>] xfs_create+0x187/0x5a0 [110926.972796] [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0 [110926.972797] [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10 [110926.972799] [<ffffffff81100332>] vfs_create+0x72/0xc0 [110926.972801] [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80 [110926.972803] [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490 [110926.972805] [<ffffffff8110184d>] do_filp_open+0x3d/0xa0 [110926.972807] [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0 [110926.972809] [<ffffffff810f223c>] sys_open+0x1c/0x20 [110926.972811] [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b [110926.972813] INITIAL USE at: [110926.972814] [<ffffffff8107ed49>] __lock_acquire+0x2e9/0x1c00 [110926.972815] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [110926.972817] [<ffffffff810f452b>] __sb_start_write+0xab/0x190 [110926.972819] [<ffffffff8122b268>] xfs_trans_alloc+0x28/0x50 [110926.972820] [<ffffffff811f5147>] xfs_create+0x187/0x5a0 [110926.972822] [<ffffffff811eca1a>] xfs_vn_mknod+0x8a/0x1b0 [110926.972824] [<ffffffff811ecb6e>] xfs_vn_create+0xe/0x10 [110926.972826] [<ffffffff81100332>] vfs_create+0x72/0xc0 [110926.972827] [<ffffffff81100b8e>] do_last.isra.69+0x80e/0xc80 [110926.972829] [<ffffffff811010ab>] path_openat.isra.70+0xab/0x490 [110926.972831] [<ffffffff8110184d>] do_filp_open+0x3d/0xa0 [110926.972833] [<ffffffff810f2139>] do_sys_open+0xf9/0x1e0 [110926.972835] [<ffffffff810f223c>] sys_open+0x1c/0x20 [110926.972837] [<ffffffff816b2d12>] system_call_fastpath+0x16/0x1b [110926.972839] } [110926.972840] ... key at: [<ffffffff81c34e40>] xfs_fs_type+0x60/0x80 [110926.972842] ... acquired at: [110926.972843] [<ffffffff8107df3b>] check_usage_forwards+0x10b/0x140 [110926.972845] [<ffffffff8107e900>] mark_lock+0x190/0x2f0 [110926.972846] [<ffffffff8107efdf>] __lock_acquire+0x57f/0x1c00 [110926.972848] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [110926.972850] [<ffffffff810f452b>] __sb_start_write+0xab/0x190 [110926.972851] [<ffffffff8122b268>] xfs_trans_alloc+0x28/0x50 [110926.972853] [<ffffffff811f3e54>] xfs_free_eofblocks+0x104/0x250 [110926.972855] [<ffffffff811f4b29>] xfs_inactive+0xa9/0x480 [110926.972856] [<ffffffff811efe00>] xfs_fs_evict_inode+0x70/0x80 [110926.972858] [<ffffffff8110cb8f>] evict+0xaf/0x1b0 [110926.972860] [<ffffffff8110d219>] dispose_list+0x39/0x50 [110926.972861] [<ffffffff8110dc73>] prune_icache_sb+0x183/0x340 [110926.972863] [<ffffffff810f5bd3>] prune_super+0xf3/0x1a0 [110926.972865] [<ffffffff810bbe0e>] shrink_slab+0x11e/0x1f0 [110926.972866] [<ffffffff810be400>] kswapd+0x690/0xa10 [110926.972868] [<ffffffff8105c246>] kthread+0xd6/0xe0 [110926.972870] [<ffffffff816b2c6c>] ret_from_fork+0x7c/0xb0 [110926.972871] [110926.972872] [110926.972872] stack backtrace: [110926.972874] Pid: 725, comm: kswapd0 Not tainted 3.7.0-rc4 #1 [110926.972875] Call Trace: [110926.972878] [<ffffffff8107de28>] print_irq_inversion_bug.part.37+0x1e8/0x1f0 [110926.972880] [<ffffffff8107df3b>] check_usage_forwards+0x10b/0x140 [110926.972883] [<ffffffff8107e900>] mark_lock+0x190/0x2f0 [110926.972885] [<ffffffff8107de30>] ? print_irq_inversion_bug.part.37+0x1f0/0x1f0 [110926.972887] [<ffffffff8107efdf>] __lock_acquire+0x57f/0x1c00 [110926.972889] [<ffffffff81220424>] ? xfs_iext_bno_to_ext+0x84/0x160 [110926.972892] [<ffffffff8120a0e3>] ? xfs_bmbt_get_all+0x13/0x20 [110926.972895] [<ffffffff81201104>] ? xfs_bmap_search_multi_extents+0xa4/0x110 [110926.972897] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [110926.972899] [<ffffffff8122b268>] ? xfs_trans_alloc+0x28/0x50 [110926.972901] [<ffffffff810f452b>] __sb_start_write+0xab/0x190 [110926.972903] [<ffffffff8122b268>] ? xfs_trans_alloc+0x28/0x50 [110926.972905] [<ffffffff8122b268>] ? xfs_trans_alloc+0x28/0x50 [110926.972907] [<ffffffff8122b268>] xfs_trans_alloc+0x28/0x50 [110926.972908] [<ffffffff811f3e54>] xfs_free_eofblocks+0x104/0x250 [110926.972910] [<ffffffff816b1efb>] ? _raw_spin_unlock_irq+0x2b/0x50 [110926.972912] [<ffffffff811f4b29>] xfs_inactive+0xa9/0x480 [110926.972914] [<ffffffff816b1efb>] ? _raw_spin_unlock_irq+0x2b/0x50 [110926.972916] [<ffffffff811efe00>] xfs_fs_evict_inode+0x70/0x80 [110926.972918] [<ffffffff8110cb8f>] evict+0xaf/0x1b0 [110926.972920] [<ffffffff8110d219>] dispose_list+0x39/0x50 [110926.972922] [<ffffffff8110dc73>] prune_icache_sb+0x183/0x340 [110926.972924] [<ffffffff810f5bd3>] prune_super+0xf3/0x1a0 [110926.972926] [<ffffffff810bbe0e>] shrink_slab+0x11e/0x1f0 [110926.972928] [<ffffffff810be400>] kswapd+0x690/0xa10 [110926.972930] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60 [110926.972932] [<ffffffff810bdd70>] ? shrink_lruvec+0x540/0x540 [110926.972934] [<ffffffff8105c246>] kthread+0xd6/0xe0 [110926.972936] [<ffffffff816b1efb>] ? _raw_spin_unlock_irq+0x2b/0x50 [110926.972938] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 [110926.972940] [<ffffffff816b2c6c>] ret_from_fork+0x7c/0xb0 [110926.972942] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 [66571.610863] [66571.610869] ========================================================= [66571.610870] [ INFO: possible irq lock inversion dependency detected ] [66571.610873] 3.7.0-rc5 #1 Not tainted [66571.610874] --------------------------------------------------------- [66571.610875] cc1/21330 just changed the state of lock: [66571.610877] (sb_internal){.+.+.?}, at: [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50 [66571.610885] but this lock took another, RECLAIM_FS-unsafe lock in the past: [66571.610886] (&(&ip->i_lock)->mr_lock/1){+.+.+.} [66571.610886] [66571.610886] and interrupts could create inverse lock ordering between them. [66571.610886] [66571.610890] [66571.610890] other info that might help us debug this: [66571.610891] Possible interrupt unsafe locking scenario: [66571.610891] [66571.610892] CPU0 CPU1 [66571.610893] ---- ---- [66571.610894] lock(&(&ip->i_lock)->mr_lock/1); [66571.610896] local_irq_disable(); [66571.610897] lock(sb_internal); [66571.610898] lock(&(&ip->i_lock)->mr_lock/1); [66571.610900] <Interrupt> [66571.610901] lock(sb_internal); [66571.610902] [66571.610902] *** DEADLOCK *** [66571.610902] [66571.610904] 3 locks held by cc1/21330: [66571.610905] #0: (&mm->mmap_sem){++++++}, at: [<ffffffff81029d8b>] __do_page_fault+0xfb/0x480 [66571.610910] #1: (shrinker_rwsem){++++..}, at: [<ffffffff810bbd02>] shrink_slab+0x32/0x1f0 [66571.610915] #2: (&type->s_umount_key#20){++++.+}, at: [<ffffffff810f5a7e>] grab_super_passive+0x3e/0x90 [66571.610921] [66571.610921] the shortest dependencies between 2nd lock and 1st lock: [66571.610927] -> (&(&ip->i_lock)->mr_lock/1){+.+.+.} ops: 169649 { [66571.610931] HARDIRQ-ON-W at: [66571.610932] [<ffffffff8107f091>] __lock_acquire+0x631/0x1c00 [66571.610935] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [66571.610937] [<ffffffff8106126a>] down_write_nested+0x4a/0x70 [66571.610941] [<ffffffff811e8164>] xfs_ilock+0x84/0xb0 [66571.610944] [<ffffffff811f51a4>] xfs_create+0x1d4/0x5a0 [66571.610946] [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 [66571.610948] [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 [66571.610950] [<ffffffff81100322>] vfs_create+0x72/0xc0 [66571.610953] [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 [66571.610955] [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 [66571.610957] [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 [66571.610959] [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 [66571.610962] [<ffffffff810f222c>] sys_open+0x1c/0x20 [66571.610964] [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b [66571.610967] SOFTIRQ-ON-W at: [66571.610968] [<ffffffff8107f0c7>] __lock_acquire+0x667/0x1c00 [66571.610970] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [66571.610972] [<ffffffff8106126a>] down_write_nested+0x4a/0x70 [66571.610974] [<ffffffff811e8164>] xfs_ilock+0x84/0xb0 [66571.610976] [<ffffffff811f51a4>] xfs_create+0x1d4/0x5a0 [66571.610977] [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 [66571.610979] [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 [66571.610981] [<ffffffff81100322>] vfs_create+0x72/0xc0 [66571.610983] [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 [66571.610985] [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 [66571.610987] [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 [66571.610989] [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 [66571.610991] [<ffffffff810f222c>] sys_open+0x1c/0x20 [66571.610993] [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b [66571.610995] RECLAIM_FS-ON-W at: [66571.610996] [<ffffffff8108137e>] mark_held_locks+0x7e/0x130 [66571.610998] [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0 [66571.610999] [<ffffffff810e9dc5>] kmem_cache_alloc+0x35/0xe0 [66571.611002] [<ffffffff810dba21>] vm_map_ram+0x271/0x770 [66571.611004] [<ffffffff811e12b6>] _xfs_buf_map_pages+0x46/0xe0 [66571.611008] [<ffffffff811e21ca>] xfs_buf_get_map+0x8a/0x130 [66571.611009] [<ffffffff81233989>] xfs_trans_get_buf_map+0xa9/0xd0 [66571.611011] [<ffffffff8121bc2d>] xfs_ialloc_inode_init+0xcd/0x1d0 [66571.611015] [<ffffffff8121c16f>] xfs_ialloc_ag_alloc+0x18f/0x500 [66571.611017] [<ffffffff8121d955>] xfs_dialloc+0x185/0x2a0 [66571.611019] [<ffffffff8121f068>] xfs_ialloc+0x58/0x650 [66571.611021] [<ffffffff811f3995>] xfs_dir_ialloc+0x65/0x270 [66571.611023] [<ffffffff811f537c>] xfs_create+0x3ac/0x5a0 [66571.611024] [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 [66571.611026] [<ffffffff811ecb61>] xfs_vn_mkdir+0x11/0x20 [66571.611028] [<ffffffff8110016f>] vfs_mkdir+0x7f/0xd0 [66571.611030] [<ffffffff81101b83>] sys_mkdirat+0x43/0x80 [66571.611032] [<ffffffff81101bd4>] sys_mkdir+0x14/0x20 [66571.611034] [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b [66571.611036] INITIAL USE at: [66571.611037] [<ffffffff8107ed49>] __lock_acquire+0x2e9/0x1c00 [66571.611038] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [66571.611040] [<ffffffff8106126a>] down_write_nested+0x4a/0x70 [66571.611042] [<ffffffff811e8164>] xfs_ilock+0x84/0xb0 [66571.611044] [<ffffffff811f51a4>] xfs_create+0x1d4/0x5a0 [66571.611046] [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 [66571.611047] [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 [66571.611049] [<ffffffff81100322>] vfs_create+0x72/0xc0 [66571.611051] [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 [66571.611053] [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 [66571.611055] [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 [66571.611057] [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 [66571.611059] [<ffffffff810f222c>] sys_open+0x1c/0x20 [66571.611061] [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b [66571.611063] } [66571.611064] ... key at: [<ffffffff825b4b81>] __key.41357+0x1/0x8 [66571.611066] ... acquired at: [66571.611067] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [66571.611069] [<ffffffff8106126a>] down_write_nested+0x4a/0x70 [66571.611071] [<ffffffff811e8164>] xfs_ilock+0x84/0xb0 [66571.611073] [<ffffffff811f51a4>] xfs_create+0x1d4/0x5a0 [66571.611074] [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 [66571.611076] [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 [66571.611078] [<ffffffff81100322>] vfs_create+0x72/0xc0 [66571.611080] [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 [66571.611082] [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 [66571.611084] [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 [66571.611086] [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 [66571.611088] [<ffffffff810f222c>] sys_open+0x1c/0x20 [66571.611090] [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b [66571.611091] [66571.611092] -> (sb_internal){.+.+.?} ops: 1341531 { [66571.611095] HARDIRQ-ON-R at: [66571.611096] [<ffffffff8107ef6a>] __lock_acquire+0x50a/0x1c00 [66571.611098] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [66571.611100] [<ffffffff810f451b>] __sb_start_write+0xab/0x190 [66571.611102] [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50 [66571.611104] [<ffffffff811f5157>] xfs_create+0x187/0x5a0 [66571.611105] [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 [66571.611107] [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 [66571.611109] [<ffffffff81100322>] vfs_create+0x72/0xc0 [66571.611111] [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 [66571.611113] [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 [66571.611115] [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 [66571.611117] [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 [66571.611119] [<ffffffff810f222c>] sys_open+0x1c/0x20 [66571.611121] [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b [66571.611123] SOFTIRQ-ON-R at: [66571.611124] [<ffffffff8107f0c7>] __lock_acquire+0x667/0x1c00 [66571.611126] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [66571.611128] [<ffffffff810f451b>] __sb_start_write+0xab/0x190 [66571.611130] [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50 [66571.611132] [<ffffffff811f5157>] xfs_create+0x187/0x5a0 [66571.611133] [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 [66571.611135] [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 [66571.611137] [<ffffffff81100322>] vfs_create+0x72/0xc0 [66571.611139] [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 [66571.611141] [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 [66571.611143] [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 [66571.611145] [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 [66571.611147] [<ffffffff810f222c>] sys_open+0x1c/0x20 [66571.611149] [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b [66571.611151] IN-RECLAIM_FS-R at: [66571.611152] [<ffffffff8107efdf>] __lock_acquire+0x57f/0x1c00 [66571.611154] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [66571.611156] [<ffffffff810f451b>] __sb_start_write+0xab/0x190 [66571.611158] [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50 [66571.611159] [<ffffffff811f3e64>] xfs_free_eofblocks+0x104/0x250 [66571.611161] [<ffffffff811f4b39>] xfs_inactive+0xa9/0x480 [66571.611163] [<ffffffff811efe10>] xfs_fs_evict_inode+0x70/0x80 [66571.611165] [<ffffffff8110cb7f>] evict+0xaf/0x1b0 [66571.611168] [<ffffffff8110d209>] dispose_list+0x39/0x50 [66571.611170] [<ffffffff8110dc63>] prune_icache_sb+0x183/0x340 [66571.611172] [<ffffffff810f5bc3>] prune_super+0xf3/0x1a0 [66571.611173] [<ffffffff810bbdee>] shrink_slab+0x11e/0x1f0 [66571.611175] [<ffffffff810be98f>] try_to_free_pages+0x21f/0x4e0 [66571.611177] [<ffffffff810b5ec6>] __alloc_pages_nodemask+0x506/0x800 [66571.611179] [<ffffffff810ce56e>] handle_pte_fault+0x5ae/0x7a0 [66571.611182] [<ffffffff810cf769>] handle_mm_fault+0x1f9/0x2a0 [66571.611184] [<ffffffff81029dfc>] __do_page_fault+0x16c/0x480 [66571.611186] [<ffffffff8102a139>] do_page_fault+0x9/0x10 [66571.611188] [<ffffffff816b287f>] page_fault+0x1f/0x30 [66571.611190] RECLAIM_FS-ON-R at: [66571.611191] [<ffffffff8108137e>] mark_held_locks+0x7e/0x130 [66571.611193] [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0 [66571.611195] [<ffffffff810e9dc5>] kmem_cache_alloc+0x35/0xe0 [66571.611197] [<ffffffff811f6d4f>] kmem_zone_alloc+0x5f/0xe0 [66571.611198] [<ffffffff811f6de8>] kmem_zone_zalloc+0x18/0x50 [66571.611200] [<ffffffff8122b0b2>] _xfs_trans_alloc+0x32/0x90 [66571.611202] [<ffffffff8122b148>] xfs_trans_alloc+0x38/0x50 [66571.611204] [<ffffffff811f5157>] xfs_create+0x187/0x5a0 [66571.611205] [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 [66571.611207] [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 [66571.611209] [<ffffffff81100322>] vfs_create+0x72/0xc0 [66571.611211] [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 [66571.611213] [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 [66571.611215] [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 [66571.611217] [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 [66571.611219] [<ffffffff810f222c>] sys_open+0x1c/0x20 [66571.611221] [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b [66571.611223] INITIAL USE at: [66571.611224] [<ffffffff8107ed49>] __lock_acquire+0x2e9/0x1c00 [66571.611225] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [66571.611227] [<ffffffff810f451b>] __sb_start_write+0xab/0x190 [66571.611229] [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50 [66571.611231] [<ffffffff811f5157>] xfs_create+0x187/0x5a0 [66571.611232] [<ffffffff811eca2a>] xfs_vn_mknod+0x8a/0x1b0 [66571.611234] [<ffffffff811ecb7e>] xfs_vn_create+0xe/0x10 [66571.611236] [<ffffffff81100322>] vfs_create+0x72/0xc0 [66571.611238] [<ffffffff81100b7e>] do_last.isra.69+0x80e/0xc80 [66571.611240] [<ffffffff8110109b>] path_openat.isra.70+0xab/0x490 [66571.611242] [<ffffffff8110183d>] do_filp_open+0x3d/0xa0 [66571.611244] [<ffffffff810f2129>] do_sys_open+0xf9/0x1e0 [66571.611246] [<ffffffff810f222c>] sys_open+0x1c/0x20 [66571.611248] [<ffffffff816b2e52>] system_call_fastpath+0x16/0x1b [66571.611250] } [66571.611251] ... key at: [<ffffffff81c34e40>] xfs_fs_type+0x60/0x80 [66571.611254] ... acquired at: [66571.611254] [<ffffffff8107df3b>] check_usage_forwards+0x10b/0x140 [66571.611256] [<ffffffff8107e900>] mark_lock+0x190/0x2f0 [66571.611258] [<ffffffff8107efdf>] __lock_acquire+0x57f/0x1c00 [66571.611260] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [66571.611261] [<ffffffff810f451b>] __sb_start_write+0xab/0x190 [66571.611263] [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50 [66571.611265] [<ffffffff811f3e64>] xfs_free_eofblocks+0x104/0x250 [66571.611266] [<ffffffff811f4b39>] xfs_inactive+0xa9/0x480 [66571.611268] [<ffffffff811efe10>] xfs_fs_evict_inode+0x70/0x80 [66571.611270] [<ffffffff8110cb7f>] evict+0xaf/0x1b0 [66571.611271] [<ffffffff8110d209>] dispose_list+0x39/0x50 [66571.611273] [<ffffffff8110dc63>] prune_icache_sb+0x183/0x340 [66571.611275] [<ffffffff810f5bc3>] prune_super+0xf3/0x1a0 [66571.611277] [<ffffffff810bbdee>] shrink_slab+0x11e/0x1f0 [66571.611278] [<ffffffff810be98f>] try_to_free_pages+0x21f/0x4e0 [66571.611280] [<ffffffff810b5ec6>] __alloc_pages_nodemask+0x506/0x800 [66571.611282] [<ffffffff810ce56e>] handle_pte_fault+0x5ae/0x7a0 [66571.611284] [<ffffffff810cf769>] handle_mm_fault+0x1f9/0x2a0 [66571.611286] [<ffffffff81029dfc>] __do_page_fault+0x16c/0x480 [66571.611287] [<ffffffff8102a139>] do_page_fault+0x9/0x10 [66571.611289] [<ffffffff816b287f>] page_fault+0x1f/0x30 [66571.611291] [66571.611292] [66571.611292] stack backtrace: [66571.611294] Pid: 21330, comm: cc1 Not tainted 3.7.0-rc5 #1 [66571.611295] Call Trace: [66571.611298] [<ffffffff8107de28>] print_irq_inversion_bug.part.37+0x1e8/0x1f0 [66571.611300] [<ffffffff8107df3b>] check_usage_forwards+0x10b/0x140 [66571.611303] [<ffffffff8107e900>] mark_lock+0x190/0x2f0 [66571.611306] [<ffffffff8150406e>] ? dm_request+0x2e/0x2a0 [66571.611308] [<ffffffff8107de30>] ? print_irq_inversion_bug.part.37+0x1f0/0x1f0 [66571.611310] [<ffffffff8107efdf>] __lock_acquire+0x57f/0x1c00 [66571.611313] [<ffffffff812202f4>] ? xfs_iext_bno_to_ext+0x84/0x160 [66571.611316] [<ffffffff8120a023>] ? xfs_bmbt_get_all+0x13/0x20 [66571.611318] [<ffffffff81200fb4>] ? xfs_bmap_search_multi_extents+0xa4/0x110 [66571.611320] [<ffffffff81080b55>] lock_acquire+0x55/0x70 [66571.611322] [<ffffffff8122b138>] ? xfs_trans_alloc+0x28/0x50 [66571.611324] [<ffffffff810f451b>] __sb_start_write+0xab/0x190 [66571.611326] [<ffffffff8122b138>] ? xfs_trans_alloc+0x28/0x50 [66571.611328] [<ffffffff8122b138>] ? xfs_trans_alloc+0x28/0x50 [66571.611330] [<ffffffff8122b138>] xfs_trans_alloc+0x28/0x50 [66571.611332] [<ffffffff811f3e64>] xfs_free_eofblocks+0x104/0x250 [66571.611334] [<ffffffff816b204b>] ? _raw_spin_unlock_irq+0x2b/0x50 [66571.611336] [<ffffffff811f4b39>] xfs_inactive+0xa9/0x480 [66571.611337] [<ffffffff816b204b>] ? _raw_spin_unlock_irq+0x2b/0x50 [66571.611340] [<ffffffff811efe10>] xfs_fs_evict_inode+0x70/0x80 [66571.611342] [<ffffffff8110cb7f>] evict+0xaf/0x1b0 [66571.611344] [<ffffffff8110d209>] dispose_list+0x39/0x50 [66571.611346] [<ffffffff8110dc63>] prune_icache_sb+0x183/0x340 [66571.611347] [<ffffffff810f5bc3>] prune_super+0xf3/0x1a0 [66571.611349] [<ffffffff810bbdee>] shrink_slab+0x11e/0x1f0 [66571.611352] [<ffffffff810be98f>] try_to_free_pages+0x21f/0x4e0 [66571.611354] [<ffffffff810b5ec6>] __alloc_pages_nodemask+0x506/0x800 [66571.611356] [<ffffffff810b9e40>] ? lru_deactivate_fn+0x1c0/0x1c0 [66571.611358] [<ffffffff810ce56e>] handle_pte_fault+0x5ae/0x7a0 [66571.611360] [<ffffffff810cf769>] handle_mm_fault+0x1f9/0x2a0 [66571.611363] [<ffffffff81029dfc>] __do_page_fault+0x16c/0x480 [66571.611366] [<ffffffff8129c7ad>] ? trace_hardirqs_off_thunk+0x3a/0x3c [66571.611368] [<ffffffff8102a139>] do_page_fault+0x9/0x10 [66571.611370] [<ffffffff816b287f>] page_fault+0x1f/0x30 [-- Attachment #3: Type: text/plain, Size: 121 bytes --] _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Hang in XFS reclaim on 3.7.0-rc3 2012-11-19 6:50 ` Torsten Kaiser @ 2012-11-19 23:53 ` Dave Chinner -1 siblings, 0 replies; 31+ messages in thread From: Dave Chinner @ 2012-11-19 23:53 UTC (permalink / raw) To: Torsten Kaiser; +Cc: xfs, Linux Kernel On Mon, Nov 19, 2012 at 07:50:06AM +0100, Torsten Kaiser wrote: > On Mon, Nov 19, 2012 at 12:51 AM, Dave Chinner <david@fromorbit.com> wrote: > > On Sun, Nov 18, 2012 at 04:29:22PM +0100, Torsten Kaiser wrote: > >> On Sun, Nov 18, 2012 at 11:24 AM, Torsten Kaiser > >> <just.for.lkml@googlemail.com> wrote: > >> > On Tue, Oct 30, 2012 at 9:37 PM, Torsten Kaiser > >> > <just.for.lkml@googlemail.com> wrote: > >> >> I will keep LOCKDEP enabled on that system, and if there really is > >> >> another splat, I will report back here. But I rather doubt that this > >> >> will be needed. > >> > > >> > After the patch, I did not see this problem again, but today I found > >> > another LOCKDEP report that also looks XFS related. > >> > I found it twice in the logs, and as both were slightly different, I > >> > will attach both versions. > >> > >> > Nov 6 21:57:09 thoregon kernel: [ 9941.104353] 3.7.0-rc4 #1 Not tainted > >> > Nov 6 21:57:09 thoregon kernel: [ 9941.104355] inconsistent > >> > {RECLAIM_FS-ON-W} -> {IN-RECLAIM_FS-W} usage. > >> > Nov 6 21:57:09 thoregon kernel: [ 9941.104430] CPU0 > >> > Nov 6 21:57:09 thoregon kernel: [ 9941.104431] ---- > >> > Nov 6 21:57:09 thoregon kernel: [ 9941.104432] lock(&(&ip->i_lock)->mr_lock); > >> > Nov 6 21:57:09 thoregon kernel: [ 9941.104433] <Interrupt> > >> > Nov 6 21:57:09 thoregon kernel: [ 9941.104434] > >> > lock(&(&ip->i_lock)->mr_lock); > >> > Nov 6 21:57:09 thoregon kernel: [ 9941.104435] > >> > Nov 6 21:57:09 thoregon kernel: [ 9941.104435] *** DEADLOCK *** > >> > >> Sorry! Copied the wrong report. Your fix only landed in -rc5, so my > >> vanilla -rc4 did (also) report the old problem again. > >> And I copy&pasted that report instead of the second appearance of the > >> new problem. > > > > Can you repost it with line wrapping turned off? The output simply > > becomes unreadable when it wraps.... > > > > Yeah, I know I can put it back together, but I've got better things > > to do with my time than stitch a couple of hundred lines of debug > > back into a readable format.... > > Sorry about that, but I can't find any option to turn that off in Gmail. Seems like you can't, as per Documentation/email-clients.txt > I have added the reports as attachment, I hope thats OK for you. Encoded as text, so it does. So, both lockdep thingy's are the same: > [110926.972482] ========================================================= > [110926.972484] [ INFO: possible irq lock inversion dependency detected ] > [110926.972486] 3.7.0-rc4 #1 Not tainted > [110926.972487] --------------------------------------------------------- > [110926.972489] kswapd0/725 just changed the state of lock: > [110926.972490] (sb_internal){.+.+.?}, at: [<ffffffff8122b268>] xfs_trans_alloc+0x28/0x50 > [110926.972499] but this lock took another, RECLAIM_FS-unsafe lock in the past: > [110926.972500] (&(&ip->i_lock)->mr_lock/1){+.+.+.} Ah, what? Since when has the ilock been reclaim unsafe? > [110926.972500] and interrupts could create inverse lock ordering between them. > [110926.972500] > [110926.972503] > [110926.972503] other info that might help us debug this: > [110926.972504] Possible interrupt unsafe locking scenario: > [110926.972504] > [110926.972505] CPU0 CPU1 > [110926.972506] ---- ---- > [110926.972507] lock(&(&ip->i_lock)->mr_lock/1); > [110926.972509] local_irq_disable(); > [110926.972509] lock(sb_internal); > [110926.972511] lock(&(&ip->i_lock)->mr_lock/1); > [110926.972512] <Interrupt> > [110926.972513] lock(sb_internal); Um, that's just bizzare. No XFS code runs with interrupts disabled, so I cannot see how this possible. ..... [<ffffffff8108137e>] mark_held_locks+0x7e/0x130 [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0 [<ffffffff810e9dd5>] kmem_cache_alloc+0x35/0xe0 [<ffffffff810dba31>] vm_map_ram+0x271/0x770 [<ffffffff811e1316>] _xfs_buf_map_pages+0x46/0xe0 [<ffffffff811e222a>] xfs_buf_get_map+0x8a/0x130 [<ffffffff81233ab9>] xfs_trans_get_buf_map+0xa9/0xd0 [<ffffffff8121bced>] xfs_ialloc_inode_init+0xcd/0x1d0 We shouldn't be mapping buffers there, there's a patch below to fix this. It's probably the source of this report, even though I cannot lockdep seems to be off with the fairies... Cheers, Dave. -- Dave Chinner david@fromorbit.com xfs: inode allocation should use unmapped buffers. From: Dave Chinner <dchinner@redhat.com> Inode buffers do not need to be mapped as inodes are read or written directly from/to the pages underlying the buffer. This fixes a regression introduced by commit 611c994 ("xfs: make XBF_MAPPED the default behaviour"). Signed-off-by: Dave Chinner <dchinner@redhat.com> --- fs/xfs/xfs_ialloc.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/fs/xfs/xfs_ialloc.c b/fs/xfs/xfs_ialloc.c index 2d6495e..a815412 100644 --- a/fs/xfs/xfs_ialloc.c +++ b/fs/xfs/xfs_ialloc.c @@ -200,7 +200,8 @@ xfs_ialloc_inode_init( */ d = XFS_AGB_TO_DADDR(mp, agno, agbno + (j * blks_per_cluster)); fbuf = xfs_trans_get_buf(tp, mp->m_ddev_targp, d, - mp->m_bsize * blks_per_cluster, 0); + mp->m_bsize * blks_per_cluster, + XBF_UNMAPPED); if (!fbuf) return ENOMEM; /* ^ permalink raw reply related [flat|nested] 31+ messages in thread
* Re: Hang in XFS reclaim on 3.7.0-rc3 @ 2012-11-19 23:53 ` Dave Chinner 0 siblings, 0 replies; 31+ messages in thread From: Dave Chinner @ 2012-11-19 23:53 UTC (permalink / raw) To: Torsten Kaiser; +Cc: Linux Kernel, xfs On Mon, Nov 19, 2012 at 07:50:06AM +0100, Torsten Kaiser wrote: > On Mon, Nov 19, 2012 at 12:51 AM, Dave Chinner <david@fromorbit.com> wrote: > > On Sun, Nov 18, 2012 at 04:29:22PM +0100, Torsten Kaiser wrote: > >> On Sun, Nov 18, 2012 at 11:24 AM, Torsten Kaiser > >> <just.for.lkml@googlemail.com> wrote: > >> > On Tue, Oct 30, 2012 at 9:37 PM, Torsten Kaiser > >> > <just.for.lkml@googlemail.com> wrote: > >> >> I will keep LOCKDEP enabled on that system, and if there really is > >> >> another splat, I will report back here. But I rather doubt that this > >> >> will be needed. > >> > > >> > After the patch, I did not see this problem again, but today I found > >> > another LOCKDEP report that also looks XFS related. > >> > I found it twice in the logs, and as both were slightly different, I > >> > will attach both versions. > >> > >> > Nov 6 21:57:09 thoregon kernel: [ 9941.104353] 3.7.0-rc4 #1 Not tainted > >> > Nov 6 21:57:09 thoregon kernel: [ 9941.104355] inconsistent > >> > {RECLAIM_FS-ON-W} -> {IN-RECLAIM_FS-W} usage. > >> > Nov 6 21:57:09 thoregon kernel: [ 9941.104430] CPU0 > >> > Nov 6 21:57:09 thoregon kernel: [ 9941.104431] ---- > >> > Nov 6 21:57:09 thoregon kernel: [ 9941.104432] lock(&(&ip->i_lock)->mr_lock); > >> > Nov 6 21:57:09 thoregon kernel: [ 9941.104433] <Interrupt> > >> > Nov 6 21:57:09 thoregon kernel: [ 9941.104434] > >> > lock(&(&ip->i_lock)->mr_lock); > >> > Nov 6 21:57:09 thoregon kernel: [ 9941.104435] > >> > Nov 6 21:57:09 thoregon kernel: [ 9941.104435] *** DEADLOCK *** > >> > >> Sorry! Copied the wrong report. Your fix only landed in -rc5, so my > >> vanilla -rc4 did (also) report the old problem again. > >> And I copy&pasted that report instead of the second appearance of the > >> new problem. > > > > Can you repost it with line wrapping turned off? The output simply > > becomes unreadable when it wraps.... > > > > Yeah, I know I can put it back together, but I've got better things > > to do with my time than stitch a couple of hundred lines of debug > > back into a readable format.... > > Sorry about that, but I can't find any option to turn that off in Gmail. Seems like you can't, as per Documentation/email-clients.txt > I have added the reports as attachment, I hope thats OK for you. Encoded as text, so it does. So, both lockdep thingy's are the same: > [110926.972482] ========================================================= > [110926.972484] [ INFO: possible irq lock inversion dependency detected ] > [110926.972486] 3.7.0-rc4 #1 Not tainted > [110926.972487] --------------------------------------------------------- > [110926.972489] kswapd0/725 just changed the state of lock: > [110926.972490] (sb_internal){.+.+.?}, at: [<ffffffff8122b268>] xfs_trans_alloc+0x28/0x50 > [110926.972499] but this lock took another, RECLAIM_FS-unsafe lock in the past: > [110926.972500] (&(&ip->i_lock)->mr_lock/1){+.+.+.} Ah, what? Since when has the ilock been reclaim unsafe? > [110926.972500] and interrupts could create inverse lock ordering between them. > [110926.972500] > [110926.972503] > [110926.972503] other info that might help us debug this: > [110926.972504] Possible interrupt unsafe locking scenario: > [110926.972504] > [110926.972505] CPU0 CPU1 > [110926.972506] ---- ---- > [110926.972507] lock(&(&ip->i_lock)->mr_lock/1); > [110926.972509] local_irq_disable(); > [110926.972509] lock(sb_internal); > [110926.972511] lock(&(&ip->i_lock)->mr_lock/1); > [110926.972512] <Interrupt> > [110926.972513] lock(sb_internal); Um, that's just bizzare. No XFS code runs with interrupts disabled, so I cannot see how this possible. ..... [<ffffffff8108137e>] mark_held_locks+0x7e/0x130 [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0 [<ffffffff810e9dd5>] kmem_cache_alloc+0x35/0xe0 [<ffffffff810dba31>] vm_map_ram+0x271/0x770 [<ffffffff811e1316>] _xfs_buf_map_pages+0x46/0xe0 [<ffffffff811e222a>] xfs_buf_get_map+0x8a/0x130 [<ffffffff81233ab9>] xfs_trans_get_buf_map+0xa9/0xd0 [<ffffffff8121bced>] xfs_ialloc_inode_init+0xcd/0x1d0 We shouldn't be mapping buffers there, there's a patch below to fix this. It's probably the source of this report, even though I cannot lockdep seems to be off with the fairies... Cheers, Dave. -- Dave Chinner david@fromorbit.com xfs: inode allocation should use unmapped buffers. From: Dave Chinner <dchinner@redhat.com> Inode buffers do not need to be mapped as inodes are read or written directly from/to the pages underlying the buffer. This fixes a regression introduced by commit 611c994 ("xfs: make XBF_MAPPED the default behaviour"). Signed-off-by: Dave Chinner <dchinner@redhat.com> --- fs/xfs/xfs_ialloc.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/fs/xfs/xfs_ialloc.c b/fs/xfs/xfs_ialloc.c index 2d6495e..a815412 100644 --- a/fs/xfs/xfs_ialloc.c +++ b/fs/xfs/xfs_ialloc.c @@ -200,7 +200,8 @@ xfs_ialloc_inode_init( */ d = XFS_AGB_TO_DADDR(mp, agno, agbno + (j * blks_per_cluster)); fbuf = xfs_trans_get_buf(tp, mp->m_ddev_targp, d, - mp->m_bsize * blks_per_cluster, 0); + mp->m_bsize * blks_per_cluster, + XBF_UNMAPPED); if (!fbuf) return ENOMEM; /* _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply related [flat|nested] 31+ messages in thread
* Re: Hang in XFS reclaim on 3.7.0-rc3 2012-11-19 23:53 ` Dave Chinner @ 2012-11-20 7:09 ` Torsten Kaiser -1 siblings, 0 replies; 31+ messages in thread From: Torsten Kaiser @ 2012-11-20 7:09 UTC (permalink / raw) To: Dave Chinner; +Cc: xfs, Linux Kernel On Tue, Nov 20, 2012 at 12:53 AM, Dave Chinner <david@fromorbit.com> wrote: > On Mon, Nov 19, 2012 at 07:50:06AM +0100, Torsten Kaiser wrote: > So, both lockdep thingy's are the same: I suspected this, but as the reports where slightly different I attached bith of them, as I couldn't decide witch one was the better/simpler report to debug this. >> [110926.972482] ========================================================= >> [110926.972484] [ INFO: possible irq lock inversion dependency detected ] >> [110926.972486] 3.7.0-rc4 #1 Not tainted >> [110926.972487] --------------------------------------------------------- >> [110926.972489] kswapd0/725 just changed the state of lock: >> [110926.972490] (sb_internal){.+.+.?}, at: [<ffffffff8122b268>] xfs_trans_alloc+0x28/0x50 >> [110926.972499] but this lock took another, RECLAIM_FS-unsafe lock in the past: >> [110926.972500] (&(&ip->i_lock)->mr_lock/1){+.+.+.} > > Ah, what? Since when has the ilock been reclaim unsafe? > >> [110926.972500] and interrupts could create inverse lock ordering between them. >> [110926.972500] >> [110926.972503] >> [110926.972503] other info that might help us debug this: >> [110926.972504] Possible interrupt unsafe locking scenario: >> [110926.972504] >> [110926.972505] CPU0 CPU1 >> [110926.972506] ---- ---- >> [110926.972507] lock(&(&ip->i_lock)->mr_lock/1); >> [110926.972509] local_irq_disable(); >> [110926.972509] lock(sb_internal); >> [110926.972511] lock(&(&ip->i_lock)->mr_lock/1); >> [110926.972512] <Interrupt> >> [110926.972513] lock(sb_internal); > > Um, that's just bizzare. No XFS code runs with interrupts disabled, > so I cannot see how this possible. > > ..... > > > [<ffffffff8108137e>] mark_held_locks+0x7e/0x130 > [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0 > [<ffffffff810e9dd5>] kmem_cache_alloc+0x35/0xe0 > [<ffffffff810dba31>] vm_map_ram+0x271/0x770 > [<ffffffff811e1316>] _xfs_buf_map_pages+0x46/0xe0 > [<ffffffff811e222a>] xfs_buf_get_map+0x8a/0x130 > [<ffffffff81233ab9>] xfs_trans_get_buf_map+0xa9/0xd0 > [<ffffffff8121bced>] xfs_ialloc_inode_init+0xcd/0x1d0 > > We shouldn't be mapping buffers there, there's a patch below to fix > this. It's probably the source of this report, even though I cannot > lockdep seems to be off with the fairies... I also tried to understand what lockdep was saying, but Documentation/lockdep-design.txt is not too helpful. I think 'CLASS'-ON-R / -ON-W means that this lock was 'ON' / held while 'CLASS' (HARDIRQ, SOFTIRQ, RECLAIM_FS) happend and that makes this lock unsafe for these contexts. IN-'CLASS'-R / -W seems to be 'lock taken in context 'CLASS'. A note that 'CLASS'-ON-? means 'CLASS'-unsafe in there would be helpful to me... Wrt. above interrupt output: I think lockdep doesn't really know about RECLAIM_FS and threats it as another interrupt. I think that output should have been something like this: CPU0 CPU1 ---- ---- lock(&(&ip->i_lock)->mr_lock/1); <Allocation enters reclaim> lock(sb_internal); lock(&(&ip->i_lock)->mr_lock/1); <Allocation enters reclaim> lock(sb_internal); Entering reclaim on CPU1 would mean that CPU1 would not enter reclaim again, so the reclaim-'interrupt' would be disabled. And instead of interrupts disrupting the normal codeflow on CPU0 it would be 'interrupted' be instead of doing a normal allocation, it would 'interrupt' the allocation to reclaim memory. print_irq_lock_scenario() would need to be taught to print a slightly different message for reclaim-'interrupts'. I will try your patch, but as I do not have a reliable reproducer to create this lockdep report, I can't really verify if this fixes it. But I will definitely mail you, if it happens again with this patch. Thanks, Torsten > Cheers, > > Dave. > -- > Dave Chinner > david@fromorbit.com > > xfs: inode allocation should use unmapped buffers. > > From: Dave Chinner <dchinner@redhat.com> > > Inode buffers do not need to be mapped as inodes are read or written > directly from/to the pages underlying the buffer. This fixes a > regression introduced by commit 611c994 ("xfs: make XBF_MAPPED the > default behaviour"). > > Signed-off-by: Dave Chinner <dchinner@redhat.com> > --- > fs/xfs/xfs_ialloc.c | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) > > diff --git a/fs/xfs/xfs_ialloc.c b/fs/xfs/xfs_ialloc.c > index 2d6495e..a815412 100644 > --- a/fs/xfs/xfs_ialloc.c > +++ b/fs/xfs/xfs_ialloc.c > @@ -200,7 +200,8 @@ xfs_ialloc_inode_init( > */ > d = XFS_AGB_TO_DADDR(mp, agno, agbno + (j * blks_per_cluster)); > fbuf = xfs_trans_get_buf(tp, mp->m_ddev_targp, d, > - mp->m_bsize * blks_per_cluster, 0); > + mp->m_bsize * blks_per_cluster, > + XBF_UNMAPPED); > if (!fbuf) > return ENOMEM; > /* ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Hang in XFS reclaim on 3.7.0-rc3 @ 2012-11-20 7:09 ` Torsten Kaiser 0 siblings, 0 replies; 31+ messages in thread From: Torsten Kaiser @ 2012-11-20 7:09 UTC (permalink / raw) To: Dave Chinner; +Cc: Linux Kernel, xfs On Tue, Nov 20, 2012 at 12:53 AM, Dave Chinner <david@fromorbit.com> wrote: > On Mon, Nov 19, 2012 at 07:50:06AM +0100, Torsten Kaiser wrote: > So, both lockdep thingy's are the same: I suspected this, but as the reports where slightly different I attached bith of them, as I couldn't decide witch one was the better/simpler report to debug this. >> [110926.972482] ========================================================= >> [110926.972484] [ INFO: possible irq lock inversion dependency detected ] >> [110926.972486] 3.7.0-rc4 #1 Not tainted >> [110926.972487] --------------------------------------------------------- >> [110926.972489] kswapd0/725 just changed the state of lock: >> [110926.972490] (sb_internal){.+.+.?}, at: [<ffffffff8122b268>] xfs_trans_alloc+0x28/0x50 >> [110926.972499] but this lock took another, RECLAIM_FS-unsafe lock in the past: >> [110926.972500] (&(&ip->i_lock)->mr_lock/1){+.+.+.} > > Ah, what? Since when has the ilock been reclaim unsafe? > >> [110926.972500] and interrupts could create inverse lock ordering between them. >> [110926.972500] >> [110926.972503] >> [110926.972503] other info that might help us debug this: >> [110926.972504] Possible interrupt unsafe locking scenario: >> [110926.972504] >> [110926.972505] CPU0 CPU1 >> [110926.972506] ---- ---- >> [110926.972507] lock(&(&ip->i_lock)->mr_lock/1); >> [110926.972509] local_irq_disable(); >> [110926.972509] lock(sb_internal); >> [110926.972511] lock(&(&ip->i_lock)->mr_lock/1); >> [110926.972512] <Interrupt> >> [110926.972513] lock(sb_internal); > > Um, that's just bizzare. No XFS code runs with interrupts disabled, > so I cannot see how this possible. > > ..... > > > [<ffffffff8108137e>] mark_held_locks+0x7e/0x130 > [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0 > [<ffffffff810e9dd5>] kmem_cache_alloc+0x35/0xe0 > [<ffffffff810dba31>] vm_map_ram+0x271/0x770 > [<ffffffff811e1316>] _xfs_buf_map_pages+0x46/0xe0 > [<ffffffff811e222a>] xfs_buf_get_map+0x8a/0x130 > [<ffffffff81233ab9>] xfs_trans_get_buf_map+0xa9/0xd0 > [<ffffffff8121bced>] xfs_ialloc_inode_init+0xcd/0x1d0 > > We shouldn't be mapping buffers there, there's a patch below to fix > this. It's probably the source of this report, even though I cannot > lockdep seems to be off with the fairies... I also tried to understand what lockdep was saying, but Documentation/lockdep-design.txt is not too helpful. I think 'CLASS'-ON-R / -ON-W means that this lock was 'ON' / held while 'CLASS' (HARDIRQ, SOFTIRQ, RECLAIM_FS) happend and that makes this lock unsafe for these contexts. IN-'CLASS'-R / -W seems to be 'lock taken in context 'CLASS'. A note that 'CLASS'-ON-? means 'CLASS'-unsafe in there would be helpful to me... Wrt. above interrupt output: I think lockdep doesn't really know about RECLAIM_FS and threats it as another interrupt. I think that output should have been something like this: CPU0 CPU1 ---- ---- lock(&(&ip->i_lock)->mr_lock/1); <Allocation enters reclaim> lock(sb_internal); lock(&(&ip->i_lock)->mr_lock/1); <Allocation enters reclaim> lock(sb_internal); Entering reclaim on CPU1 would mean that CPU1 would not enter reclaim again, so the reclaim-'interrupt' would be disabled. And instead of interrupts disrupting the normal codeflow on CPU0 it would be 'interrupted' be instead of doing a normal allocation, it would 'interrupt' the allocation to reclaim memory. print_irq_lock_scenario() would need to be taught to print a slightly different message for reclaim-'interrupts'. I will try your patch, but as I do not have a reliable reproducer to create this lockdep report, I can't really verify if this fixes it. But I will definitely mail you, if it happens again with this patch. Thanks, Torsten > Cheers, > > Dave. > -- > Dave Chinner > david@fromorbit.com > > xfs: inode allocation should use unmapped buffers. > > From: Dave Chinner <dchinner@redhat.com> > > Inode buffers do not need to be mapped as inodes are read or written > directly from/to the pages underlying the buffer. This fixes a > regression introduced by commit 611c994 ("xfs: make XBF_MAPPED the > default behaviour"). > > Signed-off-by: Dave Chinner <dchinner@redhat.com> > --- > fs/xfs/xfs_ialloc.c | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) > > diff --git a/fs/xfs/xfs_ialloc.c b/fs/xfs/xfs_ialloc.c > index 2d6495e..a815412 100644 > --- a/fs/xfs/xfs_ialloc.c > +++ b/fs/xfs/xfs_ialloc.c > @@ -200,7 +200,8 @@ xfs_ialloc_inode_init( > */ > d = XFS_AGB_TO_DADDR(mp, agno, agbno + (j * blks_per_cluster)); > fbuf = xfs_trans_get_buf(tp, mp->m_ddev_targp, d, > - mp->m_bsize * blks_per_cluster, 0); > + mp->m_bsize * blks_per_cluster, > + XBF_UNMAPPED); > if (!fbuf) > return ENOMEM; > /* _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Hang in XFS reclaim on 3.7.0-rc3 2012-11-19 23:53 ` Dave Chinner @ 2012-11-20 19:45 ` Torsten Kaiser -1 siblings, 0 replies; 31+ messages in thread From: Torsten Kaiser @ 2012-11-20 19:45 UTC (permalink / raw) To: Dave Chinner; +Cc: xfs, Linux Kernel [-- Attachment #1: Type: text/plain, Size: 2768 bytes --] On Tue, Nov 20, 2012 at 12:53 AM, Dave Chinner <david@fromorbit.com> wrote: > [<ffffffff8108137e>] mark_held_locks+0x7e/0x130 > [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0 > [<ffffffff810e9dd5>] kmem_cache_alloc+0x35/0xe0 > [<ffffffff810dba31>] vm_map_ram+0x271/0x770 > [<ffffffff811e1316>] _xfs_buf_map_pages+0x46/0xe0 > [<ffffffff811e222a>] xfs_buf_get_map+0x8a/0x130 > [<ffffffff81233ab9>] xfs_trans_get_buf_map+0xa9/0xd0 > [<ffffffff8121bced>] xfs_ialloc_inode_init+0xcd/0x1d0 > > We shouldn't be mapping buffers there, there's a patch below to fix > this. It's probably the source of this report, even though I cannot > lockdep seems to be off with the fairies... That patch seems to break my system. After it started to swap, because I was compiling seamonkey (firefox turned into the full navigator suite) on a tmpfs, several processes got stuck and triggered the hung-task-check. As a kswapd, xfsaild/md4 and flush-9:4 also got stuck, not even a shutdown worked. The attached log first contains the hung-task-notices, then the output from SysRq+W. After the shutdown got stuck trying to turn of swap, I first tries SysRq+S, but did not get a 'Done' and on SysRq+U lockdep complained about an lock imbalance wrt. sb_writer. SysRq+O also did no longer work, only SysRq+B. I don't know which one got stuck first, but I'm somewhat suspicious of the plasma-desktop and the sshd that SysRq+W reported stuck in xfs reclaim, even if these processes did never trigger the hung task check. Torsten > Cheers, > > Dave. > -- > Dave Chinner > david@fromorbit.com > > xfs: inode allocation should use unmapped buffers. > > From: Dave Chinner <dchinner@redhat.com> > > Inode buffers do not need to be mapped as inodes are read or written > directly from/to the pages underlying the buffer. This fixes a > regression introduced by commit 611c994 ("xfs: make XBF_MAPPED the > default behaviour"). > > Signed-off-by: Dave Chinner <dchinner@redhat.com> > --- > fs/xfs/xfs_ialloc.c | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) > > diff --git a/fs/xfs/xfs_ialloc.c b/fs/xfs/xfs_ialloc.c > index 2d6495e..a815412 100644 > --- a/fs/xfs/xfs_ialloc.c > +++ b/fs/xfs/xfs_ialloc.c > @@ -200,7 +200,8 @@ xfs_ialloc_inode_init( > */ > d = XFS_AGB_TO_DADDR(mp, agno, agbno + (j * blks_per_cluster)); > fbuf = xfs_trans_get_buf(tp, mp->m_ddev_targp, d, > - mp->m_bsize * blks_per_cluster, 0); > + mp->m_bsize * blks_per_cluster, > + XBF_UNMAPPED); > if (!fbuf) > return ENOMEM; > /* [-- Attachment #2: xfs-reclaim-hang-messages.txt --] [-- Type: text/plain, Size: 79805 bytes --] Nov 20 19:24:55 thoregon acpid: client 2991[0:0] has disconnected Nov 20 19:25:07 thoregon login[3002]: pam_unix(login:session): session opened for user root by LOGIN(uid=0) Nov 20 19:25:07 thoregon login[27176]: ROOT LOGIN on '/dev/tty1' Nov 20 19:27:23 thoregon kernel: [ 2160.595990] INFO: task kswapd0:725 blocked for more than 120 seconds. Nov 20 19:27:23 thoregon kernel: [ 2160.595998] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 20 19:27:23 thoregon kernel: [ 2160.596004] kswapd0 D 0000000000000001 0 725 2 0x00000000 Nov 20 19:27:23 thoregon kernel: [ 2160.596016] ffff8803280d13f8 0000000000000046 ffff880329a0ab80 ffff8803280d1fd8 Nov 20 19:27:23 thoregon kernel: [ 2160.596030] ffff8803280d1fd8 ffff8803280d1fd8 ffff880046b7c880 ffff880329a0ab80 Nov 20 19:27:23 thoregon kernel: [ 2160.596041] ffff8803280d1408 ffff8803278dbbd0 ffff8803278db800 00000000ffffffff Nov 20 19:27:23 thoregon kernel: [ 2160.596052] Call Trace: Nov 20 19:27:23 thoregon kernel: [ 2160.596070] [<ffffffff816b1224>] schedule+0x24/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.596080] [<ffffffff814f9dad>] md_super_wait+0x4d/0x80 Nov 20 19:27:23 thoregon kernel: [ 2160.596091] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.596100] [<ffffffff81500753>] bitmap_unplug+0x173/0x180 Nov 20 19:27:23 thoregon kernel: [ 2160.596109] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:27:23 thoregon kernel: [ 2160.596118] [<ffffffff814e8eb8>] raid1_unplug+0x98/0x110 Nov 20 19:27:23 thoregon kernel: [ 2160.596130] [<ffffffff81278a6d>] blk_flush_plug_list+0xad/0x240 Nov 20 19:27:23 thoregon kernel: [ 2160.596138] [<ffffffff816b24ca>] ? _raw_spin_unlock_irqrestore+0x3a/0x70 Nov 20 19:27:23 thoregon kernel: [ 2160.596148] [<ffffffff816b15c3>] io_schedule_timeout+0x83/0xf0 Nov 20 19:27:23 thoregon kernel: [ 2160.596156] [<ffffffff810815cd>] ? trace_hardirqs_on+0xd/0x10 Nov 20 19:27:23 thoregon kernel: [ 2160.596166] [<ffffffff810b0e1d>] mempool_alloc+0x12d/0x160 Nov 20 19:27:23 thoregon kernel: [ 2160.596174] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.596184] [<ffffffff811263da>] bvec_alloc_bs+0xda/0x100 Nov 20 19:27:23 thoregon kernel: [ 2160.596192] [<ffffffff811264ea>] bio_alloc_bioset+0xea/0x110 Nov 20 19:27:23 thoregon kernel: [ 2160.596201] [<ffffffff81126656>] bio_clone_bioset+0x16/0x40 Nov 20 19:27:23 thoregon kernel: [ 2160.596208] [<ffffffff814f471a>] bio_clone_mddev+0x1a/0x30 Nov 20 19:27:23 thoregon kernel: [ 2160.596217] [<ffffffff814edbb1>] make_request+0x551/0xde0 Nov 20 19:27:23 thoregon kernel: [ 2160.596224] [<ffffffff814ed9e0>] ? make_request+0x380/0xde0 Nov 20 19:27:23 thoregon kernel: [ 2160.596233] [<ffffffff810b0b80>] ? mempool_alloc_slab+0x10/0x20 Nov 20 19:27:23 thoregon kernel: [ 2160.596242] [<ffffffff814f80bb>] md_make_request+0x21b/0x4d0 Nov 20 19:27:23 thoregon kernel: [ 2160.596249] [<ffffffff814f7ef0>] ? md_make_request+0x50/0x4d0 Nov 20 19:27:23 thoregon kernel: [ 2160.596258] [<ffffffff81276e52>] generic_make_request+0xc2/0x100 Nov 20 19:27:23 thoregon kernel: [ 2160.596267] [<ffffffff81276ef5>] submit_bio+0x65/0x110 Nov 20 19:27:23 thoregon kernel: [ 2160.596278] [<ffffffff811e07bf>] xfs_submit_ioend_bio.isra.21+0x2f/0x40 Nov 20 19:27:23 thoregon kernel: [ 2160.596286] [<ffffffff811e088e>] xfs_submit_ioend+0xbe/0x110 Nov 20 19:27:23 thoregon kernel: [ 2160.596295] [<ffffffff811e0c91>] xfs_vm_writepage+0x3b1/0x540 Nov 20 19:27:23 thoregon kernel: [ 2160.596305] [<ffffffff810bcd84>] shrink_page_list+0x564/0x890 Nov 20 19:27:23 thoregon kernel: [ 2160.596315] [<ffffffff810bd637>] shrink_inactive_list+0x1d7/0x310 Nov 20 19:27:23 thoregon kernel: [ 2160.596325] [<ffffffff810bdb9d>] shrink_lruvec+0x42d/0x530 Nov 20 19:27:23 thoregon kernel: [ 2160.596334] [<ffffffff810be323>] kswapd+0x683/0xa20 Nov 20 19:27:23 thoregon kernel: [ 2160.596344] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.596352] [<ffffffff810bdca0>] ? shrink_lruvec+0x530/0x530 Nov 20 19:27:23 thoregon kernel: [ 2160.596360] [<ffffffff8105c246>] kthread+0xd6/0xe0 Nov 20 19:27:23 thoregon kernel: [ 2160.596367] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.596376] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 Nov 20 19:27:23 thoregon kernel: [ 2160.596384] [<ffffffff816b31ac>] ret_from_fork+0x7c/0xb0 Nov 20 19:27:23 thoregon kernel: [ 2160.596392] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 Nov 20 19:27:23 thoregon kernel: [ 2160.596397] no locks held by kswapd0/725. Nov 20 19:27:23 thoregon kernel: [ 2160.596424] INFO: task xfsaild/md4:1742 blocked for more than 120 seconds. Nov 20 19:27:23 thoregon kernel: [ 2160.596428] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 20 19:27:23 thoregon kernel: [ 2160.596432] xfsaild/md4 D 0000000000000003 0 1742 2 0x00000000 Nov 20 19:27:23 thoregon kernel: [ 2160.596441] ffff88032438bb68 0000000000000046 ffff880329965700 ffff88032438bfd8 Nov 20 19:27:23 thoregon kernel: [ 2160.596452] ffff88032438bfd8 ffff88032438bfd8 ffff88032827e580 ffff880329965700 Nov 20 19:27:23 thoregon kernel: [ 2160.596462] ffff88032438bb78 ffff8803278dbbd0 ffff8803278db800 00000000ffffffff Nov 20 19:27:23 thoregon kernel: [ 2160.596473] Call Trace: Nov 20 19:27:23 thoregon kernel: [ 2160.596483] [<ffffffff816b1224>] schedule+0x24/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.596490] [<ffffffff814f9dad>] md_super_wait+0x4d/0x80 Nov 20 19:27:23 thoregon kernel: [ 2160.596498] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.596507] [<ffffffff81500753>] bitmap_unplug+0x173/0x180 Nov 20 19:27:23 thoregon kernel: [ 2160.596516] [<ffffffff81278c13>] ? blk_finish_plug+0x13/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.596524] [<ffffffff814e8eb8>] raid1_unplug+0x98/0x110 Nov 20 19:27:23 thoregon kernel: [ 2160.596533] [<ffffffff81278a6d>] blk_flush_plug_list+0xad/0x240 Nov 20 19:27:23 thoregon kernel: [ 2160.596542] [<ffffffff81278c13>] blk_finish_plug+0x13/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.596550] [<ffffffff811e296a>] __xfs_buf_delwri_submit+0x1ca/0x1e0 Nov 20 19:27:23 thoregon kernel: [ 2160.596558] [<ffffffff811e2ffb>] xfs_buf_delwri_submit_nowait+0x1b/0x20 Nov 20 19:27:23 thoregon kernel: [ 2160.596566] [<ffffffff81233066>] xfsaild+0x226/0x4c0 Nov 20 19:27:23 thoregon kernel: [ 2160.596575] [<ffffffff81065dfa>] ? finish_task_switch+0x3a/0x100 Nov 20 19:27:23 thoregon kernel: [ 2160.596583] [<ffffffff81232e40>] ? xfs_trans_ail_cursor_first+0xa0/0xa0 Nov 20 19:27:23 thoregon kernel: [ 2160.596591] [<ffffffff8105c246>] kthread+0xd6/0xe0 Nov 20 19:27:23 thoregon kernel: [ 2160.596598] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.596607] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 Nov 20 19:27:23 thoregon kernel: [ 2160.596614] [<ffffffff816b31ac>] ret_from_fork+0x7c/0xb0 Nov 20 19:27:23 thoregon kernel: [ 2160.596622] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 Nov 20 19:27:23 thoregon kernel: [ 2160.596627] no locks held by xfsaild/md4/1742. Nov 20 19:27:23 thoregon kernel: [ 2160.596709] INFO: task flush-9:4:30365 blocked for more than 120 seconds. Nov 20 19:27:23 thoregon kernel: [ 2160.596712] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 20 19:27:23 thoregon kernel: [ 2160.596716] flush-9:4 D 0000000000000001 0 30365 2 0x00000000 Nov 20 19:27:23 thoregon kernel: [ 2160.596725] ffff8801d8ce1978 0000000000000046 ffff880328238e80 ffff8801d8ce1fd8 Nov 20 19:27:23 thoregon kernel: [ 2160.596735] ffff8801d8ce1fd8 ffff8801d8ce1fd8 ffff8803032cc880 ffff880328238e80 Nov 20 19:27:23 thoregon kernel: [ 2160.596745] ffff8801d8ce1988 ffff8803278dbbd0 ffff8803278db800 00000000ffffffff Nov 20 19:27:23 thoregon kernel: [ 2160.596756] Call Trace: Nov 20 19:27:23 thoregon kernel: [ 2160.596765] [<ffffffff816b1224>] schedule+0x24/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.596773] [<ffffffff814f9dad>] md_super_wait+0x4d/0x80 Nov 20 19:27:23 thoregon kernel: [ 2160.596781] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.596788] [<ffffffff81500753>] bitmap_unplug+0x173/0x180 Nov 20 19:27:23 thoregon kernel: [ 2160.596796] [<ffffffff810b6acf>] ? write_cache_pages+0x12f/0x420 Nov 20 19:27:23 thoregon kernel: [ 2160.596804] [<ffffffff810b6700>] ? set_page_dirty_lock+0x60/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.596812] [<ffffffff814e8eb8>] raid1_unplug+0x98/0x110 Nov 20 19:27:23 thoregon kernel: [ 2160.596821] [<ffffffff81278a6d>] blk_flush_plug_list+0xad/0x240 Nov 20 19:27:23 thoregon kernel: [ 2160.596830] [<ffffffff81278c13>] blk_finish_plug+0x13/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.596838] [<ffffffff810b6e0a>] generic_writepages+0x4a/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.596847] [<ffffffff811df898>] xfs_vm_writepages+0x48/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.596856] [<ffffffff8111b144>] ? writeback_sb_inodes+0x174/0x410 Nov 20 19:27:23 thoregon kernel: [ 2160.596864] [<ffffffff810b84be>] do_writepages+0x1e/0x30 Nov 20 19:27:23 thoregon kernel: [ 2160.596873] [<ffffffff8111a1ee>] __writeback_single_inode+0x3e/0x120 Nov 20 19:27:23 thoregon kernel: [ 2160.596882] [<ffffffff8111b234>] writeback_sb_inodes+0x264/0x410 Nov 20 19:27:23 thoregon kernel: [ 2160.596891] [<ffffffff8111b476>] __writeback_inodes_wb+0x96/0xc0 Nov 20 19:27:23 thoregon kernel: [ 2160.596900] [<ffffffff8111b673>] wb_writeback+0x1d3/0x1e0 Nov 20 19:27:23 thoregon kernel: [ 2160.596908] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:27:23 thoregon kernel: [ 2160.596916] [<ffffffff8111bb1c>] ? wb_do_writeback+0x7c/0x140 Nov 20 19:27:23 thoregon kernel: [ 2160.596925] [<ffffffff8111bb40>] wb_do_writeback+0xa0/0x140 Nov 20 19:27:23 thoregon kernel: [ 2160.596935] [<ffffffff8111bc52>] bdi_writeback_thread+0x72/0x150 Nov 20 19:27:23 thoregon kernel: [ 2160.596944] [<ffffffff8111bbe0>] ? wb_do_writeback+0x140/0x140 Nov 20 19:27:23 thoregon kernel: [ 2160.596951] [<ffffffff8105c246>] kthread+0xd6/0xe0 Nov 20 19:27:23 thoregon kernel: [ 2160.596958] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.596967] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 Nov 20 19:27:23 thoregon kernel: [ 2160.596975] [<ffffffff816b31ac>] ret_from_fork+0x7c/0xb0 Nov 20 19:27:23 thoregon kernel: [ 2160.596983] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 Nov 20 19:27:23 thoregon kernel: [ 2160.596988] 1 lock held by flush-9:4/30365: Nov 20 19:27:23 thoregon kernel: [ 2160.596991] #0: (&type->s_umount_key#20){++++.+}, at: [<ffffffff810f59de>] grab_super_passive+0x3e/0x90 Nov 20 19:27:23 thoregon kernel: [ 2160.597019] INFO: task x86_64-pc-linux:26516 blocked for more than 120 seconds. Nov 20 19:27:23 thoregon kernel: [ 2160.597023] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 20 19:27:23 thoregon kernel: [ 2160.597027] x86_64-pc-linux D 0000000000000000 0 26516 26114 0x00000000 Nov 20 19:27:23 thoregon kernel: [ 2160.597035] ffff8800468dddf8 0000000000000046 ffff880329303a00 ffff8800468ddfd8 Nov 20 19:27:23 thoregon kernel: [ 2160.597045] ffff8800468ddfd8 ffff8800468ddfd8 ffff88032989c880 ffff880329303a00 Nov 20 19:27:23 thoregon kernel: [ 2160.597056] 0000000000000246 ffff8800468dc000 ffff8801b1a1bc68 0000000002405150 Nov 20 19:27:23 thoregon kernel: [ 2160.597066] Call Trace: Nov 20 19:27:23 thoregon kernel: [ 2160.597076] [<ffffffff816b1224>] schedule+0x24/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.597085] [<ffffffff816b1501>] schedule_preempt_disabled+0x21/0x30 Nov 20 19:27:23 thoregon kernel: [ 2160.597094] [<ffffffff816af3a5>] mutex_lock_nested+0x165/0x330 Nov 20 19:27:23 thoregon kernel: [ 2160.597103] [<ffffffff810ff5e3>] ? do_unlinkat+0xa3/0x1b0 Nov 20 19:27:23 thoregon kernel: [ 2160.597112] [<ffffffff810fefae>] ? filename_lookup.isra.57+0x2e/0x80 Nov 20 19:27:23 thoregon kernel: [ 2160.597121] [<ffffffff810ff5e3>] do_unlinkat+0xa3/0x1b0 Nov 20 19:27:23 thoregon kernel: [ 2160.597129] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:27:23 thoregon kernel: [ 2160.597137] [<ffffffff8129ca1e>] ? trace_hardirqs_on_thunk+0x3a/0x3f Nov 20 19:27:23 thoregon kernel: [ 2160.597146] [<ffffffff81101bb1>] sys_unlink+0x11/0x20 Nov 20 19:27:23 thoregon kernel: [ 2160.597154] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b Nov 20 19:27:23 thoregon kernel: [ 2160.597159] 2 locks held by x86_64-pc-linux/26516: Nov 20 19:27:23 thoregon kernel: [ 2160.597162] #0: (sb_writers#6){.+.+.+}, at: [<ffffffff811135ef>] mnt_want_write+0x1f/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.597178] #1: (&type->i_mutex_dir_key#3/1){+.+.+.}, at: [<ffffffff810ff5e3>] do_unlinkat+0xa3/0x1b0 Nov 20 19:27:23 thoregon kernel: [ 2160.597198] INFO: task x86_64-pc-linux:26523 blocked for more than 120 seconds. Nov 20 19:27:23 thoregon kernel: [ 2160.597201] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 20 19:27:23 thoregon kernel: [ 2160.597205] x86_64-pc-linux D 0000000000000000 0 26523 26114 0x00000000 Nov 20 19:27:23 thoregon kernel: [ 2160.597212] ffff8800c0def858 0000000000000046 ffff88032986c880 ffff8800c0deffd8 Nov 20 19:27:23 thoregon kernel: [ 2160.597223] ffff8800c0deffd8 ffff8800c0deffd8 ffff88032989e580 ffff88032986c880 Nov 20 19:27:23 thoregon kernel: [ 2160.597233] ffff8800c0def928 ffff8801c57e3a30 7fffffffffffffff ffff88032986c880 Nov 20 19:27:23 thoregon kernel: [ 2160.597243] Call Trace: Nov 20 19:27:23 thoregon kernel: [ 2160.597253] [<ffffffff816b1224>] schedule+0x24/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.597261] [<ffffffff816aeddd>] schedule_timeout+0x17d/0x1c0 Nov 20 19:27:23 thoregon kernel: [ 2160.597269] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.597277] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:27:23 thoregon kernel: [ 2160.597286] [<ffffffff816b057e>] __down_common+0x97/0xe8 Nov 20 19:27:23 thoregon kernel: [ 2160.597294] [<ffffffff816b062e>] __down+0x18/0x1a Nov 20 19:27:23 thoregon kernel: [ 2160.597301] [<ffffffff8106245b>] down+0x3b/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.597308] [<ffffffff811e21b5>] xfs_buf_lock+0x25/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.597315] [<ffffffff811e22e2>] _xfs_buf_find+0xf2/0x200 Nov 20 19:27:23 thoregon kernel: [ 2160.597323] [<ffffffff811e241f>] xfs_buf_get_map+0x2f/0x130 Nov 20 19:27:23 thoregon kernel: [ 2160.597331] [<ffffffff81210417>] ? xfs_dabuf_map.isra.23+0xb7/0x2f0 Nov 20 19:27:23 thoregon kernel: [ 2160.597339] [<ffffffff810b572e>] ? get_page_from_freelist+0x1de/0x460 Nov 20 19:27:23 thoregon kernel: [ 2160.597346] [<ffffffff811e2b23>] xfs_buf_read_map+0x13/0x90 Nov 20 19:27:23 thoregon kernel: [ 2160.597353] [<ffffffff81233e8d>] xfs_trans_read_buf_map+0x1ad/0x2d0 Nov 20 19:27:23 thoregon kernel: [ 2160.597361] [<ffffffff812116b6>] xfs_da_read_buf+0xb6/0x180 Nov 20 19:27:23 thoregon kernel: [ 2160.597371] [<ffffffff8121724c>] xfs_dir2_leaf_lookup_int+0x5c/0x2c0 Nov 20 19:27:23 thoregon kernel: [ 2160.597380] [<ffffffff81217583>] xfs_dir2_leaf_removename+0x33/0x330 Nov 20 19:27:23 thoregon kernel: [ 2160.597389] [<ffffffff8120777b>] ? xfs_bmap_last_offset+0x8b/0xc0 Nov 20 19:27:23 thoregon kernel: [ 2160.597398] [<ffffffff81213b94>] xfs_dir_removename+0x134/0x150 Nov 20 19:27:23 thoregon kernel: [ 2160.597406] [<ffffffff811f5ce6>] xfs_remove+0x1f6/0x370 Nov 20 19:27:23 thoregon kernel: [ 2160.597416] [<ffffffff816af4ac>] ? mutex_lock_nested+0x26c/0x330 Nov 20 19:27:23 thoregon kernel: [ 2160.597424] [<ffffffff810ff48f>] ? vfs_unlink+0x4f/0x100 Nov 20 19:27:23 thoregon kernel: [ 2160.597432] [<ffffffff811ec7f3>] xfs_vn_unlink+0x43/0x90 Nov 20 19:27:23 thoregon kernel: [ 2160.597441] [<ffffffff810ff4cd>] vfs_unlink+0x8d/0x100 Nov 20 19:27:23 thoregon kernel: [ 2160.597449] [<ffffffff810ff63a>] do_unlinkat+0xfa/0x1b0 Nov 20 19:27:23 thoregon kernel: [ 2160.597458] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:27:23 thoregon kernel: [ 2160.597466] [<ffffffff8129ca1e>] ? trace_hardirqs_on_thunk+0x3a/0x3f Nov 20 19:27:23 thoregon kernel: [ 2160.597475] [<ffffffff81101bb1>] sys_unlink+0x11/0x20 Nov 20 19:27:23 thoregon kernel: [ 2160.597482] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b Nov 20 19:27:23 thoregon kernel: [ 2160.597487] 6 locks held by x86_64-pc-linux/26523: Nov 20 19:27:23 thoregon kernel: [ 2160.597490] #0: (sb_writers#6){.+.+.+}, at: [<ffffffff811135ef>] mnt_want_write+0x1f/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.597505] #1: (&type->i_mutex_dir_key#3/1){+.+.+.}, at: [<ffffffff810ff5e3>] do_unlinkat+0xa3/0x1b0 Nov 20 19:27:23 thoregon kernel: [ 2160.597522] #2: (&sb->s_type->i_mutex_key#7){+.+.+.}, at: [<ffffffff810ff48f>] vfs_unlink+0x4f/0x100 Nov 20 19:27:23 thoregon kernel: [ 2160.597538] #3: (sb_internal){.+.+.?}, at: [<ffffffff8122b3e8>] xfs_trans_alloc+0x28/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.597552] #4: (&(&ip->i_lock)->mr_lock/4){+.+...}, at: [<ffffffff811e8414>] xfs_ilock+0x84/0xb0 Nov 20 19:27:23 thoregon kernel: [ 2160.597567] #5: (&(&ip->i_lock)->mr_lock/5){+.+...}, at: [<ffffffff811e8414>] xfs_ilock+0x84/0xb0 Nov 20 19:27:23 thoregon kernel: [ 2160.597584] INFO: task x86_64-pc-linux:26875 blocked for more than 120 seconds. Nov 20 19:27:23 thoregon kernel: [ 2160.597587] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 20 19:27:23 thoregon kernel: [ 2160.597591] x86_64-pc-linux D 0000000000000000 0 26875 26114 0x00000000 Nov 20 19:27:23 thoregon kernel: [ 2160.597598] ffff8800c0c5ba08 0000000000000046 ffff8802f0594880 ffff8800c0c5bfd8 Nov 20 19:27:23 thoregon kernel: [ 2160.597609] ffff8800c0c5bfd8 ffff8800c0c5bfd8 ffff88032989c880 ffff8802f0594880 Nov 20 19:27:23 thoregon kernel: [ 2160.597619] 0000000000000250 ffff880205a7aa30 7fffffffffffffff ffff8802f0594880 Nov 20 19:27:23 thoregon kernel: [ 2160.597630] Call Trace: Nov 20 19:27:23 thoregon kernel: [ 2160.597639] [<ffffffff816b1224>] schedule+0x24/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.597648] [<ffffffff816aeddd>] schedule_timeout+0x17d/0x1c0 Nov 20 19:27:23 thoregon kernel: [ 2160.597655] [<ffffffff811f70b4>] ? kmem_zone_zalloc+0x34/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.597662] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.597670] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:27:23 thoregon kernel: [ 2160.597679] [<ffffffff816b057e>] __down_common+0x97/0xe8 Nov 20 19:27:23 thoregon kernel: [ 2160.597688] [<ffffffff816b062e>] __down+0x18/0x1a Nov 20 19:27:23 thoregon kernel: [ 2160.597695] [<ffffffff8106245b>] down+0x3b/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.597702] [<ffffffff811e21b5>] xfs_buf_lock+0x25/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.597708] [<ffffffff811e22e2>] _xfs_buf_find+0xf2/0x200 Nov 20 19:27:23 thoregon kernel: [ 2160.597716] [<ffffffff811e241f>] xfs_buf_get_map+0x2f/0x130 Nov 20 19:27:23 thoregon kernel: [ 2160.597723] [<ffffffff811e2b23>] xfs_buf_read_map+0x13/0x90 Nov 20 19:27:23 thoregon kernel: [ 2160.597730] [<ffffffff81233e8d>] xfs_trans_read_buf_map+0x1ad/0x2d0 Nov 20 19:27:23 thoregon kernel: [ 2160.597739] [<ffffffff8121ce6f>] xfs_read_agi+0x7f/0x110 Nov 20 19:27:23 thoregon kernel: [ 2160.597748] [<ffffffff8121fb55>] xfs_iunlink+0x45/0x150 Nov 20 19:27:23 thoregon kernel: [ 2160.597757] [<ffffffff81041fc1>] ? current_fs_time+0x11/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.597764] [<ffffffff81234352>] ? xfs_trans_ichgtime+0x22/0xa0 Nov 20 19:27:23 thoregon kernel: [ 2160.597772] [<ffffffff811f3ea0>] xfs_droplink+0x50/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.597778] [<ffffffff811f5d77>] xfs_remove+0x287/0x370 Nov 20 19:27:23 thoregon kernel: [ 2160.597788] [<ffffffff816af4ac>] ? mutex_lock_nested+0x26c/0x330 Nov 20 19:27:23 thoregon kernel: [ 2160.597796] [<ffffffff810ff48f>] ? vfs_unlink+0x4f/0x100 Nov 20 19:27:23 thoregon kernel: [ 2160.597804] [<ffffffff811ec7f3>] xfs_vn_unlink+0x43/0x90 Nov 20 19:27:23 thoregon kernel: [ 2160.597813] [<ffffffff810ff4cd>] vfs_unlink+0x8d/0x100 Nov 20 19:27:23 thoregon kernel: [ 2160.597821] [<ffffffff810ff63a>] do_unlinkat+0xfa/0x1b0 Nov 20 19:27:23 thoregon kernel: [ 2160.597830] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:27:23 thoregon kernel: [ 2160.597837] [<ffffffff8129ca1e>] ? trace_hardirqs_on_thunk+0x3a/0x3f Nov 20 19:27:23 thoregon kernel: [ 2160.597846] [<ffffffff81101bb1>] sys_unlink+0x11/0x20 Nov 20 19:27:23 thoregon kernel: [ 2160.597854] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b Nov 20 19:27:23 thoregon kernel: [ 2160.597859] 6 locks held by x86_64-pc-linux/26875: Nov 20 19:27:23 thoregon kernel: [ 2160.597862] #0: (sb_writers#6){.+.+.+}, at: [<ffffffff811135ef>] mnt_want_write+0x1f/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.597877] #1: (&type->i_mutex_dir_key#3/1){+.+.+.}, at: [<ffffffff810ff5e3>] do_unlinkat+0xa3/0x1b0 Nov 20 19:27:23 thoregon kernel: [ 2160.597894] #2: (&sb->s_type->i_mutex_key#7){+.+.+.}, at: [<ffffffff810ff48f>] vfs_unlink+0x4f/0x100 Nov 20 19:27:23 thoregon kernel: [ 2160.597909] #3: (sb_internal){.+.+.?}, at: [<ffffffff8122b3e8>] xfs_trans_alloc+0x28/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.597923] #4: (&(&ip->i_lock)->mr_lock/4){+.+...}, at: [<ffffffff811e8414>] xfs_ilock+0x84/0xb0 Nov 20 19:27:23 thoregon kernel: [ 2160.597937] #5: (&(&ip->i_lock)->mr_lock/5){+.+...}, at: [<ffffffff811e8414>] xfs_ilock+0x84/0xb0 Nov 20 19:27:23 thoregon kernel: [ 2160.597955] INFO: task x86_64-pc-linux:26897 blocked for more than 120 seconds. Nov 20 19:27:23 thoregon kernel: [ 2160.597958] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 20 19:27:23 thoregon kernel: [ 2160.597961] x86_64-pc-linux D 0000000000000001 0 26897 26114 0x00000000 Nov 20 19:27:23 thoregon kernel: [ 2160.597969] ffff880101959858 0000000000000046 ffff88032934ab80 ffff880101959fd8 Nov 20 19:27:23 thoregon kernel: [ 2160.597980] ffff880101959fd8 ffff880101959fd8 ffff880162a3e580 ffff88032934ab80 Nov 20 19:27:23 thoregon kernel: [ 2160.597990] ffff880101959928 ffff8801ca545030 7fffffffffffffff ffff88032934ab80 Nov 20 19:27:23 thoregon kernel: [ 2160.598000] Call Trace: Nov 20 19:27:23 thoregon kernel: [ 2160.598010] [<ffffffff816b1224>] schedule+0x24/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.598018] [<ffffffff816aeddd>] schedule_timeout+0x17d/0x1c0 Nov 20 19:27:23 thoregon kernel: [ 2160.598026] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.598034] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:27:23 thoregon kernel: [ 2160.598043] [<ffffffff816b057e>] __down_common+0x97/0xe8 Nov 20 19:27:23 thoregon kernel: [ 2160.598052] [<ffffffff816b062e>] __down+0x18/0x1a Nov 20 19:27:23 thoregon kernel: [ 2160.598058] [<ffffffff8106245b>] down+0x3b/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.598065] [<ffffffff811e21b5>] xfs_buf_lock+0x25/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.598072] [<ffffffff811e22e2>] _xfs_buf_find+0xf2/0x200 Nov 20 19:27:23 thoregon kernel: [ 2160.598079] [<ffffffff811e241f>] xfs_buf_get_map+0x2f/0x130 Nov 20 19:27:23 thoregon kernel: [ 2160.598087] [<ffffffff81210417>] ? xfs_dabuf_map.isra.23+0xb7/0x2f0 Nov 20 19:27:23 thoregon kernel: [ 2160.598094] [<ffffffff810b572e>] ? get_page_from_freelist+0x1de/0x460 Nov 20 19:27:23 thoregon kernel: [ 2160.598102] [<ffffffff811e2b23>] xfs_buf_read_map+0x13/0x90 Nov 20 19:27:23 thoregon kernel: [ 2160.598109] [<ffffffff81233e8d>] xfs_trans_read_buf_map+0x1ad/0x2d0 Nov 20 19:27:23 thoregon kernel: [ 2160.598117] [<ffffffff812116b6>] xfs_da_read_buf+0xb6/0x180 Nov 20 19:27:23 thoregon kernel: [ 2160.598126] [<ffffffff8121724c>] xfs_dir2_leaf_lookup_int+0x5c/0x2c0 Nov 20 19:27:23 thoregon kernel: [ 2160.598135] [<ffffffff81217583>] xfs_dir2_leaf_removename+0x33/0x330 Nov 20 19:27:23 thoregon kernel: [ 2160.598144] [<ffffffff8120777b>] ? xfs_bmap_last_offset+0x8b/0xc0 Nov 20 19:27:23 thoregon kernel: [ 2160.598152] [<ffffffff81213b94>] xfs_dir_removename+0x134/0x150 Nov 20 19:27:23 thoregon kernel: [ 2160.598160] [<ffffffff811f5ce6>] xfs_remove+0x1f6/0x370 Nov 20 19:27:23 thoregon kernel: [ 2160.598170] [<ffffffff816af4ac>] ? mutex_lock_nested+0x26c/0x330 Nov 20 19:27:23 thoregon kernel: [ 2160.598178] [<ffffffff810ff48f>] ? vfs_unlink+0x4f/0x100 Nov 20 19:27:23 thoregon kernel: [ 2160.598186] [<ffffffff811ec7f3>] xfs_vn_unlink+0x43/0x90 Nov 20 19:27:23 thoregon kernel: [ 2160.598195] [<ffffffff810ff4cd>] vfs_unlink+0x8d/0x100 Nov 20 19:27:23 thoregon kernel: [ 2160.598203] [<ffffffff810ff63a>] do_unlinkat+0xfa/0x1b0 Nov 20 19:27:23 thoregon kernel: [ 2160.598211] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:27:23 thoregon kernel: [ 2160.598219] [<ffffffff8129ca1e>] ? trace_hardirqs_on_thunk+0x3a/0x3f Nov 20 19:27:23 thoregon kernel: [ 2160.598228] [<ffffffff81101bb1>] sys_unlink+0x11/0x20 Nov 20 19:27:23 thoregon kernel: [ 2160.598236] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b Nov 20 19:27:23 thoregon kernel: [ 2160.598241] 6 locks held by x86_64-pc-linux/26897: Nov 20 19:27:23 thoregon kernel: [ 2160.598244] #0: (sb_writers#6){.+.+.+}, at: [<ffffffff811135ef>] mnt_want_write+0x1f/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.598258] #1: (&type->i_mutex_dir_key#3/1){+.+.+.}, at: [<ffffffff810ff5e3>] do_unlinkat+0xa3/0x1b0 Nov 20 19:27:23 thoregon kernel: [ 2160.598275] #2: (&sb->s_type->i_mutex_key#7){+.+.+.}, at: [<ffffffff810ff48f>] vfs_unlink+0x4f/0x100 Nov 20 19:27:23 thoregon kernel: [ 2160.598290] #3: (sb_internal){.+.+.?}, at: [<ffffffff8122b3e8>] xfs_trans_alloc+0x28/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.598304] #4: (&(&ip->i_lock)->mr_lock/4){+.+...}, at: [<ffffffff811e8414>] xfs_ilock+0x84/0xb0 Nov 20 19:27:23 thoregon kernel: [ 2160.598318] #5: (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffff811e854c>] xfs_ilock_nowait+0xbc/0x100 Nov 20 19:27:23 thoregon kernel: [ 2160.598334] INFO: task x86_64-pc-linux:26904 blocked for more than 120 seconds. Nov 20 19:27:23 thoregon kernel: [ 2160.598337] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 20 19:27:23 thoregon kernel: [ 2160.598341] x86_64-pc-linux D 0000000000000001 0 26904 26114 0x00000000 Nov 20 19:27:23 thoregon kernel: [ 2160.598348] ffff880066bff858 0000000000000046 ffff880329301d00 ffff880066bfffd8 Nov 20 19:27:23 thoregon kernel: [ 2160.598359] ffff880066bfffd8 ffff880066bfffd8 ffff88013a7ce580 ffff880329301d00 Nov 20 19:27:23 thoregon kernel: [ 2160.598369] ffff880066bff928 ffff8801c572c830 7fffffffffffffff ffff880329301d00 Nov 20 19:27:23 thoregon kernel: [ 2160.598380] Call Trace: Nov 20 19:27:23 thoregon kernel: [ 2160.598389] [<ffffffff816b1224>] schedule+0x24/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.598398] [<ffffffff816aeddd>] schedule_timeout+0x17d/0x1c0 Nov 20 19:27:23 thoregon kernel: [ 2160.598405] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.598413] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:27:23 thoregon kernel: [ 2160.598422] [<ffffffff816b057e>] __down_common+0x97/0xe8 Nov 20 19:27:23 thoregon kernel: [ 2160.598431] [<ffffffff816b062e>] __down+0x18/0x1a Nov 20 19:27:23 thoregon kernel: [ 2160.598437] [<ffffffff8106245b>] down+0x3b/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.598444] [<ffffffff811e21b5>] xfs_buf_lock+0x25/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.598451] [<ffffffff811e22e2>] _xfs_buf_find+0xf2/0x200 Nov 20 19:27:23 thoregon kernel: [ 2160.598458] [<ffffffff811e241f>] xfs_buf_get_map+0x2f/0x130 Nov 20 19:27:23 thoregon kernel: [ 2160.598466] [<ffffffff81210417>] ? xfs_dabuf_map.isra.23+0xb7/0x2f0 Nov 20 19:27:23 thoregon kernel: [ 2160.598474] [<ffffffff810b572e>] ? get_page_from_freelist+0x1de/0x460 Nov 20 19:27:23 thoregon kernel: [ 2160.598481] [<ffffffff811e2b23>] xfs_buf_read_map+0x13/0x90 Nov 20 19:27:23 thoregon kernel: [ 2160.598488] [<ffffffff81233e8d>] xfs_trans_read_buf_map+0x1ad/0x2d0 Nov 20 19:27:23 thoregon kernel: [ 2160.598496] [<ffffffff812116b6>] xfs_da_read_buf+0xb6/0x180 Nov 20 19:27:23 thoregon kernel: [ 2160.598505] [<ffffffff8121724c>] xfs_dir2_leaf_lookup_int+0x5c/0x2c0 Nov 20 19:27:23 thoregon kernel: [ 2160.598515] [<ffffffff81217583>] xfs_dir2_leaf_removename+0x33/0x330 Nov 20 19:27:23 thoregon kernel: [ 2160.598523] [<ffffffff8120777b>] ? xfs_bmap_last_offset+0x8b/0xc0 Nov 20 19:27:23 thoregon kernel: [ 2160.598532] [<ffffffff81213b94>] xfs_dir_removename+0x134/0x150 Nov 20 19:27:23 thoregon kernel: [ 2160.598540] [<ffffffff811f5ce6>] xfs_remove+0x1f6/0x370 Nov 20 19:27:23 thoregon kernel: [ 2160.598549] [<ffffffff816af4ac>] ? mutex_lock_nested+0x26c/0x330 Nov 20 19:27:23 thoregon kernel: [ 2160.598557] [<ffffffff810ff48f>] ? vfs_unlink+0x4f/0x100 Nov 20 19:27:23 thoregon kernel: [ 2160.598565] [<ffffffff811ec7f3>] xfs_vn_unlink+0x43/0x90 Nov 20 19:27:23 thoregon kernel: [ 2160.598574] [<ffffffff810ff4cd>] vfs_unlink+0x8d/0x100 Nov 20 19:27:23 thoregon kernel: [ 2160.598582] [<ffffffff810ff63a>] do_unlinkat+0xfa/0x1b0 Nov 20 19:27:23 thoregon kernel: [ 2160.598591] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:27:23 thoregon kernel: [ 2160.598598] [<ffffffff8129ca1e>] ? trace_hardirqs_on_thunk+0x3a/0x3f Nov 20 19:27:23 thoregon kernel: [ 2160.598607] [<ffffffff81101bb1>] sys_unlink+0x11/0x20 Nov 20 19:27:23 thoregon kernel: [ 2160.598615] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b Nov 20 19:27:23 thoregon kernel: [ 2160.598620] 6 locks held by x86_64-pc-linux/26904: Nov 20 19:27:23 thoregon kernel: [ 2160.598623] #0: (sb_writers#6){.+.+.+}, at: [<ffffffff811135ef>] mnt_want_write+0x1f/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.598637] #1: (&type->i_mutex_dir_key#3/1){+.+.+.}, at: [<ffffffff810ff5e3>] do_unlinkat+0xa3/0x1b0 Nov 20 19:27:23 thoregon kernel: [ 2160.598654] #2: (&sb->s_type->i_mutex_key#7){+.+.+.}, at: [<ffffffff810ff48f>] vfs_unlink+0x4f/0x100 Nov 20 19:27:23 thoregon kernel: [ 2160.598669] #3: (sb_internal){.+.+.?}, at: [<ffffffff8122b3e8>] xfs_trans_alloc+0x28/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.598683] #4: (&(&ip->i_lock)->mr_lock/4){+.+...}, at: [<ffffffff811e8414>] xfs_ilock+0x84/0xb0 Nov 20 19:27:23 thoregon kernel: [ 2160.598697] #5: (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffff811e854c>] xfs_ilock_nowait+0xbc/0x100 Nov 20 19:27:23 thoregon kernel: [ 2160.598713] INFO: task x86_64-pc-linux:26907 blocked for more than 120 seconds. Nov 20 19:27:23 thoregon kernel: [ 2160.598716] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 20 19:27:23 thoregon kernel: [ 2160.598720] x86_64-pc-linux D 0000000000000000 0 26907 26114 0x00000000 Nov 20 19:27:23 thoregon kernel: [ 2160.598727] ffff8800c0eed858 0000000000000046 ffff8802a7be6580 ffff8800c0eedfd8 Nov 20 19:27:23 thoregon kernel: [ 2160.598738] ffff8800c0eedfd8 ffff8800c0eedfd8 ffff88032989e580 ffff8802a7be6580 Nov 20 19:27:23 thoregon kernel: [ 2160.598748] ffff8800c0eed928 ffff8801c55c7630 7fffffffffffffff ffff8802a7be6580 Nov 20 19:27:23 thoregon kernel: [ 2160.598758] Call Trace: Nov 20 19:27:23 thoregon kernel: [ 2160.598768] [<ffffffff816b1224>] schedule+0x24/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.598776] [<ffffffff816aeddd>] schedule_timeout+0x17d/0x1c0 Nov 20 19:27:23 thoregon kernel: [ 2160.598784] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.598792] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:27:23 thoregon kernel: [ 2160.598801] [<ffffffff816b057e>] __down_common+0x97/0xe8 Nov 20 19:27:23 thoregon kernel: [ 2160.598809] [<ffffffff816b062e>] __down+0x18/0x1a Nov 20 19:27:23 thoregon kernel: [ 2160.598816] [<ffffffff8106245b>] down+0x3b/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.598823] [<ffffffff811e21b5>] xfs_buf_lock+0x25/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.598830] [<ffffffff811e22e2>] _xfs_buf_find+0xf2/0x200 Nov 20 19:27:23 thoregon kernel: [ 2160.598837] [<ffffffff811e241f>] xfs_buf_get_map+0x2f/0x130 Nov 20 19:27:23 thoregon kernel: [ 2160.598844] [<ffffffff81210417>] ? xfs_dabuf_map.isra.23+0xb7/0x2f0 Nov 20 19:27:23 thoregon kernel: [ 2160.598852] [<ffffffff810b572e>] ? get_page_from_freelist+0x1de/0x460 Nov 20 19:27:23 thoregon kernel: [ 2160.598859] [<ffffffff811e2b23>] xfs_buf_read_map+0x13/0x90 Nov 20 19:27:23 thoregon kernel: [ 2160.598867] [<ffffffff81233e8d>] xfs_trans_read_buf_map+0x1ad/0x2d0 Nov 20 19:27:23 thoregon kernel: [ 2160.598874] [<ffffffff812116b6>] xfs_da_read_buf+0xb6/0x180 Nov 20 19:27:23 thoregon kernel: [ 2160.598884] [<ffffffff8121724c>] xfs_dir2_leaf_lookup_int+0x5c/0x2c0 Nov 20 19:27:23 thoregon kernel: [ 2160.598893] [<ffffffff81217583>] xfs_dir2_leaf_removename+0x33/0x330 Nov 20 19:27:23 thoregon kernel: [ 2160.598901] [<ffffffff8120777b>] ? xfs_bmap_last_offset+0x8b/0xc0 Nov 20 19:27:23 thoregon kernel: [ 2160.598910] [<ffffffff81213b94>] xfs_dir_removename+0x134/0x150 Nov 20 19:27:23 thoregon kernel: [ 2160.598918] [<ffffffff811f5ce6>] xfs_remove+0x1f6/0x370 Nov 20 19:27:23 thoregon kernel: [ 2160.598927] [<ffffffff816af4ac>] ? mutex_lock_nested+0x26c/0x330 Nov 20 19:27:23 thoregon kernel: [ 2160.598935] [<ffffffff810ff48f>] ? vfs_unlink+0x4f/0x100 Nov 20 19:27:23 thoregon kernel: [ 2160.598944] [<ffffffff811ec7f3>] xfs_vn_unlink+0x43/0x90 Nov 20 19:27:23 thoregon kernel: [ 2160.598952] [<ffffffff810ff4cd>] vfs_unlink+0x8d/0x100 Nov 20 19:27:23 thoregon kernel: [ 2160.598961] [<ffffffff810ff63a>] do_unlinkat+0xfa/0x1b0 Nov 20 19:27:23 thoregon kernel: [ 2160.598969] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:27:23 thoregon kernel: [ 2160.598977] [<ffffffff8129ca1e>] ? trace_hardirqs_on_thunk+0x3a/0x3f Nov 20 19:27:23 thoregon kernel: [ 2160.598986] [<ffffffff81101bb1>] sys_unlink+0x11/0x20 Nov 20 19:27:23 thoregon kernel: [ 2160.598993] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b Nov 20 19:27:23 thoregon kernel: [ 2160.598998] 6 locks held by x86_64-pc-linux/26907: Nov 20 19:27:23 thoregon kernel: [ 2160.599001] #0: (sb_writers#6){.+.+.+}, at: [<ffffffff811135ef>] mnt_want_write+0x1f/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.599016] #1: (&type->i_mutex_dir_key#3/1){+.+.+.}, at: [<ffffffff810ff5e3>] do_unlinkat+0xa3/0x1b0 Nov 20 19:27:23 thoregon kernel: [ 2160.599033] #2: (&sb->s_type->i_mutex_key#7){+.+.+.}, at: [<ffffffff810ff48f>] vfs_unlink+0x4f/0x100 Nov 20 19:27:23 thoregon kernel: [ 2160.599048] #3: (sb_internal){.+.+.?}, at: [<ffffffff8122b3e8>] xfs_trans_alloc+0x28/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.599062] #4: (&(&ip->i_lock)->mr_lock/4){+.+...}, at: [<ffffffff811e8414>] xfs_ilock+0x84/0xb0 Nov 20 19:27:23 thoregon kernel: [ 2160.599076] #5: (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffff811e854c>] xfs_ilock_nowait+0xbc/0x100 Nov 20 19:27:41 thoregon login[3002]: pam_unix(login:session): session closed for user root Nov 20 19:27:43 thoregon acpid: client connected from 2991[0:0] Nov 20 19:27:43 thoregon acpid: 1 client rule loaded Nov 20 19:29:23 thoregon kernel: [ 2280.601023] INFO: task kswapd0:725 blocked for more than 120 seconds. Nov 20 19:29:23 thoregon kernel: [ 2280.601030] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 20 19:29:23 thoregon kernel: [ 2280.601036] kswapd0 D 0000000000000001 0 725 2 0x00000000 Nov 20 19:29:23 thoregon kernel: [ 2280.601048] ffff8803280d13f8 0000000000000046 ffff880329a0ab80 ffff8803280d1fd8 Nov 20 19:29:23 thoregon kernel: [ 2280.601063] ffff8803280d1fd8 ffff8803280d1fd8 ffff880046b7c880 ffff880329a0ab80 Nov 20 19:29:23 thoregon kernel: [ 2280.601073] ffff8803280d1408 ffff8803278dbbd0 ffff8803278db800 00000000ffffffff Nov 20 19:29:23 thoregon kernel: [ 2280.601084] Call Trace: Nov 20 19:29:23 thoregon kernel: [ 2280.601101] [<ffffffff816b1224>] schedule+0x24/0x60 Nov 20 19:29:23 thoregon kernel: [ 2280.601111] [<ffffffff814f9dad>] md_super_wait+0x4d/0x80 Nov 20 19:29:23 thoregon kernel: [ 2280.601121] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60 Nov 20 19:29:23 thoregon kernel: [ 2280.601129] [<ffffffff81500753>] bitmap_unplug+0x173/0x180 Nov 20 19:29:23 thoregon kernel: [ 2280.601138] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:29:23 thoregon kernel: [ 2280.601147] [<ffffffff814e8eb8>] raid1_unplug+0x98/0x110 Nov 20 19:29:23 thoregon kernel: [ 2280.601157] [<ffffffff81278a6d>] blk_flush_plug_list+0xad/0x240 Nov 20 19:29:23 thoregon kernel: [ 2280.601165] [<ffffffff816b24ca>] ? _raw_spin_unlock_irqrestore+0x3a/0x70 Nov 20 19:29:23 thoregon kernel: [ 2280.601175] [<ffffffff816b15c3>] io_schedule_timeout+0x83/0xf0 Nov 20 19:29:23 thoregon kernel: [ 2280.601182] [<ffffffff810815cd>] ? trace_hardirqs_on+0xd/0x10 Nov 20 19:29:23 thoregon kernel: [ 2280.601192] [<ffffffff810b0e1d>] mempool_alloc+0x12d/0x160 Nov 20 19:29:23 thoregon kernel: [ 2280.601200] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60 Nov 20 19:29:23 thoregon kernel: [ 2280.601209] [<ffffffff811263da>] bvec_alloc_bs+0xda/0x100 Nov 20 19:29:23 thoregon kernel: [ 2280.601217] [<ffffffff811264ea>] bio_alloc_bioset+0xea/0x110 Nov 20 19:29:23 thoregon kernel: [ 2280.601226] [<ffffffff81126656>] bio_clone_bioset+0x16/0x40 Nov 20 19:29:23 thoregon kernel: [ 2280.601233] [<ffffffff814f471a>] bio_clone_mddev+0x1a/0x30 Nov 20 19:29:23 thoregon kernel: [ 2280.601241] [<ffffffff814edbb1>] make_request+0x551/0xde0 Nov 20 19:29:23 thoregon kernel: [ 2280.601249] [<ffffffff814ed9e0>] ? make_request+0x380/0xde0 Nov 20 19:29:23 thoregon kernel: [ 2280.601257] [<ffffffff810b0b80>] ? mempool_alloc_slab+0x10/0x20 Nov 20 19:29:23 thoregon kernel: [ 2280.601266] [<ffffffff814f80bb>] md_make_request+0x21b/0x4d0 Nov 20 19:29:23 thoregon kernel: [ 2280.601273] [<ffffffff814f7ef0>] ? md_make_request+0x50/0x4d0 Nov 20 19:29:23 thoregon kernel: [ 2280.601282] [<ffffffff81276e52>] generic_make_request+0xc2/0x100 Nov 20 19:29:23 thoregon kernel: [ 2280.601291] [<ffffffff81276ef5>] submit_bio+0x65/0x110 Nov 20 19:29:23 thoregon kernel: [ 2280.601301] [<ffffffff811e07bf>] xfs_submit_ioend_bio.isra.21+0x2f/0x40 Nov 20 19:29:23 thoregon kernel: [ 2280.601309] [<ffffffff811e088e>] xfs_submit_ioend+0xbe/0x110 Nov 20 19:29:23 thoregon kernel: [ 2280.601318] [<ffffffff811e0c91>] xfs_vm_writepage+0x3b1/0x540 Nov 20 19:29:23 thoregon kernel: [ 2280.601328] [<ffffffff810bcd84>] shrink_page_list+0x564/0x890 Nov 20 19:29:23 thoregon kernel: [ 2280.601338] [<ffffffff810bd637>] shrink_inactive_list+0x1d7/0x310 Nov 20 19:29:23 thoregon kernel: [ 2280.601347] [<ffffffff810bdb9d>] shrink_lruvec+0x42d/0x530 Nov 20 19:29:23 thoregon kernel: [ 2280.601356] [<ffffffff810be323>] kswapd+0x683/0xa20 Nov 20 19:29:23 thoregon kernel: [ 2280.601365] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60 Nov 20 19:29:23 thoregon kernel: [ 2280.601374] [<ffffffff810bdca0>] ? shrink_lruvec+0x530/0x530 Nov 20 19:29:23 thoregon kernel: [ 2280.601381] [<ffffffff8105c246>] kthread+0xd6/0xe0 Nov 20 19:29:23 thoregon kernel: [ 2280.601388] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50 Nov 20 19:29:23 thoregon kernel: [ 2280.601397] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 Nov 20 19:29:23 thoregon kernel: [ 2280.601404] [<ffffffff816b31ac>] ret_from_fork+0x7c/0xb0 Nov 20 19:29:23 thoregon kernel: [ 2280.601412] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 Nov 20 19:29:23 thoregon kernel: [ 2280.601417] no locks held by kswapd0/725. Nov 20 19:30:01 thoregon cron[27484]: (root) CMD (test -x /usr/sbin/run-crons && /usr/sbin/run-crons) Nov 20 19:35:16 thoregon kernel: [ 2633.536440] SysRq : Show Blocked State Nov 20 19:35:16 thoregon kernel: [ 2633.536453] task PC stack pid father Nov 20 19:35:16 thoregon kernel: [ 2633.536485] kswapd0 D 0000000000000001 0 725 2 0x00000000 Nov 20 19:35:16 thoregon kernel: [ 2633.536496] ffff8803280d13f8 0000000000000046 ffff880329a0ab80 ffff8803280d1fd8 Nov 20 19:35:16 thoregon kernel: [ 2633.536507] ffff8803280d1fd8 ffff8803280d1fd8 ffff880046b7c880 ffff880329a0ab80 Nov 20 19:35:16 thoregon kernel: [ 2633.536517] ffff8803280d1408 ffff8803278dbbd0 ffff8803278db800 00000000ffffffff Nov 20 19:35:16 thoregon kernel: [ 2633.536526] Call Trace: Nov 20 19:35:16 thoregon kernel: [ 2633.536542] [<ffffffff816b1224>] schedule+0x24/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.536553] [<ffffffff814f9dad>] md_super_wait+0x4d/0x80 Nov 20 19:35:16 thoregon kernel: [ 2633.536563] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.536571] [<ffffffff81500753>] bitmap_unplug+0x173/0x180 Nov 20 19:35:16 thoregon kernel: [ 2633.536581] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:35:16 thoregon kernel: [ 2633.536589] [<ffffffff814e8eb8>] raid1_unplug+0x98/0x110 Nov 20 19:35:16 thoregon kernel: [ 2633.536599] [<ffffffff81278a6d>] blk_flush_plug_list+0xad/0x240 Nov 20 19:35:16 thoregon kernel: [ 2633.536607] [<ffffffff816b24ca>] ? _raw_spin_unlock_irqrestore+0x3a/0x70 Nov 20 19:35:16 thoregon kernel: [ 2633.536616] [<ffffffff816b15c3>] io_schedule_timeout+0x83/0xf0 Nov 20 19:35:16 thoregon kernel: [ 2633.536623] [<ffffffff810815cd>] ? trace_hardirqs_on+0xd/0x10 Nov 20 19:35:16 thoregon kernel: [ 2633.536633] [<ffffffff810b0e1d>] mempool_alloc+0x12d/0x160 Nov 20 19:35:16 thoregon kernel: [ 2633.536641] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.536650] [<ffffffff811263da>] bvec_alloc_bs+0xda/0x100 Nov 20 19:35:16 thoregon kernel: [ 2633.536658] [<ffffffff811264ea>] bio_alloc_bioset+0xea/0x110 Nov 20 19:35:16 thoregon kernel: [ 2633.536666] [<ffffffff81126656>] bio_clone_bioset+0x16/0x40 Nov 20 19:35:16 thoregon kernel: [ 2633.536673] [<ffffffff814f471a>] bio_clone_mddev+0x1a/0x30 Nov 20 19:35:16 thoregon kernel: [ 2633.536681] [<ffffffff814edbb1>] make_request+0x551/0xde0 Nov 20 19:35:16 thoregon kernel: [ 2633.536689] [<ffffffff814ed9e0>] ? make_request+0x380/0xde0 Nov 20 19:35:16 thoregon kernel: [ 2633.536697] [<ffffffff810b0b80>] ? mempool_alloc_slab+0x10/0x20 Nov 20 19:35:16 thoregon kernel: [ 2633.536706] [<ffffffff814f80bb>] md_make_request+0x21b/0x4d0 Nov 20 19:35:16 thoregon kernel: [ 2633.536712] [<ffffffff814f7ef0>] ? md_make_request+0x50/0x4d0 Nov 20 19:35:16 thoregon kernel: [ 2633.536722] [<ffffffff81276e52>] generic_make_request+0xc2/0x100 Nov 20 19:35:16 thoregon kernel: [ 2633.536730] [<ffffffff81276ef5>] submit_bio+0x65/0x110 Nov 20 19:35:16 thoregon kernel: [ 2633.536740] [<ffffffff811e07bf>] xfs_submit_ioend_bio.isra.21+0x2f/0x40 Nov 20 19:35:16 thoregon kernel: [ 2633.536748] [<ffffffff811e088e>] xfs_submit_ioend+0xbe/0x110 Nov 20 19:35:16 thoregon kernel: [ 2633.536757] [<ffffffff811e0c91>] xfs_vm_writepage+0x3b1/0x540 Nov 20 19:35:16 thoregon kernel: [ 2633.536767] [<ffffffff810bcd84>] shrink_page_list+0x564/0x890 Nov 20 19:35:16 thoregon kernel: [ 2633.536777] [<ffffffff810bd637>] shrink_inactive_list+0x1d7/0x310 Nov 20 19:35:16 thoregon kernel: [ 2633.536785] [<ffffffff810bdb9d>] shrink_lruvec+0x42d/0x530 Nov 20 19:35:16 thoregon kernel: [ 2633.536795] [<ffffffff810be323>] kswapd+0x683/0xa20 Nov 20 19:35:16 thoregon kernel: [ 2633.536804] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.536812] [<ffffffff810bdca0>] ? shrink_lruvec+0x530/0x530 Nov 20 19:35:16 thoregon kernel: [ 2633.536819] [<ffffffff8105c246>] kthread+0xd6/0xe0 Nov 20 19:35:16 thoregon kernel: [ 2633.536826] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50 Nov 20 19:35:16 thoregon kernel: [ 2633.536835] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 Nov 20 19:35:16 thoregon kernel: [ 2633.536842] [<ffffffff816b31ac>] ret_from_fork+0x7c/0xb0 Nov 20 19:35:16 thoregon kernel: [ 2633.536849] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 Nov 20 19:35:16 thoregon kernel: [ 2633.536890] xfsaild/md4 D 0000000000000003 0 1742 2 0x00000000 Nov 20 19:35:16 thoregon kernel: [ 2633.536898] ffff88032438bb68 0000000000000046 ffff880329965700 ffff88032438bfd8 Nov 20 19:35:16 thoregon kernel: [ 2633.536908] ffff88032438bfd8 ffff88032438bfd8 ffff88032827e580 ffff880329965700 Nov 20 19:35:16 thoregon kernel: [ 2633.536917] ffff88032438bb78 ffff8803278dbbd0 ffff8803278db800 00000000ffffffff Nov 20 19:35:16 thoregon kernel: [ 2633.536926] Call Trace: Nov 20 19:35:16 thoregon kernel: [ 2633.536935] [<ffffffff816b1224>] schedule+0x24/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.536942] [<ffffffff814f9dad>] md_super_wait+0x4d/0x80 Nov 20 19:35:16 thoregon kernel: [ 2633.536950] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.536957] [<ffffffff81500753>] bitmap_unplug+0x173/0x180 Nov 20 19:35:16 thoregon kernel: [ 2633.536966] [<ffffffff81278c13>] ? blk_finish_plug+0x13/0x50 Nov 20 19:35:16 thoregon kernel: [ 2633.536974] [<ffffffff814e8eb8>] raid1_unplug+0x98/0x110 Nov 20 19:35:16 thoregon kernel: [ 2633.536982] [<ffffffff81278a6d>] blk_flush_plug_list+0xad/0x240 Nov 20 19:35:16 thoregon kernel: [ 2633.536991] [<ffffffff81278c13>] blk_finish_plug+0x13/0x50 Nov 20 19:35:16 thoregon kernel: [ 2633.536998] [<ffffffff811e296a>] __xfs_buf_delwri_submit+0x1ca/0x1e0 Nov 20 19:35:16 thoregon kernel: [ 2633.537006] [<ffffffff811e2ffb>] xfs_buf_delwri_submit_nowait+0x1b/0x20 Nov 20 19:35:16 thoregon kernel: [ 2633.537014] [<ffffffff81233066>] xfsaild+0x226/0x4c0 Nov 20 19:35:16 thoregon kernel: [ 2633.537022] [<ffffffff81065dfa>] ? finish_task_switch+0x3a/0x100 Nov 20 19:35:16 thoregon kernel: [ 2633.537030] [<ffffffff81232e40>] ? xfs_trans_ail_cursor_first+0xa0/0xa0 Nov 20 19:35:16 thoregon kernel: [ 2633.537037] [<ffffffff8105c246>] kthread+0xd6/0xe0 Nov 20 19:35:16 thoregon kernel: [ 2633.537044] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50 Nov 20 19:35:16 thoregon kernel: [ 2633.537053] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 Nov 20 19:35:16 thoregon kernel: [ 2633.537060] [<ffffffff816b31ac>] ret_from_fork+0x7c/0xb0 Nov 20 19:35:16 thoregon kernel: [ 2633.537067] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 Nov 20 19:35:16 thoregon kernel: [ 2633.537158] plasma-desktop D 0000000000000001 0 3195 1 0x00000000 Nov 20 19:35:16 thoregon kernel: [ 2633.537165] ffff88030b99d448 0000000000000046 ffff8803266c1d00 ffff88030b99dfd8 Nov 20 19:35:16 thoregon kernel: [ 2633.537174] ffff88030b99dfd8 ffff88030b99dfd8 ffff880046b78e80 ffff8803266c1d00 Nov 20 19:35:16 thoregon kernel: [ 2633.537183] 0000000000000282 ffff88030b99d468 0000000100038f76 0000000100038f74 Nov 20 19:35:16 thoregon kernel: [ 2633.537192] Call Trace: Nov 20 19:35:16 thoregon kernel: [ 2633.537201] [<ffffffff816b1224>] schedule+0x24/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.537210] [<ffffffff816aed8e>] schedule_timeout+0x12e/0x1c0 Nov 20 19:35:16 thoregon kernel: [ 2633.537219] [<ffffffff81048730>] ? call_timer_fn+0xf0/0xf0 Nov 20 19:35:16 thoregon kernel: [ 2633.537228] [<ffffffff816aee39>] schedule_timeout_uninterruptible+0x19/0x20 Nov 20 19:35:16 thoregon kernel: [ 2633.537235] [<ffffffff811f1f99>] xfs_reclaim_inode+0x129/0x340 Nov 20 19:35:16 thoregon kernel: [ 2633.537242] [<ffffffff811f2d6f>] xfs_reclaim_inodes_ag+0x2bf/0x4f0 Nov 20 19:35:16 thoregon kernel: [ 2633.537249] [<ffffffff811f2b90>] ? xfs_reclaim_inodes_ag+0xe0/0x4f0 Nov 20 19:35:16 thoregon kernel: [ 2633.537258] [<ffffffff811f30de>] xfs_reclaim_inodes_nr+0x2e/0x40 Nov 20 19:35:16 thoregon kernel: [ 2633.537267] [<ffffffff811ef9b0>] xfs_fs_free_cached_objects+0x10/0x20 Nov 20 19:35:16 thoregon kernel: [ 2633.537274] [<ffffffff810f5b43>] prune_super+0x113/0x1a0 Nov 20 19:35:16 thoregon kernel: [ 2633.537282] [<ffffffff810bbd4e>] shrink_slab+0x11e/0x1f0 Nov 20 19:35:16 thoregon kernel: [ 2633.537290] [<ffffffff810b4fa5>] ? drain_pages+0x95/0xc0 Nov 20 19:35:16 thoregon kernel: [ 2633.537298] [<ffffffff810be8df>] try_to_free_pages+0x21f/0x4e0 Nov 20 19:35:16 thoregon kernel: [ 2633.537306] [<ffffffff810b5eb6>] __alloc_pages_nodemask+0x506/0x800 Nov 20 19:35:16 thoregon kernel: [ 2633.537316] [<ffffffff81570ed3>] ? __alloc_skb+0x83/0x280 Nov 20 19:35:16 thoregon kernel: [ 2633.537323] [<ffffffff810b6232>] __get_free_pages+0x12/0x40 Nov 20 19:35:16 thoregon kernel: [ 2633.537331] [<ffffffff810ebd8c>] __kmalloc_track_caller+0x14c/0x170 Nov 20 19:35:16 thoregon kernel: [ 2633.537339] [<ffffffff815703e6>] __kmalloc_reserve+0x36/0x90 Nov 20 19:35:16 thoregon kernel: [ 2633.537346] [<ffffffff81570ed3>] __alloc_skb+0x83/0x280 Nov 20 19:35:16 thoregon kernel: [ 2633.537354] [<ffffffff8105c905>] ? remove_wait_queue+0x55/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.537361] [<ffffffff81569fb1>] sock_alloc_send_pskb+0x251/0x3e0 Nov 20 19:35:16 thoregon kernel: [ 2633.537369] [<ffffffff8156a150>] sock_alloc_send_skb+0x10/0x20 Nov 20 19:35:16 thoregon kernel: [ 2633.537376] [<ffffffff81615f21>] unix_stream_sendmsg+0x2e1/0x420 Nov 20 19:35:16 thoregon kernel: [ 2633.537385] [<ffffffff81564900>] ? sock_aio_read+0x30/0x30 Nov 20 19:35:16 thoregon kernel: [ 2633.537393] [<ffffffff81564a0c>] sock_aio_write+0x10c/0x140 Nov 20 19:35:16 thoregon kernel: [ 2633.537403] [<ffffffff810f2eeb>] do_sync_readv_writev+0x9b/0xe0 Nov 20 19:35:16 thoregon kernel: [ 2633.537413] [<ffffffff810f31a8>] do_readv_writev+0xc8/0x1c0 Nov 20 19:35:16 thoregon kernel: [ 2633.537421] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:35:16 thoregon kernel: [ 2633.537428] [<ffffffff8107c899>] ? __lock_is_held+0x59/0x70 Nov 20 19:35:16 thoregon kernel: [ 2633.537438] [<ffffffff8110f892>] ? fget_light+0x42/0x4b0 Nov 20 19:35:16 thoregon kernel: [ 2633.537446] [<ffffffff810f32d7>] vfs_writev+0x37/0x50 Nov 20 19:35:16 thoregon kernel: [ 2633.537454] [<ffffffff810f341d>] sys_writev+0x4d/0x90 Nov 20 19:35:16 thoregon kernel: [ 2633.537462] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b Nov 20 19:35:16 thoregon kernel: [ 2633.537518] ssh D 0000000000000000 0 3573 1 0x00000004 Nov 20 19:35:16 thoregon kernel: [ 2633.537525] ffff8802c9f2f508 0000000000000046 ffff8802f04dd700 ffff8802c9f2ffd8 Nov 20 19:35:16 thoregon kernel: [ 2633.537534] ffff8802c9f2ffd8 ffff8802c9f2ffd8 ffffffff81c13420 ffff8802f04dd700 Nov 20 19:35:16 thoregon kernel: [ 2633.537543] 0000000000000282 ffff8802c9f2f528 0000000100038f75 0000000100038f73 Nov 20 19:35:16 thoregon kernel: [ 2633.537552] Call Trace: Nov 20 19:35:16 thoregon kernel: [ 2633.537561] [<ffffffff816b1224>] schedule+0x24/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.537569] [<ffffffff816aed8e>] schedule_timeout+0x12e/0x1c0 Nov 20 19:35:16 thoregon kernel: [ 2633.537578] [<ffffffff81048730>] ? call_timer_fn+0xf0/0xf0 Nov 20 19:35:16 thoregon kernel: [ 2633.537587] [<ffffffff816aee39>] schedule_timeout_uninterruptible+0x19/0x20 Nov 20 19:35:16 thoregon kernel: [ 2633.537594] [<ffffffff811f1f99>] xfs_reclaim_inode+0x129/0x340 Nov 20 19:35:16 thoregon kernel: [ 2633.537601] [<ffffffff811f2d6f>] xfs_reclaim_inodes_ag+0x2bf/0x4f0 Nov 20 19:35:16 thoregon kernel: [ 2633.537607] [<ffffffff811f2b90>] ? xfs_reclaim_inodes_ag+0xe0/0x4f0 Nov 20 19:35:16 thoregon kernel: [ 2633.537616] [<ffffffff811f30de>] xfs_reclaim_inodes_nr+0x2e/0x40 Nov 20 19:35:16 thoregon kernel: [ 2633.537625] [<ffffffff811ef9b0>] xfs_fs_free_cached_objects+0x10/0x20 Nov 20 19:35:16 thoregon kernel: [ 2633.537631] [<ffffffff810f5b43>] prune_super+0x113/0x1a0 Nov 20 19:35:16 thoregon kernel: [ 2633.537639] [<ffffffff810bbd4e>] shrink_slab+0x11e/0x1f0 Nov 20 19:35:16 thoregon kernel: [ 2633.537647] [<ffffffff810b4fa5>] ? drain_pages+0x95/0xc0 Nov 20 19:35:16 thoregon kernel: [ 2633.537655] [<ffffffff810be8df>] try_to_free_pages+0x21f/0x4e0 Nov 20 19:35:16 thoregon kernel: [ 2633.537663] [<ffffffff810b5eb6>] __alloc_pages_nodemask+0x506/0x800 Nov 20 19:35:16 thoregon kernel: [ 2633.537670] [<ffffffff81104d10>] ? poll_select_set_timeout+0x90/0x90 Nov 20 19:35:16 thoregon kernel: [ 2633.537679] [<ffffffff81570ed3>] ? __alloc_skb+0x83/0x280 Nov 20 19:35:16 thoregon kernel: [ 2633.537686] [<ffffffff810b6232>] __get_free_pages+0x12/0x40 Nov 20 19:35:16 thoregon kernel: [ 2633.537693] [<ffffffff810ebd8c>] __kmalloc_track_caller+0x14c/0x170 Nov 20 19:35:16 thoregon kernel: [ 2633.537701] [<ffffffff815703e6>] __kmalloc_reserve+0x36/0x90 Nov 20 19:35:16 thoregon kernel: [ 2633.537708] [<ffffffff81570ed3>] __alloc_skb+0x83/0x280 Nov 20 19:35:16 thoregon kernel: [ 2633.537715] [<ffffffff81569ac3>] ? release_sock+0x183/0x1e0 Nov 20 19:35:16 thoregon kernel: [ 2633.537722] [<ffffffff81569fb1>] sock_alloc_send_pskb+0x251/0x3e0 Nov 20 19:35:16 thoregon kernel: [ 2633.537729] [<ffffffff8156a150>] sock_alloc_send_skb+0x10/0x20 Nov 20 19:35:16 thoregon kernel: [ 2633.537736] [<ffffffff81615f21>] unix_stream_sendmsg+0x2e1/0x420 Nov 20 19:35:16 thoregon kernel: [ 2633.537745] [<ffffffff81564a0c>] sock_aio_write+0x10c/0x140 Nov 20 19:35:16 thoregon kernel: [ 2633.537753] [<ffffffff810cb5ae>] ? might_fault+0x4e/0xa0 Nov 20 19:35:16 thoregon kernel: [ 2633.537760] [<ffffffff811059aa>] ? core_sys_select+0x47a/0x4d0 Nov 20 19:35:16 thoregon kernel: [ 2633.537769] [<ffffffff810f238b>] do_sync_write+0x9b/0xe0 Nov 20 19:35:16 thoregon kernel: [ 2633.537778] [<ffffffff810f2a75>] vfs_write+0x145/0x160 Nov 20 19:35:16 thoregon kernel: [ 2633.537787] [<ffffffff810f2ccd>] sys_write+0x4d/0x90 Nov 20 19:35:16 thoregon kernel: [ 2633.537794] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b Nov 20 19:35:16 thoregon kernel: [ 2633.537801] bash D 0000000000000000 0 28378 28334 0x00000000 Nov 20 19:35:16 thoregon kernel: [ 2633.537809] ffff880299f5d5d8 0000000000000046 ffff880326a83a00 ffff880299f5dfd8 Nov 20 19:35:16 thoregon kernel: [ 2633.537818] ffff880299f5dfd8 ffff880299f5dfd8 ffff88032989ba00 ffff880326a83a00 Nov 20 19:35:16 thoregon kernel: [ 2633.537827] ffff880299f5d6a8 ffff880299c50230 7fffffffffffffff ffff880326a83a00 Nov 20 19:35:16 thoregon kernel: [ 2633.537836] Call Trace: Nov 20 19:35:16 thoregon kernel: [ 2633.537845] [<ffffffff816b1224>] schedule+0x24/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.537853] [<ffffffff816aeddd>] schedule_timeout+0x17d/0x1c0 Nov 20 19:35:16 thoregon kernel: [ 2633.537860] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50 Nov 20 19:35:16 thoregon kernel: [ 2633.537868] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:35:16 thoregon kernel: [ 2633.537876] [<ffffffff816b057e>] __down_common+0x97/0xe8 Nov 20 19:35:16 thoregon kernel: [ 2633.537885] [<ffffffff816b062e>] __down+0x18/0x1a Nov 20 19:35:16 thoregon kernel: [ 2633.537891] [<ffffffff8106245b>] down+0x3b/0x50 Nov 20 19:35:16 thoregon kernel: [ 2633.537898] [<ffffffff811e21b5>] xfs_buf_lock+0x25/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.537904] [<ffffffff811e22e2>] _xfs_buf_find+0xf2/0x200 Nov 20 19:35:16 thoregon kernel: [ 2633.537911] [<ffffffff811e241f>] xfs_buf_get_map+0x2f/0x130 Nov 20 19:35:16 thoregon kernel: [ 2633.537918] [<ffffffff811e2b23>] xfs_buf_read_map+0x13/0x90 Nov 20 19:35:16 thoregon kernel: [ 2633.537925] [<ffffffff81233e8d>] xfs_trans_read_buf_map+0x1ad/0x2d0 Nov 20 19:35:16 thoregon kernel: [ 2633.537933] [<ffffffff811f73fb>] xfs_alloc_read_agfl+0x6b/0x90 Nov 20 19:35:16 thoregon kernel: [ 2633.537941] [<ffffffff811f986d>] ? xfs_alloc_read_agf+0xcd/0xf0 Nov 20 19:35:16 thoregon kernel: [ 2633.537948] [<ffffffff810e9a93>] ? kmem_cache_free+0x83/0x150 Nov 20 19:35:16 thoregon kernel: [ 2633.537955] [<ffffffff811f9ac1>] xfs_alloc_fix_freelist+0x231/0x480 Nov 20 19:35:16 thoregon kernel: [ 2633.537962] [<ffffffff8107c899>] ? __lock_is_held+0x59/0x70 Nov 20 19:35:16 thoregon kernel: [ 2633.537971] [<ffffffff811fa279>] xfs_free_extent+0x99/0x120 Nov 20 19:35:16 thoregon kernel: [ 2633.537978] [<ffffffff811f6fff>] ? kmem_zone_alloc+0x5f/0xe0 Nov 20 19:35:16 thoregon kernel: [ 2633.537987] [<ffffffff81206edc>] xfs_bmap_finish+0x15c/0x1a0 Nov 20 19:35:16 thoregon kernel: [ 2633.537996] [<ffffffff8121f9eb>] xfs_itruncate_extents+0xdb/0x200 Nov 20 19:35:16 thoregon kernel: [ 2633.538006] [<ffffffff811ed7d5>] xfs_setattr_size+0x355/0x3b0 Nov 20 19:35:16 thoregon kernel: [ 2633.538014] [<ffffffff811ed85e>] xfs_vn_setattr+0x2e/0x40 Nov 20 19:35:16 thoregon kernel: [ 2633.538022] [<ffffffff8110dfa6>] notify_change+0x186/0x3a0 Nov 20 19:35:16 thoregon kernel: [ 2633.538030] [<ffffffff810f1229>] do_truncate+0x59/0x90 Nov 20 19:35:16 thoregon kernel: [ 2633.538039] [<ffffffff810fd777>] ? __inode_permission+0x27/0x80 Nov 20 19:35:16 thoregon kernel: [ 2633.538048] [<ffffffff811008da>] do_last.isra.69+0x60a/0xc80 Nov 20 19:35:16 thoregon kernel: [ 2633.538056] [<ffffffff810fd7e3>] ? inode_permission+0x13/0x50 Nov 20 19:35:16 thoregon kernel: [ 2633.538064] [<ffffffff810fd882>] ? link_path_walk+0x62/0x910 Nov 20 19:35:16 thoregon kernel: [ 2633.538073] [<ffffffff81100ffb>] path_openat.isra.70+0xab/0x490 Nov 20 19:35:16 thoregon kernel: [ 2633.538080] [<ffffffff8107c899>] ? __lock_is_held+0x59/0x70 Nov 20 19:35:16 thoregon kernel: [ 2633.538089] [<ffffffff8110179d>] do_filp_open+0x3d/0xa0 Nov 20 19:35:16 thoregon kernel: [ 2633.538097] [<ffffffff8111082b>] ? __alloc_fd+0x16b/0x210 Nov 20 19:35:16 thoregon kernel: [ 2633.538106] [<ffffffff810f2089>] do_sys_open+0xf9/0x1e0 Nov 20 19:35:16 thoregon kernel: [ 2633.538114] [<ffffffff810f218c>] sys_open+0x1c/0x20 Nov 20 19:35:16 thoregon kernel: [ 2633.538121] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b Nov 20 19:35:16 thoregon kernel: [ 2633.538126] flush-9:4 D 0000000000000001 0 30365 2 0x00000000 Nov 20 19:35:16 thoregon kernel: [ 2633.538134] ffff8801d8ce1978 0000000000000046 ffff880328238e80 ffff8801d8ce1fd8 Nov 20 19:35:16 thoregon kernel: [ 2633.538143] ffff8801d8ce1fd8 ffff8801d8ce1fd8 ffff8803032cc880 ffff880328238e80 Nov 20 19:35:16 thoregon kernel: [ 2633.538151] ffff8801d8ce1988 ffff8803278dbbd0 ffff8803278db800 00000000ffffffff Nov 20 19:35:16 thoregon kernel: [ 2633.538160] Call Trace: Nov 20 19:35:16 thoregon kernel: [ 2633.538169] [<ffffffff816b1224>] schedule+0x24/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.538176] [<ffffffff814f9dad>] md_super_wait+0x4d/0x80 Nov 20 19:35:16 thoregon kernel: [ 2633.538184] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.538192] [<ffffffff81500753>] bitmap_unplug+0x173/0x180 Nov 20 19:35:16 thoregon kernel: [ 2633.538199] [<ffffffff810b6acf>] ? write_cache_pages+0x12f/0x420 Nov 20 19:35:16 thoregon kernel: [ 2633.538206] [<ffffffff810b6700>] ? set_page_dirty_lock+0x60/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.538214] [<ffffffff814e8eb8>] raid1_unplug+0x98/0x110 Nov 20 19:35:16 thoregon kernel: [ 2633.538222] [<ffffffff81278a6d>] blk_flush_plug_list+0xad/0x240 Nov 20 19:35:16 thoregon kernel: [ 2633.538231] [<ffffffff81278c13>] blk_finish_plug+0x13/0x50 Nov 20 19:35:16 thoregon kernel: [ 2633.538238] [<ffffffff810b6e0a>] generic_writepages+0x4a/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.538247] [<ffffffff811df898>] xfs_vm_writepages+0x48/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.538256] [<ffffffff8111b144>] ? writeback_sb_inodes+0x174/0x410 Nov 20 19:35:16 thoregon kernel: [ 2633.538264] [<ffffffff810b84be>] do_writepages+0x1e/0x30 Nov 20 19:35:16 thoregon kernel: [ 2633.538272] [<ffffffff8111a1ee>] __writeback_single_inode+0x3e/0x120 Nov 20 19:35:16 thoregon kernel: [ 2633.538280] [<ffffffff8111b234>] writeback_sb_inodes+0x264/0x410 Nov 20 19:35:16 thoregon kernel: [ 2633.538289] [<ffffffff8111b476>] __writeback_inodes_wb+0x96/0xc0 Nov 20 19:35:16 thoregon kernel: [ 2633.538298] [<ffffffff8111b673>] wb_writeback+0x1d3/0x1e0 Nov 20 19:35:16 thoregon kernel: [ 2633.538305] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:35:16 thoregon kernel: [ 2633.538313] [<ffffffff8111bb1c>] ? wb_do_writeback+0x7c/0x140 Nov 20 19:35:16 thoregon kernel: [ 2633.538322] [<ffffffff8111bb40>] wb_do_writeback+0xa0/0x140 Nov 20 19:35:16 thoregon kernel: [ 2633.538331] [<ffffffff8111bc52>] bdi_writeback_thread+0x72/0x150 Nov 20 19:35:16 thoregon kernel: [ 2633.538339] [<ffffffff8111bbe0>] ? wb_do_writeback+0x140/0x140 Nov 20 19:35:16 thoregon kernel: [ 2633.538346] [<ffffffff8105c246>] kthread+0xd6/0xe0 Nov 20 19:35:16 thoregon kernel: [ 2633.538353] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50 Nov 20 19:35:16 thoregon kernel: [ 2633.538361] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 Nov 20 19:35:16 thoregon kernel: [ 2633.538369] [<ffffffff816b31ac>] ret_from_fork+0x7c/0xb0 Nov 20 19:35:16 thoregon kernel: [ 2633.538376] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 Nov 20 19:35:16 thoregon kernel: [ 2633.538384] x86_64-pc-linux D 0000000000000000 0 26516 1 0x00000004 Nov 20 19:35:16 thoregon kernel: [ 2633.538391] ffff8800468dddf8 0000000000000046 ffff880329303a00 ffff8800468ddfd8 Nov 20 19:35:16 thoregon kernel: [ 2633.538400] ffff8800468ddfd8 ffff8800468ddfd8 ffff88032989c880 ffff880329303a00 Nov 20 19:35:16 thoregon kernel: [ 2633.538409] 0000000000000246 ffff8800468dc000 ffff8801b1a1bc68 0000000002405150 Nov 20 19:35:16 thoregon kernel: [ 2633.538418] Call Trace: Nov 20 19:35:16 thoregon kernel: [ 2633.538427] [<ffffffff816b1224>] schedule+0x24/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.538436] [<ffffffff816b1501>] schedule_preempt_disabled+0x21/0x30 Nov 20 19:35:16 thoregon kernel: [ 2633.538444] [<ffffffff816af3a5>] mutex_lock_nested+0x165/0x330 Nov 20 19:35:16 thoregon kernel: [ 2633.538452] [<ffffffff810ff5e3>] ? do_unlinkat+0xa3/0x1b0 Nov 20 19:35:16 thoregon kernel: [ 2633.538461] [<ffffffff810fefae>] ? filename_lookup.isra.57+0x2e/0x80 Nov 20 19:35:16 thoregon kernel: [ 2633.538469] [<ffffffff810ff5e3>] do_unlinkat+0xa3/0x1b0 Nov 20 19:35:16 thoregon kernel: [ 2633.538477] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:35:16 thoregon kernel: [ 2633.538485] [<ffffffff8129ca1e>] ? trace_hardirqs_on_thunk+0x3a/0x3f Nov 20 19:35:16 thoregon kernel: [ 2633.538494] [<ffffffff81101bb1>] sys_unlink+0x11/0x20 Nov 20 19:35:16 thoregon kernel: [ 2633.538501] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b Nov 20 19:35:16 thoregon kernel: [ 2633.538505] x86_64-pc-linux D 0000000000000000 0 26523 1 0x00000004 Nov 20 19:35:16 thoregon kernel: [ 2633.538512] ffff8800c0def858 0000000000000046 ffff88032986c880 ffff8800c0deffd8 Nov 20 19:35:16 thoregon kernel: [ 2633.538521] ffff8800c0deffd8 ffff8800c0deffd8 ffff88032989e580 ffff88032986c880 Nov 20 19:35:16 thoregon kernel: [ 2633.538530] ffff8800c0def928 ffff8801c57e3a30 7fffffffffffffff ffff88032986c880 Nov 20 19:35:16 thoregon kernel: [ 2633.538539] Call Trace: Nov 20 19:35:16 thoregon kernel: [ 2633.538548] [<ffffffff816b1224>] schedule+0x24/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.538556] [<ffffffff816aeddd>] schedule_timeout+0x17d/0x1c0 Nov 20 19:35:16 thoregon kernel: [ 2633.538563] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50 Nov 20 19:35:16 thoregon kernel: [ 2633.538571] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:35:16 thoregon kernel: [ 2633.538579] [<ffffffff816b057e>] __down_common+0x97/0xe8 Nov 20 19:35:16 thoregon kernel: [ 2633.538588] [<ffffffff816b062e>] __down+0x18/0x1a Nov 20 19:35:16 thoregon kernel: [ 2633.538594] [<ffffffff8106245b>] down+0x3b/0x50 Nov 20 19:35:16 thoregon kernel: [ 2633.538601] [<ffffffff811e21b5>] xfs_buf_lock+0x25/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.538607] [<ffffffff811e22e2>] _xfs_buf_find+0xf2/0x200 Nov 20 19:35:16 thoregon kernel: [ 2633.538614] [<ffffffff811e241f>] xfs_buf_get_map+0x2f/0x130 Nov 20 19:35:16 thoregon kernel: [ 2633.538621] [<ffffffff81210417>] ? xfs_dabuf_map.isra.23+0xb7/0x2f0 Nov 20 19:35:16 thoregon kernel: [ 2633.538628] [<ffffffff810b572e>] ? get_page_from_freelist+0x1de/0x460 Nov 20 19:35:16 thoregon kernel: [ 2633.538635] [<ffffffff811e2b23>] xfs_buf_read_map+0x13/0x90 Nov 20 19:35:16 thoregon kernel: [ 2633.538642] [<ffffffff81233e8d>] xfs_trans_read_buf_map+0x1ad/0x2d0 Nov 20 19:35:16 thoregon kernel: [ 2633.538650] [<ffffffff812116b6>] xfs_da_read_buf+0xb6/0x180 Nov 20 19:35:16 thoregon kernel: [ 2633.538659] [<ffffffff8121724c>] xfs_dir2_leaf_lookup_int+0x5c/0x2c0 Nov 20 19:35:16 thoregon kernel: [ 2633.538668] [<ffffffff81217583>] xfs_dir2_leaf_removename+0x33/0x330 Nov 20 19:35:16 thoregon kernel: [ 2633.538676] [<ffffffff8120777b>] ? xfs_bmap_last_offset+0x8b/0xc0 Nov 20 19:35:16 thoregon kernel: [ 2633.538684] [<ffffffff81213b94>] xfs_dir_removename+0x134/0x150 Nov 20 19:35:16 thoregon kernel: [ 2633.538692] [<ffffffff811f5ce6>] xfs_remove+0x1f6/0x370 Nov 20 19:35:16 thoregon kernel: [ 2633.538701] [<ffffffff816af4ac>] ? mutex_lock_nested+0x26c/0x330 Nov 20 19:35:16 thoregon kernel: [ 2633.538709] [<ffffffff810ff48f>] ? vfs_unlink+0x4f/0x100 Nov 20 19:35:16 thoregon kernel: [ 2633.538717] [<ffffffff811ec7f3>] xfs_vn_unlink+0x43/0x90 Nov 20 19:35:16 thoregon kernel: [ 2633.538725] [<ffffffff810ff4cd>] vfs_unlink+0x8d/0x100 Nov 20 19:35:16 thoregon kernel: [ 2633.538733] [<ffffffff810ff63a>] do_unlinkat+0xfa/0x1b0 Nov 20 19:35:16 thoregon kernel: [ 2633.538741] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:35:16 thoregon kernel: [ 2633.538748] [<ffffffff8129ca1e>] ? trace_hardirqs_on_thunk+0x3a/0x3f Nov 20 19:35:16 thoregon kernel: [ 2633.538757] [<ffffffff81101bb1>] sys_unlink+0x11/0x20 Nov 20 19:35:16 thoregon kernel: [ 2633.538764] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b Nov 20 19:35:16 thoregon kernel: [ 2633.538769] x86_64-pc-linux D 0000000000000000 0 26875 1 0x00000004 Nov 20 19:35:16 thoregon kernel: [ 2633.538776] ffff8800c0c5ba08 0000000000000046 ffff8802f0594880 ffff8800c0c5bfd8 Nov 20 19:35:16 thoregon kernel: [ 2633.538785] ffff8800c0c5bfd8 ffff8800c0c5bfd8 ffff88032989c880 ffff8802f0594880 Nov 20 19:35:16 thoregon kernel: [ 2633.538793] 0000000000000250 ffff880205a7aa30 7fffffffffffffff ffff8802f0594880 Nov 20 19:35:16 thoregon kernel: [ 2633.538802] Call Trace: Nov 20 19:35:16 thoregon kernel: [ 2633.538811] [<ffffffff816b1224>] schedule+0x24/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.538819] [<ffffffff816aeddd>] schedule_timeout+0x17d/0x1c0 Nov 20 19:35:16 thoregon kernel: [ 2633.538826] [<ffffffff811f70b4>] ? kmem_zone_zalloc+0x34/0x50 Nov 20 19:35:16 thoregon kernel: [ 2633.538833] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50 Nov 20 19:35:16 thoregon kernel: [ 2633.538841] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:35:16 thoregon kernel: [ 2633.538849] [<ffffffff816b057e>] __down_common+0x97/0xe8 Nov 20 19:35:16 thoregon kernel: [ 2633.538857] [<ffffffff816b062e>] __down+0x18/0x1a Nov 20 19:35:16 thoregon kernel: [ 2633.538864] [<ffffffff8106245b>] down+0x3b/0x50 Nov 20 19:35:16 thoregon kernel: [ 2633.538870] [<ffffffff811e21b5>] xfs_buf_lock+0x25/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.538877] [<ffffffff811e22e2>] _xfs_buf_find+0xf2/0x200 Nov 20 19:35:16 thoregon kernel: [ 2633.538883] [<ffffffff811e241f>] xfs_buf_get_map+0x2f/0x130 Nov 20 19:35:16 thoregon kernel: [ 2633.538890] [<ffffffff811e2b23>] xfs_buf_read_map+0x13/0x90 Nov 20 19:35:16 thoregon kernel: [ 2633.538897] [<ffffffff81233e8d>] xfs_trans_read_buf_map+0x1ad/0x2d0 Nov 20 19:35:16 thoregon kernel: [ 2633.538906] [<ffffffff8121ce6f>] xfs_read_agi+0x7f/0x110 Nov 20 19:35:16 thoregon kernel: [ 2633.538915] [<ffffffff8121fb55>] xfs_iunlink+0x45/0x150 Nov 20 19:35:16 thoregon kernel: [ 2633.538922] [<ffffffff81041fc1>] ? current_fs_time+0x11/0x50 Nov 20 19:35:16 thoregon kernel: [ 2633.538929] [<ffffffff81234352>] ? xfs_trans_ichgtime+0x22/0xa0 Nov 20 19:35:16 thoregon kernel: [ 2633.538936] [<ffffffff811f3ea0>] xfs_droplink+0x50/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.538943] [<ffffffff811f5d77>] xfs_remove+0x287/0x370 Nov 20 19:35:16 thoregon kernel: [ 2633.538951] [<ffffffff816af4ac>] ? mutex_lock_nested+0x26c/0x330 Nov 20 19:35:16 thoregon kernel: [ 2633.538959] [<ffffffff810ff48f>] ? vfs_unlink+0x4f/0x100 Nov 20 19:35:16 thoregon kernel: [ 2633.538967] [<ffffffff811ec7f3>] xfs_vn_unlink+0x43/0x90 Nov 20 19:35:16 thoregon kernel: [ 2633.538975] [<ffffffff810ff4cd>] vfs_unlink+0x8d/0x100 Nov 20 19:35:16 thoregon kernel: [ 2633.538983] [<ffffffff810ff63a>] do_unlinkat+0xfa/0x1b0 Nov 20 19:35:16 thoregon kernel: [ 2633.538991] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:35:16 thoregon kernel: [ 2633.538999] [<ffffffff8129ca1e>] ? trace_hardirqs_on_thunk+0x3a/0x3f Nov 20 19:35:16 thoregon kernel: [ 2633.539007] [<ffffffff81101bb1>] sys_unlink+0x11/0x20 Nov 20 19:35:16 thoregon kernel: [ 2633.539014] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b Nov 20 19:35:16 thoregon kernel: [ 2633.539020] x86_64-pc-linux D 0000000000000001 0 26897 1 0x00000004 Nov 20 19:35:16 thoregon kernel: [ 2633.539027] ffff880101959858 0000000000000046 ffff88032934ab80 ffff880101959fd8 Nov 20 19:35:16 thoregon kernel: [ 2633.539036] ffff880101959fd8 ffff880101959fd8 ffff880162a3e580 ffff88032934ab80 Nov 20 19:35:16 thoregon kernel: [ 2633.539045] ffff880101959928 ffff8801ca545030 7fffffffffffffff ffff88032934ab80 Nov 20 19:35:16 thoregon kernel: [ 2633.539053] Call Trace: Nov 20 19:35:16 thoregon kernel: [ 2633.539062] [<ffffffff816b1224>] schedule+0x24/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.539070] [<ffffffff816aeddd>] schedule_timeout+0x17d/0x1c0 Nov 20 19:35:16 thoregon kernel: [ 2633.539077] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50 Nov 20 19:35:16 thoregon kernel: [ 2633.539085] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:35:16 thoregon kernel: [ 2633.539094] [<ffffffff816b057e>] __down_common+0x97/0xe8 Nov 20 19:35:16 thoregon kernel: [ 2633.539102] [<ffffffff816b062e>] __down+0x18/0x1a Nov 20 19:35:16 thoregon kernel: [ 2633.539108] [<ffffffff8106245b>] down+0x3b/0x50 Nov 20 19:35:16 thoregon kernel: [ 2633.539115] [<ffffffff811e21b5>] xfs_buf_lock+0x25/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.539121] [<ffffffff811e22e2>] _xfs_buf_find+0xf2/0x200 Nov 20 19:35:16 thoregon kernel: [ 2633.539128] [<ffffffff811e241f>] xfs_buf_get_map+0x2f/0x130 Nov 20 19:35:16 thoregon kernel: [ 2633.539135] [<ffffffff81210417>] ? xfs_dabuf_map.isra.23+0xb7/0x2f0 Nov 20 19:35:16 thoregon kernel: [ 2633.539142] [<ffffffff810b572e>] ? get_page_from_freelist+0x1de/0x460 Nov 20 19:35:16 thoregon kernel: [ 2633.539149] [<ffffffff811e2b23>] xfs_buf_read_map+0x13/0x90 Nov 20 19:35:16 thoregon kernel: [ 2633.539156] [<ffffffff81233e8d>] xfs_trans_read_buf_map+0x1ad/0x2d0 Nov 20 19:35:16 thoregon kernel: [ 2633.539164] [<ffffffff812116b6>] xfs_da_read_buf+0xb6/0x180 Nov 20 19:35:16 thoregon kernel: [ 2633.539172] [<ffffffff8121724c>] xfs_dir2_leaf_lookup_int+0x5c/0x2c0 Nov 20 19:35:16 thoregon kernel: [ 2633.539181] [<ffffffff81217583>] xfs_dir2_leaf_removename+0x33/0x330 Nov 20 19:35:16 thoregon kernel: [ 2633.539189] [<ffffffff8120777b>] ? xfs_bmap_last_offset+0x8b/0xc0 Nov 20 19:35:16 thoregon kernel: [ 2633.539197] [<ffffffff81213b94>] xfs_dir_removename+0x134/0x150 Nov 20 19:35:16 thoregon kernel: [ 2633.539205] [<ffffffff811f5ce6>] xfs_remove+0x1f6/0x370 Nov 20 19:35:16 thoregon kernel: [ 2633.539214] [<ffffffff816af4ac>] ? mutex_lock_nested+0x26c/0x330 Nov 20 19:35:16 thoregon kernel: [ 2633.539222] [<ffffffff810ff48f>] ? vfs_unlink+0x4f/0x100 Nov 20 19:35:16 thoregon kernel: [ 2633.539230] [<ffffffff811ec7f3>] xfs_vn_unlink+0x43/0x90 Nov 20 19:35:16 thoregon kernel: [ 2633.539238] [<ffffffff810ff4cd>] vfs_unlink+0x8d/0x100 Nov 20 19:35:16 thoregon kernel: [ 2633.539246] [<ffffffff810ff63a>] do_unlinkat+0xfa/0x1b0 Nov 20 19:35:16 thoregon kernel: [ 2633.539254] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:35:16 thoregon kernel: [ 2633.539261] [<ffffffff8129ca1e>] ? trace_hardirqs_on_thunk+0x3a/0x3f Nov 20 19:35:16 thoregon kernel: [ 2633.539270] [<ffffffff81101bb1>] sys_unlink+0x11/0x20 Nov 20 19:35:16 thoregon kernel: [ 2633.539277] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b Nov 20 19:35:16 thoregon kernel: [ 2633.539282] x86_64-pc-linux D 0000000000000001 0 26904 1 0x00000004 Nov 20 19:35:16 thoregon kernel: [ 2633.539290] ffff880066bff858 0000000000000046 ffff880329301d00 ffff880066bfffd8 Nov 20 19:35:16 thoregon kernel: [ 2633.539299] ffff880066bfffd8 ffff880066bfffd8 ffff88013a7ce580 ffff880329301d00 Nov 20 19:35:16 thoregon kernel: [ 2633.539307] ffff880066bff928 ffff8801c572c830 7fffffffffffffff ffff880329301d00 Nov 20 19:35:16 thoregon kernel: [ 2633.539316] Call Trace: Nov 20 19:35:16 thoregon kernel: [ 2633.539325] [<ffffffff816b1224>] schedule+0x24/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.539333] [<ffffffff816aeddd>] schedule_timeout+0x17d/0x1c0 Nov 20 19:35:16 thoregon kernel: [ 2633.539340] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50 Nov 20 19:35:16 thoregon kernel: [ 2633.539348] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:35:16 thoregon kernel: [ 2633.539356] [<ffffffff816b057e>] __down_common+0x97/0xe8 Nov 20 19:35:16 thoregon kernel: [ 2633.539365] [<ffffffff816b062e>] __down+0x18/0x1a Nov 20 19:35:16 thoregon kernel: [ 2633.539371] [<ffffffff8106245b>] down+0x3b/0x50 Nov 20 19:35:16 thoregon kernel: [ 2633.539377] [<ffffffff811e21b5>] xfs_buf_lock+0x25/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.539384] [<ffffffff811e22e2>] _xfs_buf_find+0xf2/0x200 Nov 20 19:35:16 thoregon kernel: [ 2633.539390] [<ffffffff811e241f>] xfs_buf_get_map+0x2f/0x130 Nov 20 19:35:16 thoregon kernel: [ 2633.539398] [<ffffffff81210417>] ? xfs_dabuf_map.isra.23+0xb7/0x2f0 Nov 20 19:35:16 thoregon kernel: [ 2633.539405] [<ffffffff810b572e>] ? get_page_from_freelist+0x1de/0x460 Nov 20 19:35:16 thoregon kernel: [ 2633.539412] [<ffffffff811e2b23>] xfs_buf_read_map+0x13/0x90 Nov 20 19:35:16 thoregon kernel: [ 2633.539419] [<ffffffff81233e8d>] xfs_trans_read_buf_map+0x1ad/0x2d0 Nov 20 19:35:16 thoregon kernel: [ 2633.539426] [<ffffffff812116b6>] xfs_da_read_buf+0xb6/0x180 Nov 20 19:35:16 thoregon kernel: [ 2633.539435] [<ffffffff8121724c>] xfs_dir2_leaf_lookup_int+0x5c/0x2c0 Nov 20 19:35:16 thoregon kernel: [ 2633.539444] [<ffffffff81217583>] xfs_dir2_leaf_removename+0x33/0x330 Nov 20 19:35:16 thoregon kernel: [ 2633.539452] [<ffffffff8120777b>] ? xfs_bmap_last_offset+0x8b/0xc0 Nov 20 19:35:16 thoregon kernel: [ 2633.539460] [<ffffffff81213b94>] xfs_dir_removename+0x134/0x150 Nov 20 19:35:16 thoregon kernel: [ 2633.539468] [<ffffffff811f5ce6>] xfs_remove+0x1f6/0x370 Nov 20 19:35:16 thoregon kernel: [ 2633.539477] [<ffffffff816af4ac>] ? mutex_lock_nested+0x26c/0x330 Nov 20 19:35:16 thoregon kernel: [ 2633.539484] [<ffffffff810ff48f>] ? vfs_unlink+0x4f/0x100 Nov 20 19:35:16 thoregon kernel: [ 2633.539493] [<ffffffff811ec7f3>] xfs_vn_unlink+0x43/0x90 Nov 20 19:35:16 thoregon kernel: [ 2633.539501] [<ffffffff810ff4cd>] vfs_unlink+0x8d/0x100 Nov 20 19:35:16 thoregon kernel: [ 2633.539509] [<ffffffff810ff63a>] do_unlinkat+0xfa/0x1b0 Nov 20 19:35:16 thoregon kernel: [ 2633.539517] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:35:16 thoregon kernel: [ 2633.539524] [<ffffffff8129ca1e>] ? trace_hardirqs_on_thunk+0x3a/0x3f Nov 20 19:35:16 thoregon kernel: [ 2633.539532] [<ffffffff81101bb1>] sys_unlink+0x11/0x20 Nov 20 19:35:16 thoregon kernel: [ 2633.539540] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b Nov 20 19:35:16 thoregon kernel: [ 2633.539544] x86_64-pc-linux D 0000000000000000 0 26907 1 0x00000004 Nov 20 19:35:16 thoregon kernel: [ 2633.539552] ffff8800c0eed858 0000000000000046 ffff8802a7be6580 ffff8800c0eedfd8 Nov 20 19:35:16 thoregon kernel: [ 2633.539560] ffff8800c0eedfd8 ffff8800c0eedfd8 ffff88032989e580 ffff8802a7be6580 Nov 20 19:35:16 thoregon kernel: [ 2633.539569] ffff8800c0eed928 ffff8801c55c7630 7fffffffffffffff ffff8802a7be6580 Nov 20 19:35:16 thoregon kernel: [ 2633.539578] Call Trace: Nov 20 19:35:16 thoregon kernel: [ 2633.539587] [<ffffffff816b1224>] schedule+0x24/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.539595] [<ffffffff816aeddd>] schedule_timeout+0x17d/0x1c0 Nov 20 19:35:16 thoregon kernel: [ 2633.539602] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50 Nov 20 19:35:16 thoregon kernel: [ 2633.539610] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:35:16 thoregon kernel: [ 2633.539618] [<ffffffff816b057e>] __down_common+0x97/0xe8 Nov 20 19:35:16 thoregon kernel: [ 2633.539627] [<ffffffff816b062e>] __down+0x18/0x1a Nov 20 19:35:16 thoregon kernel: [ 2633.539633] [<ffffffff8106245b>] down+0x3b/0x50 Nov 20 19:35:16 thoregon kernel: [ 2633.539639] [<ffffffff811e21b5>] xfs_buf_lock+0x25/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.539646] [<ffffffff811e22e2>] _xfs_buf_find+0xf2/0x200 Nov 20 19:35:16 thoregon kernel: [ 2633.539652] [<ffffffff811e241f>] xfs_buf_get_map+0x2f/0x130 Nov 20 19:35:16 thoregon kernel: [ 2633.539660] [<ffffffff81210417>] ? xfs_dabuf_map.isra.23+0xb7/0x2f0 Nov 20 19:35:16 thoregon kernel: [ 2633.539667] [<ffffffff810b572e>] ? get_page_from_freelist+0x1de/0x460 Nov 20 19:35:16 thoregon kernel: [ 2633.539674] [<ffffffff811e2b23>] xfs_buf_read_map+0x13/0x90 Nov 20 19:35:16 thoregon kernel: [ 2633.539681] [<ffffffff81233e8d>] xfs_trans_read_buf_map+0x1ad/0x2d0 Nov 20 19:35:16 thoregon kernel: [ 2633.539688] [<ffffffff812116b6>] xfs_da_read_buf+0xb6/0x180 Nov 20 19:35:16 thoregon kernel: [ 2633.539697] [<ffffffff8121724c>] xfs_dir2_leaf_lookup_int+0x5c/0x2c0 Nov 20 19:35:16 thoregon kernel: [ 2633.539706] [<ffffffff81217583>] xfs_dir2_leaf_removename+0x33/0x330 Nov 20 19:35:16 thoregon kernel: [ 2633.539714] [<ffffffff8120777b>] ? xfs_bmap_last_offset+0x8b/0xc0 Nov 20 19:35:16 thoregon kernel: [ 2633.539722] [<ffffffff81213b94>] xfs_dir_removename+0x134/0x150 Nov 20 19:35:16 thoregon kernel: [ 2633.539730] [<ffffffff811f5ce6>] xfs_remove+0x1f6/0x370 Nov 20 19:35:16 thoregon kernel: [ 2633.539738] [<ffffffff816af4ac>] ? mutex_lock_nested+0x26c/0x330 Nov 20 19:35:16 thoregon kernel: [ 2633.539746] [<ffffffff810ff48f>] ? vfs_unlink+0x4f/0x100 Nov 20 19:35:16 thoregon kernel: [ 2633.539754] [<ffffffff811ec7f3>] xfs_vn_unlink+0x43/0x90 Nov 20 19:35:16 thoregon kernel: [ 2633.539763] [<ffffffff810ff4cd>] vfs_unlink+0x8d/0x100 Nov 20 19:35:16 thoregon kernel: [ 2633.539771] [<ffffffff810ff63a>] do_unlinkat+0xfa/0x1b0 Nov 20 19:35:16 thoregon kernel: [ 2633.539779] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:35:16 thoregon kernel: [ 2633.539786] [<ffffffff8129ca1e>] ? trace_hardirqs_on_thunk+0x3a/0x3f Nov 20 19:35:16 thoregon kernel: [ 2633.539794] [<ffffffff81101bb1>] sys_unlink+0x11/0x20 Nov 20 19:35:16 thoregon kernel: [ 2633.539802] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Hang in XFS reclaim on 3.7.0-rc3 @ 2012-11-20 19:45 ` Torsten Kaiser 0 siblings, 0 replies; 31+ messages in thread From: Torsten Kaiser @ 2012-11-20 19:45 UTC (permalink / raw) To: Dave Chinner; +Cc: Linux Kernel, xfs [-- Attachment #1: Type: text/plain, Size: 2768 bytes --] On Tue, Nov 20, 2012 at 12:53 AM, Dave Chinner <david@fromorbit.com> wrote: > [<ffffffff8108137e>] mark_held_locks+0x7e/0x130 > [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0 > [<ffffffff810e9dd5>] kmem_cache_alloc+0x35/0xe0 > [<ffffffff810dba31>] vm_map_ram+0x271/0x770 > [<ffffffff811e1316>] _xfs_buf_map_pages+0x46/0xe0 > [<ffffffff811e222a>] xfs_buf_get_map+0x8a/0x130 > [<ffffffff81233ab9>] xfs_trans_get_buf_map+0xa9/0xd0 > [<ffffffff8121bced>] xfs_ialloc_inode_init+0xcd/0x1d0 > > We shouldn't be mapping buffers there, there's a patch below to fix > this. It's probably the source of this report, even though I cannot > lockdep seems to be off with the fairies... That patch seems to break my system. After it started to swap, because I was compiling seamonkey (firefox turned into the full navigator suite) on a tmpfs, several processes got stuck and triggered the hung-task-check. As a kswapd, xfsaild/md4 and flush-9:4 also got stuck, not even a shutdown worked. The attached log first contains the hung-task-notices, then the output from SysRq+W. After the shutdown got stuck trying to turn of swap, I first tries SysRq+S, but did not get a 'Done' and on SysRq+U lockdep complained about an lock imbalance wrt. sb_writer. SysRq+O also did no longer work, only SysRq+B. I don't know which one got stuck first, but I'm somewhat suspicious of the plasma-desktop and the sshd that SysRq+W reported stuck in xfs reclaim, even if these processes did never trigger the hung task check. Torsten > Cheers, > > Dave. > -- > Dave Chinner > david@fromorbit.com > > xfs: inode allocation should use unmapped buffers. > > From: Dave Chinner <dchinner@redhat.com> > > Inode buffers do not need to be mapped as inodes are read or written > directly from/to the pages underlying the buffer. This fixes a > regression introduced by commit 611c994 ("xfs: make XBF_MAPPED the > default behaviour"). > > Signed-off-by: Dave Chinner <dchinner@redhat.com> > --- > fs/xfs/xfs_ialloc.c | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) > > diff --git a/fs/xfs/xfs_ialloc.c b/fs/xfs/xfs_ialloc.c > index 2d6495e..a815412 100644 > --- a/fs/xfs/xfs_ialloc.c > +++ b/fs/xfs/xfs_ialloc.c > @@ -200,7 +200,8 @@ xfs_ialloc_inode_init( > */ > d = XFS_AGB_TO_DADDR(mp, agno, agbno + (j * blks_per_cluster)); > fbuf = xfs_trans_get_buf(tp, mp->m_ddev_targp, d, > - mp->m_bsize * blks_per_cluster, 0); > + mp->m_bsize * blks_per_cluster, > + XBF_UNMAPPED); > if (!fbuf) > return ENOMEM; > /* [-- Attachment #2: xfs-reclaim-hang-messages.txt --] [-- Type: text/plain, Size: 79805 bytes --] Nov 20 19:24:55 thoregon acpid: client 2991[0:0] has disconnected Nov 20 19:25:07 thoregon login[3002]: pam_unix(login:session): session opened for user root by LOGIN(uid=0) Nov 20 19:25:07 thoregon login[27176]: ROOT LOGIN on '/dev/tty1' Nov 20 19:27:23 thoregon kernel: [ 2160.595990] INFO: task kswapd0:725 blocked for more than 120 seconds. Nov 20 19:27:23 thoregon kernel: [ 2160.595998] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 20 19:27:23 thoregon kernel: [ 2160.596004] kswapd0 D 0000000000000001 0 725 2 0x00000000 Nov 20 19:27:23 thoregon kernel: [ 2160.596016] ffff8803280d13f8 0000000000000046 ffff880329a0ab80 ffff8803280d1fd8 Nov 20 19:27:23 thoregon kernel: [ 2160.596030] ffff8803280d1fd8 ffff8803280d1fd8 ffff880046b7c880 ffff880329a0ab80 Nov 20 19:27:23 thoregon kernel: [ 2160.596041] ffff8803280d1408 ffff8803278dbbd0 ffff8803278db800 00000000ffffffff Nov 20 19:27:23 thoregon kernel: [ 2160.596052] Call Trace: Nov 20 19:27:23 thoregon kernel: [ 2160.596070] [<ffffffff816b1224>] schedule+0x24/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.596080] [<ffffffff814f9dad>] md_super_wait+0x4d/0x80 Nov 20 19:27:23 thoregon kernel: [ 2160.596091] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.596100] [<ffffffff81500753>] bitmap_unplug+0x173/0x180 Nov 20 19:27:23 thoregon kernel: [ 2160.596109] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:27:23 thoregon kernel: [ 2160.596118] [<ffffffff814e8eb8>] raid1_unplug+0x98/0x110 Nov 20 19:27:23 thoregon kernel: [ 2160.596130] [<ffffffff81278a6d>] blk_flush_plug_list+0xad/0x240 Nov 20 19:27:23 thoregon kernel: [ 2160.596138] [<ffffffff816b24ca>] ? _raw_spin_unlock_irqrestore+0x3a/0x70 Nov 20 19:27:23 thoregon kernel: [ 2160.596148] [<ffffffff816b15c3>] io_schedule_timeout+0x83/0xf0 Nov 20 19:27:23 thoregon kernel: [ 2160.596156] [<ffffffff810815cd>] ? trace_hardirqs_on+0xd/0x10 Nov 20 19:27:23 thoregon kernel: [ 2160.596166] [<ffffffff810b0e1d>] mempool_alloc+0x12d/0x160 Nov 20 19:27:23 thoregon kernel: [ 2160.596174] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.596184] [<ffffffff811263da>] bvec_alloc_bs+0xda/0x100 Nov 20 19:27:23 thoregon kernel: [ 2160.596192] [<ffffffff811264ea>] bio_alloc_bioset+0xea/0x110 Nov 20 19:27:23 thoregon kernel: [ 2160.596201] [<ffffffff81126656>] bio_clone_bioset+0x16/0x40 Nov 20 19:27:23 thoregon kernel: [ 2160.596208] [<ffffffff814f471a>] bio_clone_mddev+0x1a/0x30 Nov 20 19:27:23 thoregon kernel: [ 2160.596217] [<ffffffff814edbb1>] make_request+0x551/0xde0 Nov 20 19:27:23 thoregon kernel: [ 2160.596224] [<ffffffff814ed9e0>] ? make_request+0x380/0xde0 Nov 20 19:27:23 thoregon kernel: [ 2160.596233] [<ffffffff810b0b80>] ? mempool_alloc_slab+0x10/0x20 Nov 20 19:27:23 thoregon kernel: [ 2160.596242] [<ffffffff814f80bb>] md_make_request+0x21b/0x4d0 Nov 20 19:27:23 thoregon kernel: [ 2160.596249] [<ffffffff814f7ef0>] ? md_make_request+0x50/0x4d0 Nov 20 19:27:23 thoregon kernel: [ 2160.596258] [<ffffffff81276e52>] generic_make_request+0xc2/0x100 Nov 20 19:27:23 thoregon kernel: [ 2160.596267] [<ffffffff81276ef5>] submit_bio+0x65/0x110 Nov 20 19:27:23 thoregon kernel: [ 2160.596278] [<ffffffff811e07bf>] xfs_submit_ioend_bio.isra.21+0x2f/0x40 Nov 20 19:27:23 thoregon kernel: [ 2160.596286] [<ffffffff811e088e>] xfs_submit_ioend+0xbe/0x110 Nov 20 19:27:23 thoregon kernel: [ 2160.596295] [<ffffffff811e0c91>] xfs_vm_writepage+0x3b1/0x540 Nov 20 19:27:23 thoregon kernel: [ 2160.596305] [<ffffffff810bcd84>] shrink_page_list+0x564/0x890 Nov 20 19:27:23 thoregon kernel: [ 2160.596315] [<ffffffff810bd637>] shrink_inactive_list+0x1d7/0x310 Nov 20 19:27:23 thoregon kernel: [ 2160.596325] [<ffffffff810bdb9d>] shrink_lruvec+0x42d/0x530 Nov 20 19:27:23 thoregon kernel: [ 2160.596334] [<ffffffff810be323>] kswapd+0x683/0xa20 Nov 20 19:27:23 thoregon kernel: [ 2160.596344] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.596352] [<ffffffff810bdca0>] ? shrink_lruvec+0x530/0x530 Nov 20 19:27:23 thoregon kernel: [ 2160.596360] [<ffffffff8105c246>] kthread+0xd6/0xe0 Nov 20 19:27:23 thoregon kernel: [ 2160.596367] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.596376] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 Nov 20 19:27:23 thoregon kernel: [ 2160.596384] [<ffffffff816b31ac>] ret_from_fork+0x7c/0xb0 Nov 20 19:27:23 thoregon kernel: [ 2160.596392] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 Nov 20 19:27:23 thoregon kernel: [ 2160.596397] no locks held by kswapd0/725. Nov 20 19:27:23 thoregon kernel: [ 2160.596424] INFO: task xfsaild/md4:1742 blocked for more than 120 seconds. Nov 20 19:27:23 thoregon kernel: [ 2160.596428] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 20 19:27:23 thoregon kernel: [ 2160.596432] xfsaild/md4 D 0000000000000003 0 1742 2 0x00000000 Nov 20 19:27:23 thoregon kernel: [ 2160.596441] ffff88032438bb68 0000000000000046 ffff880329965700 ffff88032438bfd8 Nov 20 19:27:23 thoregon kernel: [ 2160.596452] ffff88032438bfd8 ffff88032438bfd8 ffff88032827e580 ffff880329965700 Nov 20 19:27:23 thoregon kernel: [ 2160.596462] ffff88032438bb78 ffff8803278dbbd0 ffff8803278db800 00000000ffffffff Nov 20 19:27:23 thoregon kernel: [ 2160.596473] Call Trace: Nov 20 19:27:23 thoregon kernel: [ 2160.596483] [<ffffffff816b1224>] schedule+0x24/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.596490] [<ffffffff814f9dad>] md_super_wait+0x4d/0x80 Nov 20 19:27:23 thoregon kernel: [ 2160.596498] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.596507] [<ffffffff81500753>] bitmap_unplug+0x173/0x180 Nov 20 19:27:23 thoregon kernel: [ 2160.596516] [<ffffffff81278c13>] ? blk_finish_plug+0x13/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.596524] [<ffffffff814e8eb8>] raid1_unplug+0x98/0x110 Nov 20 19:27:23 thoregon kernel: [ 2160.596533] [<ffffffff81278a6d>] blk_flush_plug_list+0xad/0x240 Nov 20 19:27:23 thoregon kernel: [ 2160.596542] [<ffffffff81278c13>] blk_finish_plug+0x13/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.596550] [<ffffffff811e296a>] __xfs_buf_delwri_submit+0x1ca/0x1e0 Nov 20 19:27:23 thoregon kernel: [ 2160.596558] [<ffffffff811e2ffb>] xfs_buf_delwri_submit_nowait+0x1b/0x20 Nov 20 19:27:23 thoregon kernel: [ 2160.596566] [<ffffffff81233066>] xfsaild+0x226/0x4c0 Nov 20 19:27:23 thoregon kernel: [ 2160.596575] [<ffffffff81065dfa>] ? finish_task_switch+0x3a/0x100 Nov 20 19:27:23 thoregon kernel: [ 2160.596583] [<ffffffff81232e40>] ? xfs_trans_ail_cursor_first+0xa0/0xa0 Nov 20 19:27:23 thoregon kernel: [ 2160.596591] [<ffffffff8105c246>] kthread+0xd6/0xe0 Nov 20 19:27:23 thoregon kernel: [ 2160.596598] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.596607] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 Nov 20 19:27:23 thoregon kernel: [ 2160.596614] [<ffffffff816b31ac>] ret_from_fork+0x7c/0xb0 Nov 20 19:27:23 thoregon kernel: [ 2160.596622] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 Nov 20 19:27:23 thoregon kernel: [ 2160.596627] no locks held by xfsaild/md4/1742. Nov 20 19:27:23 thoregon kernel: [ 2160.596709] INFO: task flush-9:4:30365 blocked for more than 120 seconds. Nov 20 19:27:23 thoregon kernel: [ 2160.596712] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 20 19:27:23 thoregon kernel: [ 2160.596716] flush-9:4 D 0000000000000001 0 30365 2 0x00000000 Nov 20 19:27:23 thoregon kernel: [ 2160.596725] ffff8801d8ce1978 0000000000000046 ffff880328238e80 ffff8801d8ce1fd8 Nov 20 19:27:23 thoregon kernel: [ 2160.596735] ffff8801d8ce1fd8 ffff8801d8ce1fd8 ffff8803032cc880 ffff880328238e80 Nov 20 19:27:23 thoregon kernel: [ 2160.596745] ffff8801d8ce1988 ffff8803278dbbd0 ffff8803278db800 00000000ffffffff Nov 20 19:27:23 thoregon kernel: [ 2160.596756] Call Trace: Nov 20 19:27:23 thoregon kernel: [ 2160.596765] [<ffffffff816b1224>] schedule+0x24/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.596773] [<ffffffff814f9dad>] md_super_wait+0x4d/0x80 Nov 20 19:27:23 thoregon kernel: [ 2160.596781] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.596788] [<ffffffff81500753>] bitmap_unplug+0x173/0x180 Nov 20 19:27:23 thoregon kernel: [ 2160.596796] [<ffffffff810b6acf>] ? write_cache_pages+0x12f/0x420 Nov 20 19:27:23 thoregon kernel: [ 2160.596804] [<ffffffff810b6700>] ? set_page_dirty_lock+0x60/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.596812] [<ffffffff814e8eb8>] raid1_unplug+0x98/0x110 Nov 20 19:27:23 thoregon kernel: [ 2160.596821] [<ffffffff81278a6d>] blk_flush_plug_list+0xad/0x240 Nov 20 19:27:23 thoregon kernel: [ 2160.596830] [<ffffffff81278c13>] blk_finish_plug+0x13/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.596838] [<ffffffff810b6e0a>] generic_writepages+0x4a/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.596847] [<ffffffff811df898>] xfs_vm_writepages+0x48/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.596856] [<ffffffff8111b144>] ? writeback_sb_inodes+0x174/0x410 Nov 20 19:27:23 thoregon kernel: [ 2160.596864] [<ffffffff810b84be>] do_writepages+0x1e/0x30 Nov 20 19:27:23 thoregon kernel: [ 2160.596873] [<ffffffff8111a1ee>] __writeback_single_inode+0x3e/0x120 Nov 20 19:27:23 thoregon kernel: [ 2160.596882] [<ffffffff8111b234>] writeback_sb_inodes+0x264/0x410 Nov 20 19:27:23 thoregon kernel: [ 2160.596891] [<ffffffff8111b476>] __writeback_inodes_wb+0x96/0xc0 Nov 20 19:27:23 thoregon kernel: [ 2160.596900] [<ffffffff8111b673>] wb_writeback+0x1d3/0x1e0 Nov 20 19:27:23 thoregon kernel: [ 2160.596908] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:27:23 thoregon kernel: [ 2160.596916] [<ffffffff8111bb1c>] ? wb_do_writeback+0x7c/0x140 Nov 20 19:27:23 thoregon kernel: [ 2160.596925] [<ffffffff8111bb40>] wb_do_writeback+0xa0/0x140 Nov 20 19:27:23 thoregon kernel: [ 2160.596935] [<ffffffff8111bc52>] bdi_writeback_thread+0x72/0x150 Nov 20 19:27:23 thoregon kernel: [ 2160.596944] [<ffffffff8111bbe0>] ? wb_do_writeback+0x140/0x140 Nov 20 19:27:23 thoregon kernel: [ 2160.596951] [<ffffffff8105c246>] kthread+0xd6/0xe0 Nov 20 19:27:23 thoregon kernel: [ 2160.596958] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.596967] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 Nov 20 19:27:23 thoregon kernel: [ 2160.596975] [<ffffffff816b31ac>] ret_from_fork+0x7c/0xb0 Nov 20 19:27:23 thoregon kernel: [ 2160.596983] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 Nov 20 19:27:23 thoregon kernel: [ 2160.596988] 1 lock held by flush-9:4/30365: Nov 20 19:27:23 thoregon kernel: [ 2160.596991] #0: (&type->s_umount_key#20){++++.+}, at: [<ffffffff810f59de>] grab_super_passive+0x3e/0x90 Nov 20 19:27:23 thoregon kernel: [ 2160.597019] INFO: task x86_64-pc-linux:26516 blocked for more than 120 seconds. Nov 20 19:27:23 thoregon kernel: [ 2160.597023] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 20 19:27:23 thoregon kernel: [ 2160.597027] x86_64-pc-linux D 0000000000000000 0 26516 26114 0x00000000 Nov 20 19:27:23 thoregon kernel: [ 2160.597035] ffff8800468dddf8 0000000000000046 ffff880329303a00 ffff8800468ddfd8 Nov 20 19:27:23 thoregon kernel: [ 2160.597045] ffff8800468ddfd8 ffff8800468ddfd8 ffff88032989c880 ffff880329303a00 Nov 20 19:27:23 thoregon kernel: [ 2160.597056] 0000000000000246 ffff8800468dc000 ffff8801b1a1bc68 0000000002405150 Nov 20 19:27:23 thoregon kernel: [ 2160.597066] Call Trace: Nov 20 19:27:23 thoregon kernel: [ 2160.597076] [<ffffffff816b1224>] schedule+0x24/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.597085] [<ffffffff816b1501>] schedule_preempt_disabled+0x21/0x30 Nov 20 19:27:23 thoregon kernel: [ 2160.597094] [<ffffffff816af3a5>] mutex_lock_nested+0x165/0x330 Nov 20 19:27:23 thoregon kernel: [ 2160.597103] [<ffffffff810ff5e3>] ? do_unlinkat+0xa3/0x1b0 Nov 20 19:27:23 thoregon kernel: [ 2160.597112] [<ffffffff810fefae>] ? filename_lookup.isra.57+0x2e/0x80 Nov 20 19:27:23 thoregon kernel: [ 2160.597121] [<ffffffff810ff5e3>] do_unlinkat+0xa3/0x1b0 Nov 20 19:27:23 thoregon kernel: [ 2160.597129] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:27:23 thoregon kernel: [ 2160.597137] [<ffffffff8129ca1e>] ? trace_hardirqs_on_thunk+0x3a/0x3f Nov 20 19:27:23 thoregon kernel: [ 2160.597146] [<ffffffff81101bb1>] sys_unlink+0x11/0x20 Nov 20 19:27:23 thoregon kernel: [ 2160.597154] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b Nov 20 19:27:23 thoregon kernel: [ 2160.597159] 2 locks held by x86_64-pc-linux/26516: Nov 20 19:27:23 thoregon kernel: [ 2160.597162] #0: (sb_writers#6){.+.+.+}, at: [<ffffffff811135ef>] mnt_want_write+0x1f/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.597178] #1: (&type->i_mutex_dir_key#3/1){+.+.+.}, at: [<ffffffff810ff5e3>] do_unlinkat+0xa3/0x1b0 Nov 20 19:27:23 thoregon kernel: [ 2160.597198] INFO: task x86_64-pc-linux:26523 blocked for more than 120 seconds. Nov 20 19:27:23 thoregon kernel: [ 2160.597201] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 20 19:27:23 thoregon kernel: [ 2160.597205] x86_64-pc-linux D 0000000000000000 0 26523 26114 0x00000000 Nov 20 19:27:23 thoregon kernel: [ 2160.597212] ffff8800c0def858 0000000000000046 ffff88032986c880 ffff8800c0deffd8 Nov 20 19:27:23 thoregon kernel: [ 2160.597223] ffff8800c0deffd8 ffff8800c0deffd8 ffff88032989e580 ffff88032986c880 Nov 20 19:27:23 thoregon kernel: [ 2160.597233] ffff8800c0def928 ffff8801c57e3a30 7fffffffffffffff ffff88032986c880 Nov 20 19:27:23 thoregon kernel: [ 2160.597243] Call Trace: Nov 20 19:27:23 thoregon kernel: [ 2160.597253] [<ffffffff816b1224>] schedule+0x24/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.597261] [<ffffffff816aeddd>] schedule_timeout+0x17d/0x1c0 Nov 20 19:27:23 thoregon kernel: [ 2160.597269] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.597277] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:27:23 thoregon kernel: [ 2160.597286] [<ffffffff816b057e>] __down_common+0x97/0xe8 Nov 20 19:27:23 thoregon kernel: [ 2160.597294] [<ffffffff816b062e>] __down+0x18/0x1a Nov 20 19:27:23 thoregon kernel: [ 2160.597301] [<ffffffff8106245b>] down+0x3b/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.597308] [<ffffffff811e21b5>] xfs_buf_lock+0x25/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.597315] [<ffffffff811e22e2>] _xfs_buf_find+0xf2/0x200 Nov 20 19:27:23 thoregon kernel: [ 2160.597323] [<ffffffff811e241f>] xfs_buf_get_map+0x2f/0x130 Nov 20 19:27:23 thoregon kernel: [ 2160.597331] [<ffffffff81210417>] ? xfs_dabuf_map.isra.23+0xb7/0x2f0 Nov 20 19:27:23 thoregon kernel: [ 2160.597339] [<ffffffff810b572e>] ? get_page_from_freelist+0x1de/0x460 Nov 20 19:27:23 thoregon kernel: [ 2160.597346] [<ffffffff811e2b23>] xfs_buf_read_map+0x13/0x90 Nov 20 19:27:23 thoregon kernel: [ 2160.597353] [<ffffffff81233e8d>] xfs_trans_read_buf_map+0x1ad/0x2d0 Nov 20 19:27:23 thoregon kernel: [ 2160.597361] [<ffffffff812116b6>] xfs_da_read_buf+0xb6/0x180 Nov 20 19:27:23 thoregon kernel: [ 2160.597371] [<ffffffff8121724c>] xfs_dir2_leaf_lookup_int+0x5c/0x2c0 Nov 20 19:27:23 thoregon kernel: [ 2160.597380] [<ffffffff81217583>] xfs_dir2_leaf_removename+0x33/0x330 Nov 20 19:27:23 thoregon kernel: [ 2160.597389] [<ffffffff8120777b>] ? xfs_bmap_last_offset+0x8b/0xc0 Nov 20 19:27:23 thoregon kernel: [ 2160.597398] [<ffffffff81213b94>] xfs_dir_removename+0x134/0x150 Nov 20 19:27:23 thoregon kernel: [ 2160.597406] [<ffffffff811f5ce6>] xfs_remove+0x1f6/0x370 Nov 20 19:27:23 thoregon kernel: [ 2160.597416] [<ffffffff816af4ac>] ? mutex_lock_nested+0x26c/0x330 Nov 20 19:27:23 thoregon kernel: [ 2160.597424] [<ffffffff810ff48f>] ? vfs_unlink+0x4f/0x100 Nov 20 19:27:23 thoregon kernel: [ 2160.597432] [<ffffffff811ec7f3>] xfs_vn_unlink+0x43/0x90 Nov 20 19:27:23 thoregon kernel: [ 2160.597441] [<ffffffff810ff4cd>] vfs_unlink+0x8d/0x100 Nov 20 19:27:23 thoregon kernel: [ 2160.597449] [<ffffffff810ff63a>] do_unlinkat+0xfa/0x1b0 Nov 20 19:27:23 thoregon kernel: [ 2160.597458] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:27:23 thoregon kernel: [ 2160.597466] [<ffffffff8129ca1e>] ? trace_hardirqs_on_thunk+0x3a/0x3f Nov 20 19:27:23 thoregon kernel: [ 2160.597475] [<ffffffff81101bb1>] sys_unlink+0x11/0x20 Nov 20 19:27:23 thoregon kernel: [ 2160.597482] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b Nov 20 19:27:23 thoregon kernel: [ 2160.597487] 6 locks held by x86_64-pc-linux/26523: Nov 20 19:27:23 thoregon kernel: [ 2160.597490] #0: (sb_writers#6){.+.+.+}, at: [<ffffffff811135ef>] mnt_want_write+0x1f/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.597505] #1: (&type->i_mutex_dir_key#3/1){+.+.+.}, at: [<ffffffff810ff5e3>] do_unlinkat+0xa3/0x1b0 Nov 20 19:27:23 thoregon kernel: [ 2160.597522] #2: (&sb->s_type->i_mutex_key#7){+.+.+.}, at: [<ffffffff810ff48f>] vfs_unlink+0x4f/0x100 Nov 20 19:27:23 thoregon kernel: [ 2160.597538] #3: (sb_internal){.+.+.?}, at: [<ffffffff8122b3e8>] xfs_trans_alloc+0x28/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.597552] #4: (&(&ip->i_lock)->mr_lock/4){+.+...}, at: [<ffffffff811e8414>] xfs_ilock+0x84/0xb0 Nov 20 19:27:23 thoregon kernel: [ 2160.597567] #5: (&(&ip->i_lock)->mr_lock/5){+.+...}, at: [<ffffffff811e8414>] xfs_ilock+0x84/0xb0 Nov 20 19:27:23 thoregon kernel: [ 2160.597584] INFO: task x86_64-pc-linux:26875 blocked for more than 120 seconds. Nov 20 19:27:23 thoregon kernel: [ 2160.597587] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 20 19:27:23 thoregon kernel: [ 2160.597591] x86_64-pc-linux D 0000000000000000 0 26875 26114 0x00000000 Nov 20 19:27:23 thoregon kernel: [ 2160.597598] ffff8800c0c5ba08 0000000000000046 ffff8802f0594880 ffff8800c0c5bfd8 Nov 20 19:27:23 thoregon kernel: [ 2160.597609] ffff8800c0c5bfd8 ffff8800c0c5bfd8 ffff88032989c880 ffff8802f0594880 Nov 20 19:27:23 thoregon kernel: [ 2160.597619] 0000000000000250 ffff880205a7aa30 7fffffffffffffff ffff8802f0594880 Nov 20 19:27:23 thoregon kernel: [ 2160.597630] Call Trace: Nov 20 19:27:23 thoregon kernel: [ 2160.597639] [<ffffffff816b1224>] schedule+0x24/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.597648] [<ffffffff816aeddd>] schedule_timeout+0x17d/0x1c0 Nov 20 19:27:23 thoregon kernel: [ 2160.597655] [<ffffffff811f70b4>] ? kmem_zone_zalloc+0x34/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.597662] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.597670] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:27:23 thoregon kernel: [ 2160.597679] [<ffffffff816b057e>] __down_common+0x97/0xe8 Nov 20 19:27:23 thoregon kernel: [ 2160.597688] [<ffffffff816b062e>] __down+0x18/0x1a Nov 20 19:27:23 thoregon kernel: [ 2160.597695] [<ffffffff8106245b>] down+0x3b/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.597702] [<ffffffff811e21b5>] xfs_buf_lock+0x25/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.597708] [<ffffffff811e22e2>] _xfs_buf_find+0xf2/0x200 Nov 20 19:27:23 thoregon kernel: [ 2160.597716] [<ffffffff811e241f>] xfs_buf_get_map+0x2f/0x130 Nov 20 19:27:23 thoregon kernel: [ 2160.597723] [<ffffffff811e2b23>] xfs_buf_read_map+0x13/0x90 Nov 20 19:27:23 thoregon kernel: [ 2160.597730] [<ffffffff81233e8d>] xfs_trans_read_buf_map+0x1ad/0x2d0 Nov 20 19:27:23 thoregon kernel: [ 2160.597739] [<ffffffff8121ce6f>] xfs_read_agi+0x7f/0x110 Nov 20 19:27:23 thoregon kernel: [ 2160.597748] [<ffffffff8121fb55>] xfs_iunlink+0x45/0x150 Nov 20 19:27:23 thoregon kernel: [ 2160.597757] [<ffffffff81041fc1>] ? current_fs_time+0x11/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.597764] [<ffffffff81234352>] ? xfs_trans_ichgtime+0x22/0xa0 Nov 20 19:27:23 thoregon kernel: [ 2160.597772] [<ffffffff811f3ea0>] xfs_droplink+0x50/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.597778] [<ffffffff811f5d77>] xfs_remove+0x287/0x370 Nov 20 19:27:23 thoregon kernel: [ 2160.597788] [<ffffffff816af4ac>] ? mutex_lock_nested+0x26c/0x330 Nov 20 19:27:23 thoregon kernel: [ 2160.597796] [<ffffffff810ff48f>] ? vfs_unlink+0x4f/0x100 Nov 20 19:27:23 thoregon kernel: [ 2160.597804] [<ffffffff811ec7f3>] xfs_vn_unlink+0x43/0x90 Nov 20 19:27:23 thoregon kernel: [ 2160.597813] [<ffffffff810ff4cd>] vfs_unlink+0x8d/0x100 Nov 20 19:27:23 thoregon kernel: [ 2160.597821] [<ffffffff810ff63a>] do_unlinkat+0xfa/0x1b0 Nov 20 19:27:23 thoregon kernel: [ 2160.597830] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:27:23 thoregon kernel: [ 2160.597837] [<ffffffff8129ca1e>] ? trace_hardirqs_on_thunk+0x3a/0x3f Nov 20 19:27:23 thoregon kernel: [ 2160.597846] [<ffffffff81101bb1>] sys_unlink+0x11/0x20 Nov 20 19:27:23 thoregon kernel: [ 2160.597854] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b Nov 20 19:27:23 thoregon kernel: [ 2160.597859] 6 locks held by x86_64-pc-linux/26875: Nov 20 19:27:23 thoregon kernel: [ 2160.597862] #0: (sb_writers#6){.+.+.+}, at: [<ffffffff811135ef>] mnt_want_write+0x1f/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.597877] #1: (&type->i_mutex_dir_key#3/1){+.+.+.}, at: [<ffffffff810ff5e3>] do_unlinkat+0xa3/0x1b0 Nov 20 19:27:23 thoregon kernel: [ 2160.597894] #2: (&sb->s_type->i_mutex_key#7){+.+.+.}, at: [<ffffffff810ff48f>] vfs_unlink+0x4f/0x100 Nov 20 19:27:23 thoregon kernel: [ 2160.597909] #3: (sb_internal){.+.+.?}, at: [<ffffffff8122b3e8>] xfs_trans_alloc+0x28/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.597923] #4: (&(&ip->i_lock)->mr_lock/4){+.+...}, at: [<ffffffff811e8414>] xfs_ilock+0x84/0xb0 Nov 20 19:27:23 thoregon kernel: [ 2160.597937] #5: (&(&ip->i_lock)->mr_lock/5){+.+...}, at: [<ffffffff811e8414>] xfs_ilock+0x84/0xb0 Nov 20 19:27:23 thoregon kernel: [ 2160.597955] INFO: task x86_64-pc-linux:26897 blocked for more than 120 seconds. Nov 20 19:27:23 thoregon kernel: [ 2160.597958] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 20 19:27:23 thoregon kernel: [ 2160.597961] x86_64-pc-linux D 0000000000000001 0 26897 26114 0x00000000 Nov 20 19:27:23 thoregon kernel: [ 2160.597969] ffff880101959858 0000000000000046 ffff88032934ab80 ffff880101959fd8 Nov 20 19:27:23 thoregon kernel: [ 2160.597980] ffff880101959fd8 ffff880101959fd8 ffff880162a3e580 ffff88032934ab80 Nov 20 19:27:23 thoregon kernel: [ 2160.597990] ffff880101959928 ffff8801ca545030 7fffffffffffffff ffff88032934ab80 Nov 20 19:27:23 thoregon kernel: [ 2160.598000] Call Trace: Nov 20 19:27:23 thoregon kernel: [ 2160.598010] [<ffffffff816b1224>] schedule+0x24/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.598018] [<ffffffff816aeddd>] schedule_timeout+0x17d/0x1c0 Nov 20 19:27:23 thoregon kernel: [ 2160.598026] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.598034] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:27:23 thoregon kernel: [ 2160.598043] [<ffffffff816b057e>] __down_common+0x97/0xe8 Nov 20 19:27:23 thoregon kernel: [ 2160.598052] [<ffffffff816b062e>] __down+0x18/0x1a Nov 20 19:27:23 thoregon kernel: [ 2160.598058] [<ffffffff8106245b>] down+0x3b/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.598065] [<ffffffff811e21b5>] xfs_buf_lock+0x25/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.598072] [<ffffffff811e22e2>] _xfs_buf_find+0xf2/0x200 Nov 20 19:27:23 thoregon kernel: [ 2160.598079] [<ffffffff811e241f>] xfs_buf_get_map+0x2f/0x130 Nov 20 19:27:23 thoregon kernel: [ 2160.598087] [<ffffffff81210417>] ? xfs_dabuf_map.isra.23+0xb7/0x2f0 Nov 20 19:27:23 thoregon kernel: [ 2160.598094] [<ffffffff810b572e>] ? get_page_from_freelist+0x1de/0x460 Nov 20 19:27:23 thoregon kernel: [ 2160.598102] [<ffffffff811e2b23>] xfs_buf_read_map+0x13/0x90 Nov 20 19:27:23 thoregon kernel: [ 2160.598109] [<ffffffff81233e8d>] xfs_trans_read_buf_map+0x1ad/0x2d0 Nov 20 19:27:23 thoregon kernel: [ 2160.598117] [<ffffffff812116b6>] xfs_da_read_buf+0xb6/0x180 Nov 20 19:27:23 thoregon kernel: [ 2160.598126] [<ffffffff8121724c>] xfs_dir2_leaf_lookup_int+0x5c/0x2c0 Nov 20 19:27:23 thoregon kernel: [ 2160.598135] [<ffffffff81217583>] xfs_dir2_leaf_removename+0x33/0x330 Nov 20 19:27:23 thoregon kernel: [ 2160.598144] [<ffffffff8120777b>] ? xfs_bmap_last_offset+0x8b/0xc0 Nov 20 19:27:23 thoregon kernel: [ 2160.598152] [<ffffffff81213b94>] xfs_dir_removename+0x134/0x150 Nov 20 19:27:23 thoregon kernel: [ 2160.598160] [<ffffffff811f5ce6>] xfs_remove+0x1f6/0x370 Nov 20 19:27:23 thoregon kernel: [ 2160.598170] [<ffffffff816af4ac>] ? mutex_lock_nested+0x26c/0x330 Nov 20 19:27:23 thoregon kernel: [ 2160.598178] [<ffffffff810ff48f>] ? vfs_unlink+0x4f/0x100 Nov 20 19:27:23 thoregon kernel: [ 2160.598186] [<ffffffff811ec7f3>] xfs_vn_unlink+0x43/0x90 Nov 20 19:27:23 thoregon kernel: [ 2160.598195] [<ffffffff810ff4cd>] vfs_unlink+0x8d/0x100 Nov 20 19:27:23 thoregon kernel: [ 2160.598203] [<ffffffff810ff63a>] do_unlinkat+0xfa/0x1b0 Nov 20 19:27:23 thoregon kernel: [ 2160.598211] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:27:23 thoregon kernel: [ 2160.598219] [<ffffffff8129ca1e>] ? trace_hardirqs_on_thunk+0x3a/0x3f Nov 20 19:27:23 thoregon kernel: [ 2160.598228] [<ffffffff81101bb1>] sys_unlink+0x11/0x20 Nov 20 19:27:23 thoregon kernel: [ 2160.598236] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b Nov 20 19:27:23 thoregon kernel: [ 2160.598241] 6 locks held by x86_64-pc-linux/26897: Nov 20 19:27:23 thoregon kernel: [ 2160.598244] #0: (sb_writers#6){.+.+.+}, at: [<ffffffff811135ef>] mnt_want_write+0x1f/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.598258] #1: (&type->i_mutex_dir_key#3/1){+.+.+.}, at: [<ffffffff810ff5e3>] do_unlinkat+0xa3/0x1b0 Nov 20 19:27:23 thoregon kernel: [ 2160.598275] #2: (&sb->s_type->i_mutex_key#7){+.+.+.}, at: [<ffffffff810ff48f>] vfs_unlink+0x4f/0x100 Nov 20 19:27:23 thoregon kernel: [ 2160.598290] #3: (sb_internal){.+.+.?}, at: [<ffffffff8122b3e8>] xfs_trans_alloc+0x28/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.598304] #4: (&(&ip->i_lock)->mr_lock/4){+.+...}, at: [<ffffffff811e8414>] xfs_ilock+0x84/0xb0 Nov 20 19:27:23 thoregon kernel: [ 2160.598318] #5: (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffff811e854c>] xfs_ilock_nowait+0xbc/0x100 Nov 20 19:27:23 thoregon kernel: [ 2160.598334] INFO: task x86_64-pc-linux:26904 blocked for more than 120 seconds. Nov 20 19:27:23 thoregon kernel: [ 2160.598337] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 20 19:27:23 thoregon kernel: [ 2160.598341] x86_64-pc-linux D 0000000000000001 0 26904 26114 0x00000000 Nov 20 19:27:23 thoregon kernel: [ 2160.598348] ffff880066bff858 0000000000000046 ffff880329301d00 ffff880066bfffd8 Nov 20 19:27:23 thoregon kernel: [ 2160.598359] ffff880066bfffd8 ffff880066bfffd8 ffff88013a7ce580 ffff880329301d00 Nov 20 19:27:23 thoregon kernel: [ 2160.598369] ffff880066bff928 ffff8801c572c830 7fffffffffffffff ffff880329301d00 Nov 20 19:27:23 thoregon kernel: [ 2160.598380] Call Trace: Nov 20 19:27:23 thoregon kernel: [ 2160.598389] [<ffffffff816b1224>] schedule+0x24/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.598398] [<ffffffff816aeddd>] schedule_timeout+0x17d/0x1c0 Nov 20 19:27:23 thoregon kernel: [ 2160.598405] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.598413] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:27:23 thoregon kernel: [ 2160.598422] [<ffffffff816b057e>] __down_common+0x97/0xe8 Nov 20 19:27:23 thoregon kernel: [ 2160.598431] [<ffffffff816b062e>] __down+0x18/0x1a Nov 20 19:27:23 thoregon kernel: [ 2160.598437] [<ffffffff8106245b>] down+0x3b/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.598444] [<ffffffff811e21b5>] xfs_buf_lock+0x25/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.598451] [<ffffffff811e22e2>] _xfs_buf_find+0xf2/0x200 Nov 20 19:27:23 thoregon kernel: [ 2160.598458] [<ffffffff811e241f>] xfs_buf_get_map+0x2f/0x130 Nov 20 19:27:23 thoregon kernel: [ 2160.598466] [<ffffffff81210417>] ? xfs_dabuf_map.isra.23+0xb7/0x2f0 Nov 20 19:27:23 thoregon kernel: [ 2160.598474] [<ffffffff810b572e>] ? get_page_from_freelist+0x1de/0x460 Nov 20 19:27:23 thoregon kernel: [ 2160.598481] [<ffffffff811e2b23>] xfs_buf_read_map+0x13/0x90 Nov 20 19:27:23 thoregon kernel: [ 2160.598488] [<ffffffff81233e8d>] xfs_trans_read_buf_map+0x1ad/0x2d0 Nov 20 19:27:23 thoregon kernel: [ 2160.598496] [<ffffffff812116b6>] xfs_da_read_buf+0xb6/0x180 Nov 20 19:27:23 thoregon kernel: [ 2160.598505] [<ffffffff8121724c>] xfs_dir2_leaf_lookup_int+0x5c/0x2c0 Nov 20 19:27:23 thoregon kernel: [ 2160.598515] [<ffffffff81217583>] xfs_dir2_leaf_removename+0x33/0x330 Nov 20 19:27:23 thoregon kernel: [ 2160.598523] [<ffffffff8120777b>] ? xfs_bmap_last_offset+0x8b/0xc0 Nov 20 19:27:23 thoregon kernel: [ 2160.598532] [<ffffffff81213b94>] xfs_dir_removename+0x134/0x150 Nov 20 19:27:23 thoregon kernel: [ 2160.598540] [<ffffffff811f5ce6>] xfs_remove+0x1f6/0x370 Nov 20 19:27:23 thoregon kernel: [ 2160.598549] [<ffffffff816af4ac>] ? mutex_lock_nested+0x26c/0x330 Nov 20 19:27:23 thoregon kernel: [ 2160.598557] [<ffffffff810ff48f>] ? vfs_unlink+0x4f/0x100 Nov 20 19:27:23 thoregon kernel: [ 2160.598565] [<ffffffff811ec7f3>] xfs_vn_unlink+0x43/0x90 Nov 20 19:27:23 thoregon kernel: [ 2160.598574] [<ffffffff810ff4cd>] vfs_unlink+0x8d/0x100 Nov 20 19:27:23 thoregon kernel: [ 2160.598582] [<ffffffff810ff63a>] do_unlinkat+0xfa/0x1b0 Nov 20 19:27:23 thoregon kernel: [ 2160.598591] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:27:23 thoregon kernel: [ 2160.598598] [<ffffffff8129ca1e>] ? trace_hardirqs_on_thunk+0x3a/0x3f Nov 20 19:27:23 thoregon kernel: [ 2160.598607] [<ffffffff81101bb1>] sys_unlink+0x11/0x20 Nov 20 19:27:23 thoregon kernel: [ 2160.598615] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b Nov 20 19:27:23 thoregon kernel: [ 2160.598620] 6 locks held by x86_64-pc-linux/26904: Nov 20 19:27:23 thoregon kernel: [ 2160.598623] #0: (sb_writers#6){.+.+.+}, at: [<ffffffff811135ef>] mnt_want_write+0x1f/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.598637] #1: (&type->i_mutex_dir_key#3/1){+.+.+.}, at: [<ffffffff810ff5e3>] do_unlinkat+0xa3/0x1b0 Nov 20 19:27:23 thoregon kernel: [ 2160.598654] #2: (&sb->s_type->i_mutex_key#7){+.+.+.}, at: [<ffffffff810ff48f>] vfs_unlink+0x4f/0x100 Nov 20 19:27:23 thoregon kernel: [ 2160.598669] #3: (sb_internal){.+.+.?}, at: [<ffffffff8122b3e8>] xfs_trans_alloc+0x28/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.598683] #4: (&(&ip->i_lock)->mr_lock/4){+.+...}, at: [<ffffffff811e8414>] xfs_ilock+0x84/0xb0 Nov 20 19:27:23 thoregon kernel: [ 2160.598697] #5: (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffff811e854c>] xfs_ilock_nowait+0xbc/0x100 Nov 20 19:27:23 thoregon kernel: [ 2160.598713] INFO: task x86_64-pc-linux:26907 blocked for more than 120 seconds. Nov 20 19:27:23 thoregon kernel: [ 2160.598716] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 20 19:27:23 thoregon kernel: [ 2160.598720] x86_64-pc-linux D 0000000000000000 0 26907 26114 0x00000000 Nov 20 19:27:23 thoregon kernel: [ 2160.598727] ffff8800c0eed858 0000000000000046 ffff8802a7be6580 ffff8800c0eedfd8 Nov 20 19:27:23 thoregon kernel: [ 2160.598738] ffff8800c0eedfd8 ffff8800c0eedfd8 ffff88032989e580 ffff8802a7be6580 Nov 20 19:27:23 thoregon kernel: [ 2160.598748] ffff8800c0eed928 ffff8801c55c7630 7fffffffffffffff ffff8802a7be6580 Nov 20 19:27:23 thoregon kernel: [ 2160.598758] Call Trace: Nov 20 19:27:23 thoregon kernel: [ 2160.598768] [<ffffffff816b1224>] schedule+0x24/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.598776] [<ffffffff816aeddd>] schedule_timeout+0x17d/0x1c0 Nov 20 19:27:23 thoregon kernel: [ 2160.598784] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.598792] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:27:23 thoregon kernel: [ 2160.598801] [<ffffffff816b057e>] __down_common+0x97/0xe8 Nov 20 19:27:23 thoregon kernel: [ 2160.598809] [<ffffffff816b062e>] __down+0x18/0x1a Nov 20 19:27:23 thoregon kernel: [ 2160.598816] [<ffffffff8106245b>] down+0x3b/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.598823] [<ffffffff811e21b5>] xfs_buf_lock+0x25/0x60 Nov 20 19:27:23 thoregon kernel: [ 2160.598830] [<ffffffff811e22e2>] _xfs_buf_find+0xf2/0x200 Nov 20 19:27:23 thoregon kernel: [ 2160.598837] [<ffffffff811e241f>] xfs_buf_get_map+0x2f/0x130 Nov 20 19:27:23 thoregon kernel: [ 2160.598844] [<ffffffff81210417>] ? xfs_dabuf_map.isra.23+0xb7/0x2f0 Nov 20 19:27:23 thoregon kernel: [ 2160.598852] [<ffffffff810b572e>] ? get_page_from_freelist+0x1de/0x460 Nov 20 19:27:23 thoregon kernel: [ 2160.598859] [<ffffffff811e2b23>] xfs_buf_read_map+0x13/0x90 Nov 20 19:27:23 thoregon kernel: [ 2160.598867] [<ffffffff81233e8d>] xfs_trans_read_buf_map+0x1ad/0x2d0 Nov 20 19:27:23 thoregon kernel: [ 2160.598874] [<ffffffff812116b6>] xfs_da_read_buf+0xb6/0x180 Nov 20 19:27:23 thoregon kernel: [ 2160.598884] [<ffffffff8121724c>] xfs_dir2_leaf_lookup_int+0x5c/0x2c0 Nov 20 19:27:23 thoregon kernel: [ 2160.598893] [<ffffffff81217583>] xfs_dir2_leaf_removename+0x33/0x330 Nov 20 19:27:23 thoregon kernel: [ 2160.598901] [<ffffffff8120777b>] ? xfs_bmap_last_offset+0x8b/0xc0 Nov 20 19:27:23 thoregon kernel: [ 2160.598910] [<ffffffff81213b94>] xfs_dir_removename+0x134/0x150 Nov 20 19:27:23 thoregon kernel: [ 2160.598918] [<ffffffff811f5ce6>] xfs_remove+0x1f6/0x370 Nov 20 19:27:23 thoregon kernel: [ 2160.598927] [<ffffffff816af4ac>] ? mutex_lock_nested+0x26c/0x330 Nov 20 19:27:23 thoregon kernel: [ 2160.598935] [<ffffffff810ff48f>] ? vfs_unlink+0x4f/0x100 Nov 20 19:27:23 thoregon kernel: [ 2160.598944] [<ffffffff811ec7f3>] xfs_vn_unlink+0x43/0x90 Nov 20 19:27:23 thoregon kernel: [ 2160.598952] [<ffffffff810ff4cd>] vfs_unlink+0x8d/0x100 Nov 20 19:27:23 thoregon kernel: [ 2160.598961] [<ffffffff810ff63a>] do_unlinkat+0xfa/0x1b0 Nov 20 19:27:23 thoregon kernel: [ 2160.598969] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:27:23 thoregon kernel: [ 2160.598977] [<ffffffff8129ca1e>] ? trace_hardirqs_on_thunk+0x3a/0x3f Nov 20 19:27:23 thoregon kernel: [ 2160.598986] [<ffffffff81101bb1>] sys_unlink+0x11/0x20 Nov 20 19:27:23 thoregon kernel: [ 2160.598993] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b Nov 20 19:27:23 thoregon kernel: [ 2160.598998] 6 locks held by x86_64-pc-linux/26907: Nov 20 19:27:23 thoregon kernel: [ 2160.599001] #0: (sb_writers#6){.+.+.+}, at: [<ffffffff811135ef>] mnt_want_write+0x1f/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.599016] #1: (&type->i_mutex_dir_key#3/1){+.+.+.}, at: [<ffffffff810ff5e3>] do_unlinkat+0xa3/0x1b0 Nov 20 19:27:23 thoregon kernel: [ 2160.599033] #2: (&sb->s_type->i_mutex_key#7){+.+.+.}, at: [<ffffffff810ff48f>] vfs_unlink+0x4f/0x100 Nov 20 19:27:23 thoregon kernel: [ 2160.599048] #3: (sb_internal){.+.+.?}, at: [<ffffffff8122b3e8>] xfs_trans_alloc+0x28/0x50 Nov 20 19:27:23 thoregon kernel: [ 2160.599062] #4: (&(&ip->i_lock)->mr_lock/4){+.+...}, at: [<ffffffff811e8414>] xfs_ilock+0x84/0xb0 Nov 20 19:27:23 thoregon kernel: [ 2160.599076] #5: (&(&ip->i_lock)->mr_lock){++++--}, at: [<ffffffff811e854c>] xfs_ilock_nowait+0xbc/0x100 Nov 20 19:27:41 thoregon login[3002]: pam_unix(login:session): session closed for user root Nov 20 19:27:43 thoregon acpid: client connected from 2991[0:0] Nov 20 19:27:43 thoregon acpid: 1 client rule loaded Nov 20 19:29:23 thoregon kernel: [ 2280.601023] INFO: task kswapd0:725 blocked for more than 120 seconds. Nov 20 19:29:23 thoregon kernel: [ 2280.601030] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Nov 20 19:29:23 thoregon kernel: [ 2280.601036] kswapd0 D 0000000000000001 0 725 2 0x00000000 Nov 20 19:29:23 thoregon kernel: [ 2280.601048] ffff8803280d13f8 0000000000000046 ffff880329a0ab80 ffff8803280d1fd8 Nov 20 19:29:23 thoregon kernel: [ 2280.601063] ffff8803280d1fd8 ffff8803280d1fd8 ffff880046b7c880 ffff880329a0ab80 Nov 20 19:29:23 thoregon kernel: [ 2280.601073] ffff8803280d1408 ffff8803278dbbd0 ffff8803278db800 00000000ffffffff Nov 20 19:29:23 thoregon kernel: [ 2280.601084] Call Trace: Nov 20 19:29:23 thoregon kernel: [ 2280.601101] [<ffffffff816b1224>] schedule+0x24/0x60 Nov 20 19:29:23 thoregon kernel: [ 2280.601111] [<ffffffff814f9dad>] md_super_wait+0x4d/0x80 Nov 20 19:29:23 thoregon kernel: [ 2280.601121] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60 Nov 20 19:29:23 thoregon kernel: [ 2280.601129] [<ffffffff81500753>] bitmap_unplug+0x173/0x180 Nov 20 19:29:23 thoregon kernel: [ 2280.601138] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:29:23 thoregon kernel: [ 2280.601147] [<ffffffff814e8eb8>] raid1_unplug+0x98/0x110 Nov 20 19:29:23 thoregon kernel: [ 2280.601157] [<ffffffff81278a6d>] blk_flush_plug_list+0xad/0x240 Nov 20 19:29:23 thoregon kernel: [ 2280.601165] [<ffffffff816b24ca>] ? _raw_spin_unlock_irqrestore+0x3a/0x70 Nov 20 19:29:23 thoregon kernel: [ 2280.601175] [<ffffffff816b15c3>] io_schedule_timeout+0x83/0xf0 Nov 20 19:29:23 thoregon kernel: [ 2280.601182] [<ffffffff810815cd>] ? trace_hardirqs_on+0xd/0x10 Nov 20 19:29:23 thoregon kernel: [ 2280.601192] [<ffffffff810b0e1d>] mempool_alloc+0x12d/0x160 Nov 20 19:29:23 thoregon kernel: [ 2280.601200] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60 Nov 20 19:29:23 thoregon kernel: [ 2280.601209] [<ffffffff811263da>] bvec_alloc_bs+0xda/0x100 Nov 20 19:29:23 thoregon kernel: [ 2280.601217] [<ffffffff811264ea>] bio_alloc_bioset+0xea/0x110 Nov 20 19:29:23 thoregon kernel: [ 2280.601226] [<ffffffff81126656>] bio_clone_bioset+0x16/0x40 Nov 20 19:29:23 thoregon kernel: [ 2280.601233] [<ffffffff814f471a>] bio_clone_mddev+0x1a/0x30 Nov 20 19:29:23 thoregon kernel: [ 2280.601241] [<ffffffff814edbb1>] make_request+0x551/0xde0 Nov 20 19:29:23 thoregon kernel: [ 2280.601249] [<ffffffff814ed9e0>] ? make_request+0x380/0xde0 Nov 20 19:29:23 thoregon kernel: [ 2280.601257] [<ffffffff810b0b80>] ? mempool_alloc_slab+0x10/0x20 Nov 20 19:29:23 thoregon kernel: [ 2280.601266] [<ffffffff814f80bb>] md_make_request+0x21b/0x4d0 Nov 20 19:29:23 thoregon kernel: [ 2280.601273] [<ffffffff814f7ef0>] ? md_make_request+0x50/0x4d0 Nov 20 19:29:23 thoregon kernel: [ 2280.601282] [<ffffffff81276e52>] generic_make_request+0xc2/0x100 Nov 20 19:29:23 thoregon kernel: [ 2280.601291] [<ffffffff81276ef5>] submit_bio+0x65/0x110 Nov 20 19:29:23 thoregon kernel: [ 2280.601301] [<ffffffff811e07bf>] xfs_submit_ioend_bio.isra.21+0x2f/0x40 Nov 20 19:29:23 thoregon kernel: [ 2280.601309] [<ffffffff811e088e>] xfs_submit_ioend+0xbe/0x110 Nov 20 19:29:23 thoregon kernel: [ 2280.601318] [<ffffffff811e0c91>] xfs_vm_writepage+0x3b1/0x540 Nov 20 19:29:23 thoregon kernel: [ 2280.601328] [<ffffffff810bcd84>] shrink_page_list+0x564/0x890 Nov 20 19:29:23 thoregon kernel: [ 2280.601338] [<ffffffff810bd637>] shrink_inactive_list+0x1d7/0x310 Nov 20 19:29:23 thoregon kernel: [ 2280.601347] [<ffffffff810bdb9d>] shrink_lruvec+0x42d/0x530 Nov 20 19:29:23 thoregon kernel: [ 2280.601356] [<ffffffff810be323>] kswapd+0x683/0xa20 Nov 20 19:29:23 thoregon kernel: [ 2280.601365] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60 Nov 20 19:29:23 thoregon kernel: [ 2280.601374] [<ffffffff810bdca0>] ? shrink_lruvec+0x530/0x530 Nov 20 19:29:23 thoregon kernel: [ 2280.601381] [<ffffffff8105c246>] kthread+0xd6/0xe0 Nov 20 19:29:23 thoregon kernel: [ 2280.601388] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50 Nov 20 19:29:23 thoregon kernel: [ 2280.601397] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 Nov 20 19:29:23 thoregon kernel: [ 2280.601404] [<ffffffff816b31ac>] ret_from_fork+0x7c/0xb0 Nov 20 19:29:23 thoregon kernel: [ 2280.601412] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 Nov 20 19:29:23 thoregon kernel: [ 2280.601417] no locks held by kswapd0/725. Nov 20 19:30:01 thoregon cron[27484]: (root) CMD (test -x /usr/sbin/run-crons && /usr/sbin/run-crons) Nov 20 19:35:16 thoregon kernel: [ 2633.536440] SysRq : Show Blocked State Nov 20 19:35:16 thoregon kernel: [ 2633.536453] task PC stack pid father Nov 20 19:35:16 thoregon kernel: [ 2633.536485] kswapd0 D 0000000000000001 0 725 2 0x00000000 Nov 20 19:35:16 thoregon kernel: [ 2633.536496] ffff8803280d13f8 0000000000000046 ffff880329a0ab80 ffff8803280d1fd8 Nov 20 19:35:16 thoregon kernel: [ 2633.536507] ffff8803280d1fd8 ffff8803280d1fd8 ffff880046b7c880 ffff880329a0ab80 Nov 20 19:35:16 thoregon kernel: [ 2633.536517] ffff8803280d1408 ffff8803278dbbd0 ffff8803278db800 00000000ffffffff Nov 20 19:35:16 thoregon kernel: [ 2633.536526] Call Trace: Nov 20 19:35:16 thoregon kernel: [ 2633.536542] [<ffffffff816b1224>] schedule+0x24/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.536553] [<ffffffff814f9dad>] md_super_wait+0x4d/0x80 Nov 20 19:35:16 thoregon kernel: [ 2633.536563] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.536571] [<ffffffff81500753>] bitmap_unplug+0x173/0x180 Nov 20 19:35:16 thoregon kernel: [ 2633.536581] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:35:16 thoregon kernel: [ 2633.536589] [<ffffffff814e8eb8>] raid1_unplug+0x98/0x110 Nov 20 19:35:16 thoregon kernel: [ 2633.536599] [<ffffffff81278a6d>] blk_flush_plug_list+0xad/0x240 Nov 20 19:35:16 thoregon kernel: [ 2633.536607] [<ffffffff816b24ca>] ? _raw_spin_unlock_irqrestore+0x3a/0x70 Nov 20 19:35:16 thoregon kernel: [ 2633.536616] [<ffffffff816b15c3>] io_schedule_timeout+0x83/0xf0 Nov 20 19:35:16 thoregon kernel: [ 2633.536623] [<ffffffff810815cd>] ? trace_hardirqs_on+0xd/0x10 Nov 20 19:35:16 thoregon kernel: [ 2633.536633] [<ffffffff810b0e1d>] mempool_alloc+0x12d/0x160 Nov 20 19:35:16 thoregon kernel: [ 2633.536641] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.536650] [<ffffffff811263da>] bvec_alloc_bs+0xda/0x100 Nov 20 19:35:16 thoregon kernel: [ 2633.536658] [<ffffffff811264ea>] bio_alloc_bioset+0xea/0x110 Nov 20 19:35:16 thoregon kernel: [ 2633.536666] [<ffffffff81126656>] bio_clone_bioset+0x16/0x40 Nov 20 19:35:16 thoregon kernel: [ 2633.536673] [<ffffffff814f471a>] bio_clone_mddev+0x1a/0x30 Nov 20 19:35:16 thoregon kernel: [ 2633.536681] [<ffffffff814edbb1>] make_request+0x551/0xde0 Nov 20 19:35:16 thoregon kernel: [ 2633.536689] [<ffffffff814ed9e0>] ? make_request+0x380/0xde0 Nov 20 19:35:16 thoregon kernel: [ 2633.536697] [<ffffffff810b0b80>] ? mempool_alloc_slab+0x10/0x20 Nov 20 19:35:16 thoregon kernel: [ 2633.536706] [<ffffffff814f80bb>] md_make_request+0x21b/0x4d0 Nov 20 19:35:16 thoregon kernel: [ 2633.536712] [<ffffffff814f7ef0>] ? md_make_request+0x50/0x4d0 Nov 20 19:35:16 thoregon kernel: [ 2633.536722] [<ffffffff81276e52>] generic_make_request+0xc2/0x100 Nov 20 19:35:16 thoregon kernel: [ 2633.536730] [<ffffffff81276ef5>] submit_bio+0x65/0x110 Nov 20 19:35:16 thoregon kernel: [ 2633.536740] [<ffffffff811e07bf>] xfs_submit_ioend_bio.isra.21+0x2f/0x40 Nov 20 19:35:16 thoregon kernel: [ 2633.536748] [<ffffffff811e088e>] xfs_submit_ioend+0xbe/0x110 Nov 20 19:35:16 thoregon kernel: [ 2633.536757] [<ffffffff811e0c91>] xfs_vm_writepage+0x3b1/0x540 Nov 20 19:35:16 thoregon kernel: [ 2633.536767] [<ffffffff810bcd84>] shrink_page_list+0x564/0x890 Nov 20 19:35:16 thoregon kernel: [ 2633.536777] [<ffffffff810bd637>] shrink_inactive_list+0x1d7/0x310 Nov 20 19:35:16 thoregon kernel: [ 2633.536785] [<ffffffff810bdb9d>] shrink_lruvec+0x42d/0x530 Nov 20 19:35:16 thoregon kernel: [ 2633.536795] [<ffffffff810be323>] kswapd+0x683/0xa20 Nov 20 19:35:16 thoregon kernel: [ 2633.536804] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.536812] [<ffffffff810bdca0>] ? shrink_lruvec+0x530/0x530 Nov 20 19:35:16 thoregon kernel: [ 2633.536819] [<ffffffff8105c246>] kthread+0xd6/0xe0 Nov 20 19:35:16 thoregon kernel: [ 2633.536826] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50 Nov 20 19:35:16 thoregon kernel: [ 2633.536835] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 Nov 20 19:35:16 thoregon kernel: [ 2633.536842] [<ffffffff816b31ac>] ret_from_fork+0x7c/0xb0 Nov 20 19:35:16 thoregon kernel: [ 2633.536849] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 Nov 20 19:35:16 thoregon kernel: [ 2633.536890] xfsaild/md4 D 0000000000000003 0 1742 2 0x00000000 Nov 20 19:35:16 thoregon kernel: [ 2633.536898] ffff88032438bb68 0000000000000046 ffff880329965700 ffff88032438bfd8 Nov 20 19:35:16 thoregon kernel: [ 2633.536908] ffff88032438bfd8 ffff88032438bfd8 ffff88032827e580 ffff880329965700 Nov 20 19:35:16 thoregon kernel: [ 2633.536917] ffff88032438bb78 ffff8803278dbbd0 ffff8803278db800 00000000ffffffff Nov 20 19:35:16 thoregon kernel: [ 2633.536926] Call Trace: Nov 20 19:35:16 thoregon kernel: [ 2633.536935] [<ffffffff816b1224>] schedule+0x24/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.536942] [<ffffffff814f9dad>] md_super_wait+0x4d/0x80 Nov 20 19:35:16 thoregon kernel: [ 2633.536950] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.536957] [<ffffffff81500753>] bitmap_unplug+0x173/0x180 Nov 20 19:35:16 thoregon kernel: [ 2633.536966] [<ffffffff81278c13>] ? blk_finish_plug+0x13/0x50 Nov 20 19:35:16 thoregon kernel: [ 2633.536974] [<ffffffff814e8eb8>] raid1_unplug+0x98/0x110 Nov 20 19:35:16 thoregon kernel: [ 2633.536982] [<ffffffff81278a6d>] blk_flush_plug_list+0xad/0x240 Nov 20 19:35:16 thoregon kernel: [ 2633.536991] [<ffffffff81278c13>] blk_finish_plug+0x13/0x50 Nov 20 19:35:16 thoregon kernel: [ 2633.536998] [<ffffffff811e296a>] __xfs_buf_delwri_submit+0x1ca/0x1e0 Nov 20 19:35:16 thoregon kernel: [ 2633.537006] [<ffffffff811e2ffb>] xfs_buf_delwri_submit_nowait+0x1b/0x20 Nov 20 19:35:16 thoregon kernel: [ 2633.537014] [<ffffffff81233066>] xfsaild+0x226/0x4c0 Nov 20 19:35:16 thoregon kernel: [ 2633.537022] [<ffffffff81065dfa>] ? finish_task_switch+0x3a/0x100 Nov 20 19:35:16 thoregon kernel: [ 2633.537030] [<ffffffff81232e40>] ? xfs_trans_ail_cursor_first+0xa0/0xa0 Nov 20 19:35:16 thoregon kernel: [ 2633.537037] [<ffffffff8105c246>] kthread+0xd6/0xe0 Nov 20 19:35:16 thoregon kernel: [ 2633.537044] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50 Nov 20 19:35:16 thoregon kernel: [ 2633.537053] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 Nov 20 19:35:16 thoregon kernel: [ 2633.537060] [<ffffffff816b31ac>] ret_from_fork+0x7c/0xb0 Nov 20 19:35:16 thoregon kernel: [ 2633.537067] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 Nov 20 19:35:16 thoregon kernel: [ 2633.537158] plasma-desktop D 0000000000000001 0 3195 1 0x00000000 Nov 20 19:35:16 thoregon kernel: [ 2633.537165] ffff88030b99d448 0000000000000046 ffff8803266c1d00 ffff88030b99dfd8 Nov 20 19:35:16 thoregon kernel: [ 2633.537174] ffff88030b99dfd8 ffff88030b99dfd8 ffff880046b78e80 ffff8803266c1d00 Nov 20 19:35:16 thoregon kernel: [ 2633.537183] 0000000000000282 ffff88030b99d468 0000000100038f76 0000000100038f74 Nov 20 19:35:16 thoregon kernel: [ 2633.537192] Call Trace: Nov 20 19:35:16 thoregon kernel: [ 2633.537201] [<ffffffff816b1224>] schedule+0x24/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.537210] [<ffffffff816aed8e>] schedule_timeout+0x12e/0x1c0 Nov 20 19:35:16 thoregon kernel: [ 2633.537219] [<ffffffff81048730>] ? call_timer_fn+0xf0/0xf0 Nov 20 19:35:16 thoregon kernel: [ 2633.537228] [<ffffffff816aee39>] schedule_timeout_uninterruptible+0x19/0x20 Nov 20 19:35:16 thoregon kernel: [ 2633.537235] [<ffffffff811f1f99>] xfs_reclaim_inode+0x129/0x340 Nov 20 19:35:16 thoregon kernel: [ 2633.537242] [<ffffffff811f2d6f>] xfs_reclaim_inodes_ag+0x2bf/0x4f0 Nov 20 19:35:16 thoregon kernel: [ 2633.537249] [<ffffffff811f2b90>] ? xfs_reclaim_inodes_ag+0xe0/0x4f0 Nov 20 19:35:16 thoregon kernel: [ 2633.537258] [<ffffffff811f30de>] xfs_reclaim_inodes_nr+0x2e/0x40 Nov 20 19:35:16 thoregon kernel: [ 2633.537267] [<ffffffff811ef9b0>] xfs_fs_free_cached_objects+0x10/0x20 Nov 20 19:35:16 thoregon kernel: [ 2633.537274] [<ffffffff810f5b43>] prune_super+0x113/0x1a0 Nov 20 19:35:16 thoregon kernel: [ 2633.537282] [<ffffffff810bbd4e>] shrink_slab+0x11e/0x1f0 Nov 20 19:35:16 thoregon kernel: [ 2633.537290] [<ffffffff810b4fa5>] ? drain_pages+0x95/0xc0 Nov 20 19:35:16 thoregon kernel: [ 2633.537298] [<ffffffff810be8df>] try_to_free_pages+0x21f/0x4e0 Nov 20 19:35:16 thoregon kernel: [ 2633.537306] [<ffffffff810b5eb6>] __alloc_pages_nodemask+0x506/0x800 Nov 20 19:35:16 thoregon kernel: [ 2633.537316] [<ffffffff81570ed3>] ? __alloc_skb+0x83/0x280 Nov 20 19:35:16 thoregon kernel: [ 2633.537323] [<ffffffff810b6232>] __get_free_pages+0x12/0x40 Nov 20 19:35:16 thoregon kernel: [ 2633.537331] [<ffffffff810ebd8c>] __kmalloc_track_caller+0x14c/0x170 Nov 20 19:35:16 thoregon kernel: [ 2633.537339] [<ffffffff815703e6>] __kmalloc_reserve+0x36/0x90 Nov 20 19:35:16 thoregon kernel: [ 2633.537346] [<ffffffff81570ed3>] __alloc_skb+0x83/0x280 Nov 20 19:35:16 thoregon kernel: [ 2633.537354] [<ffffffff8105c905>] ? remove_wait_queue+0x55/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.537361] [<ffffffff81569fb1>] sock_alloc_send_pskb+0x251/0x3e0 Nov 20 19:35:16 thoregon kernel: [ 2633.537369] [<ffffffff8156a150>] sock_alloc_send_skb+0x10/0x20 Nov 20 19:35:16 thoregon kernel: [ 2633.537376] [<ffffffff81615f21>] unix_stream_sendmsg+0x2e1/0x420 Nov 20 19:35:16 thoregon kernel: [ 2633.537385] [<ffffffff81564900>] ? sock_aio_read+0x30/0x30 Nov 20 19:35:16 thoregon kernel: [ 2633.537393] [<ffffffff81564a0c>] sock_aio_write+0x10c/0x140 Nov 20 19:35:16 thoregon kernel: [ 2633.537403] [<ffffffff810f2eeb>] do_sync_readv_writev+0x9b/0xe0 Nov 20 19:35:16 thoregon kernel: [ 2633.537413] [<ffffffff810f31a8>] do_readv_writev+0xc8/0x1c0 Nov 20 19:35:16 thoregon kernel: [ 2633.537421] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:35:16 thoregon kernel: [ 2633.537428] [<ffffffff8107c899>] ? __lock_is_held+0x59/0x70 Nov 20 19:35:16 thoregon kernel: [ 2633.537438] [<ffffffff8110f892>] ? fget_light+0x42/0x4b0 Nov 20 19:35:16 thoregon kernel: [ 2633.537446] [<ffffffff810f32d7>] vfs_writev+0x37/0x50 Nov 20 19:35:16 thoregon kernel: [ 2633.537454] [<ffffffff810f341d>] sys_writev+0x4d/0x90 Nov 20 19:35:16 thoregon kernel: [ 2633.537462] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b Nov 20 19:35:16 thoregon kernel: [ 2633.537518] ssh D 0000000000000000 0 3573 1 0x00000004 Nov 20 19:35:16 thoregon kernel: [ 2633.537525] ffff8802c9f2f508 0000000000000046 ffff8802f04dd700 ffff8802c9f2ffd8 Nov 20 19:35:16 thoregon kernel: [ 2633.537534] ffff8802c9f2ffd8 ffff8802c9f2ffd8 ffffffff81c13420 ffff8802f04dd700 Nov 20 19:35:16 thoregon kernel: [ 2633.537543] 0000000000000282 ffff8802c9f2f528 0000000100038f75 0000000100038f73 Nov 20 19:35:16 thoregon kernel: [ 2633.537552] Call Trace: Nov 20 19:35:16 thoregon kernel: [ 2633.537561] [<ffffffff816b1224>] schedule+0x24/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.537569] [<ffffffff816aed8e>] schedule_timeout+0x12e/0x1c0 Nov 20 19:35:16 thoregon kernel: [ 2633.537578] [<ffffffff81048730>] ? call_timer_fn+0xf0/0xf0 Nov 20 19:35:16 thoregon kernel: [ 2633.537587] [<ffffffff816aee39>] schedule_timeout_uninterruptible+0x19/0x20 Nov 20 19:35:16 thoregon kernel: [ 2633.537594] [<ffffffff811f1f99>] xfs_reclaim_inode+0x129/0x340 Nov 20 19:35:16 thoregon kernel: [ 2633.537601] [<ffffffff811f2d6f>] xfs_reclaim_inodes_ag+0x2bf/0x4f0 Nov 20 19:35:16 thoregon kernel: [ 2633.537607] [<ffffffff811f2b90>] ? xfs_reclaim_inodes_ag+0xe0/0x4f0 Nov 20 19:35:16 thoregon kernel: [ 2633.537616] [<ffffffff811f30de>] xfs_reclaim_inodes_nr+0x2e/0x40 Nov 20 19:35:16 thoregon kernel: [ 2633.537625] [<ffffffff811ef9b0>] xfs_fs_free_cached_objects+0x10/0x20 Nov 20 19:35:16 thoregon kernel: [ 2633.537631] [<ffffffff810f5b43>] prune_super+0x113/0x1a0 Nov 20 19:35:16 thoregon kernel: [ 2633.537639] [<ffffffff810bbd4e>] shrink_slab+0x11e/0x1f0 Nov 20 19:35:16 thoregon kernel: [ 2633.537647] [<ffffffff810b4fa5>] ? drain_pages+0x95/0xc0 Nov 20 19:35:16 thoregon kernel: [ 2633.537655] [<ffffffff810be8df>] try_to_free_pages+0x21f/0x4e0 Nov 20 19:35:16 thoregon kernel: [ 2633.537663] [<ffffffff810b5eb6>] __alloc_pages_nodemask+0x506/0x800 Nov 20 19:35:16 thoregon kernel: [ 2633.537670] [<ffffffff81104d10>] ? poll_select_set_timeout+0x90/0x90 Nov 20 19:35:16 thoregon kernel: [ 2633.537679] [<ffffffff81570ed3>] ? __alloc_skb+0x83/0x280 Nov 20 19:35:16 thoregon kernel: [ 2633.537686] [<ffffffff810b6232>] __get_free_pages+0x12/0x40 Nov 20 19:35:16 thoregon kernel: [ 2633.537693] [<ffffffff810ebd8c>] __kmalloc_track_caller+0x14c/0x170 Nov 20 19:35:16 thoregon kernel: [ 2633.537701] [<ffffffff815703e6>] __kmalloc_reserve+0x36/0x90 Nov 20 19:35:16 thoregon kernel: [ 2633.537708] [<ffffffff81570ed3>] __alloc_skb+0x83/0x280 Nov 20 19:35:16 thoregon kernel: [ 2633.537715] [<ffffffff81569ac3>] ? release_sock+0x183/0x1e0 Nov 20 19:35:16 thoregon kernel: [ 2633.537722] [<ffffffff81569fb1>] sock_alloc_send_pskb+0x251/0x3e0 Nov 20 19:35:16 thoregon kernel: [ 2633.537729] [<ffffffff8156a150>] sock_alloc_send_skb+0x10/0x20 Nov 20 19:35:16 thoregon kernel: [ 2633.537736] [<ffffffff81615f21>] unix_stream_sendmsg+0x2e1/0x420 Nov 20 19:35:16 thoregon kernel: [ 2633.537745] [<ffffffff81564a0c>] sock_aio_write+0x10c/0x140 Nov 20 19:35:16 thoregon kernel: [ 2633.537753] [<ffffffff810cb5ae>] ? might_fault+0x4e/0xa0 Nov 20 19:35:16 thoregon kernel: [ 2633.537760] [<ffffffff811059aa>] ? core_sys_select+0x47a/0x4d0 Nov 20 19:35:16 thoregon kernel: [ 2633.537769] [<ffffffff810f238b>] do_sync_write+0x9b/0xe0 Nov 20 19:35:16 thoregon kernel: [ 2633.537778] [<ffffffff810f2a75>] vfs_write+0x145/0x160 Nov 20 19:35:16 thoregon kernel: [ 2633.537787] [<ffffffff810f2ccd>] sys_write+0x4d/0x90 Nov 20 19:35:16 thoregon kernel: [ 2633.537794] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b Nov 20 19:35:16 thoregon kernel: [ 2633.537801] bash D 0000000000000000 0 28378 28334 0x00000000 Nov 20 19:35:16 thoregon kernel: [ 2633.537809] ffff880299f5d5d8 0000000000000046 ffff880326a83a00 ffff880299f5dfd8 Nov 20 19:35:16 thoregon kernel: [ 2633.537818] ffff880299f5dfd8 ffff880299f5dfd8 ffff88032989ba00 ffff880326a83a00 Nov 20 19:35:16 thoregon kernel: [ 2633.537827] ffff880299f5d6a8 ffff880299c50230 7fffffffffffffff ffff880326a83a00 Nov 20 19:35:16 thoregon kernel: [ 2633.537836] Call Trace: Nov 20 19:35:16 thoregon kernel: [ 2633.537845] [<ffffffff816b1224>] schedule+0x24/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.537853] [<ffffffff816aeddd>] schedule_timeout+0x17d/0x1c0 Nov 20 19:35:16 thoregon kernel: [ 2633.537860] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50 Nov 20 19:35:16 thoregon kernel: [ 2633.537868] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:35:16 thoregon kernel: [ 2633.537876] [<ffffffff816b057e>] __down_common+0x97/0xe8 Nov 20 19:35:16 thoregon kernel: [ 2633.537885] [<ffffffff816b062e>] __down+0x18/0x1a Nov 20 19:35:16 thoregon kernel: [ 2633.537891] [<ffffffff8106245b>] down+0x3b/0x50 Nov 20 19:35:16 thoregon kernel: [ 2633.537898] [<ffffffff811e21b5>] xfs_buf_lock+0x25/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.537904] [<ffffffff811e22e2>] _xfs_buf_find+0xf2/0x200 Nov 20 19:35:16 thoregon kernel: [ 2633.537911] [<ffffffff811e241f>] xfs_buf_get_map+0x2f/0x130 Nov 20 19:35:16 thoregon kernel: [ 2633.537918] [<ffffffff811e2b23>] xfs_buf_read_map+0x13/0x90 Nov 20 19:35:16 thoregon kernel: [ 2633.537925] [<ffffffff81233e8d>] xfs_trans_read_buf_map+0x1ad/0x2d0 Nov 20 19:35:16 thoregon kernel: [ 2633.537933] [<ffffffff811f73fb>] xfs_alloc_read_agfl+0x6b/0x90 Nov 20 19:35:16 thoregon kernel: [ 2633.537941] [<ffffffff811f986d>] ? xfs_alloc_read_agf+0xcd/0xf0 Nov 20 19:35:16 thoregon kernel: [ 2633.537948] [<ffffffff810e9a93>] ? kmem_cache_free+0x83/0x150 Nov 20 19:35:16 thoregon kernel: [ 2633.537955] [<ffffffff811f9ac1>] xfs_alloc_fix_freelist+0x231/0x480 Nov 20 19:35:16 thoregon kernel: [ 2633.537962] [<ffffffff8107c899>] ? __lock_is_held+0x59/0x70 Nov 20 19:35:16 thoregon kernel: [ 2633.537971] [<ffffffff811fa279>] xfs_free_extent+0x99/0x120 Nov 20 19:35:16 thoregon kernel: [ 2633.537978] [<ffffffff811f6fff>] ? kmem_zone_alloc+0x5f/0xe0 Nov 20 19:35:16 thoregon kernel: [ 2633.537987] [<ffffffff81206edc>] xfs_bmap_finish+0x15c/0x1a0 Nov 20 19:35:16 thoregon kernel: [ 2633.537996] [<ffffffff8121f9eb>] xfs_itruncate_extents+0xdb/0x200 Nov 20 19:35:16 thoregon kernel: [ 2633.538006] [<ffffffff811ed7d5>] xfs_setattr_size+0x355/0x3b0 Nov 20 19:35:16 thoregon kernel: [ 2633.538014] [<ffffffff811ed85e>] xfs_vn_setattr+0x2e/0x40 Nov 20 19:35:16 thoregon kernel: [ 2633.538022] [<ffffffff8110dfa6>] notify_change+0x186/0x3a0 Nov 20 19:35:16 thoregon kernel: [ 2633.538030] [<ffffffff810f1229>] do_truncate+0x59/0x90 Nov 20 19:35:16 thoregon kernel: [ 2633.538039] [<ffffffff810fd777>] ? __inode_permission+0x27/0x80 Nov 20 19:35:16 thoregon kernel: [ 2633.538048] [<ffffffff811008da>] do_last.isra.69+0x60a/0xc80 Nov 20 19:35:16 thoregon kernel: [ 2633.538056] [<ffffffff810fd7e3>] ? inode_permission+0x13/0x50 Nov 20 19:35:16 thoregon kernel: [ 2633.538064] [<ffffffff810fd882>] ? link_path_walk+0x62/0x910 Nov 20 19:35:16 thoregon kernel: [ 2633.538073] [<ffffffff81100ffb>] path_openat.isra.70+0xab/0x490 Nov 20 19:35:16 thoregon kernel: [ 2633.538080] [<ffffffff8107c899>] ? __lock_is_held+0x59/0x70 Nov 20 19:35:16 thoregon kernel: [ 2633.538089] [<ffffffff8110179d>] do_filp_open+0x3d/0xa0 Nov 20 19:35:16 thoregon kernel: [ 2633.538097] [<ffffffff8111082b>] ? __alloc_fd+0x16b/0x210 Nov 20 19:35:16 thoregon kernel: [ 2633.538106] [<ffffffff810f2089>] do_sys_open+0xf9/0x1e0 Nov 20 19:35:16 thoregon kernel: [ 2633.538114] [<ffffffff810f218c>] sys_open+0x1c/0x20 Nov 20 19:35:16 thoregon kernel: [ 2633.538121] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b Nov 20 19:35:16 thoregon kernel: [ 2633.538126] flush-9:4 D 0000000000000001 0 30365 2 0x00000000 Nov 20 19:35:16 thoregon kernel: [ 2633.538134] ffff8801d8ce1978 0000000000000046 ffff880328238e80 ffff8801d8ce1fd8 Nov 20 19:35:16 thoregon kernel: [ 2633.538143] ffff8801d8ce1fd8 ffff8801d8ce1fd8 ffff8803032cc880 ffff880328238e80 Nov 20 19:35:16 thoregon kernel: [ 2633.538151] ffff8801d8ce1988 ffff8803278dbbd0 ffff8803278db800 00000000ffffffff Nov 20 19:35:16 thoregon kernel: [ 2633.538160] Call Trace: Nov 20 19:35:16 thoregon kernel: [ 2633.538169] [<ffffffff816b1224>] schedule+0x24/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.538176] [<ffffffff814f9dad>] md_super_wait+0x4d/0x80 Nov 20 19:35:16 thoregon kernel: [ 2633.538184] [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.538192] [<ffffffff81500753>] bitmap_unplug+0x173/0x180 Nov 20 19:35:16 thoregon kernel: [ 2633.538199] [<ffffffff810b6acf>] ? write_cache_pages+0x12f/0x420 Nov 20 19:35:16 thoregon kernel: [ 2633.538206] [<ffffffff810b6700>] ? set_page_dirty_lock+0x60/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.538214] [<ffffffff814e8eb8>] raid1_unplug+0x98/0x110 Nov 20 19:35:16 thoregon kernel: [ 2633.538222] [<ffffffff81278a6d>] blk_flush_plug_list+0xad/0x240 Nov 20 19:35:16 thoregon kernel: [ 2633.538231] [<ffffffff81278c13>] blk_finish_plug+0x13/0x50 Nov 20 19:35:16 thoregon kernel: [ 2633.538238] [<ffffffff810b6e0a>] generic_writepages+0x4a/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.538247] [<ffffffff811df898>] xfs_vm_writepages+0x48/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.538256] [<ffffffff8111b144>] ? writeback_sb_inodes+0x174/0x410 Nov 20 19:35:16 thoregon kernel: [ 2633.538264] [<ffffffff810b84be>] do_writepages+0x1e/0x30 Nov 20 19:35:16 thoregon kernel: [ 2633.538272] [<ffffffff8111a1ee>] __writeback_single_inode+0x3e/0x120 Nov 20 19:35:16 thoregon kernel: [ 2633.538280] [<ffffffff8111b234>] writeback_sb_inodes+0x264/0x410 Nov 20 19:35:16 thoregon kernel: [ 2633.538289] [<ffffffff8111b476>] __writeback_inodes_wb+0x96/0xc0 Nov 20 19:35:16 thoregon kernel: [ 2633.538298] [<ffffffff8111b673>] wb_writeback+0x1d3/0x1e0 Nov 20 19:35:16 thoregon kernel: [ 2633.538305] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:35:16 thoregon kernel: [ 2633.538313] [<ffffffff8111bb1c>] ? wb_do_writeback+0x7c/0x140 Nov 20 19:35:16 thoregon kernel: [ 2633.538322] [<ffffffff8111bb40>] wb_do_writeback+0xa0/0x140 Nov 20 19:35:16 thoregon kernel: [ 2633.538331] [<ffffffff8111bc52>] bdi_writeback_thread+0x72/0x150 Nov 20 19:35:16 thoregon kernel: [ 2633.538339] [<ffffffff8111bbe0>] ? wb_do_writeback+0x140/0x140 Nov 20 19:35:16 thoregon kernel: [ 2633.538346] [<ffffffff8105c246>] kthread+0xd6/0xe0 Nov 20 19:35:16 thoregon kernel: [ 2633.538353] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50 Nov 20 19:35:16 thoregon kernel: [ 2633.538361] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 Nov 20 19:35:16 thoregon kernel: [ 2633.538369] [<ffffffff816b31ac>] ret_from_fork+0x7c/0xb0 Nov 20 19:35:16 thoregon kernel: [ 2633.538376] [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 Nov 20 19:35:16 thoregon kernel: [ 2633.538384] x86_64-pc-linux D 0000000000000000 0 26516 1 0x00000004 Nov 20 19:35:16 thoregon kernel: [ 2633.538391] ffff8800468dddf8 0000000000000046 ffff880329303a00 ffff8800468ddfd8 Nov 20 19:35:16 thoregon kernel: [ 2633.538400] ffff8800468ddfd8 ffff8800468ddfd8 ffff88032989c880 ffff880329303a00 Nov 20 19:35:16 thoregon kernel: [ 2633.538409] 0000000000000246 ffff8800468dc000 ffff8801b1a1bc68 0000000002405150 Nov 20 19:35:16 thoregon kernel: [ 2633.538418] Call Trace: Nov 20 19:35:16 thoregon kernel: [ 2633.538427] [<ffffffff816b1224>] schedule+0x24/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.538436] [<ffffffff816b1501>] schedule_preempt_disabled+0x21/0x30 Nov 20 19:35:16 thoregon kernel: [ 2633.538444] [<ffffffff816af3a5>] mutex_lock_nested+0x165/0x330 Nov 20 19:35:16 thoregon kernel: [ 2633.538452] [<ffffffff810ff5e3>] ? do_unlinkat+0xa3/0x1b0 Nov 20 19:35:16 thoregon kernel: [ 2633.538461] [<ffffffff810fefae>] ? filename_lookup.isra.57+0x2e/0x80 Nov 20 19:35:16 thoregon kernel: [ 2633.538469] [<ffffffff810ff5e3>] do_unlinkat+0xa3/0x1b0 Nov 20 19:35:16 thoregon kernel: [ 2633.538477] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:35:16 thoregon kernel: [ 2633.538485] [<ffffffff8129ca1e>] ? trace_hardirqs_on_thunk+0x3a/0x3f Nov 20 19:35:16 thoregon kernel: [ 2633.538494] [<ffffffff81101bb1>] sys_unlink+0x11/0x20 Nov 20 19:35:16 thoregon kernel: [ 2633.538501] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b Nov 20 19:35:16 thoregon kernel: [ 2633.538505] x86_64-pc-linux D 0000000000000000 0 26523 1 0x00000004 Nov 20 19:35:16 thoregon kernel: [ 2633.538512] ffff8800c0def858 0000000000000046 ffff88032986c880 ffff8800c0deffd8 Nov 20 19:35:16 thoregon kernel: [ 2633.538521] ffff8800c0deffd8 ffff8800c0deffd8 ffff88032989e580 ffff88032986c880 Nov 20 19:35:16 thoregon kernel: [ 2633.538530] ffff8800c0def928 ffff8801c57e3a30 7fffffffffffffff ffff88032986c880 Nov 20 19:35:16 thoregon kernel: [ 2633.538539] Call Trace: Nov 20 19:35:16 thoregon kernel: [ 2633.538548] [<ffffffff816b1224>] schedule+0x24/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.538556] [<ffffffff816aeddd>] schedule_timeout+0x17d/0x1c0 Nov 20 19:35:16 thoregon kernel: [ 2633.538563] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50 Nov 20 19:35:16 thoregon kernel: [ 2633.538571] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:35:16 thoregon kernel: [ 2633.538579] [<ffffffff816b057e>] __down_common+0x97/0xe8 Nov 20 19:35:16 thoregon kernel: [ 2633.538588] [<ffffffff816b062e>] __down+0x18/0x1a Nov 20 19:35:16 thoregon kernel: [ 2633.538594] [<ffffffff8106245b>] down+0x3b/0x50 Nov 20 19:35:16 thoregon kernel: [ 2633.538601] [<ffffffff811e21b5>] xfs_buf_lock+0x25/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.538607] [<ffffffff811e22e2>] _xfs_buf_find+0xf2/0x200 Nov 20 19:35:16 thoregon kernel: [ 2633.538614] [<ffffffff811e241f>] xfs_buf_get_map+0x2f/0x130 Nov 20 19:35:16 thoregon kernel: [ 2633.538621] [<ffffffff81210417>] ? xfs_dabuf_map.isra.23+0xb7/0x2f0 Nov 20 19:35:16 thoregon kernel: [ 2633.538628] [<ffffffff810b572e>] ? get_page_from_freelist+0x1de/0x460 Nov 20 19:35:16 thoregon kernel: [ 2633.538635] [<ffffffff811e2b23>] xfs_buf_read_map+0x13/0x90 Nov 20 19:35:16 thoregon kernel: [ 2633.538642] [<ffffffff81233e8d>] xfs_trans_read_buf_map+0x1ad/0x2d0 Nov 20 19:35:16 thoregon kernel: [ 2633.538650] [<ffffffff812116b6>] xfs_da_read_buf+0xb6/0x180 Nov 20 19:35:16 thoregon kernel: [ 2633.538659] [<ffffffff8121724c>] xfs_dir2_leaf_lookup_int+0x5c/0x2c0 Nov 20 19:35:16 thoregon kernel: [ 2633.538668] [<ffffffff81217583>] xfs_dir2_leaf_removename+0x33/0x330 Nov 20 19:35:16 thoregon kernel: [ 2633.538676] [<ffffffff8120777b>] ? xfs_bmap_last_offset+0x8b/0xc0 Nov 20 19:35:16 thoregon kernel: [ 2633.538684] [<ffffffff81213b94>] xfs_dir_removename+0x134/0x150 Nov 20 19:35:16 thoregon kernel: [ 2633.538692] [<ffffffff811f5ce6>] xfs_remove+0x1f6/0x370 Nov 20 19:35:16 thoregon kernel: [ 2633.538701] [<ffffffff816af4ac>] ? mutex_lock_nested+0x26c/0x330 Nov 20 19:35:16 thoregon kernel: [ 2633.538709] [<ffffffff810ff48f>] ? vfs_unlink+0x4f/0x100 Nov 20 19:35:16 thoregon kernel: [ 2633.538717] [<ffffffff811ec7f3>] xfs_vn_unlink+0x43/0x90 Nov 20 19:35:16 thoregon kernel: [ 2633.538725] [<ffffffff810ff4cd>] vfs_unlink+0x8d/0x100 Nov 20 19:35:16 thoregon kernel: [ 2633.538733] [<ffffffff810ff63a>] do_unlinkat+0xfa/0x1b0 Nov 20 19:35:16 thoregon kernel: [ 2633.538741] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:35:16 thoregon kernel: [ 2633.538748] [<ffffffff8129ca1e>] ? trace_hardirqs_on_thunk+0x3a/0x3f Nov 20 19:35:16 thoregon kernel: [ 2633.538757] [<ffffffff81101bb1>] sys_unlink+0x11/0x20 Nov 20 19:35:16 thoregon kernel: [ 2633.538764] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b Nov 20 19:35:16 thoregon kernel: [ 2633.538769] x86_64-pc-linux D 0000000000000000 0 26875 1 0x00000004 Nov 20 19:35:16 thoregon kernel: [ 2633.538776] ffff8800c0c5ba08 0000000000000046 ffff8802f0594880 ffff8800c0c5bfd8 Nov 20 19:35:16 thoregon kernel: [ 2633.538785] ffff8800c0c5bfd8 ffff8800c0c5bfd8 ffff88032989c880 ffff8802f0594880 Nov 20 19:35:16 thoregon kernel: [ 2633.538793] 0000000000000250 ffff880205a7aa30 7fffffffffffffff ffff8802f0594880 Nov 20 19:35:16 thoregon kernel: [ 2633.538802] Call Trace: Nov 20 19:35:16 thoregon kernel: [ 2633.538811] [<ffffffff816b1224>] schedule+0x24/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.538819] [<ffffffff816aeddd>] schedule_timeout+0x17d/0x1c0 Nov 20 19:35:16 thoregon kernel: [ 2633.538826] [<ffffffff811f70b4>] ? kmem_zone_zalloc+0x34/0x50 Nov 20 19:35:16 thoregon kernel: [ 2633.538833] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50 Nov 20 19:35:16 thoregon kernel: [ 2633.538841] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:35:16 thoregon kernel: [ 2633.538849] [<ffffffff816b057e>] __down_common+0x97/0xe8 Nov 20 19:35:16 thoregon kernel: [ 2633.538857] [<ffffffff816b062e>] __down+0x18/0x1a Nov 20 19:35:16 thoregon kernel: [ 2633.538864] [<ffffffff8106245b>] down+0x3b/0x50 Nov 20 19:35:16 thoregon kernel: [ 2633.538870] [<ffffffff811e21b5>] xfs_buf_lock+0x25/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.538877] [<ffffffff811e22e2>] _xfs_buf_find+0xf2/0x200 Nov 20 19:35:16 thoregon kernel: [ 2633.538883] [<ffffffff811e241f>] xfs_buf_get_map+0x2f/0x130 Nov 20 19:35:16 thoregon kernel: [ 2633.538890] [<ffffffff811e2b23>] xfs_buf_read_map+0x13/0x90 Nov 20 19:35:16 thoregon kernel: [ 2633.538897] [<ffffffff81233e8d>] xfs_trans_read_buf_map+0x1ad/0x2d0 Nov 20 19:35:16 thoregon kernel: [ 2633.538906] [<ffffffff8121ce6f>] xfs_read_agi+0x7f/0x110 Nov 20 19:35:16 thoregon kernel: [ 2633.538915] [<ffffffff8121fb55>] xfs_iunlink+0x45/0x150 Nov 20 19:35:16 thoregon kernel: [ 2633.538922] [<ffffffff81041fc1>] ? current_fs_time+0x11/0x50 Nov 20 19:35:16 thoregon kernel: [ 2633.538929] [<ffffffff81234352>] ? xfs_trans_ichgtime+0x22/0xa0 Nov 20 19:35:16 thoregon kernel: [ 2633.538936] [<ffffffff811f3ea0>] xfs_droplink+0x50/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.538943] [<ffffffff811f5d77>] xfs_remove+0x287/0x370 Nov 20 19:35:16 thoregon kernel: [ 2633.538951] [<ffffffff816af4ac>] ? mutex_lock_nested+0x26c/0x330 Nov 20 19:35:16 thoregon kernel: [ 2633.538959] [<ffffffff810ff48f>] ? vfs_unlink+0x4f/0x100 Nov 20 19:35:16 thoregon kernel: [ 2633.538967] [<ffffffff811ec7f3>] xfs_vn_unlink+0x43/0x90 Nov 20 19:35:16 thoregon kernel: [ 2633.538975] [<ffffffff810ff4cd>] vfs_unlink+0x8d/0x100 Nov 20 19:35:16 thoregon kernel: [ 2633.538983] [<ffffffff810ff63a>] do_unlinkat+0xfa/0x1b0 Nov 20 19:35:16 thoregon kernel: [ 2633.538991] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:35:16 thoregon kernel: [ 2633.538999] [<ffffffff8129ca1e>] ? trace_hardirqs_on_thunk+0x3a/0x3f Nov 20 19:35:16 thoregon kernel: [ 2633.539007] [<ffffffff81101bb1>] sys_unlink+0x11/0x20 Nov 20 19:35:16 thoregon kernel: [ 2633.539014] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b Nov 20 19:35:16 thoregon kernel: [ 2633.539020] x86_64-pc-linux D 0000000000000001 0 26897 1 0x00000004 Nov 20 19:35:16 thoregon kernel: [ 2633.539027] ffff880101959858 0000000000000046 ffff88032934ab80 ffff880101959fd8 Nov 20 19:35:16 thoregon kernel: [ 2633.539036] ffff880101959fd8 ffff880101959fd8 ffff880162a3e580 ffff88032934ab80 Nov 20 19:35:16 thoregon kernel: [ 2633.539045] ffff880101959928 ffff8801ca545030 7fffffffffffffff ffff88032934ab80 Nov 20 19:35:16 thoregon kernel: [ 2633.539053] Call Trace: Nov 20 19:35:16 thoregon kernel: [ 2633.539062] [<ffffffff816b1224>] schedule+0x24/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.539070] [<ffffffff816aeddd>] schedule_timeout+0x17d/0x1c0 Nov 20 19:35:16 thoregon kernel: [ 2633.539077] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50 Nov 20 19:35:16 thoregon kernel: [ 2633.539085] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:35:16 thoregon kernel: [ 2633.539094] [<ffffffff816b057e>] __down_common+0x97/0xe8 Nov 20 19:35:16 thoregon kernel: [ 2633.539102] [<ffffffff816b062e>] __down+0x18/0x1a Nov 20 19:35:16 thoregon kernel: [ 2633.539108] [<ffffffff8106245b>] down+0x3b/0x50 Nov 20 19:35:16 thoregon kernel: [ 2633.539115] [<ffffffff811e21b5>] xfs_buf_lock+0x25/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.539121] [<ffffffff811e22e2>] _xfs_buf_find+0xf2/0x200 Nov 20 19:35:16 thoregon kernel: [ 2633.539128] [<ffffffff811e241f>] xfs_buf_get_map+0x2f/0x130 Nov 20 19:35:16 thoregon kernel: [ 2633.539135] [<ffffffff81210417>] ? xfs_dabuf_map.isra.23+0xb7/0x2f0 Nov 20 19:35:16 thoregon kernel: [ 2633.539142] [<ffffffff810b572e>] ? get_page_from_freelist+0x1de/0x460 Nov 20 19:35:16 thoregon kernel: [ 2633.539149] [<ffffffff811e2b23>] xfs_buf_read_map+0x13/0x90 Nov 20 19:35:16 thoregon kernel: [ 2633.539156] [<ffffffff81233e8d>] xfs_trans_read_buf_map+0x1ad/0x2d0 Nov 20 19:35:16 thoregon kernel: [ 2633.539164] [<ffffffff812116b6>] xfs_da_read_buf+0xb6/0x180 Nov 20 19:35:16 thoregon kernel: [ 2633.539172] [<ffffffff8121724c>] xfs_dir2_leaf_lookup_int+0x5c/0x2c0 Nov 20 19:35:16 thoregon kernel: [ 2633.539181] [<ffffffff81217583>] xfs_dir2_leaf_removename+0x33/0x330 Nov 20 19:35:16 thoregon kernel: [ 2633.539189] [<ffffffff8120777b>] ? xfs_bmap_last_offset+0x8b/0xc0 Nov 20 19:35:16 thoregon kernel: [ 2633.539197] [<ffffffff81213b94>] xfs_dir_removename+0x134/0x150 Nov 20 19:35:16 thoregon kernel: [ 2633.539205] [<ffffffff811f5ce6>] xfs_remove+0x1f6/0x370 Nov 20 19:35:16 thoregon kernel: [ 2633.539214] [<ffffffff816af4ac>] ? mutex_lock_nested+0x26c/0x330 Nov 20 19:35:16 thoregon kernel: [ 2633.539222] [<ffffffff810ff48f>] ? vfs_unlink+0x4f/0x100 Nov 20 19:35:16 thoregon kernel: [ 2633.539230] [<ffffffff811ec7f3>] xfs_vn_unlink+0x43/0x90 Nov 20 19:35:16 thoregon kernel: [ 2633.539238] [<ffffffff810ff4cd>] vfs_unlink+0x8d/0x100 Nov 20 19:35:16 thoregon kernel: [ 2633.539246] [<ffffffff810ff63a>] do_unlinkat+0xfa/0x1b0 Nov 20 19:35:16 thoregon kernel: [ 2633.539254] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:35:16 thoregon kernel: [ 2633.539261] [<ffffffff8129ca1e>] ? trace_hardirqs_on_thunk+0x3a/0x3f Nov 20 19:35:16 thoregon kernel: [ 2633.539270] [<ffffffff81101bb1>] sys_unlink+0x11/0x20 Nov 20 19:35:16 thoregon kernel: [ 2633.539277] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b Nov 20 19:35:16 thoregon kernel: [ 2633.539282] x86_64-pc-linux D 0000000000000001 0 26904 1 0x00000004 Nov 20 19:35:16 thoregon kernel: [ 2633.539290] ffff880066bff858 0000000000000046 ffff880329301d00 ffff880066bfffd8 Nov 20 19:35:16 thoregon kernel: [ 2633.539299] ffff880066bfffd8 ffff880066bfffd8 ffff88013a7ce580 ffff880329301d00 Nov 20 19:35:16 thoregon kernel: [ 2633.539307] ffff880066bff928 ffff8801c572c830 7fffffffffffffff ffff880329301d00 Nov 20 19:35:16 thoregon kernel: [ 2633.539316] Call Trace: Nov 20 19:35:16 thoregon kernel: [ 2633.539325] [<ffffffff816b1224>] schedule+0x24/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.539333] [<ffffffff816aeddd>] schedule_timeout+0x17d/0x1c0 Nov 20 19:35:16 thoregon kernel: [ 2633.539340] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50 Nov 20 19:35:16 thoregon kernel: [ 2633.539348] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:35:16 thoregon kernel: [ 2633.539356] [<ffffffff816b057e>] __down_common+0x97/0xe8 Nov 20 19:35:16 thoregon kernel: [ 2633.539365] [<ffffffff816b062e>] __down+0x18/0x1a Nov 20 19:35:16 thoregon kernel: [ 2633.539371] [<ffffffff8106245b>] down+0x3b/0x50 Nov 20 19:35:16 thoregon kernel: [ 2633.539377] [<ffffffff811e21b5>] xfs_buf_lock+0x25/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.539384] [<ffffffff811e22e2>] _xfs_buf_find+0xf2/0x200 Nov 20 19:35:16 thoregon kernel: [ 2633.539390] [<ffffffff811e241f>] xfs_buf_get_map+0x2f/0x130 Nov 20 19:35:16 thoregon kernel: [ 2633.539398] [<ffffffff81210417>] ? xfs_dabuf_map.isra.23+0xb7/0x2f0 Nov 20 19:35:16 thoregon kernel: [ 2633.539405] [<ffffffff810b572e>] ? get_page_from_freelist+0x1de/0x460 Nov 20 19:35:16 thoregon kernel: [ 2633.539412] [<ffffffff811e2b23>] xfs_buf_read_map+0x13/0x90 Nov 20 19:35:16 thoregon kernel: [ 2633.539419] [<ffffffff81233e8d>] xfs_trans_read_buf_map+0x1ad/0x2d0 Nov 20 19:35:16 thoregon kernel: [ 2633.539426] [<ffffffff812116b6>] xfs_da_read_buf+0xb6/0x180 Nov 20 19:35:16 thoregon kernel: [ 2633.539435] [<ffffffff8121724c>] xfs_dir2_leaf_lookup_int+0x5c/0x2c0 Nov 20 19:35:16 thoregon kernel: [ 2633.539444] [<ffffffff81217583>] xfs_dir2_leaf_removename+0x33/0x330 Nov 20 19:35:16 thoregon kernel: [ 2633.539452] [<ffffffff8120777b>] ? xfs_bmap_last_offset+0x8b/0xc0 Nov 20 19:35:16 thoregon kernel: [ 2633.539460] [<ffffffff81213b94>] xfs_dir_removename+0x134/0x150 Nov 20 19:35:16 thoregon kernel: [ 2633.539468] [<ffffffff811f5ce6>] xfs_remove+0x1f6/0x370 Nov 20 19:35:16 thoregon kernel: [ 2633.539477] [<ffffffff816af4ac>] ? mutex_lock_nested+0x26c/0x330 Nov 20 19:35:16 thoregon kernel: [ 2633.539484] [<ffffffff810ff48f>] ? vfs_unlink+0x4f/0x100 Nov 20 19:35:16 thoregon kernel: [ 2633.539493] [<ffffffff811ec7f3>] xfs_vn_unlink+0x43/0x90 Nov 20 19:35:16 thoregon kernel: [ 2633.539501] [<ffffffff810ff4cd>] vfs_unlink+0x8d/0x100 Nov 20 19:35:16 thoregon kernel: [ 2633.539509] [<ffffffff810ff63a>] do_unlinkat+0xfa/0x1b0 Nov 20 19:35:16 thoregon kernel: [ 2633.539517] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:35:16 thoregon kernel: [ 2633.539524] [<ffffffff8129ca1e>] ? trace_hardirqs_on_thunk+0x3a/0x3f Nov 20 19:35:16 thoregon kernel: [ 2633.539532] [<ffffffff81101bb1>] sys_unlink+0x11/0x20 Nov 20 19:35:16 thoregon kernel: [ 2633.539540] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b Nov 20 19:35:16 thoregon kernel: [ 2633.539544] x86_64-pc-linux D 0000000000000000 0 26907 1 0x00000004 Nov 20 19:35:16 thoregon kernel: [ 2633.539552] ffff8800c0eed858 0000000000000046 ffff8802a7be6580 ffff8800c0eedfd8 Nov 20 19:35:16 thoregon kernel: [ 2633.539560] ffff8800c0eedfd8 ffff8800c0eedfd8 ffff88032989e580 ffff8802a7be6580 Nov 20 19:35:16 thoregon kernel: [ 2633.539569] ffff8800c0eed928 ffff8801c55c7630 7fffffffffffffff ffff8802a7be6580 Nov 20 19:35:16 thoregon kernel: [ 2633.539578] Call Trace: Nov 20 19:35:16 thoregon kernel: [ 2633.539587] [<ffffffff816b1224>] schedule+0x24/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.539595] [<ffffffff816aeddd>] schedule_timeout+0x17d/0x1c0 Nov 20 19:35:16 thoregon kernel: [ 2633.539602] [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50 Nov 20 19:35:16 thoregon kernel: [ 2633.539610] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:35:16 thoregon kernel: [ 2633.539618] [<ffffffff816b057e>] __down_common+0x97/0xe8 Nov 20 19:35:16 thoregon kernel: [ 2633.539627] [<ffffffff816b062e>] __down+0x18/0x1a Nov 20 19:35:16 thoregon kernel: [ 2633.539633] [<ffffffff8106245b>] down+0x3b/0x50 Nov 20 19:35:16 thoregon kernel: [ 2633.539639] [<ffffffff811e21b5>] xfs_buf_lock+0x25/0x60 Nov 20 19:35:16 thoregon kernel: [ 2633.539646] [<ffffffff811e22e2>] _xfs_buf_find+0xf2/0x200 Nov 20 19:35:16 thoregon kernel: [ 2633.539652] [<ffffffff811e241f>] xfs_buf_get_map+0x2f/0x130 Nov 20 19:35:16 thoregon kernel: [ 2633.539660] [<ffffffff81210417>] ? xfs_dabuf_map.isra.23+0xb7/0x2f0 Nov 20 19:35:16 thoregon kernel: [ 2633.539667] [<ffffffff810b572e>] ? get_page_from_freelist+0x1de/0x460 Nov 20 19:35:16 thoregon kernel: [ 2633.539674] [<ffffffff811e2b23>] xfs_buf_read_map+0x13/0x90 Nov 20 19:35:16 thoregon kernel: [ 2633.539681] [<ffffffff81233e8d>] xfs_trans_read_buf_map+0x1ad/0x2d0 Nov 20 19:35:16 thoregon kernel: [ 2633.539688] [<ffffffff812116b6>] xfs_da_read_buf+0xb6/0x180 Nov 20 19:35:16 thoregon kernel: [ 2633.539697] [<ffffffff8121724c>] xfs_dir2_leaf_lookup_int+0x5c/0x2c0 Nov 20 19:35:16 thoregon kernel: [ 2633.539706] [<ffffffff81217583>] xfs_dir2_leaf_removename+0x33/0x330 Nov 20 19:35:16 thoregon kernel: [ 2633.539714] [<ffffffff8120777b>] ? xfs_bmap_last_offset+0x8b/0xc0 Nov 20 19:35:16 thoregon kernel: [ 2633.539722] [<ffffffff81213b94>] xfs_dir_removename+0x134/0x150 Nov 20 19:35:16 thoregon kernel: [ 2633.539730] [<ffffffff811f5ce6>] xfs_remove+0x1f6/0x370 Nov 20 19:35:16 thoregon kernel: [ 2633.539738] [<ffffffff816af4ac>] ? mutex_lock_nested+0x26c/0x330 Nov 20 19:35:16 thoregon kernel: [ 2633.539746] [<ffffffff810ff48f>] ? vfs_unlink+0x4f/0x100 Nov 20 19:35:16 thoregon kernel: [ 2633.539754] [<ffffffff811ec7f3>] xfs_vn_unlink+0x43/0x90 Nov 20 19:35:16 thoregon kernel: [ 2633.539763] [<ffffffff810ff4cd>] vfs_unlink+0x8d/0x100 Nov 20 19:35:16 thoregon kernel: [ 2633.539771] [<ffffffff810ff63a>] do_unlinkat+0xfa/0x1b0 Nov 20 19:35:16 thoregon kernel: [ 2633.539779] [<ffffffff8108152d>] ? trace_hardirqs_on_caller+0xfd/0x190 Nov 20 19:35:16 thoregon kernel: [ 2633.539786] [<ffffffff8129ca1e>] ? trace_hardirqs_on_thunk+0x3a/0x3f Nov 20 19:35:16 thoregon kernel: [ 2633.539794] [<ffffffff81101bb1>] sys_unlink+0x11/0x20 Nov 20 19:35:16 thoregon kernel: [ 2633.539802] [<ffffffff816b3252>] system_call_fastpath+0x16/0x1b [-- Attachment #3: Type: text/plain, Size: 121 bytes --] _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Hang in XFS reclaim on 3.7.0-rc3 2012-11-20 19:45 ` Torsten Kaiser @ 2012-11-20 20:27 ` Dave Chinner -1 siblings, 0 replies; 31+ messages in thread From: Dave Chinner @ 2012-11-20 20:27 UTC (permalink / raw) To: Torsten Kaiser; +Cc: xfs, Linux Kernel On Tue, Nov 20, 2012 at 08:45:03PM +0100, Torsten Kaiser wrote: > On Tue, Nov 20, 2012 at 12:53 AM, Dave Chinner <david@fromorbit.com> wrote: > > [<ffffffff8108137e>] mark_held_locks+0x7e/0x130 > > [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0 > > [<ffffffff810e9dd5>] kmem_cache_alloc+0x35/0xe0 > > [<ffffffff810dba31>] vm_map_ram+0x271/0x770 > > [<ffffffff811e1316>] _xfs_buf_map_pages+0x46/0xe0 > > [<ffffffff811e222a>] xfs_buf_get_map+0x8a/0x130 > > [<ffffffff81233ab9>] xfs_trans_get_buf_map+0xa9/0xd0 > > [<ffffffff8121bced>] xfs_ialloc_inode_init+0xcd/0x1d0 > > > > We shouldn't be mapping buffers there, there's a patch below to fix > > this. It's probably the source of this report, even though I cannot > > lockdep seems to be off with the fairies... > > That patch seems to break my system. You've got an IO problem, not an XFS problem. Everything is hung up on MD. INFO: task kswapd0:725 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. kswapd0 D 0000000000000001 0 725 2 0x00000000 ffff8803280d13f8 0000000000000046 ffff880329a0ab80 ffff8803280d1fd8 ffff8803280d1fd8 ffff8803280d1fd8 ffff880046b7c880 ffff880329a0ab80 ffff8803280d1408 ffff8803278dbbd0 ffff8803278db800 00000000ffffffff Call Trace: [<ffffffff816b1224>] schedule+0x24/0x60 [<ffffffff814f9dad>] md_super_wait+0x4d/0x80 [<ffffffff81500753>] bitmap_unplug+0x173/0x180 [<ffffffff814e8eb8>] raid1_unplug+0x98/0x110 [<ffffffff81278a6d>] blk_flush_plug_list+0xad/0x240 [<ffffffff816b15c3>] io_schedule_timeout+0x83/0xf0 [<ffffffff810b0e1d>] mempool_alloc+0x12d/0x160 [<ffffffff811263da>] bvec_alloc_bs+0xda/0x100 [<ffffffff811264ea>] bio_alloc_bioset+0xea/0x110 [<ffffffff81126656>] bio_clone_bioset+0x16/0x40 [<ffffffff814f471a>] bio_clone_mddev+0x1a/0x30 [<ffffffff814edbb1>] make_request+0x551/0xde0 [<ffffffff814f80bb>] md_make_request+0x21b/0x4d0 [<ffffffff81276e52>] generic_make_request+0xc2/0x100 [<ffffffff81276ef5>] submit_bio+0x65/0x110 [<ffffffff811e07bf>] xfs_submit_ioend_bio.isra.21+0x2f/0x40 [<ffffffff811e088e>] xfs_submit_ioend+0xbe/0x110 [<ffffffff811e0c91>] xfs_vm_writepage+0x3b1/0x540 [<ffffffff810bcd84>] shrink_page_list+0x564/0x890 [<ffffffff810bd637>] shrink_inactive_list+0x1d7/0x310 [<ffffffff810bdb9d>] shrink_lruvec+0x42d/0x530 [<ffffffff810be323>] kswapd+0x683/0xa20 [<ffffffff8105c246>] kthread+0xd6/0xe0 [<ffffffff816b31ac>] ret_from_fork+0x7c/0xb0 no locks held by kswapd0/725. So kswapd is trying to clean pages, but it's blocked in an unplug during IO submission. Probably one to report to the linux-raid list. INFO: task xfsaild/md4:1742 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. xfsaild/md4 D 0000000000000003 0 1742 2 0x00000000 ffff88032438bb68 0000000000000046 ffff880329965700 ffff88032438bfd8 ffff88032438bfd8 ffff88032438bfd8 ffff88032827e580 ffff880329965700 ffff88032438bb78 ffff8803278dbbd0 ffff8803278db800 00000000ffffffff Call Trace: [<ffffffff816b1224>] schedule+0x24/0x60 [<ffffffff814f9dad>] md_super_wait+0x4d/0x80 [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60 [<ffffffff81500753>] bitmap_unplug+0x173/0x180 [<ffffffff81278c13>] ? blk_finish_plug+0x13/0x50 [<ffffffff814e8eb8>] raid1_unplug+0x98/0x110 [<ffffffff81278a6d>] blk_flush_plug_list+0xad/0x240 [<ffffffff81278c13>] blk_finish_plug+0x13/0x50 [<ffffffff811e296a>] __xfs_buf_delwri_submit+0x1ca/0x1e0 [<ffffffff811e2ffb>] xfs_buf_delwri_submit_nowait+0x1b/0x20 [<ffffffff81233066>] xfsaild+0x226/0x4c0 [<ffffffff81065dfa>] ? finish_task_switch+0x3a/0x100 [<ffffffff81232e40>] ? xfs_trans_ail_cursor_first+0xa0/0xa0 [<ffffffff8105c246>] kthread+0xd6/0xe0 [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50 [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 [<ffffffff816b31ac>] ret_from_fork+0x7c/0xb0 [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 no locks held by xfsaild/md4/1742. Same here - metadata writes are backed up waiting for MD to submit IO. Everything else is stuck on thesei or MD, too... Cheers, Dave. -- Dave Chinner david@fromorbit.com ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Hang in XFS reclaim on 3.7.0-rc3 @ 2012-11-20 20:27 ` Dave Chinner 0 siblings, 0 replies; 31+ messages in thread From: Dave Chinner @ 2012-11-20 20:27 UTC (permalink / raw) To: Torsten Kaiser; +Cc: Linux Kernel, xfs On Tue, Nov 20, 2012 at 08:45:03PM +0100, Torsten Kaiser wrote: > On Tue, Nov 20, 2012 at 12:53 AM, Dave Chinner <david@fromorbit.com> wrote: > > [<ffffffff8108137e>] mark_held_locks+0x7e/0x130 > > [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0 > > [<ffffffff810e9dd5>] kmem_cache_alloc+0x35/0xe0 > > [<ffffffff810dba31>] vm_map_ram+0x271/0x770 > > [<ffffffff811e1316>] _xfs_buf_map_pages+0x46/0xe0 > > [<ffffffff811e222a>] xfs_buf_get_map+0x8a/0x130 > > [<ffffffff81233ab9>] xfs_trans_get_buf_map+0xa9/0xd0 > > [<ffffffff8121bced>] xfs_ialloc_inode_init+0xcd/0x1d0 > > > > We shouldn't be mapping buffers there, there's a patch below to fix > > this. It's probably the source of this report, even though I cannot > > lockdep seems to be off with the fairies... > > That patch seems to break my system. You've got an IO problem, not an XFS problem. Everything is hung up on MD. INFO: task kswapd0:725 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. kswapd0 D 0000000000000001 0 725 2 0x00000000 ffff8803280d13f8 0000000000000046 ffff880329a0ab80 ffff8803280d1fd8 ffff8803280d1fd8 ffff8803280d1fd8 ffff880046b7c880 ffff880329a0ab80 ffff8803280d1408 ffff8803278dbbd0 ffff8803278db800 00000000ffffffff Call Trace: [<ffffffff816b1224>] schedule+0x24/0x60 [<ffffffff814f9dad>] md_super_wait+0x4d/0x80 [<ffffffff81500753>] bitmap_unplug+0x173/0x180 [<ffffffff814e8eb8>] raid1_unplug+0x98/0x110 [<ffffffff81278a6d>] blk_flush_plug_list+0xad/0x240 [<ffffffff816b15c3>] io_schedule_timeout+0x83/0xf0 [<ffffffff810b0e1d>] mempool_alloc+0x12d/0x160 [<ffffffff811263da>] bvec_alloc_bs+0xda/0x100 [<ffffffff811264ea>] bio_alloc_bioset+0xea/0x110 [<ffffffff81126656>] bio_clone_bioset+0x16/0x40 [<ffffffff814f471a>] bio_clone_mddev+0x1a/0x30 [<ffffffff814edbb1>] make_request+0x551/0xde0 [<ffffffff814f80bb>] md_make_request+0x21b/0x4d0 [<ffffffff81276e52>] generic_make_request+0xc2/0x100 [<ffffffff81276ef5>] submit_bio+0x65/0x110 [<ffffffff811e07bf>] xfs_submit_ioend_bio.isra.21+0x2f/0x40 [<ffffffff811e088e>] xfs_submit_ioend+0xbe/0x110 [<ffffffff811e0c91>] xfs_vm_writepage+0x3b1/0x540 [<ffffffff810bcd84>] shrink_page_list+0x564/0x890 [<ffffffff810bd637>] shrink_inactive_list+0x1d7/0x310 [<ffffffff810bdb9d>] shrink_lruvec+0x42d/0x530 [<ffffffff810be323>] kswapd+0x683/0xa20 [<ffffffff8105c246>] kthread+0xd6/0xe0 [<ffffffff816b31ac>] ret_from_fork+0x7c/0xb0 no locks held by kswapd0/725. So kswapd is trying to clean pages, but it's blocked in an unplug during IO submission. Probably one to report to the linux-raid list. INFO: task xfsaild/md4:1742 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. xfsaild/md4 D 0000000000000003 0 1742 2 0x00000000 ffff88032438bb68 0000000000000046 ffff880329965700 ffff88032438bfd8 ffff88032438bfd8 ffff88032438bfd8 ffff88032827e580 ffff880329965700 ffff88032438bb78 ffff8803278dbbd0 ffff8803278db800 00000000ffffffff Call Trace: [<ffffffff816b1224>] schedule+0x24/0x60 [<ffffffff814f9dad>] md_super_wait+0x4d/0x80 [<ffffffff8105ca30>] ? __init_waitqueue_head+0x60/0x60 [<ffffffff81500753>] bitmap_unplug+0x173/0x180 [<ffffffff81278c13>] ? blk_finish_plug+0x13/0x50 [<ffffffff814e8eb8>] raid1_unplug+0x98/0x110 [<ffffffff81278a6d>] blk_flush_plug_list+0xad/0x240 [<ffffffff81278c13>] blk_finish_plug+0x13/0x50 [<ffffffff811e296a>] __xfs_buf_delwri_submit+0x1ca/0x1e0 [<ffffffff811e2ffb>] xfs_buf_delwri_submit_nowait+0x1b/0x20 [<ffffffff81233066>] xfsaild+0x226/0x4c0 [<ffffffff81065dfa>] ? finish_task_switch+0x3a/0x100 [<ffffffff81232e40>] ? xfs_trans_ail_cursor_first+0xa0/0xa0 [<ffffffff8105c246>] kthread+0xd6/0xe0 [<ffffffff816b246b>] ? _raw_spin_unlock_irq+0x2b/0x50 [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 [<ffffffff816b31ac>] ret_from_fork+0x7c/0xb0 [<ffffffff8105c170>] ? flush_kthread_worker+0xe0/0xe0 no locks held by xfsaild/md4/1742. Same here - metadata writes are backed up waiting for MD to submit IO. Everything else is stuck on thesei or MD, too... Cheers, Dave. -- Dave Chinner david@fromorbit.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Hang in XFS reclaim on 3.7.0-rc3 2012-10-29 22:26 ` Dave Chinner @ 2012-11-01 21:30 ` Ben Myers -1 siblings, 0 replies; 31+ messages in thread From: Ben Myers @ 2012-11-01 21:30 UTC (permalink / raw) To: Dave Chinner; +Cc: Torsten Kaiser, Linux Kernel, xfs Hi Dave, On Tue, Oct 30, 2012 at 09:26:13AM +1100, Dave Chinner wrote: > On Mon, Oct 29, 2012 at 09:03:15PM +0100, Torsten Kaiser wrote: > > After experiencing a hang of all IO yesterday ( > > http://marc.info/?l=linux-kernel&m=135142236520624&w=2 ), I turned on > > LOCKDEP after upgrading to -rc3. > > > > I then tried to replicate the load that hung yesterday and got the > > following lockdep report, implicating XFS instead of by stacking swap > > onto dm-crypt and md. > > > > [ 2844.971913] > > [ 2844.971920] ================================= > > [ 2844.971921] [ INFO: inconsistent lock state ] > > [ 2844.971924] 3.7.0-rc3 #1 Not tainted > > [ 2844.971925] --------------------------------- > > [ 2844.971927] inconsistent {RECLAIM_FS-ON-W} -> {IN-RECLAIM_FS-W} usage. > > [ 2844.971929] kswapd0/725 [HC0[0]:SC0[0]:HE1:SE1] takes: > > [ 2844.971931] (&(&ip->i_lock)->mr_lock){++++?.}, at: [<ffffffff811e7ef4>] xfs_ilock+0x84/0xb0 > > [ 2844.971941] {RECLAIM_FS-ON-W} state was registered at: > > [ 2844.971942] [<ffffffff8108137e>] mark_held_locks+0x7e/0x130 > > [ 2844.971947] [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0 > > [ 2844.971949] [<ffffffff810e9dd5>] kmem_cache_alloc+0x35/0xe0 > > [ 2844.971952] [<ffffffff810dba31>] vm_map_ram+0x271/0x770 > > [ 2844.971955] [<ffffffff811e10a6>] _xfs_buf_map_pages+0x46/0xe0 > > [ 2844.971959] [<ffffffff811e1fba>] xfs_buf_get_map+0x8a/0x130 > > [ 2844.971961] [<ffffffff81233849>] xfs_trans_get_buf_map+0xa9/0xd0 > > [ 2844.971964] [<ffffffff8121e339>] xfs_ifree_cluster+0x129/0x670 > > [ 2844.971967] [<ffffffff8121f959>] xfs_ifree+0xe9/0xf0 > > [ 2844.971969] [<ffffffff811f4abf>] xfs_inactive+0x2af/0x480 > > [ 2844.971972] [<ffffffff811efb90>] xfs_fs_evict_inode+0x70/0x80 > > [ 2844.971974] [<ffffffff8110cb8f>] evict+0xaf/0x1b0 > > [ 2844.971977] [<ffffffff8110cd95>] iput+0x105/0x210 > > [ 2844.971979] [<ffffffff811070d0>] dentry_iput+0xa0/0xe0 > > [ 2844.971981] [<ffffffff81108310>] dput+0x150/0x280 > > [ 2844.971983] [<ffffffff811020fb>] sys_renameat+0x21b/0x290 > > [ 2844.971986] [<ffffffff81102186>] sys_rename+0x16/0x20 > > [ 2844.971988] [<ffffffff816b2292>] system_call_fastpath+0x16/0x1b > > We shouldn't be mapping pages there. See if the patch below fixes > it. > > Fundamentally, though, the lockdep warning has come about because > vm_map_ram is doing a GFP_KERNEL allocation when we need it to be > doing GFP_NOFS - we are within a transaction here, so memory reclaim > is not allowed to recurse back into the filesystem. > > mm-folk: can we please get this vmalloc/gfp_flags passing API > fixed once and for all? This is the fourth time in the last month or > so that I've seen XFS bug reports with silent hangs and associated > lockdep output that implicate GFP_KERNEL allocations from vm_map_ram > in GFP_NOFS conditions as the potential cause.... > > Cheers, > > Dave. > -- > Dave Chinner > david@fromorbit.com > > xfs: don't vmap inode cluster buffers during free Could you write up a little more background for the commit message? Regards, Ben ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Hang in XFS reclaim on 3.7.0-rc3 @ 2012-11-01 21:30 ` Ben Myers 0 siblings, 0 replies; 31+ messages in thread From: Ben Myers @ 2012-11-01 21:30 UTC (permalink / raw) To: Dave Chinner; +Cc: Linux Kernel, Torsten Kaiser, xfs Hi Dave, On Tue, Oct 30, 2012 at 09:26:13AM +1100, Dave Chinner wrote: > On Mon, Oct 29, 2012 at 09:03:15PM +0100, Torsten Kaiser wrote: > > After experiencing a hang of all IO yesterday ( > > http://marc.info/?l=linux-kernel&m=135142236520624&w=2 ), I turned on > > LOCKDEP after upgrading to -rc3. > > > > I then tried to replicate the load that hung yesterday and got the > > following lockdep report, implicating XFS instead of by stacking swap > > onto dm-crypt and md. > > > > [ 2844.971913] > > [ 2844.971920] ================================= > > [ 2844.971921] [ INFO: inconsistent lock state ] > > [ 2844.971924] 3.7.0-rc3 #1 Not tainted > > [ 2844.971925] --------------------------------- > > [ 2844.971927] inconsistent {RECLAIM_FS-ON-W} -> {IN-RECLAIM_FS-W} usage. > > [ 2844.971929] kswapd0/725 [HC0[0]:SC0[0]:HE1:SE1] takes: > > [ 2844.971931] (&(&ip->i_lock)->mr_lock){++++?.}, at: [<ffffffff811e7ef4>] xfs_ilock+0x84/0xb0 > > [ 2844.971941] {RECLAIM_FS-ON-W} state was registered at: > > [ 2844.971942] [<ffffffff8108137e>] mark_held_locks+0x7e/0x130 > > [ 2844.971947] [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0 > > [ 2844.971949] [<ffffffff810e9dd5>] kmem_cache_alloc+0x35/0xe0 > > [ 2844.971952] [<ffffffff810dba31>] vm_map_ram+0x271/0x770 > > [ 2844.971955] [<ffffffff811e10a6>] _xfs_buf_map_pages+0x46/0xe0 > > [ 2844.971959] [<ffffffff811e1fba>] xfs_buf_get_map+0x8a/0x130 > > [ 2844.971961] [<ffffffff81233849>] xfs_trans_get_buf_map+0xa9/0xd0 > > [ 2844.971964] [<ffffffff8121e339>] xfs_ifree_cluster+0x129/0x670 > > [ 2844.971967] [<ffffffff8121f959>] xfs_ifree+0xe9/0xf0 > > [ 2844.971969] [<ffffffff811f4abf>] xfs_inactive+0x2af/0x480 > > [ 2844.971972] [<ffffffff811efb90>] xfs_fs_evict_inode+0x70/0x80 > > [ 2844.971974] [<ffffffff8110cb8f>] evict+0xaf/0x1b0 > > [ 2844.971977] [<ffffffff8110cd95>] iput+0x105/0x210 > > [ 2844.971979] [<ffffffff811070d0>] dentry_iput+0xa0/0xe0 > > [ 2844.971981] [<ffffffff81108310>] dput+0x150/0x280 > > [ 2844.971983] [<ffffffff811020fb>] sys_renameat+0x21b/0x290 > > [ 2844.971986] [<ffffffff81102186>] sys_rename+0x16/0x20 > > [ 2844.971988] [<ffffffff816b2292>] system_call_fastpath+0x16/0x1b > > We shouldn't be mapping pages there. See if the patch below fixes > it. > > Fundamentally, though, the lockdep warning has come about because > vm_map_ram is doing a GFP_KERNEL allocation when we need it to be > doing GFP_NOFS - we are within a transaction here, so memory reclaim > is not allowed to recurse back into the filesystem. > > mm-folk: can we please get this vmalloc/gfp_flags passing API > fixed once and for all? This is the fourth time in the last month or > so that I've seen XFS bug reports with silent hangs and associated > lockdep output that implicate GFP_KERNEL allocations from vm_map_ram > in GFP_NOFS conditions as the potential cause.... > > Cheers, > > Dave. > -- > Dave Chinner > david@fromorbit.com > > xfs: don't vmap inode cluster buffers during free Could you write up a little more background for the commit message? Regards, Ben _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Hang in XFS reclaim on 3.7.0-rc3 2012-11-01 21:30 ` Ben Myers @ 2012-11-01 22:32 ` Dave Chinner -1 siblings, 0 replies; 31+ messages in thread From: Dave Chinner @ 2012-11-01 22:32 UTC (permalink / raw) To: Ben Myers; +Cc: Torsten Kaiser, Linux Kernel, xfs On Thu, Nov 01, 2012 at 04:30:10PM -0500, Ben Myers wrote: > Hi Dave, > > On Tue, Oct 30, 2012 at 09:26:13AM +1100, Dave Chinner wrote: > > On Mon, Oct 29, 2012 at 09:03:15PM +0100, Torsten Kaiser wrote: > > > After experiencing a hang of all IO yesterday ( > > > http://marc.info/?l=linux-kernel&m=135142236520624&w=2 ), I turned on > > > LOCKDEP after upgrading to -rc3. > > > > > > I then tried to replicate the load that hung yesterday and got the > > > following lockdep report, implicating XFS instead of by stacking swap > > > onto dm-crypt and md. > > > > > > [ 2844.971913] > > > [ 2844.971920] ================================= > > > [ 2844.971921] [ INFO: inconsistent lock state ] > > > [ 2844.971924] 3.7.0-rc3 #1 Not tainted > > > [ 2844.971925] --------------------------------- > > > [ 2844.971927] inconsistent {RECLAIM_FS-ON-W} -> {IN-RECLAIM_FS-W} usage. > > > [ 2844.971929] kswapd0/725 [HC0[0]:SC0[0]:HE1:SE1] takes: > > > [ 2844.971931] (&(&ip->i_lock)->mr_lock){++++?.}, at: [<ffffffff811e7ef4>] xfs_ilock+0x84/0xb0 > > > [ 2844.971941] {RECLAIM_FS-ON-W} state was registered at: > > > [ 2844.971942] [<ffffffff8108137e>] mark_held_locks+0x7e/0x130 > > > [ 2844.971947] [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0 > > > [ 2844.971949] [<ffffffff810e9dd5>] kmem_cache_alloc+0x35/0xe0 > > > [ 2844.971952] [<ffffffff810dba31>] vm_map_ram+0x271/0x770 > > > [ 2844.971955] [<ffffffff811e10a6>] _xfs_buf_map_pages+0x46/0xe0 ..... > > We shouldn't be mapping pages there. See if the patch below fixes > > it. > > > > Fundamentally, though, the lockdep warning has come about because > > vm_map_ram is doing a GFP_KERNEL allocation when we need it to be > > doing GFP_NOFS - we are within a transaction here, so memory reclaim > > is not allowed to recurse back into the filesystem. > > > > mm-folk: can we please get this vmalloc/gfp_flags passing API > > fixed once and for all? This is the fourth time in the last month or > > so that I've seen XFS bug reports with silent hangs and associated > > lockdep output that implicate GFP_KERNEL allocations from vm_map_ram > > in GFP_NOFS conditions as the potential cause.... > > > > xfs: don't vmap inode cluster buffers during free > > Could you write up a little more background for the commit message? Sure, that was just a test patch and often I don't bother putting a detailed description in them until I know they fix the problem. My current tree has: xfs: don't vmap inode cluster buffers during free Inode buffers do not need to be mapped as inodes are read or written directly from/to the pages underlying the buffer. This fixes a regression introduced by commit 611c994 ("xfs: make XBF_MAPPED the default behaviour"). Cheers, Dave. -- Dave Chinner david@fromorbit.com ^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Hang in XFS reclaim on 3.7.0-rc3 @ 2012-11-01 22:32 ` Dave Chinner 0 siblings, 0 replies; 31+ messages in thread From: Dave Chinner @ 2012-11-01 22:32 UTC (permalink / raw) To: Ben Myers; +Cc: Linux Kernel, Torsten Kaiser, xfs On Thu, Nov 01, 2012 at 04:30:10PM -0500, Ben Myers wrote: > Hi Dave, > > On Tue, Oct 30, 2012 at 09:26:13AM +1100, Dave Chinner wrote: > > On Mon, Oct 29, 2012 at 09:03:15PM +0100, Torsten Kaiser wrote: > > > After experiencing a hang of all IO yesterday ( > > > http://marc.info/?l=linux-kernel&m=135142236520624&w=2 ), I turned on > > > LOCKDEP after upgrading to -rc3. > > > > > > I then tried to replicate the load that hung yesterday and got the > > > following lockdep report, implicating XFS instead of by stacking swap > > > onto dm-crypt and md. > > > > > > [ 2844.971913] > > > [ 2844.971920] ================================= > > > [ 2844.971921] [ INFO: inconsistent lock state ] > > > [ 2844.971924] 3.7.0-rc3 #1 Not tainted > > > [ 2844.971925] --------------------------------- > > > [ 2844.971927] inconsistent {RECLAIM_FS-ON-W} -> {IN-RECLAIM_FS-W} usage. > > > [ 2844.971929] kswapd0/725 [HC0[0]:SC0[0]:HE1:SE1] takes: > > > [ 2844.971931] (&(&ip->i_lock)->mr_lock){++++?.}, at: [<ffffffff811e7ef4>] xfs_ilock+0x84/0xb0 > > > [ 2844.971941] {RECLAIM_FS-ON-W} state was registered at: > > > [ 2844.971942] [<ffffffff8108137e>] mark_held_locks+0x7e/0x130 > > > [ 2844.971947] [<ffffffff81081a63>] lockdep_trace_alloc+0x63/0xc0 > > > [ 2844.971949] [<ffffffff810e9dd5>] kmem_cache_alloc+0x35/0xe0 > > > [ 2844.971952] [<ffffffff810dba31>] vm_map_ram+0x271/0x770 > > > [ 2844.971955] [<ffffffff811e10a6>] _xfs_buf_map_pages+0x46/0xe0 ..... > > We shouldn't be mapping pages there. See if the patch below fixes > > it. > > > > Fundamentally, though, the lockdep warning has come about because > > vm_map_ram is doing a GFP_KERNEL allocation when we need it to be > > doing GFP_NOFS - we are within a transaction here, so memory reclaim > > is not allowed to recurse back into the filesystem. > > > > mm-folk: can we please get this vmalloc/gfp_flags passing API > > fixed once and for all? This is the fourth time in the last month or > > so that I've seen XFS bug reports with silent hangs and associated > > lockdep output that implicate GFP_KERNEL allocations from vm_map_ram > > in GFP_NOFS conditions as the potential cause.... > > > > xfs: don't vmap inode cluster buffers during free > > Could you write up a little more background for the commit message? Sure, that was just a test patch and often I don't bother putting a detailed description in them until I know they fix the problem. My current tree has: xfs: don't vmap inode cluster buffers during free Inode buffers do not need to be mapped as inodes are read or written directly from/to the pages underlying the buffer. This fixes a regression introduced by commit 611c994 ("xfs: make XBF_MAPPED the default behaviour"). Cheers, Dave. -- Dave Chinner david@fromorbit.com _______________________________________________ xfs mailing list xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs ^ permalink raw reply [flat|nested] 31+ messages in thread
end of thread, other threads:[~2012-11-20 20:27 UTC | newest] Thread overview: 31+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2012-10-29 20:03 Hang in XFS reclaim on 3.7.0-rc3 Torsten Kaiser 2012-10-29 20:03 ` Torsten Kaiser 2012-10-29 22:26 ` Dave Chinner 2012-10-29 22:26 ` Dave Chinner 2012-10-29 22:41 ` Dave Chinner 2012-10-29 22:41 ` Dave Chinner 2012-10-29 22:41 ` Dave Chinner 2012-10-30 20:37 ` Torsten Kaiser 2012-10-30 20:37 ` Torsten Kaiser 2012-10-30 20:46 ` Christoph Hellwig 2012-10-30 20:46 ` Christoph Hellwig 2012-11-18 10:24 ` Torsten Kaiser 2012-11-18 10:24 ` Torsten Kaiser 2012-11-18 15:29 ` Torsten Kaiser 2012-11-18 15:29 ` Torsten Kaiser 2012-11-18 23:51 ` Dave Chinner 2012-11-18 23:51 ` Dave Chinner 2012-11-19 6:50 ` Torsten Kaiser 2012-11-19 6:50 ` Torsten Kaiser 2012-11-19 23:53 ` Dave Chinner 2012-11-19 23:53 ` Dave Chinner 2012-11-20 7:09 ` Torsten Kaiser 2012-11-20 7:09 ` Torsten Kaiser 2012-11-20 19:45 ` Torsten Kaiser 2012-11-20 19:45 ` Torsten Kaiser 2012-11-20 20:27 ` Dave Chinner 2012-11-20 20:27 ` Dave Chinner 2012-11-01 21:30 ` Ben Myers 2012-11-01 21:30 ` Ben Myers 2012-11-01 22:32 ` Dave Chinner 2012-11-01 22:32 ` Dave Chinner
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.