From: Dave Chinner <david@fromorbit.com> To: Michal Hocko <mhocko@suse.cz> Cc: Glauber Costa <glommer@gmail.com>, Andrew Morton <akpm@linux-foundation.org>, linux-mm@kvack.org, LKML <linux-kernel@vger.kernel.org> Subject: Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92 Date: Tue, 2 Jul 2013 22:19:47 +1000 [thread overview] Message-ID: <20130702121947.GE14996@dastard> (raw) In-Reply-To: <20130702092200.GB16815@dhcp22.suse.cz> On Tue, Jul 02, 2013 at 11:22:00AM +0200, Michal Hocko wrote: > On Mon 01-07-13 18:10:56, Dave Chinner wrote: > > On Mon, Jul 01, 2013 at 09:50:05AM +0200, Michal Hocko wrote: > > > On Mon 01-07-13 11:25:58, Dave Chinner wrote: > > That is the recycle stat, which indicates we've found an inode being > > reclaimed. When it's found an inode that have been evicted, but not > > yet reclaimed at the XFS level, that stat will increase. If the > > inode is still valid at the VFS level, and igrab() fails, then we'll > > get EAGAIN without that stat being increased. So, igrab() is > > failing, and that means I_FREEING|I_WILL_FREE are set. > > > > So, it looks to be the same case as the ext4 hang, and it's likely > > that we have some dangling inode dispose list somewhere. So, here's > > the fun part. Use tracing to grab the inode number that is stuck > > (tracepoint xfs::xfs_iget_skip), > > $ cat /sys/kernel/debug/tracing/trace_pipe > demon.trace.log & > $ pid=$! > $ sleep 10s ; kill $pid > $ awk '{print $1, $9}' demon.trace.log | sort -u > cc1-7561 0xf78d4f > cc1-9100 0x80b2a35 ..... > > > and the dispose list that it is on should be the on the > > inode->i_lru_list. > > crash> struct inode.i_lru ffff88000c09e2f8 > i_lru = { > next = 0xffff88000c09e3e8, > prev = 0xffff88000c09e3e8 > } Hmmm, that's empty. > crash> struct inode.i_flags ffff88000c09e2f8 > i_flags = 4096 I asked for the wrong field, I wanted i_state, but seeing as you: > The full xfs_inode dump is attached. Dumped the whole inode, I got it from below :) > i_state = 32, so, i_state = I_FREEING. IOWs, we've got an inode marked I_FREEING that isn't on a dispose list but hasn't passed through evict() correctly. > crash> struct xfs_inode ffff88000c09e1c0 > struct xfs_inode { ..... > i_flags = 0, XFS doesn't see the inode as reclaimable yet, either. Ok, so it's been leaked from a dispose list somehow. Thanks for the info, Michal, it's time to go look at the code.... Cheers, Dave. -- Dave Chinner david@fromorbit.com
WARNING: multiple messages have this Message-ID (diff)
From: Dave Chinner <david@fromorbit.com> To: Michal Hocko <mhocko@suse.cz> Cc: Glauber Costa <glommer@gmail.com>, Andrew Morton <akpm@linux-foundation.org>, linux-mm@kvack.org, LKML <linux-kernel@vger.kernel.org> Subject: Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92 Date: Tue, 2 Jul 2013 22:19:47 +1000 [thread overview] Message-ID: <20130702121947.GE14996@dastard> (raw) In-Reply-To: <20130702092200.GB16815@dhcp22.suse.cz> On Tue, Jul 02, 2013 at 11:22:00AM +0200, Michal Hocko wrote: > On Mon 01-07-13 18:10:56, Dave Chinner wrote: > > On Mon, Jul 01, 2013 at 09:50:05AM +0200, Michal Hocko wrote: > > > On Mon 01-07-13 11:25:58, Dave Chinner wrote: > > That is the recycle stat, which indicates we've found an inode being > > reclaimed. When it's found an inode that have been evicted, but not > > yet reclaimed at the XFS level, that stat will increase. If the > > inode is still valid at the VFS level, and igrab() fails, then we'll > > get EAGAIN without that stat being increased. So, igrab() is > > failing, and that means I_FREEING|I_WILL_FREE are set. > > > > So, it looks to be the same case as the ext4 hang, and it's likely > > that we have some dangling inode dispose list somewhere. So, here's > > the fun part. Use tracing to grab the inode number that is stuck > > (tracepoint xfs::xfs_iget_skip), > > $ cat /sys/kernel/debug/tracing/trace_pipe > demon.trace.log & > $ pid=$! > $ sleep 10s ; kill $pid > $ awk '{print $1, $9}' demon.trace.log | sort -u > cc1-7561 0xf78d4f > cc1-9100 0x80b2a35 ..... > > > and the dispose list that it is on should be the on the > > inode->i_lru_list. > > crash> struct inode.i_lru ffff88000c09e2f8 > i_lru = { > next = 0xffff88000c09e3e8, > prev = 0xffff88000c09e3e8 > } Hmmm, that's empty. > crash> struct inode.i_flags ffff88000c09e2f8 > i_flags = 4096 I asked for the wrong field, I wanted i_state, but seeing as you: > The full xfs_inode dump is attached. Dumped the whole inode, I got it from below :) > i_state = 32, so, i_state = I_FREEING. IOWs, we've got an inode marked I_FREEING that isn't on a dispose list but hasn't passed through evict() correctly. > crash> struct xfs_inode ffff88000c09e1c0 > struct xfs_inode { ..... > i_flags = 0, XFS doesn't see the inode as reclaimable yet, either. Ok, so it's been leaked from a dispose list somehow. Thanks for the info, Michal, it's time to go look at the code.... Cheers, Dave. -- Dave Chinner david@fromorbit.com -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2013-07-02 12:19 UTC|newest] Thread overview: 127+ messages / expand[flat|nested] mbox.gz Atom feed top 2013-06-17 14:18 linux-next: slab shrinkers: BUG at mm/list_lru.c:92 Michal Hocko 2013-06-17 14:18 ` Michal Hocko 2013-06-17 15:14 ` Glauber Costa 2013-06-17 15:14 ` Glauber Costa 2013-06-17 15:33 ` Michal Hocko 2013-06-17 15:33 ` Michal Hocko 2013-06-17 16:54 ` Glauber Costa 2013-06-17 16:54 ` Glauber Costa 2013-06-18 7:42 ` Michal Hocko 2013-06-18 7:42 ` Michal Hocko 2013-06-17 21:35 ` Andrew Morton 2013-06-17 21:35 ` Andrew Morton 2013-06-17 22:30 ` Glauber Costa 2013-06-18 2:46 ` Dave Chinner 2013-06-18 2:46 ` Dave Chinner 2013-06-18 6:31 ` Glauber Costa 2013-06-18 6:31 ` Glauber Costa 2013-06-18 8:24 ` Michal Hocko 2013-06-18 8:24 ` Michal Hocko 2013-06-18 10:44 ` Michal Hocko 2013-06-18 10:44 ` Michal Hocko 2013-06-18 13:50 ` Michal Hocko 2013-06-18 13:50 ` Michal Hocko 2013-06-25 2:27 ` Dave Chinner 2013-06-25 2:27 ` Dave Chinner 2013-06-26 8:15 ` Michal Hocko 2013-06-26 8:15 ` Michal Hocko 2013-06-26 23:24 ` Dave Chinner 2013-06-26 23:24 ` Dave Chinner 2013-06-27 14:54 ` Michal Hocko 2013-06-27 14:54 ` Michal Hocko 2013-06-28 8:39 ` Michal Hocko 2013-06-28 8:39 ` Michal Hocko 2013-06-28 14:31 ` Glauber Costa 2013-06-28 14:31 ` Glauber Costa 2013-06-28 15:12 ` Michal Hocko 2013-06-28 15:12 ` Michal Hocko 2013-06-29 2:55 ` Dave Chinner 2013-06-29 2:55 ` Dave Chinner 2013-06-30 18:33 ` Michal Hocko 2013-07-01 1:25 ` Dave Chinner 2013-07-01 1:25 ` Dave Chinner 2013-07-01 7:50 ` Michal Hocko 2013-07-01 7:50 ` Michal Hocko 2013-07-01 8:10 ` Dave Chinner 2013-07-01 8:10 ` Dave Chinner 2013-07-02 9:22 ` Michal Hocko 2013-07-02 12:19 ` Dave Chinner [this message] 2013-07-02 12:19 ` Dave Chinner 2013-07-02 12:44 ` Michal Hocko 2013-07-02 12:44 ` Michal Hocko 2013-07-03 11:24 ` Dave Chinner 2013-07-03 11:24 ` Dave Chinner 2013-07-03 14:08 ` Glauber Costa 2013-07-03 14:08 ` Glauber Costa 2013-07-04 16:36 ` Michal Hocko 2013-07-04 16:36 ` Michal Hocko 2013-07-08 12:53 ` Michal Hocko 2013-07-08 21:04 ` Andrew Morton 2013-07-08 21:04 ` Andrew Morton 2013-07-09 17:34 ` Glauber Costa 2013-07-09 17:34 ` Glauber Costa 2013-07-09 17:51 ` Andrew Morton 2013-07-09 17:51 ` Andrew Morton 2013-07-09 17:32 ` Glauber Costa 2013-07-09 17:32 ` Glauber Costa 2013-07-09 17:50 ` Andrew Morton 2013-07-09 17:50 ` Andrew Morton 2013-07-09 17:57 ` Glauber Costa 2013-07-09 17:57 ` Glauber Costa 2013-07-09 17:57 ` Michal Hocko 2013-07-09 17:57 ` Michal Hocko 2013-07-09 21:39 ` Andrew Morton 2013-07-09 21:39 ` Andrew Morton 2013-07-10 2:31 ` Dave Chinner 2013-07-10 2:31 ` Dave Chinner 2013-07-10 7:34 ` Michal Hocko 2013-07-10 7:34 ` Michal Hocko 2013-07-10 8:06 ` Michal Hocko 2013-07-10 8:06 ` Michal Hocko 2013-07-11 2:26 ` Dave Chinner 2013-07-11 2:26 ` Dave Chinner 2013-07-11 3:03 ` Andrew Morton 2013-07-11 3:03 ` Andrew Morton 2013-07-11 13:23 ` Michal Hocko 2013-07-11 13:23 ` Michal Hocko 2013-07-12 1:42 ` Hugh Dickins 2013-07-12 1:42 ` Hugh Dickins 2013-07-13 3:29 ` Dave Chinner 2013-07-13 3:29 ` Dave Chinner 2013-07-15 9:14 ` Michal Hocko 2013-07-15 9:14 ` Michal Hocko 2013-06-18 6:26 ` Glauber Costa 2013-06-18 8:25 ` Michal Hocko 2013-06-18 8:25 ` Michal Hocko 2013-06-19 7:13 ` Michal Hocko 2013-06-19 7:13 ` Michal Hocko 2013-06-19 7:35 ` Glauber Costa 2013-06-19 7:35 ` Glauber Costa 2013-06-19 8:52 ` Glauber Costa 2013-06-19 8:52 ` Glauber Costa 2013-06-19 13:57 ` Michal Hocko 2013-06-19 13:57 ` Michal Hocko 2013-06-19 14:02 ` Glauber Costa 2013-06-19 14:02 ` Glauber Costa 2013-06-19 14:28 ` Michal Hocko 2013-06-19 14:28 ` Michal Hocko 2013-06-20 14:11 ` Glauber Costa 2013-06-20 14:11 ` Glauber Costa 2013-06-20 15:12 ` Michal Hocko 2013-06-20 15:16 ` Michal Hocko 2013-06-20 15:16 ` Michal Hocko 2013-06-21 9:00 ` Michal Hocko 2013-06-21 9:00 ` Michal Hocko 2013-06-23 11:51 ` Glauber Costa 2013-06-23 11:51 ` Glauber Costa 2013-06-23 11:55 ` Glauber Costa 2013-06-25 2:29 ` Dave Chinner 2013-06-25 2:29 ` Dave Chinner 2013-06-26 8:22 ` Michal Hocko 2013-06-26 8:22 ` Michal Hocko 2013-06-18 8:19 ` Michal Hocko 2013-06-18 8:19 ` Michal Hocko 2013-06-18 8:21 ` Glauber Costa 2013-06-18 8:21 ` Glauber Costa 2013-06-18 8:26 ` Michal Hocko 2013-06-18 8:26 ` Michal Hocko
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20130702121947.GE14996@dastard \ --to=david@fromorbit.com \ --cc=akpm@linux-foundation.org \ --cc=glommer@gmail.com \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=mhocko@suse.cz \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.