From: Michal Hocko <mhocko@suse.cz> To: Dave Chinner <david@fromorbit.com> Cc: Glauber Costa <glommer@gmail.com>, Andrew Morton <akpm@linux-foundation.org>, linux-mm@kvack.org, LKML <linux-kernel@vger.kernel.org> Subject: Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92 Date: Wed, 26 Jun 2013 10:15:09 +0200 [thread overview] Message-ID: <20130626081509.GF28748@dhcp22.suse.cz> (raw) In-Reply-To: <20130625022754.GP29376@dastard> On Tue 25-06-13 12:27:54, Dave Chinner wrote: > On Tue, Jun 18, 2013 at 03:50:25PM +0200, Michal Hocko wrote: > > And again, another hang. It looks like the inode deletion never > > finishes. The good thing is that I do not see any LRU related BUG_ONs > > anymore. I am going to test with the other patch in the thread. > > > > 2476 [<ffffffff8118325e>] __wait_on_freeing_inode+0x9e/0xc0 <<< waiting for an inode to go away > > [<ffffffff81183321>] find_inode_fast+0xa1/0xc0 > > [<ffffffff8118525f>] iget_locked+0x4f/0x180 > > [<ffffffff811ef9e3>] ext4_iget+0x33/0x9f0 > > [<ffffffff811f6a1c>] ext4_lookup+0xbc/0x160 > > [<ffffffff81174ad0>] lookup_real+0x20/0x60 > > [<ffffffff81177e25>] lookup_open+0x175/0x1d0 > > [<ffffffff8117815e>] do_last+0x2de/0x780 <<< holds i_mutex > > [<ffffffff8117ae9a>] path_openat+0xda/0x400 > > [<ffffffff8117b303>] do_filp_open+0x43/0xa0 > > [<ffffffff81168ee0>] do_sys_open+0x160/0x1e0 > > [<ffffffff81168f9c>] sys_open+0x1c/0x20 > > [<ffffffff81582fe9>] system_call_fastpath+0x16/0x1b > > [<ffffffffffffffff>] 0xffffffffffffffff > > I don't think this has anything to do with LRUs. I am not claiming that. It might be a timing issue which never mattered but it is strange I can reproduce this so easily and repeatedly with the shrinkers patchset applied. As I said earlier, this might be breakage in my -mm tree as well (missing some patch which didn't go via Andrew or misapplied patch). The situation is worsen by the state of linux-next which has some unrelated issues. I really do not want to delay the whole patchset just because of some problem on my side. Do you have any tree that I should try to test? > __wait_on_freeing_inode() only blocks once the inode is being freed > (i.e. I_FREEING is set), and that happens when a lookup is done when > the inode is still in the inode hash. > > I_FREEING is set on the inode at the same time it is removed from > the LRU, and from that point onwards the LRUs play no part in the > inode being freed and anyone waiting on the inode being freed > getting woken. > > The only way I can see this happening, is if there is a dispose list > that is not getting processed properly. e.g., we move a bunch on > inodes to the dispose list setting I_FREEING, then for some reason > it gets dropped on the ground and so the wakeup call doesn't happen > when the inode has been removed from the hash. > > I can't see anywhere in the code that this happens, though, but it > might be some pre-existing race in the inode hash that you are now > triggering because freeing will be happening in parallel on multiple > nodes rather than serialising on a global lock... > > I won't have seen this on XFS stress testing, because it doesn't use > the VFS inode hashes for inode lookups. Given that XFS is not > triggering either problem you are seeing, that makes me think I haven't tested with xfs. > that it might be a pre-existing inode hash lookup/reclaim race > condition, not a LRU problem. > > Cheers, > > Dave. > -- > Dave Chinner > david@fromorbit.com -- Michal Hocko SUSE Labs
WARNING: multiple messages have this Message-ID (diff)
From: Michal Hocko <mhocko@suse.cz> To: Dave Chinner <david@fromorbit.com> Cc: Glauber Costa <glommer@gmail.com>, Andrew Morton <akpm@linux-foundation.org>, linux-mm@kvack.org, LKML <linux-kernel@vger.kernel.org> Subject: Re: linux-next: slab shrinkers: BUG at mm/list_lru.c:92 Date: Wed, 26 Jun 2013 10:15:09 +0200 [thread overview] Message-ID: <20130626081509.GF28748@dhcp22.suse.cz> (raw) In-Reply-To: <20130625022754.GP29376@dastard> On Tue 25-06-13 12:27:54, Dave Chinner wrote: > On Tue, Jun 18, 2013 at 03:50:25PM +0200, Michal Hocko wrote: > > And again, another hang. It looks like the inode deletion never > > finishes. The good thing is that I do not see any LRU related BUG_ONs > > anymore. I am going to test with the other patch in the thread. > > > > 2476 [<ffffffff8118325e>] __wait_on_freeing_inode+0x9e/0xc0 <<< waiting for an inode to go away > > [<ffffffff81183321>] find_inode_fast+0xa1/0xc0 > > [<ffffffff8118525f>] iget_locked+0x4f/0x180 > > [<ffffffff811ef9e3>] ext4_iget+0x33/0x9f0 > > [<ffffffff811f6a1c>] ext4_lookup+0xbc/0x160 > > [<ffffffff81174ad0>] lookup_real+0x20/0x60 > > [<ffffffff81177e25>] lookup_open+0x175/0x1d0 > > [<ffffffff8117815e>] do_last+0x2de/0x780 <<< holds i_mutex > > [<ffffffff8117ae9a>] path_openat+0xda/0x400 > > [<ffffffff8117b303>] do_filp_open+0x43/0xa0 > > [<ffffffff81168ee0>] do_sys_open+0x160/0x1e0 > > [<ffffffff81168f9c>] sys_open+0x1c/0x20 > > [<ffffffff81582fe9>] system_call_fastpath+0x16/0x1b > > [<ffffffffffffffff>] 0xffffffffffffffff > > I don't think this has anything to do with LRUs. I am not claiming that. It might be a timing issue which never mattered but it is strange I can reproduce this so easily and repeatedly with the shrinkers patchset applied. As I said earlier, this might be breakage in my -mm tree as well (missing some patch which didn't go via Andrew or misapplied patch). The situation is worsen by the state of linux-next which has some unrelated issues. I really do not want to delay the whole patchset just because of some problem on my side. Do you have any tree that I should try to test? > __wait_on_freeing_inode() only blocks once the inode is being freed > (i.e. I_FREEING is set), and that happens when a lookup is done when > the inode is still in the inode hash. > > I_FREEING is set on the inode at the same time it is removed from > the LRU, and from that point onwards the LRUs play no part in the > inode being freed and anyone waiting on the inode being freed > getting woken. > > The only way I can see this happening, is if there is a dispose list > that is not getting processed properly. e.g., we move a bunch on > inodes to the dispose list setting I_FREEING, then for some reason > it gets dropped on the ground and so the wakeup call doesn't happen > when the inode has been removed from the hash. > > I can't see anywhere in the code that this happens, though, but it > might be some pre-existing race in the inode hash that you are now > triggering because freeing will be happening in parallel on multiple > nodes rather than serialising on a global lock... > > I won't have seen this on XFS stress testing, because it doesn't use > the VFS inode hashes for inode lookups. Given that XFS is not > triggering either problem you are seeing, that makes me think I haven't tested with xfs. > that it might be a pre-existing inode hash lookup/reclaim race > condition, not a LRU problem. > > Cheers, > > Dave. > -- > Dave Chinner > david@fromorbit.com -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2013-06-26 8:15 UTC|newest] Thread overview: 127+ messages / expand[flat|nested] mbox.gz Atom feed top 2013-06-17 14:18 linux-next: slab shrinkers: BUG at mm/list_lru.c:92 Michal Hocko 2013-06-17 14:18 ` Michal Hocko 2013-06-17 15:14 ` Glauber Costa 2013-06-17 15:14 ` Glauber Costa 2013-06-17 15:33 ` Michal Hocko 2013-06-17 15:33 ` Michal Hocko 2013-06-17 16:54 ` Glauber Costa 2013-06-17 16:54 ` Glauber Costa 2013-06-18 7:42 ` Michal Hocko 2013-06-18 7:42 ` Michal Hocko 2013-06-17 21:35 ` Andrew Morton 2013-06-17 21:35 ` Andrew Morton 2013-06-17 22:30 ` Glauber Costa 2013-06-18 2:46 ` Dave Chinner 2013-06-18 2:46 ` Dave Chinner 2013-06-18 6:31 ` Glauber Costa 2013-06-18 6:31 ` Glauber Costa 2013-06-18 8:24 ` Michal Hocko 2013-06-18 8:24 ` Michal Hocko 2013-06-18 10:44 ` Michal Hocko 2013-06-18 10:44 ` Michal Hocko 2013-06-18 13:50 ` Michal Hocko 2013-06-18 13:50 ` Michal Hocko 2013-06-25 2:27 ` Dave Chinner 2013-06-25 2:27 ` Dave Chinner 2013-06-26 8:15 ` Michal Hocko [this message] 2013-06-26 8:15 ` Michal Hocko 2013-06-26 23:24 ` Dave Chinner 2013-06-26 23:24 ` Dave Chinner 2013-06-27 14:54 ` Michal Hocko 2013-06-27 14:54 ` Michal Hocko 2013-06-28 8:39 ` Michal Hocko 2013-06-28 8:39 ` Michal Hocko 2013-06-28 14:31 ` Glauber Costa 2013-06-28 14:31 ` Glauber Costa 2013-06-28 15:12 ` Michal Hocko 2013-06-28 15:12 ` Michal Hocko 2013-06-29 2:55 ` Dave Chinner 2013-06-29 2:55 ` Dave Chinner 2013-06-30 18:33 ` Michal Hocko 2013-07-01 1:25 ` Dave Chinner 2013-07-01 1:25 ` Dave Chinner 2013-07-01 7:50 ` Michal Hocko 2013-07-01 7:50 ` Michal Hocko 2013-07-01 8:10 ` Dave Chinner 2013-07-01 8:10 ` Dave Chinner 2013-07-02 9:22 ` Michal Hocko 2013-07-02 12:19 ` Dave Chinner 2013-07-02 12:19 ` Dave Chinner 2013-07-02 12:44 ` Michal Hocko 2013-07-02 12:44 ` Michal Hocko 2013-07-03 11:24 ` Dave Chinner 2013-07-03 11:24 ` Dave Chinner 2013-07-03 14:08 ` Glauber Costa 2013-07-03 14:08 ` Glauber Costa 2013-07-04 16:36 ` Michal Hocko 2013-07-04 16:36 ` Michal Hocko 2013-07-08 12:53 ` Michal Hocko 2013-07-08 21:04 ` Andrew Morton 2013-07-08 21:04 ` Andrew Morton 2013-07-09 17:34 ` Glauber Costa 2013-07-09 17:34 ` Glauber Costa 2013-07-09 17:51 ` Andrew Morton 2013-07-09 17:51 ` Andrew Morton 2013-07-09 17:32 ` Glauber Costa 2013-07-09 17:32 ` Glauber Costa 2013-07-09 17:50 ` Andrew Morton 2013-07-09 17:50 ` Andrew Morton 2013-07-09 17:57 ` Glauber Costa 2013-07-09 17:57 ` Glauber Costa 2013-07-09 17:57 ` Michal Hocko 2013-07-09 17:57 ` Michal Hocko 2013-07-09 21:39 ` Andrew Morton 2013-07-09 21:39 ` Andrew Morton 2013-07-10 2:31 ` Dave Chinner 2013-07-10 2:31 ` Dave Chinner 2013-07-10 7:34 ` Michal Hocko 2013-07-10 7:34 ` Michal Hocko 2013-07-10 8:06 ` Michal Hocko 2013-07-10 8:06 ` Michal Hocko 2013-07-11 2:26 ` Dave Chinner 2013-07-11 2:26 ` Dave Chinner 2013-07-11 3:03 ` Andrew Morton 2013-07-11 3:03 ` Andrew Morton 2013-07-11 13:23 ` Michal Hocko 2013-07-11 13:23 ` Michal Hocko 2013-07-12 1:42 ` Hugh Dickins 2013-07-12 1:42 ` Hugh Dickins 2013-07-13 3:29 ` Dave Chinner 2013-07-13 3:29 ` Dave Chinner 2013-07-15 9:14 ` Michal Hocko 2013-07-15 9:14 ` Michal Hocko 2013-06-18 6:26 ` Glauber Costa 2013-06-18 8:25 ` Michal Hocko 2013-06-18 8:25 ` Michal Hocko 2013-06-19 7:13 ` Michal Hocko 2013-06-19 7:13 ` Michal Hocko 2013-06-19 7:35 ` Glauber Costa 2013-06-19 7:35 ` Glauber Costa 2013-06-19 8:52 ` Glauber Costa 2013-06-19 8:52 ` Glauber Costa 2013-06-19 13:57 ` Michal Hocko 2013-06-19 13:57 ` Michal Hocko 2013-06-19 14:02 ` Glauber Costa 2013-06-19 14:02 ` Glauber Costa 2013-06-19 14:28 ` Michal Hocko 2013-06-19 14:28 ` Michal Hocko 2013-06-20 14:11 ` Glauber Costa 2013-06-20 14:11 ` Glauber Costa 2013-06-20 15:12 ` Michal Hocko 2013-06-20 15:16 ` Michal Hocko 2013-06-20 15:16 ` Michal Hocko 2013-06-21 9:00 ` Michal Hocko 2013-06-21 9:00 ` Michal Hocko 2013-06-23 11:51 ` Glauber Costa 2013-06-23 11:51 ` Glauber Costa 2013-06-23 11:55 ` Glauber Costa 2013-06-25 2:29 ` Dave Chinner 2013-06-25 2:29 ` Dave Chinner 2013-06-26 8:22 ` Michal Hocko 2013-06-26 8:22 ` Michal Hocko 2013-06-18 8:19 ` Michal Hocko 2013-06-18 8:19 ` Michal Hocko 2013-06-18 8:21 ` Glauber Costa 2013-06-18 8:21 ` Glauber Costa 2013-06-18 8:26 ` Michal Hocko 2013-06-18 8:26 ` Michal Hocko
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20130626081509.GF28748@dhcp22.suse.cz \ --to=mhocko@suse.cz \ --cc=akpm@linux-foundation.org \ --cc=david@fromorbit.com \ --cc=glommer@gmail.com \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.