From: James Bottomley <James.Bottomley@HansenPartnership.com> To: Waiman Long <longman@redhat.com>, Michal Hocko <mhocko@kernel.org> Cc: Alexander Viro <viro@zeniv.linux.org.uk>, Jonathan Corbet <corbet@lwn.net>, "Luis R. Rodriguez" <mcgrof@kernel.org>, Kees Cook <keescook@chromium.org>, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-doc@vger.kernel.org, Linus Torvalds <torvalds@linux-foundation.org>, Jan Kara <jack@suse.cz>, "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>, Andrew Morton <akpm@linux-foundation.org>, Ingo Molnar <mingo@kernel.org>, Miklos Szeredi <mszeredi@redhat.com>, Matthew Wilcox <willy@infradead.org>, Larry Woodman <lwoodman@redhat.com>, "Wangkai (Kevin C)" <wangkai86@huawei.com> Subject: Re: [PATCH v6 0/7] fs/dcache: Track & limit # of negative dentries Date: Wed, 11 Jul 2018 10:42:27 -0700 [thread overview] Message-ID: <1531330947.3260.13.camel@HansenPartnership.com> (raw) In-Reply-To: <9f24c043-1fca-ee86-d609-873a7a8f7a64@redhat.com> On Wed, 2018-07-11 at 11:13 -0400, Waiman Long wrote: > On 07/11/2018 06:21 AM, Michal Hocko wrote: > > On Tue 10-07-18 12:09:17, Waiman Long wrote: [...] > > > I am going to reduce the granularity of each unit to 1/1000 of > > > the total system memory so that for large system with TB of > > > memory, a smaller amount of memory can be specified. > > > > It is just a matter of time for this to be too coarse as well. > > The goal is to not have too much memory being consumed by negative > dentries and also the limit won't be reached by regular daily > activities. So a limit of 1/1000 of the total system memory will be > good enough on large memory system even if the absolute number is > really big. OK, I think the reason we're going round and round here without converging is that one of the goals of the mm subsystem is to manage all of our cached objects and to it the negative (and positive) dentries simply look like a clean cache of objects. Right at the moment mm manages them in the same way it manages all the other caches, a lot of which suffer from the "you can cause lots of allocations to artificially grow them" problem. So the main question is why doesn't the current mm control of the caches work well enough for dentries? What are the problems you're seeing that mm should be catching? If you can answer this, then we could get on to whether a separate shrinker, cache separation or some fix in mm itself is the right answer. What you say above is based on a conclusion: limiting dentries improves the system performance. What we're asking for is evidence for that conclusion so we can explore whether the same would go for any of our other system caches (so do we have a global cache management problem or is it only the dentry cache?) James
WARNING: multiple messages have this Message-ID (diff)
From: James Bottomley <James.Bottomley@HansenPartnership.com> To: Waiman Long <longman@redhat.com>, Michal Hocko <mhocko@kernel.org> Cc: Alexander Viro <viro@zeniv.linux.org.uk>, Jonathan Corbet <corbet@lwn.net>, "Luis R. Rodriguez" <mcgrof@kernel.org>, Kees Cook <keescook@chromium.org>, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-doc@vger.kernel.org, Linus Torvalds <torvalds@linux-foundation.org>, Jan Kara <jack@suse.cz>, "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>, Andrew Morton <akpm@linux-foundation.org>, Ingo Molnar <mingo@kernel.org>, Miklos Szeredi <mszeredi@redhat.com>, Matthew Wilcox <willy@infradead.org>, Larry Woodman <lwoodman@redhat.com>, "Wangkai (Kevin C)" <wangkai86@huawei.com> Subject: Re: [PATCH v6 0/7] fs/dcache: Track & limit # of negative dentries Date: Wed, 11 Jul 2018 10:42:27 -0700 [thread overview] Message-ID: <1531330947.3260.13.camel@HansenPartnership.com> (raw) In-Reply-To: <9f24c043-1fca-ee86-d609-873a7a8f7a64@redhat.com> On Wed, 2018-07-11 at 11:13 -0400, Waiman Long wrote: > On 07/11/2018 06:21 AM, Michal Hocko wrote: > > On Tue 10-07-18 12:09:17, Waiman Long wrote: [...] > > > I am going to reduce the granularity of each unit to 1/1000 of > > > the total system memory so that for large system with TB of > > > memory, a smaller amount of memory can be specified. > > > > It is just a matter of time for this to be too coarse as well. > > The goal is to not have too much memory being consumed by negative > dentries and also the limit won't be reached by regular daily > activities. So a limit of 1/1000 of the total system memory will be > good enough on large memory system even if the absolute number is > really big. OK, I think the reason we're going round and round here without converging is that one of the goals of the mm subsystem is to manage all of our cached objects and to it the negative (and positive) dentries simply look like a clean cache of objects. Right at the moment mm manages them in the same way it manages all the other caches, a lot of which suffer from the "you can cause lots of allocations to artificially grow them" problem. So the main question is why doesn't the current mm control of the caches work well enough for dentries? What are the problems you're seeing that mm should be catching? If you can answer this, then we could get on to whether a separate shrinker, cache separation or some fix in mm itself is the right answer. What you say above is based on a conclusion: limiting dentries improves the system performance. What we're asking for is evidence for that conclusion so we can explore whether the same would go for any of our other system caches (so do we have a global cache management problem or is it only the dentry cache?) James -- To unsubscribe from this list: send the line "unsubscribe linux-doc" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
next prev parent reply other threads:[~2018-07-11 17:42 UTC|newest] Thread overview: 114+ messages / expand[flat|nested] mbox.gz Atom feed top 2018-07-06 19:32 [PATCH v6 0/7] fs/dcache: Track & limit # of negative dentries Waiman Long 2018-07-06 19:32 ` Waiman Long 2018-07-06 19:32 ` [PATCH v6 1/7] fs/dcache: Track & report number " Waiman Long 2018-07-06 19:32 ` Waiman Long 2018-07-06 19:32 ` [PATCH v6 2/7] fs/dcache: Add sysctl parameter neg-dentry-pc as a soft limit on " Waiman Long 2018-07-06 19:32 ` Waiman Long 2018-07-06 19:32 ` [PATCH v6 3/7] fs/dcache: Enable automatic pruning of " Waiman Long 2018-07-06 19:32 ` Waiman Long 2018-07-06 19:32 ` [PATCH v6 4/7] fs/dcache: Spread negative dentry pruning across multiple CPUs Waiman Long 2018-07-06 19:32 ` Waiman Long 2018-07-06 19:32 ` [PATCH v6 5/7] fs/dcache: Add negative dentries to LRU head initially Waiman Long 2018-07-06 19:32 ` Waiman Long 2018-07-06 19:32 ` [PATCH v6 6/7] fs/dcache: Allow optional enforcement of negative dentry limit Waiman Long 2018-07-06 19:32 ` Waiman Long 2018-07-06 19:32 ` [PATCH v6 7/7] fs/dcache: Allow deconfiguration of negative dentry code to reduce kernel size Waiman Long 2018-07-06 19:32 ` Waiman Long 2018-07-06 21:54 ` Eric Biggers 2018-07-06 21:54 ` Eric Biggers 2018-07-06 22:28 ` [PATCH v6 0/7] fs/dcache: Track & limit # of negative dentries Al Viro 2018-07-06 22:28 ` Al Viro 2018-07-07 3:02 ` Waiman Long 2018-07-07 3:02 ` Waiman Long 2018-07-09 8:19 ` Michal Hocko 2018-07-09 8:19 ` Michal Hocko 2018-07-09 16:01 ` Waiman Long 2018-07-09 16:01 ` Waiman Long 2018-07-10 14:27 ` Michal Hocko 2018-07-10 14:27 ` Michal Hocko 2018-07-10 16:09 ` Waiman Long 2018-07-10 16:09 ` Waiman Long 2018-07-11 10:21 ` Michal Hocko 2018-07-11 10:21 ` Michal Hocko 2018-07-11 15:13 ` Waiman Long 2018-07-11 15:13 ` Waiman Long 2018-07-11 17:42 ` James Bottomley [this message] 2018-07-11 17:42 ` James Bottomley 2018-07-11 17:42 ` James Bottomley 2018-07-11 19:07 ` Waiman Long 2018-07-11 19:07 ` Waiman Long 2018-07-11 19:21 ` James Bottomley 2018-07-11 19:21 ` James Bottomley 2018-07-11 19:21 ` James Bottomley 2018-07-11 19:21 ` James Bottomley 2018-07-12 15:54 ` Waiman Long 2018-07-12 15:54 ` Waiman Long 2018-07-12 16:04 ` James Bottomley 2018-07-12 16:04 ` James Bottomley 2018-07-12 16:04 ` James Bottomley 2018-07-12 16:04 ` James Bottomley 2018-07-12 16:26 ` Waiman Long 2018-07-12 16:26 ` Waiman Long 2018-07-12 17:33 ` James Bottomley 2018-07-12 17:33 ` James Bottomley 2018-07-12 17:33 ` James Bottomley 2018-07-12 17:33 ` James Bottomley 2018-07-13 15:32 ` Waiman Long 2018-07-13 15:32 ` Waiman Long 2018-07-12 16:49 ` Matthew Wilcox 2018-07-12 16:49 ` Matthew Wilcox 2018-07-12 17:21 ` James Bottomley 2018-07-12 17:21 ` James Bottomley 2018-07-12 17:21 ` James Bottomley 2018-07-12 17:21 ` James Bottomley 2018-07-12 18:06 ` Linus Torvalds 2018-07-12 19:57 ` James Bottomley 2018-07-12 19:57 ` James Bottomley 2018-07-12 19:57 ` James Bottomley 2018-07-12 19:57 ` James Bottomley 2018-07-13 0:36 ` Dave Chinner 2018-07-13 0:36 ` Dave Chinner 2018-07-13 15:46 ` James Bottomley 2018-07-13 15:46 ` James Bottomley 2018-07-13 15:46 ` James Bottomley 2018-07-13 15:46 ` James Bottomley 2018-07-13 23:17 ` Dave Chinner 2018-07-13 23:17 ` Dave Chinner 2018-07-13 23:17 ` Dave Chinner 2018-07-13 23:17 ` Dave Chinner 2018-07-16 9:10 ` Michal Hocko 2018-07-16 9:10 ` Michal Hocko 2018-07-16 14:42 ` James Bottomley 2018-07-16 14:42 ` James Bottomley 2018-07-16 14:42 ` James Bottomley 2018-07-16 14:42 ` James Bottomley 2018-07-16 9:09 ` Michal Hocko 2018-07-16 9:09 ` Michal Hocko 2018-07-16 9:12 ` Michal Hocko 2018-07-16 9:12 ` Michal Hocko 2018-07-16 12:41 ` Matthew Wilcox 2018-07-16 12:41 ` Matthew Wilcox 2018-07-16 23:40 ` Andrew Morton 2018-07-16 23:40 ` Andrew Morton 2018-07-17 1:30 ` Matthew Wilcox 2018-07-17 1:30 ` Matthew Wilcox 2018-07-17 8:33 ` Michal Hocko 2018-07-17 8:33 ` Michal Hocko 2018-07-19 0:33 ` Dave Chinner 2018-07-19 0:33 ` Dave Chinner 2018-07-19 8:45 ` Michal Hocko 2018-07-19 8:45 ` Michal Hocko 2018-07-19 9:13 ` Jan Kara 2018-07-19 9:13 ` Jan Kara 2018-07-18 18:39 ` Waiman Long 2018-07-18 18:39 ` Waiman Long 2018-07-18 16:17 ` Waiman Long 2018-07-18 16:17 ` Waiman Long 2018-07-19 8:48 ` Michal Hocko 2018-07-19 8:48 ` Michal Hocko 2018-07-12 8:48 ` Michal Hocko 2018-07-12 8:48 ` Michal Hocko 2018-07-12 16:12 ` Waiman Long 2018-07-12 16:12 ` Waiman Long 2018-07-12 23:16 ` Andrew Morton 2018-07-12 23:16 ` Andrew Morton
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=1531330947.3260.13.camel@HansenPartnership.com \ --to=james.bottomley@hansenpartnership.com \ --cc=akpm@linux-foundation.org \ --cc=corbet@lwn.net \ --cc=jack@suse.cz \ --cc=keescook@chromium.org \ --cc=linux-doc@vger.kernel.org \ --cc=linux-fsdevel@vger.kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=longman@redhat.com \ --cc=lwoodman@redhat.com \ --cc=mcgrof@kernel.org \ --cc=mhocko@kernel.org \ --cc=mingo@kernel.org \ --cc=mszeredi@redhat.com \ --cc=paulmck@linux.vnet.ibm.com \ --cc=torvalds@linux-foundation.org \ --cc=viro@zeniv.linux.org.uk \ --cc=wangkai86@huawei.com \ --cc=willy@infradead.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.