From: Rik van Riel <riel@redhat.com> To: Mandeep Singh Baines <msb@chromium.org> Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>, Andrew Morton <akpm@linux-foundation.org>, Mel Gorman <mel@csn.ul.ie>, Minchan Kim <minchan.kim@gmail.com>, Johannes Weiner <hannes@cmpxchg.org>, linux-kernel@vger.kernel.org, linux-mm@kvack.org, wad@chromium.org, olofj@chromium.org, hughd@chromium.org Subject: Re: [PATCH] RFC: vmscan: add min_filelist_kbytes sysctl for protecting the working set Date: Mon, 01 Nov 2010 14:50:11 -0400 [thread overview] Message-ID: <4CCF0BE3.2090700@redhat.com> (raw) In-Reply-To: <20101101182416.GB31189@google.com> On 11/01/2010 02:24 PM, Mandeep Singh Baines wrote: > Under memory pressure, I see the active list get smaller and smaller. Its > getting smaller because we're scanning it faster and faster, causing more > and more page faults which slows forward progress resulting in the active > list getting smaller still. One way to approach this might to make the > scan rate constant and configurable. It doesn't seem right that we scan > memory faster and faster under low memory. For us, we'd rather OOM than > evict pages that are likely to be accessed again so we'd prefer to make > a conservative estimate as to what belongs in the working set. Other > folks (long computations) might want to reclaim more aggressively. Have you actually read the code? The active file list is only ever scanned when it is larger than the inactive file list. >> Q2: In the above you used min_filelist_kbytes=50000. How do you decide >> such value? Do other users can calculate proper value? >> > > 50M was small enough that we were comfortable with keeping 50M of file pages > in memory and large enough that it is bigger than the working set. I tested > by loading up a bunch of popular web sites in chrome and then observing what > happend when I ran out of memory. With 50M, I saw almost no thrashing and > the system stayed responsive even under low memory. but I wanted to be > conservative since I'm really just guessing. > > Other users could calculate their value by doing something similar. Maybe we can scale this by memory amount? Say, make sure the total amount of page cache in the system is at least 2* as much as the sum of all the zone->pages_high watermarks, and refuse to evict page cache if we have less than that? This may need to be tunable for a few special use cases, like HPC and virtual machine hosting nodes, but it may just do the right thing for everybody else. Another alternative could be to really slow down the reclaiming of page cache once we hit this level, so virt hosts and HPC nodes can still decrease the page cache to something really small ... but only if it is not being used. Andrew, could a hack like the above be "good enough"? Anybody - does the above hack inspire you to come up with an even better idea?
WARNING: multiple messages have this Message-ID (diff)
From: Rik van Riel <riel@redhat.com> To: Mandeep Singh Baines <msb@chromium.org> Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>, Andrew Morton <akpm@linux-foundation.org>, Mel Gorman <mel@csn.ul.ie>, Minchan Kim <minchan.kim@gmail.com>, Johannes Weiner <hannes@cmpxchg.org>, linux-kernel@vger.kernel.org, linux-mm@kvack.org, wad@chromium.org, olofj@chromium.org, hughd@chromium.org Subject: Re: [PATCH] RFC: vmscan: add min_filelist_kbytes sysctl for protecting the working set Date: Mon, 01 Nov 2010 14:50:11 -0400 [thread overview] Message-ID: <4CCF0BE3.2090700@redhat.com> (raw) In-Reply-To: <20101101182416.GB31189@google.com> On 11/01/2010 02:24 PM, Mandeep Singh Baines wrote: > Under memory pressure, I see the active list get smaller and smaller. Its > getting smaller because we're scanning it faster and faster, causing more > and more page faults which slows forward progress resulting in the active > list getting smaller still. One way to approach this might to make the > scan rate constant and configurable. It doesn't seem right that we scan > memory faster and faster under low memory. For us, we'd rather OOM than > evict pages that are likely to be accessed again so we'd prefer to make > a conservative estimate as to what belongs in the working set. Other > folks (long computations) might want to reclaim more aggressively. Have you actually read the code? The active file list is only ever scanned when it is larger than the inactive file list. >> Q2: In the above you used min_filelist_kbytes=50000. How do you decide >> such value? Do other users can calculate proper value? >> > > 50M was small enough that we were comfortable with keeping 50M of file pages > in memory and large enough that it is bigger than the working set. I tested > by loading up a bunch of popular web sites in chrome and then observing what > happend when I ran out of memory. With 50M, I saw almost no thrashing and > the system stayed responsive even under low memory. but I wanted to be > conservative since I'm really just guessing. > > Other users could calculate their value by doing something similar. Maybe we can scale this by memory amount? Say, make sure the total amount of page cache in the system is at least 2* as much as the sum of all the zone->pages_high watermarks, and refuse to evict page cache if we have less than that? This may need to be tunable for a few special use cases, like HPC and virtual machine hosting nodes, but it may just do the right thing for everybody else. Another alternative could be to really slow down the reclaiming of page cache once we hit this level, so virt hosts and HPC nodes can still decrease the page cache to something really small ... but only if it is not being used. Andrew, could a hack like the above be "good enough"? Anybody - does the above hack inspire you to come up with an even better idea? -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/ Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2010-11-01 18:51 UTC|newest] Thread overview: 55+ messages / expand[flat|nested] mbox.gz Atom feed top 2010-10-28 19:15 [PATCH] RFC: vmscan: add min_filelist_kbytes sysctl for protecting the working set Mandeep Singh Baines 2010-10-28 19:15 ` Mandeep Singh Baines 2010-10-28 20:10 ` Andrew Morton 2010-10-28 20:10 ` Andrew Morton 2010-10-28 22:03 ` Mandeep Singh Baines 2010-10-28 22:03 ` Mandeep Singh Baines 2010-10-28 23:28 ` Minchan Kim 2010-10-28 23:28 ` Minchan Kim 2010-10-28 23:29 ` Minchan Kim 2010-10-28 23:29 ` Minchan Kim 2010-10-29 0:04 ` KAMEZAWA Hiroyuki 2010-10-29 0:04 ` KAMEZAWA Hiroyuki 2010-10-29 0:28 ` Minchan Kim 2010-10-29 0:28 ` Minchan Kim 2010-10-28 21:30 ` Rik van Riel 2010-10-28 21:30 ` Rik van Riel 2010-10-28 22:13 ` Mandeep Singh Baines 2010-10-28 22:13 ` Mandeep Singh Baines 2010-11-01 7:05 ` KOSAKI Motohiro 2010-11-01 7:05 ` KOSAKI Motohiro 2010-11-01 18:24 ` Mandeep Singh Baines 2010-11-01 18:24 ` Mandeep Singh Baines 2010-11-01 18:50 ` Rik van Riel [this message] 2010-11-01 18:50 ` Rik van Riel 2010-11-01 19:43 ` Mandeep Singh Baines 2010-11-01 19:43 ` Mandeep Singh Baines 2010-11-02 3:11 ` Rik van Riel 2010-11-02 3:11 ` Rik van Riel 2010-11-03 0:48 ` Minchan Kim 2010-11-03 2:00 ` Rik van Riel 2010-11-03 2:00 ` Rik van Riel 2010-11-03 3:03 ` Minchan Kim 2010-11-03 3:03 ` Minchan Kim 2010-11-03 11:41 ` Rik van Riel 2010-11-03 11:41 ` Rik van Riel 2010-11-03 15:42 ` Minchan Kim 2010-11-03 15:42 ` Minchan Kim 2010-11-03 22:40 ` Mandeep Singh Baines 2010-11-03 22:40 ` Mandeep Singh Baines 2010-11-03 23:49 ` Minchan Kim 2010-11-03 23:49 ` Minchan Kim 2010-11-04 15:30 ` Rik van Riel 2010-11-04 15:30 ` Rik van Riel 2010-11-08 21:55 ` Mandeep Singh Baines 2010-11-08 21:55 ` Mandeep Singh Baines 2010-11-09 2:49 ` KOSAKI Motohiro 2010-11-09 2:49 ` KOSAKI Motohiro 2010-11-01 23:46 ` Minchan Kim 2010-11-01 23:46 ` Minchan Kim 2010-11-04 1:52 ` Mandeep Singh Baines 2010-11-04 1:52 ` Mandeep Singh Baines 2010-11-05 2:36 ` Minchan Kim 2010-11-05 2:36 ` Minchan Kim 2010-11-09 2:53 ` KOSAKI Motohiro 2010-11-09 2:53 ` KOSAKI Motohiro
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=4CCF0BE3.2090700@redhat.com \ --to=riel@redhat.com \ --cc=akpm@linux-foundation.org \ --cc=hannes@cmpxchg.org \ --cc=hughd@chromium.org \ --cc=kosaki.motohiro@jp.fujitsu.com \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=mel@csn.ul.ie \ --cc=minchan.kim@gmail.com \ --cc=msb@chromium.org \ --cc=olofj@chromium.org \ --cc=wad@chromium.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.