linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [patch 0/5] refault distance-based file cache sizing
@ 2012-05-01  8:41 Johannes Weiner
  2012-05-01  8:41 ` [patch 1/5] mm: readahead: move radix tree hole searching here Johannes Weiner
                   ` (6 more replies)
  0 siblings, 7 replies; 35+ messages in thread
From: Johannes Weiner @ 2012-05-01  8:41 UTC (permalink / raw)
  To: linux-mm
  Cc: Rik van Riel, Andrea Arcangeli, Peter Zijlstra, Mel Gorman,
	Andrew Morton, Minchan Kim, Hugh Dickins, KOSAKI Motohiro,
	linux-fsdevel, linux-kernel

Hi,

our file data caching implementation is done by having an inactive
list of pages that have yet to prove worth keeping around and an
active list of pages that already did.  The question is how to balance
those two lists against each other.

On one hand, the space for inactive pages needs to be big enough so
that they have the necessary time in memory to gather the references
required for an activation.  On the other hand, we want an active list
big enough to hold all data that is frequently used, if possible, to
protect it from streams of less frequently used/once used pages.

Our current balancing ("active can't grow larger than inactive") does
not really work too well.  We have people complaining that the working
set is not well protected from used-once file cache, and other people
complaining that we don't adapt to changes in the workingset and
protect stale pages in other cases.

This series stores file cache eviction information in the vacated page
cache radix tree slots and uses it on refault to see if the pages
currently on the active list need to have their status challenged.

A fully activated file set that occupies 85% of memory is successfully
detected as stale when another file set of equal size is accessed for
a few times (4-5).  The old kernel would never adapt to the second
one.  If the new set is bigger than memory, the old set is left
untouched, where the old kernel would shrink the old set to half of
memory and leave it at that.  Tested on a multi-zone single-node
machine.

More testing is obviously required, but I first wanted some opinions
at this point.  Is there fundamental disagreement with the concept?
With the implementation?

Memcg hard limit reclaim is not converted (anymore, ripped it out to
focus on the global case first) and it still does the 50/50 balancing
between lists, but this will be re-added in the next version.

Patches are based on 3.3.

 fs/btrfs/compression.c     |   10 +-
 fs/btrfs/extent_io.c       |    3 +-
 fs/cachefiles/rdwr.c       |   26 +++--
 fs/ceph/xattr.c            |    2 +-
 fs/inode.c                 |    7 +-
 fs/logfs/readwrite.c       |    9 +-
 fs/nilfs2/inode.c          |    6 +-
 fs/ntfs/file.c             |   11 ++-
 fs/splice.c                |   10 +-
 include/linux/mm.h         |    8 ++
 include/linux/mmzone.h     |    7 ++
 include/linux/pagemap.h    |   54 ++++++++---
 include/linux/pagevec.h    |    3 +
 include/linux/radix-tree.h |    4 -
 include/linux/sched.h      |    1 +
 include/linux/shmem_fs.h   |    1 +
 include/linux/swap.h       |    7 ++
 lib/radix-tree.c           |   75 ---------------
 mm/Makefile                |    1 +
 mm/filemap.c               |  222 ++++++++++++++++++++++++++++++++++----------
 mm/memcontrol.c            |    3 +
 mm/mincore.c               |   20 +++-
 mm/page_alloc.c            |    7 ++
 mm/readahead.c             |   51 +++++++++-
 mm/shmem.c                 |   89 +++---------------
 mm/swap.c                  |   23 +++++
 mm/truncate.c              |   73 +++++++++++---
 mm/vmscan.c                |   80 +++++++++-------
 mm/vmstat.c                |    4 +
 mm/workingset.c            |  174 ++++++++++++++++++++++++++++++++++
 net/ceph/messenger.c       |    2 +-
 net/ceph/pagelist.c        |    4 +-
 net/ceph/pagevec.c         |    2 +-
 33 files changed, 682 insertions(+), 317 deletions(-)

Thanks,
Johannes

^ permalink raw reply	[flat|nested] 35+ messages in thread

end of thread, other threads:[~2012-05-18 15:31 UTC | newest]

Thread overview: 35+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-05-01  8:41 [patch 0/5] refault distance-based file cache sizing Johannes Weiner
2012-05-01  8:41 ` [patch 1/5] mm: readahead: move radix tree hole searching here Johannes Weiner
2012-05-01 21:06   ` Rik van Riel
2012-05-01  8:41 ` [patch 2/5] mm + fs: prepare for non-page entries in page cache Johannes Weiner
2012-05-01 19:02   ` Andrew Morton
2012-05-01 20:15     ` Johannes Weiner
2012-05-01 20:24       ` Andrew Morton
2012-05-01 21:14         ` Rik van Riel
2012-05-01 21:29         ` Johannes Weiner
2012-05-01  8:41 ` [patch 3/5] mm + fs: store shadow pages " Johannes Weiner
2012-05-01  8:41 ` [patch 4/5] mm + fs: provide refault distance to page cache instantiations Johannes Weiner
2012-05-01  9:30   ` Peter Zijlstra
2012-05-01  9:55     ` Johannes Weiner
2012-05-01  9:58       ` Peter Zijlstra
2012-05-01  8:41 ` [patch 5/5] mm: refault distance-based file cache sizing Johannes Weiner
2012-05-01 14:13   ` Minchan Kim
2012-05-01 15:38     ` Johannes Weiner
2012-05-02  5:21       ` Minchan Kim
2012-05-02  1:57   ` Andrea Arcangeli
2012-05-02  6:23     ` Johannes Weiner
2012-05-02 15:11       ` Andrea Arcangeli
2012-05-01 19:08 ` [patch 0/5] " Andrew Morton
2012-05-01 21:19   ` Rik van Riel
2012-05-01 21:26     ` Andrew Morton
2012-05-02  1:10       ` Andrea Arcangeli
2012-05-03 13:15       ` Johannes Weiner
2012-05-16  5:25 ` nai.xia
2012-05-16  6:51   ` Johannes Weiner
2012-05-16 12:56     ` nai.xia
2012-05-17 21:08       ` Johannes Weiner
2012-05-18  3:44         ` Nai Xia
2012-05-18 15:07           ` Rik van Riel
2012-05-18 15:30             ` Nai Xia
2012-05-17 13:11   ` Rik van Riel
2012-05-18  5:03     ` Nai Xia

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).