Re: [patch 0/9] mm: thrash detection-based file cache sizing v3

From: Ozgun Erdogan <ozgun@citusdata.com>
To: Andrew Morton <akpm@linux-foundation.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>,
	linux-mm@kvack.org, Andi Kleen <andi@firstfloor.org>,
	Andrea Arcangeli <aarcange@redhat.com>,
	Greg Thelen <gthelen@google.com>,
	Christoph Hellwig <hch@infradead.org>,
	Hugh Dickins <hughd@google.com>, Jan Kara <jack@suse.cz>,
	KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>,
	Mel Gorman <mgorman@suse.de>, Minchan Kim <minchan.kim@gmail.com>,
	Peter Zijlstra <peterz@infradead.org>,
	Rik van Riel <riel@redhat.com>,
	Michel Lespinasse <walken@google.com>,
	Seth Jennings <sjenning@linux.vnet.ibm.com>,
	Roman Gushchin <klamm@yandex-team.ru>,
	Metin Doslu <metin@citusdata.com>,
	linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: [patch 0/9] mm: thrash detection-based file cache sizing v3
Date: Fri, 9 Aug 2013 18:39:20 -0700	[thread overview]
Message-ID: <CAAxz3Xsn_m5CxudayR+ChTZhS04rGChK+9QM2SWwt1vV_1aDdA@mail.gmail.com> (raw)
In-Reply-To: <20130809155309.71d93380425ef8e19c0ff44c@linux-foundation.org>

[-- Attachment #1: Type: text/plain, Size: 1835 bytes --]

Hi Andrew,

One common use case where this is really helpful is in data analytics.
Assume that you regularly analyze some chunk of data, say one month's
worth, and you run SQL queries or MapReduce jobs on this data. Let's also
assume you want to serve the current month's data from memory.

Going with an example, let's say data for March takes 60% of total memory.
You run queries over that data, and it gets pulled into the active list.
Comes next month, you want to query April's data (which again holds 60% of
memory). Since analytic queries sequentially walk over data, April's data
never becomes active, doesn't get pulled into memory, and you're stuck with
serving queries from disk.

To overcome this issue, you could regularly drop the page cache, or advise
customers to provision clusters whose cumulative memory is 2x the working
set. Neither are that ideal. My understanding is that this patch resolves
this issue, but then again my knowledge of the Linux memory manager is
pretty limited. So please call off if I'm off here.

Thanks,
Ozgun

On Fri, Aug 9, 2013 at 3:53 PM, Andrew Morton <akpm@linux-foundation.org>wrote:

> On Tue,  6 Aug 2013 18:44:01 -0400 Johannes Weiner <hannes@cmpxchg.org>
> wrote:
>
> > This series solves the problem by maintaining a history of pages
> > evicted from the inactive list, enabling the VM to tell streaming IO
> > from thrashing and rebalance the page cache lists when appropriate.
>
> Looks nice. The lack of testing results is conspicuous ;)
>
> It only really solves the problem in the case where
>
>         size-of-inactive-list < size-of-working-set < size-of-total-memory
>
> yes?  In fact less than that, because the active list presumably
> doesn't get shrunk to zero (how far *can* it go?).  I wonder how many
> workloads fit into those constraints in the real world.
>
>

[-- Attachment #2: Type: text/html, Size: 2408 bytes --]