From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754056Ab0HRQHw (ORCPT ); Wed, 18 Aug 2010 12:07:52 -0400 Received: from mga01.intel.com ([192.55.52.88]:17545 "EHLO mga01.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753783Ab0HRQHu (ORCPT ); Wed, 18 Aug 2010 12:07:50 -0400 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="4.56,228,1280732400"; d="scan'208";a="829390281" Date: Thu, 19 Aug 2010 00:07:31 +0800 From: Wu Fengguang To: Johannes Weiner Cc: Matthew Wilcox , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Li Shaohua Subject: Re: [TESTCASE] Clean pages clogging the VM Message-ID: <20100818160731.GA15002@localhost> References: <20100809133000.GB6981@wil.cx> <20100817195001.GA18817@linux.intel.com> <20100818141308.GD1779@cmpxchg.org> <20100818160613.GE9431@localhost> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20100818160613.GE9431@localhost> User-Agent: Mutt/1.5.20 (2009-06-14) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, Aug 19, 2010 at 12:06:13AM +0800, Wu Fengguang wrote: > On Wed, Aug 18, 2010 at 04:13:08PM +0200, Johannes Weiner wrote: > > Hi Matthew, > > > > On Tue, Aug 17, 2010 at 03:50:01PM -0400, Matthew Wilcox wrote: > > > > > > No comment on this? Was it just that I posted it during the VM summit? > > > > I have not forgotten about it. I just have a hard time reproducing > > those extreme stalls you observed. > > > > Running that test on a 2.5GHz machine with 2G of memory gives me > > stalls of up to half a second. The patchset I am experimenting with > > gets me down to peaks of 70ms, but it needs further work. > > > > Mapped file pages get two rounds on the LRU list, so once the VM > > starts scanning, it has to go through all of them twice and can only > > reclaim them on the second encounter. > > > > At that point, since we scan without making progress, we start waiting > > for IO, which is not happening in this case, so we sit there until a > > timeout expires. > > Right, this could lead to some 1s stall. Shaohua and me also noticed > this when investigating the responsiveness issues. And we are wondering > if it makes sense to do congestion_wait() only when the bdi is really > congested? There are no IO underway anyway in this case. > > > This stupid-waiting can be improved, and I am working on that. But > > Yeah, stupid waiting :) > > > since I can not reproduce your observations, I don't know if this is > > the (sole) source of the problem. Can I send you patches? > > Sure. > > > > On Mon, Aug 09, 2010 at 09:30:00AM -0400, Matthew Wilcox wrote: > > > > > > > > This testcase shows some odd behaviour from the Linux VM. > > > > > > > > It creates a 1TB sparse file, mmaps it, and randomly reads locations > > > > in it. Due to the file being entirely sparse, the VM allocates new pages > > > > and zeroes them. Initially, it runs very fast, taking on the order of > > > > 2.7 to 4us per page fault. Eventually, the VM runs out of free pages, > > > > and starts doing huge amounts of work trying to figure out which of > > > > these clean pages to throw away. > > > > This is similar to one of my test cases for: > > > > 6457474 vmscan: detect mapped file pages used only once > > 31c0569 vmscan: drop page_mapping_inuse() > > dfc8d63 vmscan: factor out page reference checks > > > > because the situation was even worse before (see the series > > description in dfc8d63). Maybe asking the obvious, but the kernel you > > tested on did include those commits, right? > > > > And just to be sure, I sent you a test-patch to disable the used-once > > detection on IRC the other day. Did you have time to run it yet? > > Here it is again: > > > > diff --git a/mm/vmscan.c b/mm/vmscan.c > > index 9c7e57c..c757bba 100644 > > --- a/mm/vmscan.c > > +++ b/mm/vmscan.c > > @@ -584,6 +584,7 @@ static enum page_references page_check_references(struct page *page, > > return PAGEREF_RECLAIM; > > > > if (referenced_ptes) { > > + return PAGEREF_ACTIVATE; > > How come page activation helps? > > > if (PageAnon(page)) > > return PAGEREF_ACTIVATE; > > /* > > > > > > > > In my testing with a 6GB machine and 2.9GHz CPU, one in every > > > > 15,000 page faults takes over a second, and one in every 40,000 > > > > page faults take over seven seconds! > > > > > > > > This test-case demonstrates a problem that occurs with a read-mostly > > > > mmap of a file on very fast media. I wouldn't like to see a solution > > > > that special-cases zeroed pages. I think userspace has done its part > > > > to tell the kernel what's it's doing by calling madvise(MADV_RANDOM). > > > > This ought to be enough to hint to the kernel that it should be eagerly > > > > throwing away pages in this VMA. > > > > We can probably do something like the following, but I am not sure > > this is a good fix, either. How many applications are using > > madvise()? > > Heh, it sounds crazy to rip random read pages, though it does help to > produce a FAST test case. > > > --- a/mm/rmap.c > > +++ b/mm/rmap.c > > @@ -495,7 +495,7 @@ int page_referenced_one(struct page *pag > > * mapping is already gone, the unmap path will have > > * set PG_referenced or activated the page. > > */ > > - if (likely(!VM_SequentialReadHint(vma))) > > + if (likely(!(vma->vm_flags & (VM_SEQ_READ|VM_RAND_READ)))) > > referenced++; > > } > > Thanks, > Fengguang > From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail190.messagelabs.com (mail190.messagelabs.com [216.82.249.51]) by kanga.kvack.org (Postfix) with SMTP id AA78D6B01F1 for ; Wed, 18 Aug 2010 12:08:53 -0400 (EDT) Date: Thu, 19 Aug 2010 00:07:31 +0800 From: Wu Fengguang Subject: Re: [TESTCASE] Clean pages clogging the VM Message-ID: <20100818160731.GA15002@localhost> References: <20100809133000.GB6981@wil.cx> <20100817195001.GA18817@linux.intel.com> <20100818141308.GD1779@cmpxchg.org> <20100818160613.GE9431@localhost> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20100818160613.GE9431@localhost> Sender: owner-linux-mm@kvack.org To: Johannes Weiner Cc: Matthew Wilcox , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Li Shaohua List-ID: On Thu, Aug 19, 2010 at 12:06:13AM +0800, Wu Fengguang wrote: > On Wed, Aug 18, 2010 at 04:13:08PM +0200, Johannes Weiner wrote: > > Hi Matthew, > > > > On Tue, Aug 17, 2010 at 03:50:01PM -0400, Matthew Wilcox wrote: > > > > > > No comment on this? Was it just that I posted it during the VM summit? > > > > I have not forgotten about it. I just have a hard time reproducing > > those extreme stalls you observed. > > > > Running that test on a 2.5GHz machine with 2G of memory gives me > > stalls of up to half a second. The patchset I am experimenting with > > gets me down to peaks of 70ms, but it needs further work. > > > > Mapped file pages get two rounds on the LRU list, so once the VM > > starts scanning, it has to go through all of them twice and can only > > reclaim them on the second encounter. > > > > At that point, since we scan without making progress, we start waiting > > for IO, which is not happening in this case, so we sit there until a > > timeout expires. > > Right, this could lead to some 1s stall. Shaohua and me also noticed > this when investigating the responsiveness issues. And we are wondering > if it makes sense to do congestion_wait() only when the bdi is really > congested? There are no IO underway anyway in this case. > > > This stupid-waiting can be improved, and I am working on that. But > > Yeah, stupid waiting :) > > > since I can not reproduce your observations, I don't know if this is > > the (sole) source of the problem. Can I send you patches? > > Sure. > > > > On Mon, Aug 09, 2010 at 09:30:00AM -0400, Matthew Wilcox wrote: > > > > > > > > This testcase shows some odd behaviour from the Linux VM. > > > > > > > > It creates a 1TB sparse file, mmaps it, and randomly reads locations > > > > in it. Due to the file being entirely sparse, the VM allocates new pages > > > > and zeroes them. Initially, it runs very fast, taking on the order of > > > > 2.7 to 4us per page fault. Eventually, the VM runs out of free pages, > > > > and starts doing huge amounts of work trying to figure out which of > > > > these clean pages to throw away. > > > > This is similar to one of my test cases for: > > > > 6457474 vmscan: detect mapped file pages used only once > > 31c0569 vmscan: drop page_mapping_inuse() > > dfc8d63 vmscan: factor out page reference checks > > > > because the situation was even worse before (see the series > > description in dfc8d63). Maybe asking the obvious, but the kernel you > > tested on did include those commits, right? > > > > And just to be sure, I sent you a test-patch to disable the used-once > > detection on IRC the other day. Did you have time to run it yet? > > Here it is again: > > > > diff --git a/mm/vmscan.c b/mm/vmscan.c > > index 9c7e57c..c757bba 100644 > > --- a/mm/vmscan.c > > +++ b/mm/vmscan.c > > @@ -584,6 +584,7 @@ static enum page_references page_check_references(struct page *page, > > return PAGEREF_RECLAIM; > > > > if (referenced_ptes) { > > + return PAGEREF_ACTIVATE; > > How come page activation helps? > > > if (PageAnon(page)) > > return PAGEREF_ACTIVATE; > > /* > > > > > > > > In my testing with a 6GB machine and 2.9GHz CPU, one in every > > > > 15,000 page faults takes over a second, and one in every 40,000 > > > > page faults take over seven seconds! > > > > > > > > This test-case demonstrates a problem that occurs with a read-mostly > > > > mmap of a file on very fast media. I wouldn't like to see a solution > > > > that special-cases zeroed pages. I think userspace has done its part > > > > to tell the kernel what's it's doing by calling madvise(MADV_RANDOM). > > > > This ought to be enough to hint to the kernel that it should be eagerly > > > > throwing away pages in this VMA. > > > > We can probably do something like the following, but I am not sure > > this is a good fix, either. How many applications are using > > madvise()? > > Heh, it sounds crazy to rip random read pages, though it does help to > produce a FAST test case. > > > --- a/mm/rmap.c > > +++ b/mm/rmap.c > > @@ -495,7 +495,7 @@ int page_referenced_one(struct page *pag > > * mapping is already gone, the unmap path will have > > * set PG_referenced or activated the page. > > */ > > - if (likely(!VM_SequentialReadHint(vma))) > > + if (likely(!(vma->vm_flags & (VM_SEQ_READ|VM_RAND_READ)))) > > referenced++; > > } > > Thanks, > Fengguang > -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org