From: Peter Zijlstra <a.p.zijlstra@chello.nl> To: Andrea Arcangeli <aarcange@redhat.com>, Avi Kivity <avi@redhat.com>, Thomas Gleixner <tglx@linutronix.de>, Rik van Riel <riel@redhat.com>, Ingo Molnar <mingo@elte.hu>, akpm@linux-foundation.org, Linus Torvalds <torvalds@linux-foundation.org> Cc: linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, Benjamin Herrenschmidt <benh@kernel.crashing.org>, David Miller <davem@davemloft.net>, Hugh Dickins <hugh.dickins@tiscali.co.uk>, Mel Gorman <mel@csn.ul.ie>, Nick Piggin <npiggin@suse.de>, Peter Zijlstra <a.p.zijlstra@chello.nl>, Paul McKenney <paulmck@linux.vnet.ibm.com>, Yanmin Zhang <yanmin_zhang@linux.intel.com>, Stephen Rothwell <sfr@canb.auug.org.au> Subject: [PATCH 20/20] mm: Optimize page_lock_anon_vma() fast-path Date: Sat, 28 Aug 2010 16:16:57 +0200 [thread overview] Message-ID: <20100828142456.804579956@chello.nl> (raw) In-Reply-To: 20100828141637.421594670@chello.nl [-- Attachment #1: mm-opt-page_lock_anon_vma.patch --] [-- Type: text/plain, Size: 2724 bytes --] Optimize the page_lock_anon_vma() fast path to be one LOCKed op, instead of two. Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl> --- mm/rmap.c | 65 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 61 insertions(+), 4 deletions(-) Index: linux-2.6/mm/rmap.c =================================================================== --- linux-2.6.orig/mm/rmap.c +++ linux-2.6/mm/rmap.c @@ -353,20 +353,69 @@ out: return anon_vma; } +/* + * Similar to page_get_anon_vma() except it locks the anon_vma. + * + * Its a little more complex as it tries to keep the fast path to a single + * atomic op -- the trylock. If we fail the trylock, we fall back to getting a + * reference like with page_get_anon_vma() and then block on the mutex. + */ struct anon_vma *page_lock_anon_vma(struct page *page) { - struct anon_vma *anon_vma = page_get_anon_vma(page); + struct anon_vma *anon_vma = NULL; + unsigned long anon_mapping; + + rcu_read_lock(); + anon_mapping = (unsigned long) ACCESS_ONCE(page->mapping); + if ((anon_mapping & PAGE_MAPPING_FLAGS) != PAGE_MAPPING_ANON) + goto out; + if (!page_mapped(page)) + goto out; + + anon_vma = (struct anon_vma *) (anon_mapping - PAGE_MAPPING_ANON); + if (mutex_trylock(&anon_vma->root->lock)) { + /* + * If we observe a !0 refcount, then holding the lock ensures + * the anon_vma will not go away, see __put_anon_vma(). + */ + if (!atomic_read(&anon_vma->refcount)) { + anon_vma_unlock(anon_vma); + anon_vma = NULL; + } + goto out; + } + + /* trylock failed, we got to sleep */ + if (!atomic_inc_not_zero(&anon_vma->refcount)) { + anon_vma = NULL; + goto out; + } + + /* we pinned the anon_vma, its safe to sleep */ + rcu_read_unlock(); + anon_vma_lock(anon_vma); + + if (atomic_dec_and_test(&anon_vma->refcount)) { + /* + * Oops, we held the last refcount, release the lock + * and bail -- can't simply use put_anon_vma() because + * we'll deadlock on the anon_vma_lock() recursion. + */ + anon_vma_unlock(anon_vma); + __put_anon_vma(anon_vma); + anon_vma = NULL; + } - if (anon_vma) - anon_vma_lock(anon_vma); + return anon_vma; +out: + rcu_read_unlock(); return anon_vma; } void page_unlock_anon_vma(struct anon_vma *anon_vma) { anon_vma_unlock(anon_vma); - put_anon_vma(anon_vma); } /* @@ -1458,6 +1507,14 @@ int try_to_munlock(struct page *page) void __put_anon_vma(struct anon_vma *anon_vma) { + /* + * Synchronize against page_lock_anon_vma() such that + * we can safely hold the lock without the anon_vma getting + * freed. + */ + anon_vma_lock(anon_vma); + anon_vma_unlock(anon_vma); + if (anon_vma->root != anon_vma) put_anon_vma(anon_vma->root); anon_vma_free(anon_vma);
WARNING: multiple messages have this Message-ID (diff)
From: Peter Zijlstra <a.p.zijlstra@chello.nl> To: Andrea Arcangeli <aarcange@redhat.com>, Avi Kivity <avi@redhat.com>, Thomas Gleixner <tglx@linutronix.de>, Rik van Riel <riel@redhat.com>, Ingo Molnar <mingo@elte.hu>, akpm@linux-fou Cc: linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, Benjamin Herrenschmidt <benh@kernel.crashing.org>, David Miller <davem@davemloft.net>, Hugh Dickins <hugh.dickins@tiscali.co.uk>, Mel Gorman <mel@csn.ul.ie>, Nick Piggin <npiggin@suse.de>, Peter Zijlstra <a.p.zijlstra@chello.nl>, Paul McKenney <paulmck@linux.vnet.ibm.com>, Yanmin Zhang <yanmin_zhang@linux.intel.com>, Stephen Rothwell <sfr@canb.auug.org.au> Subject: [PATCH 20/20] mm: Optimize page_lock_anon_vma() fast-path Date: Sat, 28 Aug 2010 16:16:57 +0200 [thread overview] Message-ID: <20100828142456.804579956@chello.nl> (raw) In-Reply-To: 20100828141637.421594670@chello.nl [-- Attachment #1: mm-opt-page_lock_anon_vma.patch --] [-- Type: text/plain, Size: 2722 bytes --] Optimize the page_lock_anon_vma() fast path to be one LOCKed op, instead of two. Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl> --- mm/rmap.c | 65 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 61 insertions(+), 4 deletions(-) Index: linux-2.6/mm/rmap.c =================================================================== --- linux-2.6.orig/mm/rmap.c +++ linux-2.6/mm/rmap.c @@ -353,20 +353,69 @@ out: return anon_vma; } +/* + * Similar to page_get_anon_vma() except it locks the anon_vma. + * + * Its a little more complex as it tries to keep the fast path to a single + * atomic op -- the trylock. If we fail the trylock, we fall back to getting a + * reference like with page_get_anon_vma() and then block on the mutex. + */ struct anon_vma *page_lock_anon_vma(struct page *page) { - struct anon_vma *anon_vma = page_get_anon_vma(page); + struct anon_vma *anon_vma = NULL; + unsigned long anon_mapping; + + rcu_read_lock(); + anon_mapping = (unsigned long) ACCESS_ONCE(page->mapping); + if ((anon_mapping & PAGE_MAPPING_FLAGS) != PAGE_MAPPING_ANON) + goto out; + if (!page_mapped(page)) + goto out; + + anon_vma = (struct anon_vma *) (anon_mapping - PAGE_MAPPING_ANON); + if (mutex_trylock(&anon_vma->root->lock)) { + /* + * If we observe a !0 refcount, then holding the lock ensures + * the anon_vma will not go away, see __put_anon_vma(). + */ + if (!atomic_read(&anon_vma->refcount)) { + anon_vma_unlock(anon_vma); + anon_vma = NULL; + } + goto out; + } + + /* trylock failed, we got to sleep */ + if (!atomic_inc_not_zero(&anon_vma->refcount)) { + anon_vma = NULL; + goto out; + } + + /* we pinned the anon_vma, its safe to sleep */ + rcu_read_unlock(); + anon_vma_lock(anon_vma); + + if (atomic_dec_and_test(&anon_vma->refcount)) { + /* + * Oops, we held the last refcount, release the lock + * and bail -- can't simply use put_anon_vma() because + * we'll deadlock on the anon_vma_lock() recursion. + */ + anon_vma_unlock(anon_vma); + __put_anon_vma(anon_vma); + anon_vma = NULL; + } - if (anon_vma) - anon_vma_lock(anon_vma); + return anon_vma; +out: + rcu_read_unlock(); return anon_vma; } void page_unlock_anon_vma(struct anon_vma *anon_vma) { anon_vma_unlock(anon_vma); - put_anon_vma(anon_vma); } /* @@ -1458,6 +1507,14 @@ int try_to_munlock(struct page *page) void __put_anon_vma(struct anon_vma *anon_vma) { + /* + * Synchronize against page_lock_anon_vma() such that + * we can safely hold the lock without the anon_vma getting + * freed. + */ + anon_vma_lock(anon_vma); + anon_vma_unlock(anon_vma); + if (anon_vma->root != anon_vma) put_anon_vma(anon_vma->root); anon_vma_free(anon_vma);
next prev parent reply other threads:[~2010-08-28 14:28 UTC|newest] Thread overview: 78+ messages / expand[flat|nested] mbox.gz Atom feed top 2010-08-28 14:16 [PATCH 00/20] mm: Preemptibility -v4 Peter Zijlstra 2010-08-28 14:16 ` Peter Zijlstra 2010-08-28 14:16 ` [PATCH 01/20] powerpc: Use call_rcu_sched() for pagetables Peter Zijlstra 2010-08-28 14:16 ` Peter Zijlstra 2010-08-31 6:10 ` Benjamin Herrenschmidt 2010-08-28 14:16 ` [PATCH 02/20] mm: Improve page_lock_anon_vma() comment Peter Zijlstra 2010-08-28 14:16 ` Peter Zijlstra 2010-08-28 14:16 ` [PATCH 03/20] mm: Rename drop_anon_vma to put_anon_vma Peter Zijlstra 2010-08-28 14:16 ` Peter Zijlstra 2010-08-28 15:08 ` Pekka Enberg 2010-08-28 14:16 ` [PATCH 04/20] mm: Move anon_vma ref out from under CONFIG_KSM Peter Zijlstra 2010-08-28 14:16 ` Peter Zijlstra 2010-08-28 14:16 ` [PATCH 05/20] mm: Simplify anon_vma refcounts Peter Zijlstra 2010-08-28 14:16 ` Peter Zijlstra 2010-08-28 15:13 ` Pekka Enberg 2010-08-28 14:16 ` [PATCH 06/20] mm: Use refcounts for page_lock_anon_vma() Peter Zijlstra 2010-08-28 14:16 ` Peter Zijlstra 2010-08-28 14:16 ` [PATCH 07/20] mm: Preemptible mmu_gather Peter Zijlstra 2010-08-28 14:16 ` Peter Zijlstra 2010-08-28 14:16 ` [PATCH 08/20] powerpc: " Peter Zijlstra 2010-08-28 14:16 ` Peter Zijlstra 2010-08-31 6:26 ` Benjamin Herrenschmidt 2010-08-31 6:31 ` Benjamin Herrenschmidt 2010-08-31 9:14 ` Peter Zijlstra 2010-08-28 14:16 ` [PATCH 09/20] sparc: " Peter Zijlstra 2010-08-28 14:16 ` Peter Zijlstra 2010-08-28 14:16 ` [PATCH 10/20] s390: preemptible mmu_gather Peter Zijlstra 2010-08-28 14:16 ` Peter Zijlstra 2010-08-28 14:16 ` [PATCH 11/20] arm: Preemptible mmu_gather Peter Zijlstra 2010-08-28 14:16 ` Peter Zijlstra 2010-08-28 14:16 ` [PATCH 12/20] sh: " Peter Zijlstra 2010-08-28 14:16 ` Peter Zijlstra 2010-08-28 14:16 ` [PATCH 13/20] um: " Peter Zijlstra 2010-08-28 14:16 ` Peter Zijlstra 2010-08-28 14:16 ` [PATCH 14/20] ia64: " Peter Zijlstra 2010-08-28 14:16 ` Peter Zijlstra 2010-08-30 15:44 ` Peter Zijlstra 2010-08-28 14:16 ` [PATCH 15/20] mm, powerpc: Move the RCU page-table freeing into generic code Peter Zijlstra 2010-08-28 14:16 ` Peter Zijlstra 2010-08-28 14:16 ` [PATCH 16/20] lockdep, mutex: Provide mutex_lock_nest_lock Peter Zijlstra 2010-08-28 14:16 ` Peter Zijlstra 2010-08-28 14:16 ` [PATCH 17/20] mutex: Provide mutex_is_contended Peter Zijlstra 2010-08-28 14:16 ` Peter Zijlstra 2010-08-28 14:16 ` [PATCH 18/20] mm: Convert i_mmap_lock and anon_vma->lock to mutexes Peter Zijlstra 2010-08-28 14:16 ` Peter Zijlstra 2010-08-28 14:16 ` [PATCH 19/20] mm: Extended batches for generic mmu_gather Peter Zijlstra 2010-08-28 14:16 ` Peter Zijlstra 2010-08-28 14:16 ` Peter Zijlstra [this message] 2010-08-28 14:16 ` [PATCH 20/20] mm: Optimize page_lock_anon_vma() fast-path Peter Zijlstra 2010-08-28 14:32 ` [PATCH 00/20] mm: Preemptibility -v4 Peter Zijlstra 2010-08-28 22:28 ` David Miller 2010-08-28 22:41 ` Peter Zijlstra 2010-08-28 14:56 ` Piotr Hosowicz 2010-08-28 15:10 ` Peter Zijlstra 2010-08-28 15:17 ` Piotr Hosowicz 2010-08-28 15:23 ` Peter Zijlstra 2010-08-28 16:01 ` Piotr Hosowicz 2010-08-29 12:46 ` Piotr Hosowicz 2010-08-29 13:37 ` Peter Zijlstra 2010-08-29 13:43 ` Piotr Hosowicz 2010-08-31 14:02 ` Piotr Hosowicz 2010-08-31 14:14 ` Piotr Hosowicz 2010-09-02 14:53 ` Piotr Hosowicz 2010-08-28 15:19 ` Pekka Enberg 2010-08-28 15:27 ` Peter Zijlstra [not found] ` <AANLkTikSm2Mq8hGNac9rpFH-3pvryw2kW57EP45Ny6Vp@mail.gmail.com> 2010-09-14 5:36 ` Alex,Shi 2010-09-14 7:42 ` Peter Zijlstra 2010-10-18 11:24 [PATCH 00/20] mm: Preemptibility -v5 Peter Zijlstra 2010-10-18 11:24 ` [PATCH 20/20] mm: Optimize page_lock_anon_vma() fast-path Peter Zijlstra 2010-10-18 11:24 ` Peter Zijlstra 2011-04-01 12:12 [PATCH 00/20] mm: Preemptibility -v10 Peter Zijlstra 2011-04-01 12:13 ` [PATCH 20/20] mm: Optimize page_lock_anon_vma() fast-path Peter Zijlstra 2011-04-01 12:13 ` Peter Zijlstra 2011-04-01 12:13 ` Peter Zijlstra 2011-04-19 20:08 ` Andrew Morton 2011-04-19 20:08 ` Andrew Morton 2011-04-20 12:38 ` Peter Zijlstra 2011-04-20 12:38 ` Peter Zijlstra 2011-04-20 15:00 ` Peter Zijlstra 2011-04-20 15:00 ` Peter Zijlstra
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20100828142456.804579956@chello.nl \ --to=a.p.zijlstra@chello.nl \ --cc=aarcange@redhat.com \ --cc=akpm@linux-foundation.org \ --cc=avi@redhat.com \ --cc=benh@kernel.crashing.org \ --cc=davem@davemloft.net \ --cc=hugh.dickins@tiscali.co.uk \ --cc=linux-arch@vger.kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=mel@csn.ul.ie \ --cc=mingo@elte.hu \ --cc=npiggin@suse.de \ --cc=paulmck@linux.vnet.ibm.com \ --cc=riel@redhat.com \ --cc=sfr@canb.auug.org.au \ --cc=tglx@linutronix.de \ --cc=torvalds@linux-foundation.org \ --cc=yanmin_zhang@linux.intel.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.