From: "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
To: linux-mm@kvack.org
Cc: linux-mips@linux-mips.org, linux-sh@vger.kernel.org,
Jan Beulich <jbeulich@novell.com>,
"H. Peter Anvin" <hpa@zytor.com>,
sparclinux@vger.kernel.org,
Andrea Arcangeli <aarcange@redhat.com>,
Andi Kleen <ak@linux.intel.com>,
Robert Richter <robert.richter@amd.com>,
x86@kernel.org, Hugh Dickins <hughd@google.com>,
Ingo Molnar <mingo@redhat.com>, Mel Gorman <mgorman@suse.de>,
Alex Shi <alex.shu@intel.com>,
Thomas Gleixner <tglx@linutronix.de>,
KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>,
Tim Chen <tim.c.chen@linux.intel.com>,
linux-kernel@vger.kernel.org,
Andy Lutomirski <luto@amacapital.net>,
Johannes Weiner <hannes@cmpxchg.org>,
Andrew Morton <akpm@linux-foundation.org>,
linuxppc-dev@lists.ozlabs.org,
"Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
Subject: [PATCH v2 5/6] mm: make clear_huge_page cache clear only around the fault address
Date: Thu, 9 Aug 2012 18:03:02 +0300 [thread overview]
Message-ID: <1344524583-1096-6-git-send-email-kirill.shutemov@linux.intel.com> (raw)
In-Reply-To: <1344524583-1096-1-git-send-email-kirill.shutemov@linux.intel.com>
From: Andi Kleen <ak@linux.intel.com>
Clearing a 2MB huge page will typically blow away several levels
of CPU caches. To avoid this only cache clear the 4K area
around the fault address and use a cache avoiding clears
for the rest of the 2MB area.
Signed-off-by: Andi Kleen <ak@linux.intel.com>
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
mm/memory.c | 30 +++++++++++++++++++++++++++---
1 files changed, 27 insertions(+), 3 deletions(-)
diff --git a/mm/memory.c b/mm/memory.c
index b47199a..e9a75c2 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -3969,18 +3969,35 @@ EXPORT_SYMBOL(might_fault);
#endif
#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS)
+
+#ifndef ARCH_HAS_USER_NOCACHE
+#define ARCH_HAS_USER_NOCACHE 0
+#endif
+
+#if ARCH_HAS_USER_NOCACHE == 0
+#define clear_user_highpage_nocache clear_user_highpage
+#endif
+
static void clear_gigantic_page(struct page *page,
unsigned long addr,
unsigned int pages_per_huge_page)
{
int i;
struct page *p = page;
+ unsigned long vaddr;
+ unsigned long haddr = addr & HPAGE_PMD_MASK;
+ int target = (addr - haddr) >> PAGE_SHIFT;
might_sleep();
+ vaddr = haddr;
for (i = 0; i < pages_per_huge_page;
i++, p = mem_map_next(p, page, i)) {
cond_resched();
- clear_user_highpage(p, addr + i * PAGE_SIZE);
+ vaddr = haddr + i*PAGE_SIZE;
+ if (!ARCH_HAS_USER_NOCACHE || i == target)
+ clear_user_highpage(p, vaddr);
+ else
+ clear_user_highpage_nocache(p, vaddr);
}
}
void clear_huge_page(struct page *page,
@@ -3988,16 +4005,23 @@ void clear_huge_page(struct page *page,
{
int i;
unsigned long haddr = addr & HPAGE_PMD_MASK;
+ unsigned long vaddr;
+ int target = (addr - haddr) >> PAGE_SHIFT;
if (unlikely(pages_per_huge_page > MAX_ORDER_NR_PAGES)) {
- clear_gigantic_page(page, haddr, pages_per_huge_page);
+ clear_gigantic_page(page, addr, pages_per_huge_page);
return;
}
might_sleep();
+ vaddr = haddr;
for (i = 0; i < pages_per_huge_page; i++) {
cond_resched();
- clear_user_highpage(page + i, haddr + i * PAGE_SIZE);
+ vaddr = haddr + i*PAGE_SIZE;
+ if (!ARCH_HAS_USER_NOCACHE || i == target)
+ clear_user_highpage(page + i, vaddr);
+ else
+ clear_user_highpage_nocache(page + i, vaddr);
}
}
--
1.7.7.6
next prev parent reply other threads:[~2012-08-09 15:03 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-08-09 15:02 [PATCH v2 0/6] Avoid cache trashing on clearing huge/gigantic page Kirill A. Shutemov
2012-08-09 15:02 ` [PATCH v2 1/6] THP: Use real address for NUMA policy Kirill A. Shutemov
2012-08-09 15:02 ` [PATCH v2 2/6] mm: make clear_huge_page tolerate non aligned address Kirill A. Shutemov
2012-08-09 15:03 ` [PATCH v2 3/6] THP: Pass real, not rounded, address to clear_huge_page Kirill A. Shutemov
2012-08-09 15:03 ` [PATCH v2 4/6] x86: Add clear_page_nocache Kirill A. Shutemov
2012-08-09 15:22 ` Jan Beulich
2012-08-13 11:43 ` Kirill A. Shutemov
2012-08-13 12:02 ` Jan Beulich
2012-08-13 16:27 ` Andi Kleen
2012-08-13 17:04 ` Borislav Petkov
2012-08-13 19:07 ` Kirill A. Shutemov
2012-08-09 15:23 ` H. Peter Anvin
2012-08-09 15:03 ` Kirill A. Shutemov [this message]
2012-08-09 15:03 ` [PATCH v2 6/6] x86: switch the 64bit uncached page clear to SSE/AVX v2 Kirill A. Shutemov
2012-08-09 15:28 ` Jan Beulich
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1344524583-1096-6-git-send-email-kirill.shutemov@linux.intel.com \
--to=kirill.shutemov@linux.intel.com \
--cc=aarcange@redhat.com \
--cc=ak@linux.intel.com \
--cc=akpm@linux-foundation.org \
--cc=alex.shu@intel.com \
--cc=hannes@cmpxchg.org \
--cc=hpa@zytor.com \
--cc=hughd@google.com \
--cc=jbeulich@novell.com \
--cc=kamezawa.hiroyu@jp.fujitsu.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mips@linux-mips.org \
--cc=linux-mm@kvack.org \
--cc=linux-sh@vger.kernel.org \
--cc=linuxppc-dev@lists.ozlabs.org \
--cc=luto@amacapital.net \
--cc=mgorman@suse.de \
--cc=mingo@redhat.com \
--cc=robert.richter@amd.com \
--cc=sparclinux@vger.kernel.org \
--cc=tglx@linutronix.de \
--cc=tim.c.chen@linux.intel.com \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).