From: Scott Wood <scottwood@freescale.com>
To: <benh@kernel.crashing.org>
Cc: linuxppc-dev@lists.ozlabs.org
Subject: [PATCH 2/7] powerpc/mm: 64-bit 4k: use a PMD-based virtual page table
Date: Wed, 18 May 2011 16:05:29 -0500 [thread overview]
Message-ID: <20110518210528.GA29524@schlenkerla.am.freescale.net> (raw)
In-Reply-To: <20110518210453.GA29500@schlenkerla.am.freescale.net>
Loads with non-linear access patterns were producing a very high
ratio of recursive pt faults to regular tlb misses. Rather than
choose between a 4-level table walk or a 1-level virtual page table
lookup, use a hybrid scheme with a virtual linear pmd, followed by a
2-level lookup in the normal handler.
This adds about 5 cycles (assuming no cache misses, and e5500 timing)
to a normal TLB miss, but greatly reduces the recursive fault rate
for loads which don't have locality within 2 MiB regions but do have
significant locality within 1 GiB regions. Improvements of close to 50%
were seen on such benchmarks.
Signed-off-by: Scott Wood <scottwood@freescale.com>
---
arch/powerpc/mm/tlb_low_64e.S | 23 +++++++++++++++--------
1 files changed, 15 insertions(+), 8 deletions(-)
diff --git a/arch/powerpc/mm/tlb_low_64e.S b/arch/powerpc/mm/tlb_low_64e.S
index af08922..17726d3 100644
--- a/arch/powerpc/mm/tlb_low_64e.S
+++ b/arch/powerpc/mm/tlb_low_64e.S
@@ -24,7 +24,7 @@
#ifdef CONFIG_PPC_64K_PAGES
#define VPTE_PMD_SHIFT (PTE_INDEX_SIZE+1)
#else
-#define VPTE_PMD_SHIFT (PTE_INDEX_SIZE)
+#define VPTE_PMD_SHIFT 0
#endif
#define VPTE_PUD_SHIFT (VPTE_PMD_SHIFT + PMD_INDEX_SIZE)
#define VPTE_PGD_SHIFT (VPTE_PUD_SHIFT + PUD_INDEX_SIZE)
@@ -185,7 +185,7 @@ normal_tlb_miss:
/* Insert the bottom bits in */
rlwimi r14,r15,0,16,31
#else
- rldicl r14,r16,64-(PAGE_SHIFT-3),PAGE_SHIFT-3+4
+ rldicl r14,r16,64-(PMD_SHIFT-3),PMD_SHIFT-3+4
#endif
sldi r15,r10,60
clrrdi r14,r14,3
@@ -202,6 +202,16 @@ MMU_FTR_SECTION_ELSE
ld r14,0(r10)
ALT_MMU_FTR_SECTION_END_IFSET(MMU_FTR_USE_TLBRSRV)
+#ifndef CONFIG_PPC_64K_PAGES
+ rldicl r15,r16,64-PAGE_SHIFT+3,64-PTE_INDEX_SIZE-3
+ clrrdi r15,r15,3
+
+ cmpldi cr0,r14,0
+ beq normal_tlb_miss_access_fault
+
+ ldx r14,r14,r15
+#endif
+
finish_normal_tlb_miss:
/* Check if required permissions are met */
andc. r15,r11,r14
@@ -353,14 +363,11 @@ END_MMU_FTR_SECTION_IFSET(MMU_FTR_USE_TLBRSRV)
#ifndef CONFIG_PPC_64K_PAGES
/* Get to PUD entry */
rldicl r11,r16,64-VPTE_PUD_SHIFT,64-PUD_INDEX_SIZE-3
- clrrdi r10,r11,3
- ldx r15,r10,r15
- cmpldi cr0,r15,0
- beq virt_page_table_tlb_miss_fault
-#endif /* CONFIG_PPC_64K_PAGES */
-
+#else
/* Get to PMD entry */
rldicl r11,r16,64-VPTE_PMD_SHIFT,64-PMD_INDEX_SIZE-3
+#endif
+
clrrdi r10,r11,3
ldx r15,r10,r15
cmpldi cr0,r15,0
--
1.7.4.1
next prev parent reply other threads:[~2011-05-18 21:05 UTC|newest]
Thread overview: 27+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-05-18 21:04 [PATCH 1/7] powerpc/mm: 64-bit 4k: use page-sized PMDs Scott Wood
2011-05-18 21:05 ` Scott Wood [this message]
2011-05-18 21:33 ` [PATCH 2/7] powerpc/mm: 64-bit 4k: use a PMD-based virtual page table Benjamin Herrenschmidt
2011-05-20 20:57 ` Scott Wood
2011-05-20 22:15 ` Benjamin Herrenschmidt
2011-05-23 18:54 ` Scott Wood
2011-05-23 20:51 ` Benjamin Herrenschmidt
2011-05-23 23:31 ` Scott Wood
2011-05-24 2:52 ` Benjamin Herrenschmidt
2011-05-18 21:05 ` [PATCH 3/7] powerpc/mm: 64-bit tlb miss: get PACA from memory rather than SPR Scott Wood
2011-05-18 21:05 ` [PATCH 4/7] powerpc/mm: 64-bit: Don't load PACA in normal TLB miss exceptions Scott Wood
2011-05-18 21:05 ` [PATCH 5/7] powerpc/mm: 64-bit: don't handle non-standard page sizes Scott Wood
2011-05-18 21:36 ` Benjamin Herrenschmidt
2011-05-18 21:50 ` Scott Wood
2011-05-18 21:54 ` Benjamin Herrenschmidt
2011-05-18 21:05 ` [PATCH 6/7] powerpc/mm: 64-bit: tlb handler micro-optimization Scott Wood
2011-05-18 21:37 ` Benjamin Herrenschmidt
2011-05-18 21:51 ` Scott Wood
2011-05-18 21:54 ` Benjamin Herrenschmidt
2011-05-18 22:27 ` Scott Wood
2011-05-18 21:05 ` [PATCH 7/7] powerpc/e5500: set MMU_FTR_USE_PAIRED_MAS Scott Wood
2011-05-18 21:38 ` Benjamin Herrenschmidt
2011-05-18 21:52 ` Scott Wood
2011-05-18 21:58 ` Benjamin Herrenschmidt
2011-05-18 21:32 ` [PATCH 1/7] powerpc/mm: 64-bit 4k: use page-sized PMDs Benjamin Herrenschmidt
2011-05-18 21:46 ` Scott Wood
2011-05-18 21:52 ` Benjamin Herrenschmidt
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20110518210528.GA29524@schlenkerla.am.freescale.net \
--to=scottwood@freescale.com \
--cc=benh@kernel.crashing.org \
--cc=linuxppc-dev@lists.ozlabs.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.