* [RFC PATCH 0/6] powerpc/mm/book3s64: Support for split pmd ptlock
@ 2018-02-14 13:50 Aneesh Kumar K.V
2018-02-14 13:50 ` [RFC PATCH 1/6] powerpc/mm: Rename pte fragment functions Aneesh Kumar K.V
` (5 more replies)
0 siblings, 6 replies; 7+ messages in thread
From: Aneesh Kumar K.V @ 2018-02-14 13:50 UTC (permalink / raw)
To: benh, paulus, mpe, Anton Blanchard, Nicholas Piggin
Cc: linuxppc-dev, Aneesh Kumar K.V
This patch series add split pmd pagetable lock for book3s64. nohash64 also should
be able to switch to this. I need to workout the code dependency. This series
also migh have broken the build on platforms otherthan book3s64. I am sending this early
to get feedback on whether we should continue with the approach.
We switch the pmd allocator to use something similar to what we already use for
level 4 pagetable allocation. We get an order 0 page and divide that to fragments
and hand over fragments when we get request for a pmd pagetable. The pmd lock is
now stashed in the struct page backing the allocated page.
The series should help in reducing lock contention on mm->page_table_lock.
Aneesh Kumar K.V (6):
powerpc/mm: Rename pte fragment functions
powerpc/mm/4k: Switch 4k pagesize config to use pagetable fragment
powerpc/mm: Implement helpers for pagetable fragment support at PMD
level
powerpc/mm: Simplify the rcu callback for page table free
powerpc/mm: Use page fragments for allocation page table at PMD level
enable split pmd ptlock.
arch/powerpc/include/asm/book3s/32/pgalloc.h | 2 +-
arch/powerpc/include/asm/book3s/64/hash-4k.h | 10 +-
arch/powerpc/include/asm/book3s/64/hash-64k.h | 4 +
arch/powerpc/include/asm/book3s/64/mmu.h | 7 +-
arch/powerpc/include/asm/book3s/64/pgalloc.h | 43 ++------
arch/powerpc/include/asm/book3s/64/pgtable.h | 6 ++
arch/powerpc/include/asm/book3s/64/radix-4k.h | 8 ++
arch/powerpc/include/asm/book3s/64/radix-64k.h | 4 +
arch/powerpc/include/asm/pgalloc.h | 9 ++
arch/powerpc/mm/hash_utils_64.c | 2 +
arch/powerpc/mm/init-common.c | 2 -
arch/powerpc/mm/mmu_context_book3s64.c | 39 ++++---
arch/powerpc/mm/pgtable-radix.c | 2 +
arch/powerpc/mm/pgtable_64.c | 144 +++++++++++++++++++++----
arch/powerpc/platforms/Kconfig.cputype | 4 +
15 files changed, 209 insertions(+), 77 deletions(-)
--
2.14.3
^ permalink raw reply [flat|nested] 7+ messages in thread
* [RFC PATCH 1/6] powerpc/mm: Rename pte fragment functions
2018-02-14 13:50 [RFC PATCH 0/6] powerpc/mm/book3s64: Support for split pmd ptlock Aneesh Kumar K.V
@ 2018-02-14 13:50 ` Aneesh Kumar K.V
2018-02-14 13:50 ` [RFC PATCH 2/6] powerpc/mm/4k: Switch 4k pagesize config to use pagetable fragment Aneesh Kumar K.V
` (4 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Aneesh Kumar K.V @ 2018-02-14 13:50 UTC (permalink / raw)
To: benh, paulus, mpe, Anton Blanchard, Nicholas Piggin
Cc: linuxppc-dev, Aneesh Kumar K.V
We rename the alloc and get_from_cache to indicate they operate on pte
fragments. In later patch we will add pmd fragment support.
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
---
arch/powerpc/mm/pgtable_64.c | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/arch/powerpc/mm/pgtable_64.c b/arch/powerpc/mm/pgtable_64.c
index a0f8928c0b86..330ef1d1daf5 100644
--- a/arch/powerpc/mm/pgtable_64.c
+++ b/arch/powerpc/mm/pgtable_64.c
@@ -322,7 +322,7 @@ struct page *pmd_page(pmd_t pmd)
}
#ifdef CONFIG_PPC_64K_PAGES
-static pte_t *get_from_cache(struct mm_struct *mm)
+static pte_t *get_pte_from_cache(struct mm_struct *mm)
{
void *pte_frag, *ret;
@@ -341,7 +341,7 @@ static pte_t *get_from_cache(struct mm_struct *mm)
return (pte_t *)ret;
}
-static pte_t *__alloc_for_cache(struct mm_struct *mm, int kernel)
+static pte_t *__alloc_for_ptecache(struct mm_struct *mm, int kernel)
{
void *ret = NULL;
struct page *page;
@@ -380,12 +380,13 @@ pte_t *pte_fragment_alloc(struct mm_struct *mm, unsigned long vmaddr, int kernel
{
pte_t *pte;
- pte = get_from_cache(mm);
+ pte = get_pte_from_cache(mm);
if (pte)
return pte;
- return __alloc_for_cache(mm, kernel);
+ return __alloc_for_ptecache(mm, kernel);
}
+
#endif /* CONFIG_PPC_64K_PAGES */
void pte_fragment_free(unsigned long *table, int kernel)
--
2.14.3
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [RFC PATCH 2/6] powerpc/mm/4k: Switch 4k pagesize config to use pagetable fragment
2018-02-14 13:50 [RFC PATCH 0/6] powerpc/mm/book3s64: Support for split pmd ptlock Aneesh Kumar K.V
2018-02-14 13:50 ` [RFC PATCH 1/6] powerpc/mm: Rename pte fragment functions Aneesh Kumar K.V
@ 2018-02-14 13:50 ` Aneesh Kumar K.V
2018-02-14 13:50 ` [RFC PATCH 3/6] powerpc/mm: Implement helpers for pagetable fragment support at PMD level Aneesh Kumar K.V
` (3 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Aneesh Kumar K.V @ 2018-02-14 13:50 UTC (permalink / raw)
To: benh, paulus, mpe, Anton Blanchard, Nicholas Piggin
Cc: linuxppc-dev, Aneesh Kumar K.V
4K config use one full page at level 4 of the table. Add support single fragment
and use that for 4K config. This makes both 4k and 64k use the same code path.
Later we will switch pmd to use the page table fragment code
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
---
arch/powerpc/include/asm/book3s/64/hash-4k.h | 6 ++++--
arch/powerpc/include/asm/book3s/64/mmu.h | 6 +++---
arch/powerpc/include/asm/book3s/64/pgalloc.h | 26 --------------------------
arch/powerpc/include/asm/book3s/64/radix-4k.h | 6 ++++++
arch/powerpc/mm/mmu_context_book3s64.c | 10 ----------
arch/powerpc/mm/pgtable_64.c | 10 +++++++---
6 files changed, 20 insertions(+), 44 deletions(-)
diff --git a/arch/powerpc/include/asm/book3s/64/hash-4k.h b/arch/powerpc/include/asm/book3s/64/hash-4k.h
index 67c5475311ee..62098daa3af8 100644
--- a/arch/powerpc/include/asm/book3s/64/hash-4k.h
+++ b/arch/powerpc/include/asm/book3s/64/hash-4k.h
@@ -32,8 +32,10 @@
#define H_PAGE_4K_PFN 0x0
#define H_PAGE_THP_HUGE 0x0
#define H_PAGE_COMBO 0x0
-#define H_PTE_FRAG_NR 0
-#define H_PTE_FRAG_SIZE_SHIFT 0
+
+/* 8 bytes per each pte entry */
+#define H_PTE_FRAG_SIZE_SHIFT (H_PTE_INDEX_SIZE + 3)
+#define H_PTE_FRAG_NR (PAGE_SIZE >> H_PTE_FRAG_SIZE_SHIFT)
/*
* On all 4K setups, remap_4k_pfn() equates to remap_pfn_range()
*/
diff --git a/arch/powerpc/include/asm/book3s/64/mmu.h b/arch/powerpc/include/asm/book3s/64/mmu.h
index 0abeb0e2d616..00a961bc76a9 100644
--- a/arch/powerpc/include/asm/book3s/64/mmu.h
+++ b/arch/powerpc/include/asm/book3s/64/mmu.h
@@ -101,10 +101,10 @@ typedef struct {
#ifdef CONFIG_PPC_SUBPAGE_PROT
struct subpage_prot_table spt;
#endif /* CONFIG_PPC_SUBPAGE_PROT */
-#ifdef CONFIG_PPC_64K_PAGES
- /* for 4K PTE fragment support */
+ /*
+ * pagetable fragment support
+ */
void *pte_frag;
-#endif
#ifdef CONFIG_SPAPR_TCE_IOMMU
struct list_head iommu_group_mem_list;
#endif
diff --git a/arch/powerpc/include/asm/book3s/64/pgalloc.h b/arch/powerpc/include/asm/book3s/64/pgalloc.h
index d6ee7563b09d..8de75c2ae7c9 100644
--- a/arch/powerpc/include/asm/book3s/64/pgalloc.h
+++ b/arch/powerpc/include/asm/book3s/64/pgalloc.h
@@ -167,31 +167,6 @@ static inline pgtable_t pmd_pgtable(pmd_t pmd)
return (pgtable_t)pmd_page_vaddr(pmd);
}
-#ifdef CONFIG_PPC_4K_PAGES
-static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
- unsigned long address)
-{
- return (pte_t *)__get_free_page(GFP_KERNEL | __GFP_ZERO);
-}
-
-static inline pgtable_t pte_alloc_one(struct mm_struct *mm,
- unsigned long address)
-{
- struct page *page;
- pte_t *pte;
-
- pte = (pte_t *)__get_free_page(GFP_KERNEL | __GFP_ZERO | __GFP_ACCOUNT);
- if (!pte)
- return NULL;
- page = virt_to_page(pte);
- if (!pgtable_page_ctor(page)) {
- __free_page(page);
- return NULL;
- }
- return pte;
-}
-#else /* if CONFIG_PPC_64K_PAGES */
-
static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
unsigned long address)
{
@@ -203,7 +178,6 @@ static inline pgtable_t pte_alloc_one(struct mm_struct *mm,
{
return (pgtable_t)pte_fragment_alloc(mm, address, 0);
}
-#endif
static inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte)
{
diff --git a/arch/powerpc/include/asm/book3s/64/radix-4k.h b/arch/powerpc/include/asm/book3s/64/radix-4k.h
index a61aa9cd63ec..14717cfe15e2 100644
--- a/arch/powerpc/include/asm/book3s/64/radix-4k.h
+++ b/arch/powerpc/include/asm/book3s/64/radix-4k.h
@@ -9,5 +9,11 @@
#define RADIX_PMD_INDEX_SIZE 9 /* 1G huge page */
#define RADIX_PUD_INDEX_SIZE 9
#define RADIX_PGD_INDEX_SIZE 13
+/*
+ * One fragment per per page
+ */
+#define RADIX_PTE_FRAG_SIZE_SHIFT (RADIX_PTE_INDEX_SIZE + 3)
+#define RADIX_PTE_FRAG_NR (PAGE_SIZE >> RADIX_PTE_FRAG_SIZE_SHIFT)
+
#endif /* _ASM_POWERPC_PGTABLE_RADIX_4K_H */
diff --git a/arch/powerpc/mm/mmu_context_book3s64.c b/arch/powerpc/mm/mmu_context_book3s64.c
index 929d9ef7083f..b4d795b5162a 100644
--- a/arch/powerpc/mm/mmu_context_book3s64.c
+++ b/arch/powerpc/mm/mmu_context_book3s64.c
@@ -166,9 +166,7 @@ int init_new_context(struct task_struct *tsk, struct mm_struct *mm)
mm->context.id = index;
-#ifdef CONFIG_PPC_64K_PAGES
mm->context.pte_frag = NULL;
-#endif
#ifdef CONFIG_SPAPR_TCE_IOMMU
mm_iommu_init(mm);
#endif
@@ -185,7 +183,6 @@ void __destroy_context(int context_id)
}
EXPORT_SYMBOL_GPL(__destroy_context);
-#ifdef CONFIG_PPC_64K_PAGES
static void destroy_pagetable_page(struct mm_struct *mm)
{
int count;
@@ -206,13 +203,6 @@ static void destroy_pagetable_page(struct mm_struct *mm)
}
}
-#else
-static inline void destroy_pagetable_page(struct mm_struct *mm)
-{
- return;
-}
-#endif
-
void destroy_context(struct mm_struct *mm)
{
#ifdef CONFIG_SPAPR_TCE_IOMMU
diff --git a/arch/powerpc/mm/pgtable_64.c b/arch/powerpc/mm/pgtable_64.c
index 330ef1d1daf5..ff4973565abb 100644
--- a/arch/powerpc/mm/pgtable_64.c
+++ b/arch/powerpc/mm/pgtable_64.c
@@ -321,7 +321,6 @@ struct page *pmd_page(pmd_t pmd)
return virt_to_page(pmd_page_vaddr(pmd));
}
-#ifdef CONFIG_PPC_64K_PAGES
static pte_t *get_pte_from_cache(struct mm_struct *mm)
{
void *pte_frag, *ret;
@@ -360,7 +359,14 @@ static pte_t *__alloc_for_ptecache(struct mm_struct *mm, int kernel)
return NULL;
}
+
ret = page_address(page);
+ /*
+ * if we support only one fragment just return the
+ * allocated page.
+ */
+ if (PTE_FRAG_NR == 1)
+ return ret;
spin_lock(&mm->page_table_lock);
/*
* If we find pgtable_page set, we return
@@ -387,8 +393,6 @@ pte_t *pte_fragment_alloc(struct mm_struct *mm, unsigned long vmaddr, int kernel
return __alloc_for_ptecache(mm, kernel);
}
-#endif /* CONFIG_PPC_64K_PAGES */
-
void pte_fragment_free(unsigned long *table, int kernel)
{
struct page *page = virt_to_page(table);
--
2.14.3
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [RFC PATCH 3/6] powerpc/mm: Implement helpers for pagetable fragment support at PMD level
2018-02-14 13:50 [RFC PATCH 0/6] powerpc/mm/book3s64: Support for split pmd ptlock Aneesh Kumar K.V
2018-02-14 13:50 ` [RFC PATCH 1/6] powerpc/mm: Rename pte fragment functions Aneesh Kumar K.V
2018-02-14 13:50 ` [RFC PATCH 2/6] powerpc/mm/4k: Switch 4k pagesize config to use pagetable fragment Aneesh Kumar K.V
@ 2018-02-14 13:50 ` Aneesh Kumar K.V
2018-02-14 13:50 ` [RFC PATCH 4/6] powerpc/mm: Simplify the rcu callback for page table free Aneesh Kumar K.V
` (2 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Aneesh Kumar K.V @ 2018-02-14 13:50 UTC (permalink / raw)
To: benh, paulus, mpe, Anton Blanchard, Nicholas Piggin
Cc: linuxppc-dev, Aneesh Kumar K.V
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
---
arch/powerpc/include/asm/book3s/64/hash-4k.h | 4 ++
arch/powerpc/include/asm/book3s/64/hash-64k.h | 4 ++
arch/powerpc/include/asm/book3s/64/mmu.h | 1 +
arch/powerpc/include/asm/book3s/64/pgalloc.h | 2 +
arch/powerpc/include/asm/book3s/64/pgtable.h | 6 ++
arch/powerpc/include/asm/book3s/64/radix-4k.h | 2 +
arch/powerpc/include/asm/book3s/64/radix-64k.h | 4 ++
arch/powerpc/mm/hash_utils_64.c | 2 +
arch/powerpc/mm/mmu_context_book3s64.c | 37 +++++++++--
arch/powerpc/mm/pgtable-radix.c | 2 +
arch/powerpc/mm/pgtable_64.c | 85 ++++++++++++++++++++++++++
11 files changed, 143 insertions(+), 6 deletions(-)
diff --git a/arch/powerpc/include/asm/book3s/64/hash-4k.h b/arch/powerpc/include/asm/book3s/64/hash-4k.h
index 62098daa3af8..fc3dc6a93939 100644
--- a/arch/powerpc/include/asm/book3s/64/hash-4k.h
+++ b/arch/powerpc/include/asm/book3s/64/hash-4k.h
@@ -36,6 +36,10 @@
/* 8 bytes per each pte entry */
#define H_PTE_FRAG_SIZE_SHIFT (H_PTE_INDEX_SIZE + 3)
#define H_PTE_FRAG_NR (PAGE_SIZE >> H_PTE_FRAG_SIZE_SHIFT)
+
+#define H_PMD_FRAG_SIZE_SHIFT (H_PMD_INDEX_SIZE + 3)
+#define H_PMD_FRAG_NR (PAGE_SIZE >> H_PMD_FRAG_SIZE_SHIFT)
+
/*
* On all 4K setups, remap_4k_pfn() equates to remap_pfn_range()
*/
diff --git a/arch/powerpc/include/asm/book3s/64/hash-64k.h b/arch/powerpc/include/asm/book3s/64/hash-64k.h
index 0aa4f755b3f6..b8ca64fd2bea 100644
--- a/arch/powerpc/include/asm/book3s/64/hash-64k.h
+++ b/arch/powerpc/include/asm/book3s/64/hash-64k.h
@@ -33,6 +33,10 @@
#define H_PTE_FRAG_SIZE_SHIFT (H_PTE_INDEX_SIZE + 3 + 1)
#define H_PTE_FRAG_NR (PAGE_SIZE >> H_PTE_FRAG_SIZE_SHIFT)
+/* +1 for THP and hugetlb */
+#define H_PMD_FRAG_SIZE_SHIFT (H_PMD_INDEX_SIZE + 3 + 1)
+#define H_PMD_FRAG_NR (PAGE_SIZE >> H_PMD_FRAG_SIZE_SHIFT)
+
#ifndef __ASSEMBLY__
#include <asm/errno.h>
diff --git a/arch/powerpc/include/asm/book3s/64/mmu.h b/arch/powerpc/include/asm/book3s/64/mmu.h
index 00a961bc76a9..ad19651ea10e 100644
--- a/arch/powerpc/include/asm/book3s/64/mmu.h
+++ b/arch/powerpc/include/asm/book3s/64/mmu.h
@@ -105,6 +105,7 @@ typedef struct {
* pagetable fragment support
*/
void *pte_frag;
+ void *pmd_frag;
#ifdef CONFIG_SPAPR_TCE_IOMMU
struct list_head iommu_group_mem_list;
#endif
diff --git a/arch/powerpc/include/asm/book3s/64/pgalloc.h b/arch/powerpc/include/asm/book3s/64/pgalloc.h
index 8de75c2ae7c9..f3838ad0dc6c 100644
--- a/arch/powerpc/include/asm/book3s/64/pgalloc.h
+++ b/arch/powerpc/include/asm/book3s/64/pgalloc.h
@@ -42,7 +42,9 @@ extern struct kmem_cache *pgtable_cache[];
})
extern pte_t *pte_fragment_alloc(struct mm_struct *, unsigned long, int);
+extern pmd_t *pmd_fragment_alloc(struct mm_struct *, unsigned long);
extern void pte_fragment_free(unsigned long *, int);
+extern void pmd_fragment_free(unsigned long *);
extern void pgtable_free_tlb(struct mmu_gather *tlb, void *table, int shift);
#ifdef CONFIG_SMP
extern void __tlb_remove_table(void *_table);
diff --git a/arch/powerpc/include/asm/book3s/64/pgtable.h b/arch/powerpc/include/asm/book3s/64/pgtable.h
index a6b9f1d74600..88d10319adfe 100644
--- a/arch/powerpc/include/asm/book3s/64/pgtable.h
+++ b/arch/powerpc/include/asm/book3s/64/pgtable.h
@@ -265,6 +265,12 @@ extern unsigned long __pte_frag_size_shift;
#define PTE_FRAG_SIZE_SHIFT __pte_frag_size_shift
#define PTE_FRAG_SIZE (1UL << PTE_FRAG_SIZE_SHIFT)
+extern unsigned long __pmd_frag_nr;
+#define PMD_FRAG_NR __pmd_frag_nr
+extern unsigned long __pmd_frag_size_shift;
+#define PMD_FRAG_SIZE_SHIFT __pmd_frag_size_shift
+#define PMD_FRAG_SIZE (1UL << PMD_FRAG_SIZE_SHIFT)
+
#define PTRS_PER_PTE (1 << PTE_INDEX_SIZE)
#define PTRS_PER_PMD (1 << PMD_INDEX_SIZE)
#define PTRS_PER_PUD (1 << PUD_INDEX_SIZE)
diff --git a/arch/powerpc/include/asm/book3s/64/radix-4k.h b/arch/powerpc/include/asm/book3s/64/radix-4k.h
index 14717cfe15e2..863c3e8286fb 100644
--- a/arch/powerpc/include/asm/book3s/64/radix-4k.h
+++ b/arch/powerpc/include/asm/book3s/64/radix-4k.h
@@ -15,5 +15,7 @@
#define RADIX_PTE_FRAG_SIZE_SHIFT (RADIX_PTE_INDEX_SIZE + 3)
#define RADIX_PTE_FRAG_NR (PAGE_SIZE >> RADIX_PTE_FRAG_SIZE_SHIFT)
+#define RADIX_PMD_FRAG_SIZE_SHIFT (RADIX_PMD_INDEX_SIZE + 3)
+#define RADIX_PMD_FRAG_NR (PAGE_SIZE >> RADIX_PMD_FRAG_SIZE_SHIFT)
#endif /* _ASM_POWERPC_PGTABLE_RADIX_4K_H */
diff --git a/arch/powerpc/include/asm/book3s/64/radix-64k.h b/arch/powerpc/include/asm/book3s/64/radix-64k.h
index 830082496876..ccb78ca9d0c5 100644
--- a/arch/powerpc/include/asm/book3s/64/radix-64k.h
+++ b/arch/powerpc/include/asm/book3s/64/radix-64k.h
@@ -16,4 +16,8 @@
*/
#define RADIX_PTE_FRAG_SIZE_SHIFT (RADIX_PTE_INDEX_SIZE + 3)
#define RADIX_PTE_FRAG_NR (PAGE_SIZE >> RADIX_PTE_FRAG_SIZE_SHIFT)
+
+#define RADIX_PMD_FRAG_SIZE_SHIFT (RADIX_PMD_INDEX_SIZE + 3)
+#define RADIX_PMD_FRAG_NR (PAGE_SIZE >> RADIX_PMD_FRAG_SIZE_SHIFT)
+
#endif /* _ASM_POWERPC_PGTABLE_RADIX_64K_H */
diff --git a/arch/powerpc/mm/hash_utils_64.c b/arch/powerpc/mm/hash_utils_64.c
index cf290d415dcd..efdb49124b2a 100644
--- a/arch/powerpc/mm/hash_utils_64.c
+++ b/arch/powerpc/mm/hash_utils_64.c
@@ -1003,6 +1003,8 @@ void __init hash__early_init_mmu(void)
*/
__pte_frag_nr = H_PTE_FRAG_NR;
__pte_frag_size_shift = H_PTE_FRAG_SIZE_SHIFT;
+ __pmd_frag_nr = H_PMD_FRAG_NR;
+ __pmd_frag_size_shift = H_PMD_FRAG_SIZE_SHIFT;
__pte_index_size = H_PTE_INDEX_SIZE;
__pmd_index_size = H_PMD_INDEX_SIZE;
diff --git a/arch/powerpc/mm/mmu_context_book3s64.c b/arch/powerpc/mm/mmu_context_book3s64.c
index b4d795b5162a..ee2b1fa0cbe3 100644
--- a/arch/powerpc/mm/mmu_context_book3s64.c
+++ b/arch/powerpc/mm/mmu_context_book3s64.c
@@ -167,6 +167,7 @@ int init_new_context(struct task_struct *tsk, struct mm_struct *mm)
mm->context.id = index;
mm->context.pte_frag = NULL;
+ mm->context.pmd_frag = NULL;
#ifdef CONFIG_SPAPR_TCE_IOMMU
mm_iommu_init(mm);
#endif
@@ -183,16 +184,11 @@ void __destroy_context(int context_id)
}
EXPORT_SYMBOL_GPL(__destroy_context);
-static void destroy_pagetable_page(struct mm_struct *mm)
+static void pte_frag_destory(void *pte_frag)
{
int count;
- void *pte_frag;
struct page *page;
- pte_frag = mm->context.pte_frag;
- if (!pte_frag)
- return;
-
page = virt_to_page(pte_frag);
/* drop all the pending references */
count = ((unsigned long)pte_frag & ~PAGE_MASK) >> PTE_FRAG_SIZE_SHIFT;
@@ -203,6 +199,35 @@ static void destroy_pagetable_page(struct mm_struct *mm)
}
}
+static void pmd_frag_destory(void *pmd_frag)
+{
+ int count;
+ struct page *page;
+
+ page = virt_to_page(pmd_frag);
+ /* drop all the pending references */
+ count = ((unsigned long)pmd_frag & ~PAGE_MASK) >> PMD_FRAG_SIZE_SHIFT;
+ /* We allow PTE_FRAG_NR fragments from a PTE page */
+ if (page_ref_sub_and_test(page, PMD_FRAG_NR - count)) {
+ pgtable_pmd_page_dtor(page);
+ free_unref_page(page);
+ }
+}
+
+static void destroy_pagetable_page(struct mm_struct *mm)
+{
+ void *frag;
+
+ frag = mm->context.pte_frag;
+ if (frag)
+ pte_frag_destory(frag);
+
+ frag = mm->context.pmd_frag;
+ if (frag)
+ pmd_frag_destory(frag);
+ return;
+}
+
void destroy_context(struct mm_struct *mm)
{
#ifdef CONFIG_SPAPR_TCE_IOMMU
diff --git a/arch/powerpc/mm/pgtable-radix.c b/arch/powerpc/mm/pgtable-radix.c
index f8b3f4a99659..b2f89851d8eb 100644
--- a/arch/powerpc/mm/pgtable-radix.c
+++ b/arch/powerpc/mm/pgtable-radix.c
@@ -559,6 +559,8 @@ void __init radix__early_init_mmu(void)
#endif
__pte_frag_nr = RADIX_PTE_FRAG_NR;
__pte_frag_size_shift = RADIX_PTE_FRAG_SIZE_SHIFT;
+ __pmd_frag_nr = RADIX_PMD_FRAG_NR;
+ __pmd_frag_size_shift = RADIX_PMD_FRAG_SIZE_SHIFT;
if (!firmware_has_feature(FW_FEATURE_LPAR)) {
radix_init_native();
diff --git a/arch/powerpc/mm/pgtable_64.c b/arch/powerpc/mm/pgtable_64.c
index ff4973565abb..0a6859e6ef76 100644
--- a/arch/powerpc/mm/pgtable_64.c
+++ b/arch/powerpc/mm/pgtable_64.c
@@ -114,6 +114,11 @@ unsigned long __pte_frag_nr;
EXPORT_SYMBOL(__pte_frag_nr);
unsigned long __pte_frag_size_shift;
EXPORT_SYMBOL(__pte_frag_size_shift);
+unsigned long __pmd_frag_nr;
+EXPORT_SYMBOL(__pmd_frag_nr);
+unsigned long __pmd_frag_size_shift;
+EXPORT_SYMBOL(__pmd_frag_size_shift);
+
unsigned long ioremap_bot;
#else /* !CONFIG_PPC_BOOK3S_64 */
unsigned long ioremap_bot = IOREMAP_BASE;
@@ -393,6 +398,75 @@ pte_t *pte_fragment_alloc(struct mm_struct *mm, unsigned long vmaddr, int kernel
return __alloc_for_ptecache(mm, kernel);
}
+static pmd_t *get_pmd_from_cache(struct mm_struct *mm)
+{
+ void *pmd_frag, *ret;
+
+ spin_lock(&mm->page_table_lock);
+ ret = mm->context.pmd_frag;
+ if (ret) {
+ pmd_frag = ret + PMD_FRAG_SIZE;
+ /*
+ * If we have taken up all the fragments mark PTE page NULL
+ */
+ if (((unsigned long)pmd_frag & ~PAGE_MASK) == 0)
+ pmd_frag = NULL;
+ mm->context.pmd_frag = pmd_frag;
+ }
+ spin_unlock(&mm->page_table_lock);
+ return (pmd_t *)ret;
+}
+
+static pmd_t *__alloc_for_pmdcache(struct mm_struct *mm)
+{
+ void *ret = NULL;
+ struct page *page;
+ gfp_t gfp = GFP_KERNEL_ACCOUNT | __GFP_ZERO;
+
+ if (mm == &init_mm)
+ gfp &= ~__GFP_ACCOUNT;
+ page = alloc_page(gfp);
+ if (!page)
+ return NULL;
+ if (!pgtable_pmd_page_ctor(page)) {
+ __free_pages(page, 0);
+ return NULL;
+ }
+
+ ret = page_address(page);
+ /*
+ * if we support only one fragment just return the
+ * allocated page.
+ */
+ if (PMD_FRAG_NR == 1)
+ return ret;
+
+ spin_lock(&mm->page_table_lock);
+ /*
+ * If we find pgtable_page set, we return
+ * the allocated page with single fragement
+ * count.
+ */
+ if (likely(!mm->context.pmd_frag)) {
+ set_page_count(page, PMD_FRAG_NR);
+ mm->context.pmd_frag = ret + PMD_FRAG_SIZE;
+ }
+ spin_unlock(&mm->page_table_lock);
+
+ return (pmd_t *)ret;
+}
+
+pmd_t *pmd_fragment_alloc(struct mm_struct *mm, unsigned long vmaddr)
+{
+ pmd_t *pmd;
+
+ pmd = get_pmd_from_cache(mm);
+ if (pmd)
+ return pmd;
+
+ return __alloc_for_pmdcache(mm);
+}
+
void pte_fragment_free(unsigned long *table, int kernel)
{
struct page *page = virt_to_page(table);
@@ -403,6 +477,17 @@ void pte_fragment_free(unsigned long *table, int kernel)
}
}
+void pmd_fragment_free(unsigned long *pmd)
+{
+ struct page *page = virt_to_page(pmd);
+
+ if (put_page_testzero(page)) {
+ pgtable_pmd_page_dtor(page);
+ free_unref_page(page);
+ }
+}
+
+
#ifdef CONFIG_SMP
void pgtable_free_tlb(struct mmu_gather *tlb, void *table, int shift)
{
--
2.14.3
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [RFC PATCH 4/6] powerpc/mm: Simplify the rcu callback for page table free
2018-02-14 13:50 [RFC PATCH 0/6] powerpc/mm/book3s64: Support for split pmd ptlock Aneesh Kumar K.V
` (2 preceding siblings ...)
2018-02-14 13:50 ` [RFC PATCH 3/6] powerpc/mm: Implement helpers for pagetable fragment support at PMD level Aneesh Kumar K.V
@ 2018-02-14 13:50 ` Aneesh Kumar K.V
2018-02-14 13:50 ` [RFC PATCH 5/6] powerpc/mm: Use page fragments for allocation page table at PMD level Aneesh Kumar K.V
2018-02-14 13:50 ` [RFC PATCH 6/6] enable split pmd ptlock Aneesh Kumar K.V
5 siblings, 0 replies; 7+ messages in thread
From: Aneesh Kumar K.V @ 2018-02-14 13:50 UTC (permalink / raw)
To: benh, paulus, mpe, Anton Blanchard, Nicholas Piggin
Cc: linuxppc-dev, Aneesh Kumar K.V
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
---
arch/powerpc/include/asm/book3s/32/pgalloc.h | 2 +-
arch/powerpc/include/asm/book3s/64/pgalloc.h | 6 ++--
arch/powerpc/include/asm/pgalloc.h | 9 ++++++
arch/powerpc/mm/pgtable_64.c | 44 ++++++++++++++++++----------
4 files changed, 41 insertions(+), 20 deletions(-)
diff --git a/arch/powerpc/include/asm/book3s/32/pgalloc.h b/arch/powerpc/include/asm/book3s/32/pgalloc.h
index 9f5c411bce1b..f102920fade4 100644
--- a/arch/powerpc/include/asm/book3s/32/pgalloc.h
+++ b/arch/powerpc/include/asm/book3s/32/pgalloc.h
@@ -137,7 +137,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t table,
unsigned long address)
{
pgtable_page_dtor(table);
- pgtable_free_tlb(tlb, page_address(table), 0);
+ pgtable_free_tlb(tlb, page_address(table), PTE_INDEX);
}
static inline void pgd_ctor(void *addr)
diff --git a/arch/powerpc/include/asm/book3s/64/pgalloc.h b/arch/powerpc/include/asm/book3s/64/pgalloc.h
index f3838ad0dc6c..e5d104caae26 100644
--- a/arch/powerpc/include/asm/book3s/64/pgalloc.h
+++ b/arch/powerpc/include/asm/book3s/64/pgalloc.h
@@ -123,7 +123,7 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud,
* ahead and flush the page walk cache
*/
flush_tlb_pgtable(tlb, address);
- pgtable_free_tlb(tlb, pud, PUD_CACHE_INDEX);
+ pgtable_free_tlb(tlb, pud, PUD_INDEX);
}
static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long addr)
@@ -149,7 +149,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd,
* ahead and flush the page walk cache
*/
flush_tlb_pgtable(tlb, address);
- return pgtable_free_tlb(tlb, pmd, PMD_CACHE_INDEX);
+ return pgtable_free_tlb(tlb, pmd, PMD_INDEX);
}
static inline void pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd,
@@ -199,7 +199,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t table,
* ahead and flush the page walk cache
*/
flush_tlb_pgtable(tlb, address);
- pgtable_free_tlb(tlb, table, 0);
+ pgtable_free_tlb(tlb, table, PTE_INDEX);
}
#define check_pgt_cache() do { } while (0)
diff --git a/arch/powerpc/include/asm/pgalloc.h b/arch/powerpc/include/asm/pgalloc.h
index e11f03007b57..8949e73a028e 100644
--- a/arch/powerpc/include/asm/pgalloc.h
+++ b/arch/powerpc/include/asm/pgalloc.h
@@ -19,6 +19,15 @@ static inline gfp_t pgtable_gfp_flags(struct mm_struct *mm, gfp_t gfp)
#endif /* MODULE */
#define PGALLOC_GFP (GFP_KERNEL | __GFP_ZERO)
+/*
+ * Used as an indicator for rcu callback functions
+ */
+enum pgtable_index {
+ PTE_INDEX = 0,
+ PMD_INDEX,
+ PUD_INDEX,
+ PGD_INDEX,
+};
#ifdef CONFIG_PPC_BOOK3S
#include <asm/book3s/pgalloc.h>
diff --git a/arch/powerpc/mm/pgtable_64.c b/arch/powerpc/mm/pgtable_64.c
index 0a6859e6ef76..db3ee7ab8418 100644
--- a/arch/powerpc/mm/pgtable_64.c
+++ b/arch/powerpc/mm/pgtable_64.c
@@ -487,39 +487,51 @@ void pmd_fragment_free(unsigned long *pmd)
}
}
-
#ifdef CONFIG_SMP
-void pgtable_free_tlb(struct mmu_gather *tlb, void *table, int shift)
+void pgtable_free_tlb(struct mmu_gather *tlb, void *table, int index)
{
unsigned long pgf = (unsigned long)table;
- BUG_ON(shift > MAX_PGTABLE_INDEX_SIZE);
- pgf |= shift;
+ BUG_ON(index > MAX_PGTABLE_INDEX_SIZE);
+ pgf |= index;
tlb_remove_table(tlb, (void *)pgf);
}
void __tlb_remove_table(void *_table)
{
void *table = (void *)((unsigned long)_table & ~MAX_PGTABLE_INDEX_SIZE);
- unsigned shift = (unsigned long)_table & MAX_PGTABLE_INDEX_SIZE;
+ unsigned index = (unsigned long)_table & MAX_PGTABLE_INDEX_SIZE;
- if (!shift)
- /* PTE page needs special handling */
+ switch (index) {
+ case PTE_INDEX:
pte_fragment_free(table, 0);
- else {
- BUG_ON(shift > MAX_PGTABLE_INDEX_SIZE);
- kmem_cache_free(PGT_CACHE(shift), table);
+ break;
+ case PMD_INDEX:
+ pmd_fragment_free(table);
+ break;
+ case PUD_INDEX:
+ kmem_cache_free(PGT_CACHE(PUD_CACHE_INDEX), table);
+ break;
+ default:
+ BUG();
}
}
#else
-void pgtable_free_tlb(struct mmu_gather *tlb, void *table, int shift)
+void pgtable_free_tlb(struct mmu_gather *tlb, void *table, int index)
{
- if (!shift) {
- /* PTE page needs special handling */
+ switch (index) {
+ case PTE_INDEX:
pte_fragment_free(table, 0);
- } else {
- BUG_ON(shift > MAX_PGTABLE_INDEX_SIZE);
- kmem_cache_free(PGT_CACHE(shift), table);
+ break;
+ case PMD_INDEX:
+ kmem_cache_free(PGT_CACHE(PMD_CACHE_INDEX), table);
+ break;
+ case PUD_INDEX:
+ kmem_cache_free(PGT_CACHE(PUD_CACHE_INDEX), table);
+ break;
+ /* We don't free pgd table via RCU callback */
+ default:
+ BUG();
}
}
#endif
--
2.14.3
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [RFC PATCH 5/6] powerpc/mm: Use page fragments for allocation page table at PMD level
2018-02-14 13:50 [RFC PATCH 0/6] powerpc/mm/book3s64: Support for split pmd ptlock Aneesh Kumar K.V
` (3 preceding siblings ...)
2018-02-14 13:50 ` [RFC PATCH 4/6] powerpc/mm: Simplify the rcu callback for page table free Aneesh Kumar K.V
@ 2018-02-14 13:50 ` Aneesh Kumar K.V
2018-02-14 13:50 ` [RFC PATCH 6/6] enable split pmd ptlock Aneesh Kumar K.V
5 siblings, 0 replies; 7+ messages in thread
From: Aneesh Kumar K.V @ 2018-02-14 13:50 UTC (permalink / raw)
To: benh, paulus, mpe, Anton Blanchard, Nicholas Piggin
Cc: linuxppc-dev, Aneesh Kumar K.V
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
---
arch/powerpc/include/asm/book3s/64/pgalloc.h | 9 ++-------
arch/powerpc/mm/init-common.c | 2 --
arch/powerpc/mm/pgtable_64.c | 2 +-
3 files changed, 3 insertions(+), 10 deletions(-)
diff --git a/arch/powerpc/include/asm/book3s/64/pgalloc.h b/arch/powerpc/include/asm/book3s/64/pgalloc.h
index e5d104caae26..f91a8bc1d67f 100644
--- a/arch/powerpc/include/asm/book3s/64/pgalloc.h
+++ b/arch/powerpc/include/asm/book3s/64/pgalloc.h
@@ -128,17 +128,12 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud,
static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long addr)
{
- pmd_t *pmd;
- pmd = kmem_cache_alloc(PGT_CACHE(PMD_CACHE_INDEX),
- pgtable_gfp_flags(mm, GFP_KERNEL));
- memset(pmd, 0, PMD_TABLE_SIZE);
- return pmd;
-
+ return pmd_fragment_alloc(mm, addr);
}
static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd)
{
- kmem_cache_free(PGT_CACHE(PMD_CACHE_INDEX), pmd);
+ pmd_fragment_free((unsigned long *)pmd);
}
static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd,
diff --git a/arch/powerpc/mm/init-common.c b/arch/powerpc/mm/init-common.c
index f92dd8cee3c5..0382df3ef6a8 100644
--- a/arch/powerpc/mm/init-common.c
+++ b/arch/powerpc/mm/init-common.c
@@ -78,8 +78,6 @@ void pgtable_cache_init(void)
{
pgtable_cache_add(PGD_INDEX_SIZE, pgd_ctor);
- if (PMD_CACHE_INDEX && !PGT_CACHE(PMD_CACHE_INDEX))
- pgtable_cache_add(PMD_CACHE_INDEX, pmd_ctor);
/*
* In all current configs, when the PUD index exists it's the
* same size as either the pgd or pmd index except with THP enabled
diff --git a/arch/powerpc/mm/pgtable_64.c b/arch/powerpc/mm/pgtable_64.c
index db3ee7ab8418..05267a8764f5 100644
--- a/arch/powerpc/mm/pgtable_64.c
+++ b/arch/powerpc/mm/pgtable_64.c
@@ -524,7 +524,7 @@ void pgtable_free_tlb(struct mmu_gather *tlb, void *table, int index)
pte_fragment_free(table, 0);
break;
case PMD_INDEX:
- kmem_cache_free(PGT_CACHE(PMD_CACHE_INDEX), table);
+ pmd_fragment_free(table);
break;
case PUD_INDEX:
kmem_cache_free(PGT_CACHE(PUD_CACHE_INDEX), table);
--
2.14.3
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [RFC PATCH 6/6] enable split pmd ptlock.
2018-02-14 13:50 [RFC PATCH 0/6] powerpc/mm/book3s64: Support for split pmd ptlock Aneesh Kumar K.V
` (4 preceding siblings ...)
2018-02-14 13:50 ` [RFC PATCH 5/6] powerpc/mm: Use page fragments for allocation page table at PMD level Aneesh Kumar K.V
@ 2018-02-14 13:50 ` Aneesh Kumar K.V
5 siblings, 0 replies; 7+ messages in thread
From: Aneesh Kumar K.V @ 2018-02-14 13:50 UTC (permalink / raw)
To: benh, paulus, mpe, Anton Blanchard, Nicholas Piggin
Cc: linuxppc-dev, Aneesh Kumar K.V
---
arch/powerpc/platforms/Kconfig.cputype | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/arch/powerpc/platforms/Kconfig.cputype b/arch/powerpc/platforms/Kconfig.cputype
index a429d859f15d..28ba1acb7842 100644
--- a/arch/powerpc/platforms/Kconfig.cputype
+++ b/arch/powerpc/platforms/Kconfig.cputype
@@ -287,6 +287,10 @@ config PPC_STD_MMU_32
def_bool y
depends on PPC_STD_MMU && PPC32
+config ARCH_ENABLE_SPLIT_PMD_PTLOCK
+ def_bool y
+ depends on PPC_BOOK3S_64
+
config PPC_RADIX_MMU
bool "Radix MMU Support"
depends on PPC_BOOK3S_64
--
2.14.3
^ permalink raw reply related [flat|nested] 7+ messages in thread
end of thread, other threads:[~2018-02-14 13:50 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-02-14 13:50 [RFC PATCH 0/6] powerpc/mm/book3s64: Support for split pmd ptlock Aneesh Kumar K.V
2018-02-14 13:50 ` [RFC PATCH 1/6] powerpc/mm: Rename pte fragment functions Aneesh Kumar K.V
2018-02-14 13:50 ` [RFC PATCH 2/6] powerpc/mm/4k: Switch 4k pagesize config to use pagetable fragment Aneesh Kumar K.V
2018-02-14 13:50 ` [RFC PATCH 3/6] powerpc/mm: Implement helpers for pagetable fragment support at PMD level Aneesh Kumar K.V
2018-02-14 13:50 ` [RFC PATCH 4/6] powerpc/mm: Simplify the rcu callback for page table free Aneesh Kumar K.V
2018-02-14 13:50 ` [RFC PATCH 5/6] powerpc/mm: Use page fragments for allocation page table at PMD level Aneesh Kumar K.V
2018-02-14 13:50 ` [RFC PATCH 6/6] enable split pmd ptlock Aneesh Kumar K.V
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.