[PATCH v2 0/4] ARM: mm: HugeTLB + THP support.

[PATCH v2 0/4] ARM: mm: HugeTLB + THP support.

[Steve Capper]

The following patches bring both HugeTLB support and Transparent
HugePage (THP) support to ARM.

Only long descriptors (LPAE) are supported in this series.

The code has been tested on an Arndale board (Exynos 5250).

This patch set is based on 3.10-rc2.

Major changes since the Patch:
 * LPAE code has been separated from non-LPAE code (this series
   is the LPAE code).
 * PROT_NONE support for HugeTLB and THP has been implemented.

Major changes since the RFC:
 * huge pmd sharing removed from the 3-level code as this was
   found to be very rarely, if ever?, used. This allowed for some
   code simplification.

 * hardware pmd bits for 2-levels of paging are now taken from
   mmu.c. Also the mapping code now uses pte/pmd bit helper
   functions rather than the custom pre-processor logic.

Cheers,
-- 
Steve

Catalin Marinas (2):
  ARM: mm: HugeTLB support for LPAE systems.
  ARM: mm: Transparent huge page support for LPAE systems.

Steve Capper (2):
  ARM: mm: correct pte_same behaviour for LPAE.
  ARM: mm: Add support for flushing HugeTLB pages.

 arch/arm/Kconfig                            |   8 +++
 arch/arm/include/asm/hugetlb-3level.h       |  71 +++++++++++++++++++
 arch/arm/include/asm/hugetlb.h              |  84 +++++++++++++++++++++++
 arch/arm/include/asm/pgtable-3level-hwdef.h |   4 ++
 arch/arm/include/asm/pgtable-3level.h       |  88 ++++++++++++++++++++++++
 arch/arm/include/asm/pgtable.h              |   3 +
 arch/arm/include/asm/tlb.h                  |   6 ++
 arch/arm/include/asm/tlbflush.h             |   2 +
 arch/arm/mm/Makefile                        |   1 +
 arch/arm/mm/dma-mapping.c                   |   2 +-
 arch/arm/mm/flush.c                         |  25 ++++---
 arch/arm/mm/fsr-3level.c                    |   4 +-
 arch/arm/mm/hugetlbpage.c                   | 101 ++++++++++++++++++++++++++++
 13 files changed, 386 insertions(+), 13 deletions(-)
 create mode 100644 arch/arm/include/asm/hugetlb-3level.h
 create mode 100644 arch/arm/include/asm/hugetlb.h
 create mode 100644 arch/arm/mm/hugetlbpage.c

-- 
1.8.1.4

[PATCH v2 1/4] ARM: mm: correct pte_same behaviour for LPAE.

[Steve Capper]

For 3 levels of paging the PTE_EXT_NG bit will be set for user
address ptes that are written to a page table but not for ptes
created with mk_pte.

This can cause some comparison tests made by pte_same to fail
spuriously and lead to other problems.

To correct this behaviour, we mask off PTE_EXT_NG for any pte that
is present before running the comparison.

Signed-off-by: Steve Capper <steve.capper@linaro.org>
Reviewed-by: Will Deacon <will.deacon@arm.com>
---
 arch/arm/include/asm/pgtable-3level.h | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/arch/arm/include/asm/pgtable-3level.h b/arch/arm/include/asm/pgtable-3level.h
index 86b8fe3..70f041c 100644
--- a/arch/arm/include/asm/pgtable-3level.h
+++ b/arch/arm/include/asm/pgtable-3level.h
@@ -166,6 +166,23 @@ static inline pmd_t *pmd_offset(pud_t *pud, unsigned long addr)
 		clean_pmd_entry(pmdp);	\
 	} while (0)
 
+/*
+ * For 3 levels of paging the PTE_EXT_NG bit will be set for user address ptes
+ * that are written to a page table but not for ptes created with mk_pte.
+ *
+ * In hugetlb_no_page, a new huge pte (new_pte) is generated and passed to
+ * hugetlb_cow, where it is compared with an entry in a page table.
+ * This comparison test fails erroneously leading ultimately to a memory leak.
+ *
+ * To correct this behaviour, we mask off PTE_EXT_NG for any pte that is
+ * present before running the comparison.
+ */
+#define __HAVE_ARCH_PTE_SAME
+#define pte_same(pte_a,pte_b)	((pte_present(pte_a) ? pte_val(pte_a) & ~PTE_EXT_NG	\
+					: pte_val(pte_a))				\
+				== (pte_present(pte_b) ? pte_val(pte_b) & ~PTE_EXT_NG	\
+					: pte_val(pte_b)))
+
 #define set_pte_ext(ptep,pte,ext) cpu_set_pte_ext(ptep,__pte(pte_val(pte)|(ext)))
 
 #endif /* __ASSEMBLY__ */
-- 
1.8.1.4

[PATCH v2 2/4] ARM: mm: Add support for flushing HugeTLB pages.

[Steve Capper]

On ARM we use the __flush_dcache_page function to flush the dcache
of pages when needed; usually when the PG_dcache_clean bit is unset
and we are setting a PTE.

A HugeTLB page is represented as a compound page consisting of an
array of pages. Thus to flush the dcache of a HugeTLB page, one must
flush more than a single page.

This patch modifies __flush_dcache_page such that all constituent
pages of a HugeTLB page are flushed.

Signed-off-by: Steve Capper <steve.capper@linaro.org>
Reviewed-by: Will Deacon <will.deacon@arm.com>
---
 arch/arm/mm/flush.c | 25 +++++++++++++++----------
 1 file changed, 15 insertions(+), 10 deletions(-)

diff --git a/arch/arm/mm/flush.c b/arch/arm/mm/flush.c
index 0d473cc..3706407 100644
--- a/arch/arm/mm/flush.c
+++ b/arch/arm/mm/flush.c
@@ -17,6 +17,7 @@
 #include <asm/highmem.h>
 #include <asm/smp_plat.h>
 #include <asm/tlbflush.h>
+#include <linux/hugetlb.h>
 
 #include "mm.h"
 
@@ -168,19 +169,23 @@ void __flush_dcache_page(struct address_space *mapping, struct page *page)
 	 * coherent with the kernels mapping.
 	 */
 	if (!PageHighMem(page)) {
-		__cpuc_flush_dcache_area(page_address(page), PAGE_SIZE);
+		size_t page_size = PAGE_SIZE << compound_order(page);
+		__cpuc_flush_dcache_area(page_address(page), page_size);
 	} else {
-		void *addr;
-
+		unsigned long i;
 		if (cache_is_vipt_nonaliasing()) {
-			addr = kmap_atomic(page);
-			__cpuc_flush_dcache_area(addr, PAGE_SIZE);
-			kunmap_atomic(addr);
-		} else {
-			addr = kmap_high_get(page);
-			if (addr) {
+			for (i = 0; i < (1 << compound_order(page)); i++) {
+				void *addr = kmap_atomic(page);
 				__cpuc_flush_dcache_area(addr, PAGE_SIZE);
-				kunmap_high(page);
+				kunmap_atomic(addr);
+			}
+		} else {
+			for (i = 0; i < (1 << compound_order(page)); i++) {
+				void *addr = kmap_high_get(page);
+				if (addr) {
+					__cpuc_flush_dcache_area(addr, PAGE_SIZE);
+					kunmap_high(page);
+				}
 			}
 		}
 	}
-- 
1.8.1.4

[PATCH v2 3/4] ARM: mm: HugeTLB support for LPAE systems.

[Steve Capper]

From: Catalin Marinas <catalin.marinas@arm.com>

This patch adds support for hugetlbfs based on the x86 implementation.
It allows mapping of 2MB sections (see Documentation/vm/hugetlbpage.txt
for usage). The 64K pages configuration is not supported (section size
is 512MB in this case).

Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
[steve.capper at linaro.org: symbolic constants replace numbers in places.
Split up into multiple files, to simplify future non-LPAE support,
removed huge_pmd_share code, as this is very rarely executed,
Added PROT_NONE support].
Signed-off-by: Steve Capper <steve.capper@linaro.org>
Reviewed-by: Will Deacon <will.deacon@arm.com>
---
 arch/arm/Kconfig                            |   4 ++
 arch/arm/include/asm/hugetlb-3level.h       |  71 +++++++++++++++++++
 arch/arm/include/asm/hugetlb.h              |  84 +++++++++++++++++++++++
 arch/arm/include/asm/pgtable-3level-hwdef.h |   2 +
 arch/arm/include/asm/pgtable-3level.h       |  11 +++
 arch/arm/mm/Makefile                        |   1 +
 arch/arm/mm/dma-mapping.c                   |   2 +-
 arch/arm/mm/fsr-3level.c                    |   2 +-
 arch/arm/mm/hugetlbpage.c                   | 101 ++++++++++++++++++++++++++++
 9 files changed, 276 insertions(+), 2 deletions(-)
 create mode 100644 arch/arm/include/asm/hugetlb-3level.h
 create mode 100644 arch/arm/include/asm/hugetlb.h
 create mode 100644 arch/arm/mm/hugetlbpage.c

diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
index 49d993c..860f034 100644
--- a/arch/arm/Kconfig
+++ b/arch/arm/Kconfig
@@ -1707,6 +1707,10 @@ config HW_PERF_EVENTS
 	  Enable hardware performance counter support for perf events. If
 	  disabled, perf events will use software events only.
 
+config SYS_SUPPORTS_HUGETLBFS
+       def_bool y
+       depends on ARM_LPAE
+
 source "mm/Kconfig"
 
 config FORCE_MAX_ZONEORDER
diff --git a/arch/arm/include/asm/hugetlb-3level.h b/arch/arm/include/asm/hugetlb-3level.h
new file mode 100644
index 0000000..d4014fb
--- /dev/null
+++ b/arch/arm/include/asm/hugetlb-3level.h
@@ -0,0 +1,71 @@
+/*
+ * arch/arm/include/asm/hugetlb-3level.h
+ *
+ * Copyright (C) 2012 ARM Ltd.
+ *
+ * Based on arch/x86/include/asm/hugetlb.h.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+ */
+
+#ifndef _ASM_ARM_HUGETLB_3LEVEL_H
+#define _ASM_ARM_HUGETLB_3LEVEL_H
+
+
+/*
+ * If our huge pte is non-zero then mark the valid bit.
+ * This allows pte_present(huge_ptep_get(ptep)) to return true for non-zero
+ * ptes.
+ * (The valid bit is automatically cleared by set_pte_at for PROT_NONE ptes).
+ */
+static inline pte_t huge_ptep_get(pte_t *ptep)
+{
+	pte_t retval = *ptep;
+	if (pte_val(retval))
+		pte_val(retval) |= L_PTE_VALID;
+	return retval;
+}
+
+static inline void set_huge_pte_at(struct mm_struct *mm, unsigned long addr,
+				   pte_t *ptep, pte_t pte)
+{
+	set_pte_at(mm, addr, ptep, pte);
+}
+
+static inline void huge_ptep_clear_flush(struct vm_area_struct *vma,
+					 unsigned long addr, pte_t *ptep)
+{
+	ptep_clear_flush(vma, addr, ptep);
+}
+
+static inline void huge_ptep_set_wrprotect(struct mm_struct *mm,
+					   unsigned long addr, pte_t *ptep)
+{
+	ptep_set_wrprotect(mm, addr, ptep);
+}
+
+static inline pte_t huge_ptep_get_and_clear(struct mm_struct *mm,
+					    unsigned long addr, pte_t *ptep)
+{
+	return ptep_get_and_clear(mm, addr, ptep);
+}
+
+static inline int huge_ptep_set_access_flags(struct vm_area_struct *vma,
+					     unsigned long addr, pte_t *ptep,
+					     pte_t pte, int dirty)
+{
+	return ptep_set_access_flags(vma, addr, ptep, pte, dirty);
+}
+
+#endif /* _ASM_ARM_HUGETLB_3LEVEL_H */
diff --git a/arch/arm/include/asm/hugetlb.h b/arch/arm/include/asm/hugetlb.h
new file mode 100644
index 0000000..1f1b1cd
--- /dev/null
+++ b/arch/arm/include/asm/hugetlb.h
@@ -0,0 +1,84 @@
+/*
+ * arch/arm/include/asm/hugetlb.h
+ *
+ * Copyright (C) 2012 ARM Ltd.
+ *
+ * Based on arch/x86/include/asm/hugetlb.h
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+ */
+
+#ifndef _ASM_ARM_HUGETLB_H
+#define _ASM_ARM_HUGETLB_H
+
+#include <asm/page.h>
+#include <asm-generic/hugetlb.h>
+
+#include <asm/hugetlb-3level.h>
+
+static inline void hugetlb_free_pgd_range(struct mmu_gather *tlb,
+					  unsigned long addr, unsigned long end,
+					  unsigned long floor,
+					  unsigned long ceiling)
+{
+	free_pgd_range(tlb, addr, end, floor, ceiling);
+}
+
+
+static inline int is_hugepage_only_range(struct mm_struct *mm,
+					 unsigned long addr, unsigned long len)
+{
+	return 0;
+}
+
+static inline int prepare_hugepage_range(struct file *file,
+					 unsigned long addr, unsigned long len)
+{
+	struct hstate *h = hstate_file(file);
+	if (len & ~huge_page_mask(h))
+		return -EINVAL;
+	if (addr & ~huge_page_mask(h))
+		return -EINVAL;
+	return 0;
+}
+
+static inline void hugetlb_prefault_arch_hook(struct mm_struct *mm)
+{
+}
+
+static inline int huge_pte_none(pte_t pte)
+{
+	return pte_none(pte);
+}
+
+static inline pte_t huge_pte_wrprotect(pte_t pte)
+{
+	return pte_wrprotect(pte);
+}
+
+static inline int arch_prepare_hugepage(struct page *page)
+{
+	return 0;
+}
+
+static inline void arch_release_hugepage(struct page *page)
+{
+}
+
+static inline void arch_clear_hugepage_flags(struct page *page)
+{
+	clear_bit(PG_dcache_clean, &page->flags);
+}
+
+#endif /* _ASM_ARM_HUGETLB_H */
diff --git a/arch/arm/include/asm/pgtable-3level-hwdef.h b/arch/arm/include/asm/pgtable-3level-hwdef.h
index 18f5cef..42df407 100644
--- a/arch/arm/include/asm/pgtable-3level-hwdef.h
+++ b/arch/arm/include/asm/pgtable-3level-hwdef.h
@@ -30,6 +30,7 @@
 #define PMD_TYPE_FAULT		(_AT(pmdval_t, 0) << 0)
 #define PMD_TYPE_TABLE		(_AT(pmdval_t, 3) << 0)
 #define PMD_TYPE_SECT		(_AT(pmdval_t, 1) << 0)
+#define PMD_TABLE_BIT		(_AT(pmdval_t, 1) << 1)
 #define PMD_BIT4		(_AT(pmdval_t, 0))
 #define PMD_DOMAIN(x)		(_AT(pmdval_t, 0))
 #define PMD_APTABLE_SHIFT	(61)
@@ -66,6 +67,7 @@
 #define PTE_TYPE_MASK		(_AT(pteval_t, 3) << 0)
 #define PTE_TYPE_FAULT		(_AT(pteval_t, 0) << 0)
 #define PTE_TYPE_PAGE		(_AT(pteval_t, 3) << 0)
+#define PTE_TABLE_BIT		(_AT(pteval_t, 1) << 1)
 #define PTE_BUFFERABLE		(_AT(pteval_t, 1) << 2)		/* AttrIndx[0] */
 #define PTE_CACHEABLE		(_AT(pteval_t, 1) << 3)		/* AttrIndx[1] */
 #define PTE_EXT_SHARED		(_AT(pteval_t, 3) << 8)		/* SH[1:0], inner shareable */
diff --git a/arch/arm/include/asm/pgtable-3level.h b/arch/arm/include/asm/pgtable-3level.h
index 70f041c..d1bcd82 100644
--- a/arch/arm/include/asm/pgtable-3level.h
+++ b/arch/arm/include/asm/pgtable-3level.h
@@ -62,6 +62,14 @@
 #define USER_PTRS_PER_PGD	(PAGE_OFFSET / PGDIR_SIZE)
 
 /*
+ * Hugetlb definitions.
+ */
+#define HPAGE_SHIFT		PMD_SHIFT
+#define HPAGE_SIZE		(_AC(1, UL) << HPAGE_SHIFT)
+#define HPAGE_MASK		(~(HPAGE_SIZE - 1))
+#define HUGETLB_PAGE_ORDER	(HPAGE_SHIFT - PAGE_SHIFT)
+
+/*
  * "Linux" PTE definitions for LPAE.
  *
  * These bits overlap with the hardware bits but the naming is preserved for
@@ -185,6 +193,9 @@ static inline pmd_t *pmd_offset(pud_t *pud, unsigned long addr)
 
 #define set_pte_ext(ptep,pte,ext) cpu_set_pte_ext(ptep,__pte(pte_val(pte)|(ext)))
 
+#define pte_huge(pte)		(pte_val(pte) && !(pte_val(pte) & PTE_TABLE_BIT))
+#define pte_mkhuge(pte)		(__pte(pte_val(pte) & ~PTE_TABLE_BIT))
+
 #endif /* __ASSEMBLY__ */
 
 #endif /* _ASM_PGTABLE_3LEVEL_H */
diff --git a/arch/arm/mm/Makefile b/arch/arm/mm/Makefile
index 9e51be9..224a9cc 100644
--- a/arch/arm/mm/Makefile
+++ b/arch/arm/mm/Makefile
@@ -16,6 +16,7 @@ obj-$(CONFIG_MODULES)		+= proc-syms.o
 
 obj-$(CONFIG_ALIGNMENT_TRAP)	+= alignment.o
 obj-$(CONFIG_HIGHMEM)		+= highmem.o
+obj-$(CONFIG_HUGETLB_PAGE)	+= hugetlbpage.o
 
 obj-$(CONFIG_CPU_ABRT_NOMMU)	+= abort-nommu.o
 obj-$(CONFIG_CPU_ABRT_EV4)	+= abort-ev4.o
diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
index 7857f0b..38b78ef 100644
--- a/arch/arm/mm/dma-mapping.c
+++ b/arch/arm/mm/dma-mapping.c
@@ -250,7 +250,7 @@ static void __dma_free_buffer(struct page *page, size_t size)
 
 #ifdef CONFIG_MMU
 #ifdef CONFIG_HUGETLB_PAGE
-#error ARM Coherent DMA allocator does not (yet) support huge TLB
+#warning ARM Coherent DMA allocator does not (yet) support huge TLB
 #endif
 
 static void *__alloc_from_contiguous(struct device *dev, size_t size,
diff --git a/arch/arm/mm/fsr-3level.c b/arch/arm/mm/fsr-3level.c
index 05a4e94..e115fc7 100644
--- a/arch/arm/mm/fsr-3level.c
+++ b/arch/arm/mm/fsr-3level.c
@@ -13,7 +13,7 @@ static struct fsr_info fsr_info[] = {
 	{ do_page_fault,	SIGSEGV, SEGV_ACCERR,	"level 3 access flag fault"	},
 	{ do_bad,		SIGBUS,  0,		"reserved permission fault"	},
 	{ do_bad,		SIGSEGV, SEGV_ACCERR,	"level 1 permission fault"	},
-	{ do_sect_fault,	SIGSEGV, SEGV_ACCERR,	"level 2 permission fault"	},
+	{ do_page_fault,	SIGSEGV, SEGV_ACCERR,	"level 2 permission fault"	},
 	{ do_page_fault,	SIGSEGV, SEGV_ACCERR,	"level 3 permission fault"	},
 	{ do_bad,		SIGBUS,  0,		"synchronous external abort"	},
 	{ do_bad,		SIGBUS,  0,		"asynchronous external abort"	},
diff --git a/arch/arm/mm/hugetlbpage.c b/arch/arm/mm/hugetlbpage.c
new file mode 100644
index 0000000..3d1e4a2
--- /dev/null
+++ b/arch/arm/mm/hugetlbpage.c
@@ -0,0 +1,101 @@
+/*
+ * arch/arm/mm/hugetlbpage.c
+ *
+ * Copyright (C) 2012 ARM Ltd.
+ *
+ * Based on arch/x86/include/asm/hugetlb.h and Bill Carson's patches
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+ */
+
+#include <linux/init.h>
+#include <linux/fs.h>
+#include <linux/mm.h>
+#include <linux/hugetlb.h>
+#include <linux/pagemap.h>
+#include <linux/err.h>
+#include <linux/sysctl.h>
+#include <asm/mman.h>
+#include <asm/tlb.h>
+#include <asm/tlbflush.h>
+#include <asm/pgalloc.h>
+
+/*
+ * On ARM, huge pages are backed by pmd's rather than pte's, so we do a lot
+ * of type casting from pmd_t * to pte_t *.
+ */
+
+pte_t *huge_pte_offset(struct mm_struct *mm, unsigned long addr)
+{
+	pgd_t *pgd;
+	pud_t *pud;
+	pmd_t *pmd = NULL;
+
+	pgd = pgd_offset(mm, addr);
+	if (pgd_present(*pgd)) {
+		pud = pud_offset(pgd, addr);
+		if (pud_present(*pud))
+			pmd = pmd_offset(pud, addr);
+	}
+
+	return (pte_t *)pmd;
+}
+
+struct page *follow_huge_addr(struct mm_struct *mm, unsigned long address,
+			      int write)
+{
+	return ERR_PTR(-EINVAL);
+}
+
+int pud_huge(pud_t pud)
+{
+	return 0;
+}
+
+int huge_pmd_unshare(struct mm_struct *mm, unsigned long *addr, pte_t *ptep)
+{
+	return 0;
+}
+
+pte_t *huge_pte_alloc(struct mm_struct *mm,
+			unsigned long addr, unsigned long sz)
+{
+	pgd_t *pgd;
+	pud_t *pud;
+	pte_t *pte = NULL;
+
+	pgd = pgd_offset(mm, addr);
+	pud = pud_alloc(mm, pgd, addr);
+	if (pud)
+		pte = (pte_t *)pmd_alloc(mm, pud, addr);
+
+	return pte;
+}
+
+struct page *
+follow_huge_pmd(struct mm_struct *mm, unsigned long address,
+		pmd_t *pmd, int write)
+{
+	struct page *page;
+
+	page = pte_page(*(pte_t *)pmd);
+	if (page)
+		page += ((address & ~PMD_MASK) >> PAGE_SHIFT);
+	return page;
+}
+
+int pmd_huge(pmd_t pmd)
+{
+	return pmd_val(pmd) && !(pmd_val(pmd) & PMD_TABLE_BIT);
+}
-- 
1.8.1.4

[PATCH v2 4/4] ARM: mm: Transparent huge page support for LPAE systems.

[Steve Capper]

From: Catalin Marinas <catalin.marinas@arm.com>

The patch adds support for THP (transparent huge pages) to LPAE
systems. When this feature is enabled, the kernel tries to map
anonymous pages as 2MB sections where possible.

Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
[steve.capper at linaro.org: symbolic constants used, value of
PMD_SECT_SPLITTING adjusted, tlbflush.h included in pgtable.h,
added PROT_NONE support.]
Signed-off-by: Steve Capper <steve.capper@linaro.org>
Reviewed-by: Will Deacon <will.deacon@arm.com>
---
 arch/arm/Kconfig                            |  4 ++
 arch/arm/include/asm/pgtable-3level-hwdef.h |  2 +
 arch/arm/include/asm/pgtable-3level.h       | 60 +++++++++++++++++++++++++++++
 arch/arm/include/asm/pgtable.h              |  3 ++
 arch/arm/include/asm/tlb.h                  |  6 +++
 arch/arm/include/asm/tlbflush.h             |  2 +
 arch/arm/mm/fsr-3level.c                    |  2 +-
 7 files changed, 78 insertions(+), 1 deletion(-)

diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
index 860f034..f07a462 100644
--- a/arch/arm/Kconfig
+++ b/arch/arm/Kconfig
@@ -1711,6 +1711,10 @@ config SYS_SUPPORTS_HUGETLBFS
        def_bool y
        depends on ARM_LPAE
 
+config HAVE_ARCH_TRANSPARENT_HUGEPAGE
+       def_bool y
+       depends on ARM_LPAE
+
 source "mm/Kconfig"
 
 config FORCE_MAX_ZONEORDER
diff --git a/arch/arm/include/asm/pgtable-3level-hwdef.h b/arch/arm/include/asm/pgtable-3level-hwdef.h
index 42df407..f088c86 100644
--- a/arch/arm/include/asm/pgtable-3level-hwdef.h
+++ b/arch/arm/include/asm/pgtable-3level-hwdef.h
@@ -42,6 +42,8 @@
  */
 #define PMD_SECT_BUFFERABLE	(_AT(pmdval_t, 1) << 2)
 #define PMD_SECT_CACHEABLE	(_AT(pmdval_t, 1) << 3)
+#define PMD_SECT_USER		(_AT(pmdval_t, 1) << 6)		/* AP[1] */
+#define PMD_SECT_RDONLY		(_AT(pmdval_t, 1) << 7)		/* AP[2] */
 #define PMD_SECT_S		(_AT(pmdval_t, 3) << 8)
 #define PMD_SECT_AF		(_AT(pmdval_t, 1) << 10)
 #define PMD_SECT_nG		(_AT(pmdval_t, 1) << 11)
diff --git a/arch/arm/include/asm/pgtable-3level.h b/arch/arm/include/asm/pgtable-3level.h
index d1bcd82..54733e5 100644
--- a/arch/arm/include/asm/pgtable-3level.h
+++ b/arch/arm/include/asm/pgtable-3level.h
@@ -87,6 +87,11 @@
 #define L_PTE_SPECIAL		(_AT(pteval_t, 1) << 56)	/* unused */
 #define L_PTE_NONE		(_AT(pteval_t, 1) << 57)	/* PROT_NONE */
 
+#define PMD_SECT_VALID		(_AT(pmdval_t, 1) << 0)
+#define PMD_SECT_DIRTY		(_AT(pmdval_t, 1) << 55)
+#define PMD_SECT_SPLITTING	(_AT(pmdval_t, 1) << 56)
+#define PMD_SECT_NONE		(_AT(pmdval_t, 1) << 57)
+
 /*
  * To be used in assembly code with the upper page attributes.
  */
@@ -196,6 +201,61 @@ static inline pmd_t *pmd_offset(pud_t *pud, unsigned long addr)
 #define pte_huge(pte)		(pte_val(pte) && !(pte_val(pte) & PTE_TABLE_BIT))
 #define pte_mkhuge(pte)		(__pte(pte_val(pte) & ~PTE_TABLE_BIT))
 
+#define pmd_young(pmd)		(pmd_val(pmd) & PMD_SECT_AF)
+
+#define __HAVE_ARCH_PMD_WRITE
+#define pmd_write(pmd)		(!(pmd_val(pmd) & PMD_SECT_RDONLY))
+
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+#define pmd_trans_huge(pmd)	(pmd_val(pmd) && !(pmd_val(pmd) & PMD_TABLE_BIT))
+#define pmd_trans_splitting(pmd) (pmd_val(pmd) & PMD_SECT_SPLITTING)
+#endif
+
+#define PMD_BIT_FUNC(fn,op) \
+static inline pmd_t pmd_##fn(pmd_t pmd) { pmd_val(pmd) op; return pmd; }
+
+PMD_BIT_FUNC(wrprotect,	|= PMD_SECT_RDONLY);
+PMD_BIT_FUNC(mkold,	&= ~PMD_SECT_AF);
+PMD_BIT_FUNC(mksplitting, |= PMD_SECT_SPLITTING);
+PMD_BIT_FUNC(mkwrite,   &= ~PMD_SECT_RDONLY);
+PMD_BIT_FUNC(mkdirty,   |= PMD_SECT_DIRTY);
+PMD_BIT_FUNC(mkyoung,   |= PMD_SECT_AF);
+
+#define pmd_mkhuge(pmd)		(__pmd(pmd_val(pmd) & ~PMD_TABLE_BIT))
+
+#define pmd_pfn(pmd)		(((pmd_val(pmd) & PMD_MASK) & PHYS_MASK) >> PAGE_SHIFT)
+#define pfn_pmd(pfn,prot)	(__pmd(((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot)))
+#define mk_pmd(page,prot)	pfn_pmd(page_to_pfn(page),prot)
+
+/* represent a notpresent pmd by zero, this is used by pmdp_invalidate */
+#define pmd_mknotpresent(pmd)	(__pmd(0))
+
+static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot)
+{
+	const pmdval_t mask = PMD_SECT_USER | PMD_SECT_XN | PMD_SECT_RDONLY |
+				PMD_SECT_VALID | PMD_SECT_NONE;
+	pmd_val(pmd) = (pmd_val(pmd) & ~mask) | (pgprot_val(newprot) & mask);
+	return pmd;
+}
+
+static inline void set_pmd_at(struct mm_struct *mm, unsigned long addr,
+			      pmd_t *pmdp, pmd_t pmd)
+{
+	BUG_ON(addr >= TASK_SIZE);
+
+	/* create a faulting entry if PROT_NONE protected */
+	if (pmd_val(pmd) & PMD_SECT_NONE)
+		pmd_val(pmd) &= ~PMD_SECT_VALID;
+
+	*pmdp = __pmd(pmd_val(pmd) | PMD_SECT_nG);
+	flush_pmd_entry(pmdp);
+}
+
+static inline int has_transparent_hugepage(void)
+{
+	return 1;
+}
+
 #endif /* __ASSEMBLY__ */
 
 #endif /* _ASM_PGTABLE_3LEVEL_H */
diff --git a/arch/arm/include/asm/pgtable.h b/arch/arm/include/asm/pgtable.h
index 9bcd262..eaedce7 100644
--- a/arch/arm/include/asm/pgtable.h
+++ b/arch/arm/include/asm/pgtable.h
@@ -24,6 +24,9 @@
 #include <asm/memory.h>
 #include <asm/pgtable-hwdef.h>
 
+
+#include <asm/tlbflush.h>
+
 #ifdef CONFIG_ARM_LPAE
 #include <asm/pgtable-3level.h>
 #else
diff --git a/arch/arm/include/asm/tlb.h b/arch/arm/include/asm/tlb.h
index 99a1951..bdc62da 100644
--- a/arch/arm/include/asm/tlb.h
+++ b/arch/arm/include/asm/tlb.h
@@ -223,6 +223,12 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp,
 #endif
 }
 
+static inline void
+tlb_remove_pmd_tlb_entry(struct mmu_gather *tlb, pmd_t *pmdp, unsigned long addr)
+{
+	tlb_add_flush(tlb, addr);
+}
+
 #define pte_free_tlb(tlb, ptep, addr)	__pte_free_tlb(tlb, ptep, addr)
 #define pmd_free_tlb(tlb, pmdp, addr)	__pmd_free_tlb(tlb, pmdp, addr)
 #define pud_free_tlb(tlb, pudp, addr)	pud_free((tlb)->mm, pudp)
diff --git a/arch/arm/include/asm/tlbflush.h b/arch/arm/include/asm/tlbflush.h
index a3625d1..c374592 100644
--- a/arch/arm/include/asm/tlbflush.h
+++ b/arch/arm/include/asm/tlbflush.h
@@ -535,6 +535,8 @@ static inline void update_mmu_cache(struct vm_area_struct *vma,
 }
 #endif
 
+#define update_mmu_cache_pmd(vma, address, pmd) do { } while (0)
+
 #endif
 
 #endif /* CONFIG_MMU */
diff --git a/arch/arm/mm/fsr-3level.c b/arch/arm/mm/fsr-3level.c
index e115fc7..ab4409a 100644
--- a/arch/arm/mm/fsr-3level.c
+++ b/arch/arm/mm/fsr-3level.c
@@ -9,7 +9,7 @@ static struct fsr_info fsr_info[] = {
 	{ do_page_fault,	SIGSEGV, SEGV_MAPERR,	"level 3 translation fault"	},
 	{ do_bad,		SIGBUS,  0,		"reserved access flag fault"	},
 	{ do_bad,		SIGSEGV, SEGV_ACCERR,	"level 1 access flag fault"	},
-	{ do_bad,		SIGSEGV, SEGV_ACCERR,	"level 2 access flag fault"	},
+	{ do_page_fault,	SIGSEGV, SEGV_ACCERR,	"level 2 access flag fault"	},
 	{ do_page_fault,	SIGSEGV, SEGV_ACCERR,	"level 3 access flag fault"	},
 	{ do_bad,		SIGBUS,  0,		"reserved permission fault"	},
 	{ do_bad,		SIGSEGV, SEGV_ACCERR,	"level 1 permission fault"	},
-- 
1.8.1.4

[PATCH v2 1/4] ARM: mm: correct pte_same behaviour for LPAE.

[Catalin Marinas]

On Thu, May 23, 2013 at 04:31:17PM +0100, Steve Capper wrote:
> For 3 levels of paging the PTE_EXT_NG bit will be set for user
> address ptes that are written to a page table but not for ptes
> created with mk_pte.
> 
> This can cause some comparison tests made by pte_same to fail
> spuriously and lead to other problems.
> 
> To correct this behaviour, we mask off PTE_EXT_NG for any pte that
> is present before running the comparison.
> 
> Signed-off-by: Steve Capper <steve.capper@linaro.org>
> Reviewed-by: Will Deacon <will.deacon@arm.com>

Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>

[PATCH v2 2/4] ARM: mm: Add support for flushing HugeTLB pages.

[Catalin Marinas]

On Thu, May 23, 2013 at 04:31:18PM +0100, Steve Capper wrote:
> On ARM we use the __flush_dcache_page function to flush the dcache
> of pages when needed; usually when the PG_dcache_clean bit is unset
> and we are setting a PTE.
> 
> A HugeTLB page is represented as a compound page consisting of an
> array of pages. Thus to flush the dcache of a HugeTLB page, one must
> flush more than a single page.
> 
> This patch modifies __flush_dcache_page such that all constituent
> pages of a HugeTLB page are flushed.
> 
> Signed-off-by: Steve Capper <steve.capper@linaro.org>
> Reviewed-by: Will Deacon <will.deacon@arm.com>

Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>

[PATCH v2 3/4] ARM: mm: HugeTLB support for LPAE systems.

[Catalin Marinas]

On Thu, May 23, 2013 at 04:31:19PM +0100, Steve Capper wrote:
> From: Catalin Marinas <catalin.marinas@arm.com>
> 
> This patch adds support for hugetlbfs based on the x86 implementation.

Can we not share the x86 code as you are proposing for arm64 already?

-- 
Catalin

[PATCH v2 3/4] ARM: mm: HugeTLB support for LPAE systems.

[Steve Capper]

On Fri, May 22, 2013 at 12:03:06PM +0100, Catalin Marinas wrote:
> On Thu, May 23, 2013 at 04:31:19PM +0100, Steve Capper wrote:
> > From: Catalin Marinas <catalin.marinas@arm.com>
> > 
> > This patch adds support for hugetlbfs based on the x86 implementation.
> 
> Can we not share the x86 code as you are proposing for arm64 already?
>

Yes, we can. I didn't want one patch set dependent on another so opted to
make the ARMv7 one self-contained for now. I was planning on changing this
with a future patch.

Cheers,
-- 
Steve
 
> -- 
> Catalin

[PATCH v2 0/4] ARM: mm: HugeTLB + THP support.

[Steve Capper]

On Thu, May 23, 2013 at 04:31:16PM +0100, Steve Capper wrote:
> The following patches bring both HugeTLB support and Transparent
> HugePage (THP) support to ARM.
> 
> Only long descriptors (LPAE) are supported in this series.
> 
> The code has been tested on an Arndale board (Exynos 5250).
> 
> This patch set is based on 3.10-rc2.
> 
> Major changes since the Patch:
>  * LPAE code has been separated from non-LPAE code (this series
>    is the LPAE code).
>  * PROT_NONE support for HugeTLB and THP has been implemented.
> 
> Major changes since the RFC:
>  * huge pmd sharing removed from the 3-level code as this was
>    found to be very rarely, if ever?, used. This allowed for some
>    code simplification.
> 
>  * hardware pmd bits for 2-levels of paging are now taken from
>    mmu.c. Also the mapping code now uses pte/pmd bit helper
>    functions rather than the custom pre-processor logic.
> 
> Cheers,
> -- 
> Steve
> 

Hello,
I was wondering if anyone had any further comments on this series?

Russell,
Are you happy for these to go into your patch system, or should anything be
changed?

Thanks,
-- 
Steve 

[PATCH v2 0/4] ARM: mm: HugeTLB + THP support.

[Steve Capper]

On Mon, Jun 03, 2013 at 11:40:11AM +0100, Steve Capper wrote:
> On Thu, May 23, 2013 at 04:31:16PM +0100, Steve Capper wrote:
> > The following patches bring both HugeTLB support and Transparent
> > HugePage (THP) support to ARM.
> > 
> > Only long descriptors (LPAE) are supported in this series.
> > 
> > The code has been tested on an Arndale board (Exynos 5250).
> > 
> > This patch set is based on 3.10-rc2.
> > 
> > Major changes since the Patch:
> >  * LPAE code has been separated from non-LPAE code (this series
> >    is the LPAE code).
> >  * PROT_NONE support for HugeTLB and THP has been implemented.
> > 
> > Major changes since the RFC:
> >  * huge pmd sharing removed from the 3-level code as this was
> >    found to be very rarely, if ever?, used. This allowed for some
> >    code simplification.
> > 
> >  * hardware pmd bits for 2-levels of paging are now taken from
> >    mmu.c. Also the mapping code now uses pte/pmd bit helper
> >    functions rather than the custom pre-processor logic.
> > 
> > Cheers,
> > -- 
> > Steve
> > 
> 
> Hello,
> I was wondering if anyone had any further comments on this series?
> 
> Russell,
> Are you happy for these to go into your patch system, or should anything be
> changed?
> 
> Thanks,
> -- 
> Steve 

Just a polite ping on the above.

Are people happy to have this patch set merged?

Thanks,
-- 
Steve

[PATCH v2 0/4] ARM: mm: HugeTLB + THP support.

[Will Deacon]

On Tue, Jun 04, 2013 at 01:56:12PM +0100, Steve Capper wrote:
> On Mon, Jun 03, 2013 at 11:40:11AM +0100, Steve Capper wrote:
> > On Thu, May 23, 2013 at 04:31:16PM +0100, Steve Capper wrote:
> > > The following patches bring both HugeTLB support and Transparent
> > > HugePage (THP) support to ARM.
> > > 
> > > Only long descriptors (LPAE) are supported in this series.
> > > 
> > > The code has been tested on an Arndale board (Exynos 5250).
> > > 
> > > This patch set is based on 3.10-rc2.
> > > 
> > > Major changes since the Patch:
> > >  * LPAE code has been separated from non-LPAE code (this series
> > >    is the LPAE code).
> > >  * PROT_NONE support for HugeTLB and THP has been implemented.
> > > 
> > > Major changes since the RFC:
> > >  * huge pmd sharing removed from the 3-level code as this was
> > >    found to be very rarely, if ever?, used. This allowed for some
> > >    code simplification.
> > > 
> > >  * hardware pmd bits for 2-levels of paging are now taken from
> > >    mmu.c. Also the mapping code now uses pte/pmd bit helper
> > >    functions rather than the custom pre-processor logic.
> > > 
> > > Cheers,
> > > -- 
> > > Steve
> > > 
> > 
> > Hello,
> > I was wondering if anyone had any further comments on this series?
> > 
> > Russell,
> > Are you happy for these to go into your patch system, or should anything be
> > changed?
> > 
> > Thanks,
> > -- 
> > Steve 
> 
> Just a polite ping on the above.
> 
> Are people happy to have this patch set merged?

Looks good to me, but worth checking that it doesn't conflict with the LPAE
fixes I plan to send for 3.11 (since they're not terribly critical):

  https://git.kernel.org/cgit/linux/kernel/git/will/linux.git/log/?h=for-rmk/lpae

Of course, I can fix things up in my branch if necessary (just send me a
pointer to your git repo).

Cheers,

Will

[PATCH v2 0/4] ARM: mm: HugeTLB + THP support.

[Steve Capper]

On Tue, Jun 04, 2013 at 02:03:29PM +0100, Will Deacon wrote:
> On Tue, Jun 04, 2013 at 01:56:12PM +0100, Steve Capper wrote:
> > On Mon, Jun 03, 2013 at 11:40:11AM +0100, Steve Capper wrote:
> > > On Thu, May 23, 2013 at 04:31:16PM +0100, Steve Capper wrote:
> > > > The following patches bring both HugeTLB support and Transparent
> > > > HugePage (THP) support to ARM.
> > > > 
> > > > Only long descriptors (LPAE) are supported in this series.
> > > > 
> > > > The code has been tested on an Arndale board (Exynos 5250).
> > > > 
> > > > This patch set is based on 3.10-rc2.
> > > > 
> > > > Major changes since the Patch:
> > > >  * LPAE code has been separated from non-LPAE code (this series
> > > >    is the LPAE code).
> > > >  * PROT_NONE support for HugeTLB and THP has been implemented.
> > > > 
> > > > Major changes since the RFC:
> > > >  * huge pmd sharing removed from the 3-level code as this was
> > > >    found to be very rarely, if ever?, used. This allowed for some
> > > >    code simplification.
> > > > 
> > > >  * hardware pmd bits for 2-levels of paging are now taken from
> > > >    mmu.c. Also the mapping code now uses pte/pmd bit helper
> > > >    functions rather than the custom pre-processor logic.
> > > > 
> > > > Cheers,
> > > > -- 
> > > > Steve
> > > > 

[ ... ]

> > 
> > Just a polite ping on the above.
> > 
> > Are people happy to have this patch set merged?
> 
> Looks good to me, but worth checking that it doesn't conflict with the LPAE
> fixes I plan to send for 3.11 (since they're not terribly critical):
> 
>   https://git.kernel.org/cgit/linux/kernel/git/will/linux.git/log/?h=for-rmk/lpae
> 
> Of course, I can fix things up in my branch if necessary (just send me a
> pointer to your git repo).

Thanks, I'll take a look at the above now.

For reference these patches can be found at:
git://git.linaro.org/people/stevecapper/linux.git arm-hugepages-20130523

Cheers,
-- 
Steve

[PATCH v2 0/4] ARM: mm: HugeTLB + THP support.

[Steve Capper]

On Tue, Jun 04, 2013 at 02:03:29PM +0100, Will Deacon wrote:
> On Tue, Jun 04, 2013 at 01:56:12PM +0100, Steve Capper wrote:
> > On Mon, Jun 03, 2013 at 11:40:11AM +0100, Steve Capper wrote:
> > > On Thu, May 23, 2013 at 04:31:16PM +0100, Steve Capper wrote:
> > > > The following patches bring both HugeTLB support and Transparent
> > > > HugePage (THP) support to ARM.
> > > > 
> > > > Only long descriptors (LPAE) are supported in this series.
> > > > 
> > > > The code has been tested on an Arndale board (Exynos 5250).
> > > > 
> > > > This patch set is based on 3.10-rc2.
> > > > 
> > > > Major changes since the Patch:
> > > >  * LPAE code has been separated from non-LPAE code (this series
> > > >    is the LPAE code).
> > > >  * PROT_NONE support for HugeTLB and THP has been implemented.
> > > > 
> > > > Major changes since the RFC:
> > > >  * huge pmd sharing removed from the 3-level code as this was
> > > >    found to be very rarely, if ever?, used. This allowed for some
> > > >    code simplification.
> > > > 
> > > >  * hardware pmd bits for 2-levels of paging are now taken from
> > > >    mmu.c. Also the mapping code now uses pte/pmd bit helper
> > > >    functions rather than the custom pre-processor logic.
> > > > 
> > > > Cheers,
> > > > -- 
> > > > Steve
> > > > 
> > > 

[ ... ]

> > 
> > Just a polite ping on the above.
> > 
> > Are people happy to have this patch set merged?
> 
> Looks good to me, but worth checking that it doesn't conflict with the LPAE
> fixes I plan to send for 3.11 (since they're not terribly critical):
> 
>   https://git.kernel.org/cgit/linux/kernel/git/will/linux.git/log/?h=for-rmk/lpae
> 
> Of course, I can fix things up in my branch if necessary (just send me a
> pointer to your git repo).

Hey Will,
I've rebased the series against to 3.10-rc4, and merged in your for-rmk/lpae
branch.

It builds, and I've run my way through the libhugetlbfs unit tests
successfully on a Versatile Express (TC2 tile). I get THP collapses, splits
and allocs too when I rebuild the test suite.

It looks good to me so far.

Thanks,
-- 
Steve