* [PATCH v7 00/11] arm64: Support 4 levels of translation tables
@ 2014-07-16 19:09 Catalin Marinas
2014-07-16 19:09 ` [PATCH v7 01/11] arm64: Use pr_* instead of printk Catalin Marinas
` (13 more replies)
0 siblings, 14 replies; 18+ messages in thread
From: Catalin Marinas @ 2014-07-16 19:09 UTC (permalink / raw)
To: linux-arm-kernel
Hi,
I've taken over Jungseok's 4-level page table series here:
https://lkml.kernel.org/g/000001cf6dc6$03ded0c0$0b9c7240$@samsung.com
I kept the same series numbering, so this is normally version 7. Changes
since v6:
- Additional fixes for duplicate (SWAPPER|IDMAP)_DIR_SIZE definitions
- Removed the fixmap pmd from swapper_pg_dir populated in head.S
- Several clean-ups in Jungseok's patches (annotated above my
signed-off-by line)
- Removal of the pgtable-*level-hwdef.h files
- Converting levels config options to int ARM64_PGTABLE_LEVELS
Important change: I dropped the KVM support for 4 levels temporarily. I
plan to revive them but didn't have for v7.
You can also access the branch here:
git://git.kernel.org/pub/scm/linux/kernel/git/cmarinas/linux-aarch64 pgtable-4levels
The branch is on top of the arm64 for-next/core branch.
Thanks.
Catalin Marinas (6):
arm64: Remove duplicate (SWAPPER|IDMAP)_DIR_SIZE definitions
arm64: Do not initialise the fixmap page tables in head.S
arm64: Convert bool ARM64_x_LEVELS to int ARM64_PGTABLE_LEVELS
arm64: Remove asm/pgtable-*level-hwdef.h files
arm64: Clean up the initial page table creation in head.S
arm64: Determine the vmalloc/vmemmap space at build time based on
VA_BITS
Jungseok Lee (5):
arm64: Use pr_* instead of printk
arm64: Introduce VA_BITS and translation level options
arm64: Add a description on 48-bit address space with 4KB pages
arm64: Add 4 levels of page tables definition with 4KB pages
arm64: mm: Implement 4 levels of translation tables
Documentation/arm64/memory.txt | 67 ++++++++-----------------
arch/arm64/Kconfig | 51 ++++++++++++++++++-
arch/arm64/include/asm/memory.h | 6 +--
arch/arm64/include/asm/page.h | 17 +++++--
arch/arm64/include/asm/pgalloc.h | 24 ++++++++-
arch/arm64/include/asm/pgtable-2level-hwdef.h | 43 ----------------
arch/arm64/include/asm/pgtable-3level-hwdef.h | 50 -------------------
arch/arm64/include/asm/pgtable-4level-types.h | 71 +++++++++++++++++++++++++++
arch/arm64/include/asm/pgtable-hwdef.h | 42 ++++++++++++++--
arch/arm64/include/asm/pgtable.h | 56 ++++++++++++++++++---
arch/arm64/include/asm/tlb.h | 11 ++++-
arch/arm64/kernel/head.S | 52 +++++++++++++-------
arch/arm64/kernel/traps.c | 13 +++--
arch/arm64/mm/fault.c | 1 +
arch/arm64/mm/init.c | 22 ++++++---
arch/arm64/mm/ioremap.c | 30 ++++++++---
arch/arm64/mm/mmu.c | 14 ++++--
17 files changed, 363 insertions(+), 207 deletions(-)
delete mode 100644 arch/arm64/include/asm/pgtable-2level-hwdef.h
delete mode 100644 arch/arm64/include/asm/pgtable-3level-hwdef.h
create mode 100644 arch/arm64/include/asm/pgtable-4level-types.h
^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH v7 01/11] arm64: Use pr_* instead of printk
2014-07-16 19:09 [PATCH v7 00/11] arm64: Support 4 levels of translation tables Catalin Marinas
@ 2014-07-16 19:09 ` Catalin Marinas
2014-07-16 19:09 ` [PATCH v7 02/11] arm64: Remove duplicate (SWAPPER|IDMAP)_DIR_SIZE definitions Catalin Marinas
` (12 subsequent siblings)
13 siblings, 0 replies; 18+ messages in thread
From: Catalin Marinas @ 2014-07-16 19:09 UTC (permalink / raw)
To: linux-arm-kernel
From: Jungseok Lee <jays.lee@samsung.com>
This patch fixed the following checkpatch complaint as using pr_*
instead of printk.
WARNING: printk() should include KERN_ facility level
Signed-off-by: Jungseok Lee <jays.lee@samsung.com>
Reviewed-by: Sungjinn Chung <sungjinn.chung@samsung.com>
Acked-by: Kukjin Kim <kgene.kim@samsung.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
---
arch/arm64/kernel/traps.c | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c
index c43cfa9b8304..506f7814e305 100644
--- a/arch/arm64/kernel/traps.c
+++ b/arch/arm64/kernel/traps.c
@@ -156,7 +156,7 @@ static void dump_backtrace(struct pt_regs *regs, struct task_struct *tsk)
frame.pc = thread_saved_pc(tsk);
}
- printk("Call trace:\n");
+ pr_emerg("Call trace:\n");
while (1) {
unsigned long where = frame.pc;
int ret;
@@ -331,17 +331,17 @@ asmlinkage void bad_mode(struct pt_regs *regs, int reason, unsigned int esr)
void __pte_error(const char *file, int line, unsigned long val)
{
- printk("%s:%d: bad pte %016lx.\n", file, line, val);
+ pr_crit("%s:%d: bad pte %016lx.\n", file, line, val);
}
void __pmd_error(const char *file, int line, unsigned long val)
{
- printk("%s:%d: bad pmd %016lx.\n", file, line, val);
+ pr_crit("%s:%d: bad pmd %016lx.\n", file, line, val);
}
void __pgd_error(const char *file, int line, unsigned long val)
{
- printk("%s:%d: bad pgd %016lx.\n", file, line, val);
+ pr_crit("%s:%d: bad pgd %016lx.\n", file, line, val);
}
void __init trap_init(void)
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH v7 02/11] arm64: Remove duplicate (SWAPPER|IDMAP)_DIR_SIZE definitions
2014-07-16 19:09 [PATCH v7 00/11] arm64: Support 4 levels of translation tables Catalin Marinas
2014-07-16 19:09 ` [PATCH v7 01/11] arm64: Use pr_* instead of printk Catalin Marinas
@ 2014-07-16 19:09 ` Catalin Marinas
2014-07-17 10:49 ` Mark Rutland
2014-07-16 19:09 ` [PATCH v7 03/11] arm64: Do not initialise the fixmap page tables in head.S Catalin Marinas
` (11 subsequent siblings)
13 siblings, 1 reply; 18+ messages in thread
From: Catalin Marinas @ 2014-07-16 19:09 UTC (permalink / raw)
To: linux-arm-kernel
Just keep the asm/page.h definition as this is included in vmlinux.lds.S
as well.
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
---
arch/arm64/include/asm/pgtable.h | 3 ---
1 file changed, 3 deletions(-)
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index e0ccceb317d9..d7455fa83bc7 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -383,9 +383,6 @@ static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot)
extern pgd_t swapper_pg_dir[PTRS_PER_PGD];
extern pgd_t idmap_pg_dir[PTRS_PER_PGD];
-#define SWAPPER_DIR_SIZE (3 * PAGE_SIZE)
-#define IDMAP_DIR_SIZE (2 * PAGE_SIZE)
-
/*
* Encode and decode a swap entry:
* bits 0-1: present (must be zero)
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH v7 03/11] arm64: Do not initialise the fixmap page tables in head.S
2014-07-16 19:09 [PATCH v7 00/11] arm64: Support 4 levels of translation tables Catalin Marinas
2014-07-16 19:09 ` [PATCH v7 01/11] arm64: Use pr_* instead of printk Catalin Marinas
2014-07-16 19:09 ` [PATCH v7 02/11] arm64: Remove duplicate (SWAPPER|IDMAP)_DIR_SIZE definitions Catalin Marinas
@ 2014-07-16 19:09 ` Catalin Marinas
2014-07-16 23:14 ` Geoff Levand
2014-07-16 19:09 ` [PATCH v7 04/11] arm64: Introduce VA_BITS and translation level options Catalin Marinas
` (10 subsequent siblings)
13 siblings, 1 reply; 18+ messages in thread
From: Catalin Marinas @ 2014-07-16 19:09 UTC (permalink / raw)
To: linux-arm-kernel
The early_ioremap_init() function already handles fixmap pte
initialisation, so upgrade this to cover all of pud/pmd/pte and remove
one page from swapper_pg_dir.
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
---
arch/arm64/include/asm/page.h | 8 ++++----
arch/arm64/kernel/head.S | 7 -------
arch/arm64/mm/ioremap.c | 26 ++++++++++++++++++--------
3 files changed, 22 insertions(+), 19 deletions(-)
diff --git a/arch/arm64/include/asm/page.h b/arch/arm64/include/asm/page.h
index a6331e6a92b5..d3515bce4077 100644
--- a/arch/arm64/include/asm/page.h
+++ b/arch/arm64/include/asm/page.h
@@ -33,11 +33,11 @@
/*
* The idmap and swapper page tables need some space reserved in the kernel
- * image. The idmap only requires a pgd and a next level table to (section) map
- * the kernel, while the swapper also maps the FDT and requires an additional
- * table to map an early UART. See __create_page_tables for more information.
+ * image. Both require a pgd and a next level table to (section) map the
+ * kernel. The the swapper also maaps the FDT (see __create_page_tables for
+ * more information).
*/
-#define SWAPPER_DIR_SIZE (3 * PAGE_SIZE)
+#define SWAPPER_DIR_SIZE (2 * PAGE_SIZE)
#define IDMAP_DIR_SIZE (2 * PAGE_SIZE)
#ifndef __ASSEMBLY__
diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index 69dafe9621fd..fa3b7fb8a77a 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -583,13 +583,6 @@ __create_page_tables:
create_block_map x0, x7, x3, x5, x6
1:
/*
- * Create the pgd entry for the fixed mappings.
- */
- ldr x5, =FIXADDR_TOP // Fixed mapping virtual address
- add x0, x26, #2 * PAGE_SIZE // section table address
- create_pgd_entry x26, x0, x5, x6, x7
-
- /*
* Since the page tables have been populated with non-cacheable
* accesses (MMU disabled), invalidate the idmap and swapper page
* tables again to remove any speculatively loaded cache lines.
diff --git a/arch/arm64/mm/ioremap.c b/arch/arm64/mm/ioremap.c
index 7ec328392ae0..69000efa015e 100644
--- a/arch/arm64/mm/ioremap.c
+++ b/arch/arm64/mm/ioremap.c
@@ -103,19 +103,25 @@ void __iomem *ioremap_cache(phys_addr_t phys_addr, size_t size)
}
EXPORT_SYMBOL(ioremap_cache);
-#ifndef CONFIG_ARM64_64K_PAGES
static pte_t bm_pte[PTRS_PER_PTE] __page_aligned_bss;
+#ifndef CONFIG_ARM64_64K_PAGES
+static pte_t bm_pmd[PTRS_PER_PMD] __page_aligned_bss;
#endif
-static inline pmd_t * __init early_ioremap_pmd(unsigned long addr)
+static inline pud_t * __init early_ioremap_pud(unsigned long addr)
{
pgd_t *pgd;
- pud_t *pud;
pgd = pgd_offset_k(addr);
BUG_ON(pgd_none(*pgd) || pgd_bad(*pgd));
- pud = pud_offset(pgd, addr);
+ return pud_offset(pgd, addr);
+}
+
+static inline pmd_t * __init early_ioremap_pmd(unsigned long addr)
+{
+ pud_t *pud = early_ioremap_pud(addr);
+
BUG_ON(pud_none(*pud) || pud_bad(*pud));
return pmd_offset(pud, addr);
@@ -132,13 +138,17 @@ static inline pte_t * __init early_ioremap_pte(unsigned long addr)
void __init early_ioremap_init(void)
{
+ pgd_t *pgd;
+ pud_t *pud;
pmd_t *pmd;
+ unsigned long addr = fix_to_virt(FIX_BTMAP_BEGIN);
- pmd = early_ioremap_pmd(fix_to_virt(FIX_BTMAP_BEGIN));
-#ifndef CONFIG_ARM64_64K_PAGES
- /* need to populate pmd for 4k pagesize only */
+ pgd = pgd_offset_k(addr);
+ pud = pud_offset(pgd, addr);
+ pud_populate(&init_mm, pud, bm_pmd);
+ pmd = pmd_offset(pud, addr);
pmd_populate_kernel(&init_mm, pmd, bm_pte);
-#endif
+
/*
* The boot-ioremap range spans multiple pmds, for which
* we are not prepared:
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH v7 04/11] arm64: Introduce VA_BITS and translation level options
2014-07-16 19:09 [PATCH v7 00/11] arm64: Support 4 levels of translation tables Catalin Marinas
` (2 preceding siblings ...)
2014-07-16 19:09 ` [PATCH v7 03/11] arm64: Do not initialise the fixmap page tables in head.S Catalin Marinas
@ 2014-07-16 19:09 ` Catalin Marinas
2014-07-16 19:09 ` [PATCH v7 05/11] arm64: Add a description on 48-bit address space with 4KB pages Catalin Marinas
` (9 subsequent siblings)
13 siblings, 0 replies; 18+ messages in thread
From: Catalin Marinas @ 2014-07-16 19:09 UTC (permalink / raw)
To: linux-arm-kernel
From: Jungseok Lee <jays.lee@samsung.com>
This patch adds virtual address space size and a level of translation
tables to kernel configuration. It facilicates introduction of
different MMU options, such as 4KB + 4 levels, 16KB + 4 levels and
64KB + 3 levels, easily.
The idea is based on the discussion with Catalin Marinas:
http://www.spinics.net/linux/lists/arm-kernel/msg319552.html
Signed-off-by: Jungseok Lee <jays.lee@samsung.com>
Reviewed-by: Sungjinn Chung <sungjinn.chung@samsung.com>
Acked-by: Kukjin Kim <kgene.kim@samsung.com>
Reviewed-by: Christoffer Dall <christoffer.dall@linaro.org>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
---
arch/arm64/Kconfig | 45 +++++++++++++++++++++++++++++++++-
arch/arm64/include/asm/memory.h | 6 +----
arch/arm64/include/asm/page.h | 2 +-
arch/arm64/include/asm/pgalloc.h | 4 +--
arch/arm64/include/asm/pgtable-hwdef.h | 2 +-
arch/arm64/include/asm/pgtable.h | 8 +++---
arch/arm64/include/asm/tlb.h | 2 +-
7 files changed, 54 insertions(+), 15 deletions(-)
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index ce6e733e0c05..4daf11f5b403 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -158,14 +158,57 @@ endmenu
menu "Kernel Features"
+choice
+ prompt "Page size"
+ default ARM64_4K_PAGES
+ help
+ Page size (translation granule) configuration.
+
+config ARM64_4K_PAGES
+ bool "4KB"
+ help
+ This feature enables 4KB pages support.
+
config ARM64_64K_PAGES
- bool "Enable 64KB pages support"
+ bool "64KB"
help
This feature enables 64KB pages support (4KB by default)
allowing only two levels of page tables and faster TLB
look-up. AArch32 emulation is not available when this feature
is enabled.
+endchoice
+
+choice
+ prompt "Virtual address space size"
+ default ARM64_VA_BITS_39 if ARM64_4K_PAGES
+ default ARM64_VA_BITS_42 if ARM64_64K_PAGES
+ help
+ Allows choosing one of multiple possible virtual address
+ space sizes. The level of translation table is determined by
+ a combination of page size and virtual address space size.
+
+config ARM64_VA_BITS_39
+ bool "39-bit"
+ depends on ARM64_4K_PAGES
+
+config ARM64_VA_BITS_42
+ bool "42-bit"
+ depends on ARM64_64K_PAGES
+
+endchoice
+
+config ARM64_VA_BITS
+ int
+ default 39 if ARM64_VA_BITS_39
+ default 42 if ARM64_VA_BITS_42
+
+config ARM64_2_LEVELS
+ def_bool y if ARM64_64K_PAGES && ARM64_VA_BITS_42
+
+config ARM64_3_LEVELS
+ def_bool y if ARM64_4K_PAGES && ARM64_VA_BITS_39
+
config CPU_BIG_ENDIAN
bool "Build big-endian kernel"
help
diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h
index 993bce527b85..45ad6cf678dd 100644
--- a/arch/arm64/include/asm/memory.h
+++ b/arch/arm64/include/asm/memory.h
@@ -41,11 +41,7 @@
* The module space lives between the addresses given by TASK_SIZE
* and PAGE_OFFSET - it must be within 128MB of the kernel text.
*/
-#ifdef CONFIG_ARM64_64K_PAGES
-#define VA_BITS (42)
-#else
-#define VA_BITS (39)
-#endif
+#define VA_BITS (CONFIG_ARM64_VA_BITS)
#define PAGE_OFFSET (UL(0xffffffffffffffff) << (VA_BITS - 1))
#define MODULES_END (PAGE_OFFSET)
#define MODULES_VADDR (MODULES_END - SZ_64M)
diff --git a/arch/arm64/include/asm/page.h b/arch/arm64/include/asm/page.h
index d3515bce4077..6bf139188792 100644
--- a/arch/arm64/include/asm/page.h
+++ b/arch/arm64/include/asm/page.h
@@ -42,7 +42,7 @@
#ifndef __ASSEMBLY__
-#ifdef CONFIG_ARM64_64K_PAGES
+#ifdef CONFIG_ARM64_2_LEVELS
#include <asm/pgtable-2level-types.h>
#else
#include <asm/pgtable-3level-types.h>
diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h
index 9bea6e74a001..48298376e46a 100644
--- a/arch/arm64/include/asm/pgalloc.h
+++ b/arch/arm64/include/asm/pgalloc.h
@@ -26,7 +26,7 @@
#define check_pgt_cache() do { } while (0)
-#ifndef CONFIG_ARM64_64K_PAGES
+#ifndef CONFIG_ARM64_2_LEVELS
static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long addr)
{
@@ -44,7 +44,7 @@ static inline void pud_populate(struct mm_struct *mm, pud_t *pud, pmd_t *pmd)
set_pud(pud, __pud(__pa(pmd) | PMD_TYPE_TABLE));
}
-#endif /* CONFIG_ARM64_64K_PAGES */
+#endif /* CONFIG_ARM64_2_LEVELS */
extern pgd_t *pgd_alloc(struct mm_struct *mm);
extern void pgd_free(struct mm_struct *mm, pgd_t *pgd);
diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h
index 955e8c5f0afb..c7c603b489b8 100644
--- a/arch/arm64/include/asm/pgtable-hwdef.h
+++ b/arch/arm64/include/asm/pgtable-hwdef.h
@@ -16,7 +16,7 @@
#ifndef __ASM_PGTABLE_HWDEF_H
#define __ASM_PGTABLE_HWDEF_H
-#ifdef CONFIG_ARM64_64K_PAGES
+#ifdef CONFIG_ARM64_2_LEVELS
#include <asm/pgtable-2level-hwdef.h>
#else
#include <asm/pgtable-3level-hwdef.h>
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index d7455fa83bc7..6d5854972a77 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -47,7 +47,7 @@ extern void __pmd_error(const char *file, int line, unsigned long val);
extern void __pgd_error(const char *file, int line, unsigned long val);
#define pte_ERROR(pte) __pte_error(__FILE__, __LINE__, pte_val(pte))
-#ifndef CONFIG_ARM64_64K_PAGES
+#ifndef CONFIG_ARM64_2_LEVELS
#define pmd_ERROR(pmd) __pmd_error(__FILE__, __LINE__, pmd_val(pmd))
#endif
#define pgd_ERROR(pgd) __pgd_error(__FILE__, __LINE__, pgd_val(pgd))
@@ -323,7 +323,7 @@ static inline pte_t *pmd_page_vaddr(pmd_t pmd)
*/
#define mk_pte(page,prot) pfn_pte(page_to_pfn(page),prot)
-#ifndef CONFIG_ARM64_64K_PAGES
+#ifndef CONFIG_ARM64_2_LEVELS
#define pud_none(pud) (!pud_val(pud))
#define pud_bad(pud) (!(pud_val(pud) & 2))
@@ -345,7 +345,7 @@ static inline pmd_t *pud_page_vaddr(pud_t pud)
return __va(pud_val(pud) & PHYS_MASK & (s32)PAGE_MASK);
}
-#endif /* CONFIG_ARM64_64K_PAGES */
+#endif /* CONFIG_ARM64_2_LEVELS */
/* to find an entry in a page-table-directory */
#define pgd_index(addr) (((addr) >> PGDIR_SHIFT) & (PTRS_PER_PGD - 1))
@@ -356,7 +356,7 @@ static inline pmd_t *pud_page_vaddr(pud_t pud)
#define pgd_offset_k(addr) pgd_offset(&init_mm, addr)
/* Find an entry in the second-level page table.. */
-#ifndef CONFIG_ARM64_64K_PAGES
+#ifndef CONFIG_ARM64_2_LEVELS
#define pmd_index(addr) (((addr) >> PMD_SHIFT) & (PTRS_PER_PMD - 1))
static inline pmd_t *pmd_offset(pud_t *pud, unsigned long addr)
{
diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h
index 80e2c08900d6..bc19101edaeb 100644
--- a/arch/arm64/include/asm/tlb.h
+++ b/arch/arm64/include/asm/tlb.h
@@ -91,7 +91,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte,
tlb_remove_page(tlb, pte);
}
-#ifndef CONFIG_ARM64_64K_PAGES
+#ifndef CONFIG_ARM64_2_LEVELS
static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp,
unsigned long addr)
{
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH v7 05/11] arm64: Add a description on 48-bit address space with 4KB pages
2014-07-16 19:09 [PATCH v7 00/11] arm64: Support 4 levels of translation tables Catalin Marinas
` (3 preceding siblings ...)
2014-07-16 19:09 ` [PATCH v7 04/11] arm64: Introduce VA_BITS and translation level options Catalin Marinas
@ 2014-07-16 19:09 ` Catalin Marinas
2014-07-16 19:09 ` [PATCH v7 06/11] arm64: Add 4 levels of page tables definition " Catalin Marinas
` (8 subsequent siblings)
13 siblings, 0 replies; 18+ messages in thread
From: Catalin Marinas @ 2014-07-16 19:09 UTC (permalink / raw)
To: linux-arm-kernel
From: Jungseok Lee <jays.lee@samsung.com>
This patch adds memory layout and translation lookup information
about 48-bit address space with 4K pages. The description is based
on 4 levels of translation tables.
Signed-off-by: Jungseok Lee <jays.lee@samsung.com>
Reviewed-by: Sungjinn Chung <sungjinn.chung@samsung.com>
Acked-by: Kukjin Kim <kgene.kim@samsung.com>
Acked-by: Christoffer Dall <christoffer.dall@linaro.org>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
---
Documentation/arm64/memory.txt | 59 ++++++++++++++++++++++++++++++++++++------
1 file changed, 51 insertions(+), 8 deletions(-)
diff --git a/Documentation/arm64/memory.txt b/Documentation/arm64/memory.txt
index d50fa618371b..4c720d698e8e 100644
--- a/Documentation/arm64/memory.txt
+++ b/Documentation/arm64/memory.txt
@@ -8,10 +8,11 @@ This document describes the virtual memory layout used by the AArch64
Linux kernel. The architecture allows up to 4 levels of translation
tables with a 4KB page size and up to 3 levels with a 64KB page size.
-AArch64 Linux uses 3 levels of translation tables with the 4KB page
-configuration, allowing 39-bit (512GB) virtual addresses for both user
-and kernel. With 64KB pages, only 2 levels of translation tables are
-used but the memory layout is the same.
+AArch64 Linux uses either 3 levels or 4 levels of translation tables with
+the 4KB page configuration, allowing 39-bit (512GB) or 48-bit (256TB)
+virtual addresses, respectively, for both user and kernel. With 64KB
+pages, only 2 levels of translation tables, allowing 42-bit (4TB)
+virtual address, are used but the memory layout is the same.
User addresses have bits 63:39 set to 0 while the kernel addresses have
the same bits set to 1. TTBRx selection is given by bit 63 of the
@@ -21,7 +22,7 @@ The swapper_pgd_dir address is written to TTBR1 and never written to
TTBR0.
-AArch64 Linux memory layout with 4KB pages:
+AArch64 Linux memory layout with 4KB pages + 3 levels:
Start End Size Use
-----------------------------------------------------------------------
@@ -48,7 +49,34 @@ ffffffbffc000000 ffffffbfffffffff 64MB modules
ffffffc000000000 ffffffffffffffff 256GB kernel logical memory map
-AArch64 Linux memory layout with 64KB pages:
+AArch64 Linux memory layout with 4KB pages + 4 levels:
+
+Start End Size Use
+-----------------------------------------------------------------------
+0000000000000000 0000ffffffffffff 256TB user
+
+ffff000000000000 ffff7bfffffeffff ~124TB vmalloc
+
+ffff7bffffff0000 ffff7bffffffffff 64KB [guard page]
+
+ffff7c0000000000 ffff7dffffffffff 2TB vmemmap
+
+ffff7e0000000000 ffff7ffffbbfffff ~2TB [guard, future vmmemap]
+
+ffff7ffffa000000 ffff7ffffaffffff 16MB PCI I/O space
+
+ffff7ffffb000000 ffff7ffffbbfffff 12MB [guard]
+
+ffff7ffffbc00000 ffff7ffffbdfffff 2MB fixed mappings
+
+ffff7ffffbe00000 ffff7ffffbffffff 2MB [guard]
+
+ffff7ffffc000000 ffff7fffffffffff 64MB modules
+
+ffff800000000000 ffffffffffffffff 128TB kernel logical memory map
+
+
+AArch64 Linux memory layout with 64KB pages + 2 levels:
Start End Size Use
-----------------------------------------------------------------------
@@ -75,7 +103,7 @@ fffffdfffc000000 fffffdffffffffff 64MB modules
fffffe0000000000 ffffffffffffffff 2TB kernel logical memory map
-Translation table lookup with 4KB pages:
+Translation table lookup with 4KB pages + 3 levels:
+--------+--------+--------+--------+--------+--------+--------+--------+
|63 56|55 48|47 40|39 32|31 24|23 16|15 8|7 0|
@@ -90,7 +118,22 @@ Translation table lookup with 4KB pages:
+-------------------------------------------------> [63] TTBR0/1
-Translation table lookup with 64KB pages:
+Translation table lookup with 4KB pages + 4 levels:
+
++--------+--------+--------+--------+--------+--------+--------+--------+
+|63 56|55 48|47 40|39 32|31 24|23 16|15 8|7 0|
++--------+--------+--------+--------+--------+--------+--------+--------+
+ | | | | | |
+ | | | | | v
+ | | | | | [11:0] in-page offset
+ | | | | +-> [20:12] L3 index
+ | | | +-----------> [29:21] L2 index
+ | | +---------------------> [38:30] L1 index
+ | +-------------------------------> [47:39] L0 index
+ +-------------------------------------------------> [63] TTBR0/1
+
+
+Translation table lookup with 64KB pages + 2 levels:
+--------+--------+--------+--------+--------+--------+--------+--------+
|63 56|55 48|47 40|39 32|31 24|23 16|15 8|7 0|
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH v7 06/11] arm64: Add 4 levels of page tables definition with 4KB pages
2014-07-16 19:09 [PATCH v7 00/11] arm64: Support 4 levels of translation tables Catalin Marinas
` (4 preceding siblings ...)
2014-07-16 19:09 ` [PATCH v7 05/11] arm64: Add a description on 48-bit address space with 4KB pages Catalin Marinas
@ 2014-07-16 19:09 ` Catalin Marinas
2014-07-16 19:09 ` [PATCH v7 07/11] arm64: mm: Implement 4 levels of translation tables Catalin Marinas
` (7 subsequent siblings)
13 siblings, 0 replies; 18+ messages in thread
From: Catalin Marinas @ 2014-07-16 19:09 UTC (permalink / raw)
To: linux-arm-kernel
From: Jungseok Lee <jays.lee@samsung.com>
This patch adds hardware definition and types for 4 levels of
translation tables with 4KB pages.
Signed-off-by: Jungseok Lee <jays.lee@samsung.com>
Reviewed-by: Sungjinn Chung <sungjinn.chung@samsung.com>
Acked-by: Kukjin Kim <kgene.kim@samsung.com>
Reviewed-by: Christoffer Dall <christoffer.dall@linaro.org>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
---
arch/arm64/include/asm/pgtable-4level-hwdef.h | 50 +++++++++++++++++++
arch/arm64/include/asm/pgtable-4level-types.h | 71 +++++++++++++++++++++++++++
2 files changed, 121 insertions(+)
create mode 100644 arch/arm64/include/asm/pgtable-4level-hwdef.h
create mode 100644 arch/arm64/include/asm/pgtable-4level-types.h
diff --git a/arch/arm64/include/asm/pgtable-4level-hwdef.h b/arch/arm64/include/asm/pgtable-4level-hwdef.h
new file mode 100644
index 000000000000..0ec84e2274a3
--- /dev/null
+++ b/arch/arm64/include/asm/pgtable-4level-hwdef.h
@@ -0,0 +1,50 @@
+/*
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+#ifndef __ASM_PGTABLE_4LEVEL_HWDEF_H
+#define __ASM_PGTABLE_4LEVEL_HWDEF_H
+
+#define PTRS_PER_PTE 512
+#define PTRS_PER_PMD 512
+#define PTRS_PER_PUD 512
+#define PTRS_PER_PGD 512
+
+/*
+ * PGDIR_SHIFT determines the size a top-level page table entry can map.
+ */
+#define PGDIR_SHIFT 39
+#define PGDIR_SIZE (_AC(1, UL) << PGDIR_SHIFT)
+#define PGDIR_MASK (~(PGDIR_SIZE-1))
+
+/*
+ * PUD_SHIFT determines the size the second level page table entry can map.
+ */
+#define PUD_SHIFT 30
+#define PUD_SIZE (_AC(1, UL) << PUD_SHIFT)
+#define PUD_MASK (~(PUD_SIZE-1))
+
+/*
+ * PMD_SHIFT determines the size the third level page table entry can map.
+ */
+#define PMD_SHIFT 21
+#define PMD_SIZE (_AC(1, UL) << PMD_SHIFT)
+#define PMD_MASK (~(PMD_SIZE-1))
+
+/*
+ * section address mask and size definitions.
+ */
+#define SECTION_SHIFT 21
+#define SECTION_SIZE (_AC(1, UL) << SECTION_SHIFT)
+#define SECTION_MASK (~(SECTION_SIZE-1))
+
+#endif
diff --git a/arch/arm64/include/asm/pgtable-4level-types.h b/arch/arm64/include/asm/pgtable-4level-types.h
new file mode 100644
index 000000000000..7ad8dd257ea1
--- /dev/null
+++ b/arch/arm64/include/asm/pgtable-4level-types.h
@@ -0,0 +1,71 @@
+/*
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+#ifndef __ASM_PGTABLE_4LEVEL_TYPES_H
+#define __ASM_PGTABLE_4LEVEL_TYPES_H
+
+#include <asm/types.h>
+
+typedef u64 pteval_t;
+typedef u64 pmdval_t;
+typedef u64 pudval_t;
+typedef u64 pgdval_t;
+
+#undef STRICT_MM_TYPECHECKS
+
+#ifdef STRICT_MM_TYPECHECKS
+
+/*
+ * These are used to make use of C type-checking..
+ */
+typedef struct { pteval_t pte; } pte_t;
+typedef struct { pmdval_t pmd; } pmd_t;
+typedef struct { pudval_t pud; } pud_t;
+typedef struct { pgdval_t pgd; } pgd_t;
+typedef struct { pteval_t pgprot; } pgprot_t;
+
+#define pte_val(x) ((x).pte)
+#define pmd_val(x) ((x).pmd)
+#define pud_val(x) ((x).pud)
+#define pgd_val(x) ((x).pgd)
+#define pgprot_val(x) ((x).pgprot)
+
+#define __pte(x) ((pte_t) { (x) } )
+#define __pmd(x) ((pmd_t) { (x) } )
+#define __pud(x) ((pud_t) { (x) } )
+#define __pgd(x) ((pgd_t) { (x) } )
+#define __pgprot(x) ((pgprot_t) { (x) } )
+
+#else /* !STRICT_MM_TYPECHECKS */
+
+typedef pteval_t pte_t;
+typedef pmdval_t pmd_t;
+typedef pudval_t pud_t;
+typedef pgdval_t pgd_t;
+typedef pteval_t pgprot_t;
+
+#define pte_val(x) (x)
+#define pmd_val(x) (x)
+#define pud_val(x) (x)
+#define pgd_val(x) (x)
+#define pgprot_val(x) (x)
+
+#define __pte(x) (x)
+#define __pmd(x) (x)
+#define __pud(x) (x)
+#define __pgd(x) (x)
+#define __pgprot(x) (x)
+
+#endif /* STRICT_MM_TYPECHECKS */
+
+#endif /* __ASM_PGTABLE_4LEVEL_TYPES_H */
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH v7 07/11] arm64: mm: Implement 4 levels of translation tables
2014-07-16 19:09 [PATCH v7 00/11] arm64: Support 4 levels of translation tables Catalin Marinas
` (5 preceding siblings ...)
2014-07-16 19:09 ` [PATCH v7 06/11] arm64: Add 4 levels of page tables definition " Catalin Marinas
@ 2014-07-16 19:09 ` Catalin Marinas
2014-07-28 15:40 ` Joel Schopp
2014-07-16 19:09 ` [PATCH v7 08/11] arm64: Convert bool ARM64_x_LEVELS to int ARM64_PGTABLE_LEVELS Catalin Marinas
` (6 subsequent siblings)
13 siblings, 1 reply; 18+ messages in thread
From: Catalin Marinas @ 2014-07-16 19:09 UTC (permalink / raw)
To: linux-arm-kernel
From: Jungseok Lee <jays.lee@samsung.com>
This patch implements 4 levels of translation tables since 3 levels
of page tables with 4KB pages cannot support 40-bit physical address
space described in [1] due to the following issue.
It is a restriction that kernel logical memory map with 4KB + 3 levels
(0xffffffc000000000-0xffffffffffffffff) cannot cover RAM region from
544GB to 1024GB in [1]. Specifically, ARM64 kernel fails to create
mapping for this region in map_mem function since __phys_to_virt for
this region reaches to address overflow.
If SoC design follows the document, [1], over 32GB RAM would be placed
from 544GB. Even 64GB system is supposed to use the region from 544GB
to 576GB for only 32GB RAM. Naturally, it would reach to enable 4 levels
of page tables to avoid hacking __virt_to_phys and __phys_to_virt.
However, it is recommended 4 levels of page table should be only enabled
if memory map is too sparse or there is about 512GB RAM.
References
----------
[1]: Principles of ARM Memory Maps, White Paper, Issue C
Signed-off-by: Jungseok Lee <jays.lee@samsung.com>
Reviewed-by: Sungjinn Chung <sungjinn.chung@samsung.com>
Acked-by: Kukjin Kim <kgene.kim@samsung.com>
Reviewed-by: Christoffer Dall <christoffer.dall@linaro.org>
Reviewed-by: Steve Capper <steve.capper@linaro.org>
[catalin.marinas at arm.com: MEMBLOCK_INITIAL_LIMIT removed, same as PUD_SIZE]
[catalin.marinas at arm.com: early_ioremap_init() updated for 4 levels]
[catalin.marinas at arm.com: 4 page tables levels only if !KVM]
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
---
arch/arm64/Kconfig | 9 ++++++++
arch/arm64/include/asm/page.h | 13 ++++++++---
arch/arm64/include/asm/pgalloc.h | 20 ++++++++++++++++
arch/arm64/include/asm/pgtable-hwdef.h | 6 +++--
arch/arm64/include/asm/pgtable.h | 40 ++++++++++++++++++++++++++++++++
arch/arm64/include/asm/tlb.h | 9 ++++++++
arch/arm64/kernel/head.S | 42 +++++++++++++++++++++++++++-------
arch/arm64/kernel/traps.c | 5 ++++
arch/arm64/mm/fault.c | 1 +
arch/arm64/mm/ioremap.c | 6 ++++-
arch/arm64/mm/mmu.c | 14 +++++++++---
11 files changed, 148 insertions(+), 17 deletions(-)
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 4daf11f5b403..24cbe72c0da9 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -196,12 +196,18 @@ config ARM64_VA_BITS_42
bool "42-bit"
depends on ARM64_64K_PAGES
+config ARM64_VA_BITS_48
+ bool "48-bit"
+ depends on !KVM
+ depends on ARM64_4K_PAGES
+
endchoice
config ARM64_VA_BITS
int
default 39 if ARM64_VA_BITS_39
default 42 if ARM64_VA_BITS_42
+ default 48 if ARM64_VA_BITS_48
config ARM64_2_LEVELS
def_bool y if ARM64_64K_PAGES && ARM64_VA_BITS_42
@@ -209,6 +215,9 @@ config ARM64_2_LEVELS
config ARM64_3_LEVELS
def_bool y if ARM64_4K_PAGES && ARM64_VA_BITS_39
+config ARM64_4_LEVELS
+ def_bool y if ARM64_4K_PAGES && ARM64_VA_BITS_48
+
config CPU_BIG_ENDIAN
bool "Build big-endian kernel"
help
diff --git a/arch/arm64/include/asm/page.h b/arch/arm64/include/asm/page.h
index 6bf139188792..cf9afa0366b6 100644
--- a/arch/arm64/include/asm/page.h
+++ b/arch/arm64/include/asm/page.h
@@ -33,19 +33,26 @@
/*
* The idmap and swapper page tables need some space reserved in the kernel
- * image. Both require a pgd and a next level table to (section) map the
- * kernel. The the swapper also maaps the FDT (see __create_page_tables for
+ * image. Both require pgd, pud (4 levels only) and pmd tables to (section)
+ * map the kernel. The swapper also maps the FDT (see __create_page_tables for
* more information).
*/
+#ifdef CONFIG_ARM64_4_LEVELS
+#define SWAPPER_DIR_SIZE (3 * PAGE_SIZE)
+#define IDMAP_DIR_SIZE (3 * PAGE_SIZE)
+#else
#define SWAPPER_DIR_SIZE (2 * PAGE_SIZE)
#define IDMAP_DIR_SIZE (2 * PAGE_SIZE)
+#endif
#ifndef __ASSEMBLY__
#ifdef CONFIG_ARM64_2_LEVELS
#include <asm/pgtable-2level-types.h>
-#else
+#elif defined(CONFIG_ARM64_3_LEVELS)
#include <asm/pgtable-3level-types.h>
+#else
+#include <asm/pgtable-4level-types.h>
#endif
extern void __cpu_clear_user_page(void *p, unsigned long user);
diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h
index 48298376e46a..8d745fae4c2d 100644
--- a/arch/arm64/include/asm/pgalloc.h
+++ b/arch/arm64/include/asm/pgalloc.h
@@ -26,6 +26,26 @@
#define check_pgt_cache() do { } while (0)
+#ifdef CONFIG_ARM64_4_LEVELS
+
+static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long addr)
+{
+ return (pud_t *)get_zeroed_page(GFP_KERNEL | __GFP_REPEAT);
+}
+
+static inline void pud_free(struct mm_struct *mm, pud_t *pud)
+{
+ BUG_ON((unsigned long)pud & (PAGE_SIZE-1));
+ free_page((unsigned long)pud);
+}
+
+static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgd, pud_t *pud)
+{
+ set_pgd(pgd, __pgd(__pa(pud) | PUD_TYPE_TABLE));
+}
+
+#endif /* CONFIG_ARM64_4_LEVELS */
+
#ifndef CONFIG_ARM64_2_LEVELS
static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long addr)
diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h
index c7c603b489b8..fddcc3efa569 100644
--- a/arch/arm64/include/asm/pgtable-hwdef.h
+++ b/arch/arm64/include/asm/pgtable-hwdef.h
@@ -18,8 +18,10 @@
#ifdef CONFIG_ARM64_2_LEVELS
#include <asm/pgtable-2level-hwdef.h>
-#else
+#elif defined(CONFIG_ARM64_3_LEVELS)
#include <asm/pgtable-3level-hwdef.h>
+#else
+#include <asm/pgtable-4level-hwdef.h>
#endif
/*
@@ -27,7 +29,7 @@
*
* Level 1 descriptor (PUD).
*/
-
+#define PUD_TYPE_TABLE (_AT(pudval_t, 3) << 0)
#define PUD_TABLE_BIT (_AT(pgdval_t, 1) << 1)
#define PUD_TYPE_MASK (_AT(pgdval_t, 3) << 0)
#define PUD_TYPE_SECT (_AT(pgdval_t, 1) << 0)
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index 6d5854972a77..d9b23efdaded 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -35,7 +35,11 @@
* VMALLOC and SPARSEMEM_VMEMMAP ranges.
*/
#define VMALLOC_START (UL(0xffffffffffffffff) << VA_BITS)
+#ifndef CONFIG_ARM64_4_LEVELS
#define VMALLOC_END (PAGE_OFFSET - UL(0x400000000) - SZ_64K)
+#else
+#define VMALLOC_END (PAGE_OFFSET - UL(0x40000000000) - SZ_64K)
+#endif
#define vmemmap ((struct page *)(VMALLOC_END + SZ_64K))
@@ -44,12 +48,16 @@
#ifndef __ASSEMBLY__
extern void __pte_error(const char *file, int line, unsigned long val);
extern void __pmd_error(const char *file, int line, unsigned long val);
+extern void __pud_error(const char *file, int line, unsigned long val);
extern void __pgd_error(const char *file, int line, unsigned long val);
#define pte_ERROR(pte) __pte_error(__FILE__, __LINE__, pte_val(pte))
#ifndef CONFIG_ARM64_2_LEVELS
#define pmd_ERROR(pmd) __pmd_error(__FILE__, __LINE__, pmd_val(pmd))
#endif
+#ifdef CONFIG_ARM64_4_LEVELS
+#define pud_ERROR(pud) __pud_error(__FILE__, __LINE__, pud_val(pud))
+#endif
#define pgd_ERROR(pgd) __pgd_error(__FILE__, __LINE__, pgd_val(pgd))
#ifdef CONFIG_SMP
@@ -347,6 +355,30 @@ static inline pmd_t *pud_page_vaddr(pud_t pud)
#endif /* CONFIG_ARM64_2_LEVELS */
+#ifdef CONFIG_ARM64_4_LEVELS
+
+#define pgd_none(pgd) (!pgd_val(pgd))
+#define pgd_bad(pgd) (!(pgd_val(pgd) & 2))
+#define pgd_present(pgd) (pgd_val(pgd))
+
+static inline void set_pgd(pgd_t *pgdp, pgd_t pgd)
+{
+ *pgdp = pgd;
+ dsb(ishst);
+}
+
+static inline void pgd_clear(pgd_t *pgdp)
+{
+ set_pgd(pgdp, __pgd(0));
+}
+
+static inline pud_t *pgd_page_vaddr(pgd_t pgd)
+{
+ return __va(pgd_val(pgd) & PHYS_MASK & (s32)PAGE_MASK);
+}
+
+#endif /* CONFIG_ARM64_4_LEVELS */
+
/* to find an entry in a page-table-directory */
#define pgd_index(addr) (((addr) >> PGDIR_SHIFT) & (PTRS_PER_PGD - 1))
@@ -355,6 +387,14 @@ static inline pmd_t *pud_page_vaddr(pud_t pud)
/* to find an entry in a kernel page-table-directory */
#define pgd_offset_k(addr) pgd_offset(&init_mm, addr)
+#ifdef CONFIG_ARM64_4_LEVELS
+#define pud_index(addr) (((addr) >> PUD_SHIFT) & (PTRS_PER_PUD - 1))
+static inline pud_t *pud_offset(pgd_t *pgd, unsigned long addr)
+{
+ return (pud_t *)pgd_page_vaddr(*pgd) + pud_index(addr);
+}
+#endif
+
/* Find an entry in the second-level page table.. */
#ifndef CONFIG_ARM64_2_LEVELS
#define pmd_index(addr) (((addr) >> PMD_SHIFT) & (PTRS_PER_PMD - 1))
diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h
index bc19101edaeb..49dc8f03362f 100644
--- a/arch/arm64/include/asm/tlb.h
+++ b/arch/arm64/include/asm/tlb.h
@@ -100,6 +100,15 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp,
}
#endif
+#ifdef CONFIG_ARM64_4_LEVELS
+static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp,
+ unsigned long addr)
+{
+ tlb_add_flush(tlb, addr);
+ tlb_remove_page(tlb, virt_to_page(pudp));
+}
+#endif
+
static inline void __tlb_remove_pmd_tlb_entry(struct mmu_gather *tlb, pmd_t *pmdp,
unsigned long address)
{
diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index fa3b7fb8a77a..847b99daad79 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -476,16 +476,42 @@ ENDPROC(__calc_phys_offset)
.quad PAGE_OFFSET
/*
- * Macro to populate the PGD for the corresponding block entry in the next
- * level (tbl) for the given virtual address.
+ * Macro to populate the PUD for the corresponding block entry in the next
+ * level (tbl) for the given virtual address in case of 4 levels.
*
- * Preserves: pgd, tbl, virt
- * Corrupts: tmp1, tmp2
+ * Preserves: pgd, virt
+ * Corrupts: tbl, tmp1, tmp2
+ * Returns: pud
*/
- .macro create_pgd_entry, pgd, tbl, virt, tmp1, tmp2
+ .macro create_pud_entry, pgd, tbl, virt, pud, tmp1, tmp2
+#ifdef CONFIG_ARM64_4_LEVELS
+ add \tbl, \tbl, #PAGE_SIZE // bump tbl 1 page up.
+ // to make room for pud
+ add \pud, \pgd, #PAGE_SIZE // pgd points to pud which
+ // follows pgd
+ lsr \tmp1, \virt, #PUD_SHIFT
+ and \tmp1, \tmp1, #PTRS_PER_PUD - 1 // PUD index
+ orr \tmp2, \tbl, #3 // PUD entry table type
+ str \tmp2, [\pud, \tmp1, lsl #3]
+#else
+ mov \pud, \tbl
+#endif
+ .endm
+
+/*
+ * Macro to populate the PGD (and possibily PUD) for the corresponding
+ * block entry in the next level (tbl) for the given virtual address.
+ *
+ * Preserves: pgd, virt
+ * Corrupts: tmp1, tmp2, tmp3
+ * Returns: tbl -> page where block mappings can be placed
+ * (changed to make room for pud with 4 levels, preserved otherwise)
+ */
+ .macro create_pgd_entry, pgd, tbl, virt, tmp1, tmp2, tmp3
+ create_pud_entry \pgd, \tbl, \virt, \tmp3, \tmp1, \tmp2
lsr \tmp1, \virt, #PGDIR_SHIFT
and \tmp1, \tmp1, #PTRS_PER_PGD - 1 // PGD index
- orr \tmp2, \tbl, #3 // PGD entry table type
+ orr \tmp2, \tmp3, #3 // PGD entry table type
str \tmp2, [\pgd, \tmp1, lsl #3]
.endm
@@ -550,7 +576,7 @@ __create_page_tables:
add x0, x25, #PAGE_SIZE // section table address
ldr x3, =KERNEL_START
add x3, x3, x28 // __pa(KERNEL_START)
- create_pgd_entry x25, x0, x3, x5, x6
+ create_pgd_entry x25, x0, x3, x1, x5, x6
ldr x6, =KERNEL_END
mov x5, x3 // __pa(KERNEL_START)
add x6, x6, x28 // __pa(KERNEL_END)
@@ -561,7 +587,7 @@ __create_page_tables:
*/
add x0, x26, #PAGE_SIZE // section table address
mov x5, #PAGE_OFFSET
- create_pgd_entry x26, x0, x5, x3, x6
+ create_pgd_entry x26, x0, x5, x1, x3, x6
ldr x6, =KERNEL_END
mov x3, x24 // phys offset
create_block_map x0, x7, x3, x5, x6
diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c
index 506f7814e305..02cd3f023e9a 100644
--- a/arch/arm64/kernel/traps.c
+++ b/arch/arm64/kernel/traps.c
@@ -339,6 +339,11 @@ void __pmd_error(const char *file, int line, unsigned long val)
pr_crit("%s:%d: bad pmd %016lx.\n", file, line, val);
}
+void __pud_error(const char *file, int line, unsigned long val)
+{
+ pr_crit("%s:%d: bad pud %016lx.\n", file, line, val);
+}
+
void __pgd_error(const char *file, int line, unsigned long val)
{
pr_crit("%s:%d: bad pgd %016lx.\n", file, line, val);
diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
index bcc965e2cce1..41cb6d3d6075 100644
--- a/arch/arm64/mm/fault.c
+++ b/arch/arm64/mm/fault.c
@@ -62,6 +62,7 @@ void show_pte(struct mm_struct *mm, unsigned long addr)
break;
pud = pud_offset(pgd, addr);
+ printk(", *pud=%016llx", pud_val(*pud));
if (pud_none(*pud) || pud_bad(*pud))
break;
diff --git a/arch/arm64/mm/ioremap.c b/arch/arm64/mm/ioremap.c
index 69000efa015e..fa324bd5a5c4 100644
--- a/arch/arm64/mm/ioremap.c
+++ b/arch/arm64/mm/ioremap.c
@@ -104,9 +104,12 @@ void __iomem *ioremap_cache(phys_addr_t phys_addr, size_t size)
EXPORT_SYMBOL(ioremap_cache);
static pte_t bm_pte[PTRS_PER_PTE] __page_aligned_bss;
-#ifndef CONFIG_ARM64_64K_PAGES
+#if CONFIG_ARM64_PGTABLE_LEVELS > 2
static pte_t bm_pmd[PTRS_PER_PMD] __page_aligned_bss;
#endif
+#if CONFIG_ARM64_PGTABLE_LEVELS > 3
+static pte_t bm_pud[PTRS_PER_PUD] __page_aligned_bss;
+#endif
static inline pud_t * __init early_ioremap_pud(unsigned long addr)
{
@@ -144,6 +147,7 @@ void __init early_ioremap_init(void)
unsigned long addr = fix_to_virt(FIX_BTMAP_BEGIN);
pgd = pgd_offset_k(addr);
+ pgd_populate(&init_mm, pgd, bm_pud);
pud = pud_offset(pgd, addr);
pud_populate(&init_mm, pud, bm_pmd);
pmd = pmd_offset(pud, addr);
diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index c43f1dd19489..c55567283cde 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -32,6 +32,7 @@
#include <asm/setup.h>
#include <asm/sizes.h>
#include <asm/tlb.h>
+#include <asm/memblock.h>
#include <asm/mmu_context.h>
#include "mm.h"
@@ -204,9 +205,16 @@ static void __init alloc_init_pud(pgd_t *pgd, unsigned long addr,
unsigned long end, unsigned long phys,
int map_io)
{
- pud_t *pud = pud_offset(pgd, addr);
+ pud_t *pud;
unsigned long next;
+ if (pgd_none(*pgd)) {
+ pud = early_alloc(PTRS_PER_PUD * sizeof(pud_t));
+ pgd_populate(&init_mm, pgd, pud);
+ }
+ BUG_ON(pgd_bad(*pgd));
+
+ pud = pud_offset(pgd, addr);
do {
next = pud_addr_end(addr, end);
@@ -290,10 +298,10 @@ static void __init map_mem(void)
* memory addressable from the initial direct kernel mapping.
*
* The initial direct kernel mapping, located at swapper_pg_dir,
- * gives us PGDIR_SIZE memory starting from PHYS_OFFSET (which must be
+ * gives us PUD_SIZE memory starting from PHYS_OFFSET (which must be
* aligned to 2MB as per Documentation/arm64/booting.txt).
*/
- limit = PHYS_OFFSET + PGDIR_SIZE;
+ limit = PHYS_OFFSET + PUD_SIZE;
memblock_set_current_limit(limit);
/* map all the memory banks */
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH v7 08/11] arm64: Convert bool ARM64_x_LEVELS to int ARM64_PGTABLE_LEVELS
2014-07-16 19:09 [PATCH v7 00/11] arm64: Support 4 levels of translation tables Catalin Marinas
` (6 preceding siblings ...)
2014-07-16 19:09 ` [PATCH v7 07/11] arm64: mm: Implement 4 levels of translation tables Catalin Marinas
@ 2014-07-16 19:09 ` Catalin Marinas
2014-07-16 19:09 ` [PATCH v7 09/11] arm64: Remove asm/pgtable-*level-hwdef.h files Catalin Marinas
` (5 subsequent siblings)
13 siblings, 0 replies; 18+ messages in thread
From: Catalin Marinas @ 2014-07-16 19:09 UTC (permalink / raw)
To: linux-arm-kernel
Rather than having several Kconfig options, define int
ARM64_PGTABLE_LEVELS which will be also useful in converting some of the
pgtable macros.
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
---
arch/arm64/Kconfig | 13 +++++--------
arch/arm64/include/asm/page.h | 6 +++---
arch/arm64/include/asm/pgalloc.h | 8 ++++----
arch/arm64/include/asm/pgtable-hwdef.h | 4 ++--
arch/arm64/include/asm/pgtable.h | 18 +++++++++---------
arch/arm64/include/asm/tlb.h | 4 ++--
arch/arm64/kernel/head.S | 2 +-
7 files changed, 26 insertions(+), 29 deletions(-)
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 24cbe72c0da9..9fe62025776b 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -209,14 +209,11 @@ config ARM64_VA_BITS
default 42 if ARM64_VA_BITS_42
default 48 if ARM64_VA_BITS_48
-config ARM64_2_LEVELS
- def_bool y if ARM64_64K_PAGES && ARM64_VA_BITS_42
-
-config ARM64_3_LEVELS
- def_bool y if ARM64_4K_PAGES && ARM64_VA_BITS_39
-
-config ARM64_4_LEVELS
- def_bool y if ARM64_4K_PAGES && ARM64_VA_BITS_48
+config ARM64_PGTABLE_LEVELS
+ int
+ default 2 if ARM64_64K_PAGES && ARM64_VA_BITS_42
+ default 3 if ARM64_4K_PAGES && ARM64_VA_BITS_39
+ default 4 if ARM64_4K_PAGES && ARM64_VA_BITS_48
config CPU_BIG_ENDIAN
bool "Build big-endian kernel"
diff --git a/arch/arm64/include/asm/page.h b/arch/arm64/include/asm/page.h
index cf9afa0366b6..a998ff478777 100644
--- a/arch/arm64/include/asm/page.h
+++ b/arch/arm64/include/asm/page.h
@@ -37,7 +37,7 @@
* map the kernel. The swapper also maps the FDT (see __create_page_tables for
* more information).
*/
-#ifdef CONFIG_ARM64_4_LEVELS
+#if CONFIG_ARM64_PGTABLE_LEVELS == 4
#define SWAPPER_DIR_SIZE (3 * PAGE_SIZE)
#define IDMAP_DIR_SIZE (3 * PAGE_SIZE)
#else
@@ -47,9 +47,9 @@
#ifndef __ASSEMBLY__
-#ifdef CONFIG_ARM64_2_LEVELS
+#if CONFIG_ARM64_PGTABLE_LEVELS == 2
#include <asm/pgtable-2level-types.h>
-#elif defined(CONFIG_ARM64_3_LEVELS)
+#elif CONFIG_ARM64_PGTABLE_LEVELS == 3
#include <asm/pgtable-3level-types.h>
#else
#include <asm/pgtable-4level-types.h>
diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h
index 8d745fae4c2d..ffef62debe82 100644
--- a/arch/arm64/include/asm/pgalloc.h
+++ b/arch/arm64/include/asm/pgalloc.h
@@ -26,7 +26,7 @@
#define check_pgt_cache() do { } while (0)
-#ifdef CONFIG_ARM64_4_LEVELS
+#if CONFIG_ARM64_PGTABLE_LEVELS > 3
static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long addr)
{
@@ -44,9 +44,9 @@ static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgd, pud_t *pud)
set_pgd(pgd, __pgd(__pa(pud) | PUD_TYPE_TABLE));
}
-#endif /* CONFIG_ARM64_4_LEVELS */
+#endif /* CONFIG_ARM64_PGTABLE_LEVELS > 3 */
-#ifndef CONFIG_ARM64_2_LEVELS
+#if CONFIG_ARM64_PGTABLE_LEVELS > 2
static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long addr)
{
@@ -64,7 +64,7 @@ static inline void pud_populate(struct mm_struct *mm, pud_t *pud, pmd_t *pmd)
set_pud(pud, __pud(__pa(pmd) | PMD_TYPE_TABLE));
}
-#endif /* CONFIG_ARM64_2_LEVELS */
+#endif /* CONFIG_ARM64_PGTABLE_LEVELS > 2 */
extern pgd_t *pgd_alloc(struct mm_struct *mm);
extern void pgd_free(struct mm_struct *mm, pgd_t *pgd);
diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h
index fddcc3efa569..d453e8bfef06 100644
--- a/arch/arm64/include/asm/pgtable-hwdef.h
+++ b/arch/arm64/include/asm/pgtable-hwdef.h
@@ -16,9 +16,9 @@
#ifndef __ASM_PGTABLE_HWDEF_H
#define __ASM_PGTABLE_HWDEF_H
-#ifdef CONFIG_ARM64_2_LEVELS
+#if CONFIG_ARM64_PGTABLE_LEVELS == 2
#include <asm/pgtable-2level-hwdef.h>
-#elif defined(CONFIG_ARM64_3_LEVELS)
+#elif CONFIG_ARM64_PGTABLE_LEVELS == 3
#include <asm/pgtable-3level-hwdef.h>
#else
#include <asm/pgtable-4level-hwdef.h>
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index d9b23efdaded..9f862e6e9286 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -35,7 +35,7 @@
* VMALLOC and SPARSEMEM_VMEMMAP ranges.
*/
#define VMALLOC_START (UL(0xffffffffffffffff) << VA_BITS)
-#ifndef CONFIG_ARM64_4_LEVELS
+#if CONFIG_ARM64_PGTABLE_LEVELS != 4
#define VMALLOC_END (PAGE_OFFSET - UL(0x400000000) - SZ_64K)
#else
#define VMALLOC_END (PAGE_OFFSET - UL(0x40000000000) - SZ_64K)
@@ -52,10 +52,10 @@ extern void __pud_error(const char *file, int line, unsigned long val);
extern void __pgd_error(const char *file, int line, unsigned long val);
#define pte_ERROR(pte) __pte_error(__FILE__, __LINE__, pte_val(pte))
-#ifndef CONFIG_ARM64_2_LEVELS
+#if CONFIG_ARM64_PGTABLE_LEVELS > 2
#define pmd_ERROR(pmd) __pmd_error(__FILE__, __LINE__, pmd_val(pmd))
#endif
-#ifdef CONFIG_ARM64_4_LEVELS
+#if CONFIG_ARM64_PGTABLE_LEVELS > 3
#define pud_ERROR(pud) __pud_error(__FILE__, __LINE__, pud_val(pud))
#endif
#define pgd_ERROR(pgd) __pgd_error(__FILE__, __LINE__, pgd_val(pgd))
@@ -331,7 +331,7 @@ static inline pte_t *pmd_page_vaddr(pmd_t pmd)
*/
#define mk_pte(page,prot) pfn_pte(page_to_pfn(page),prot)
-#ifndef CONFIG_ARM64_2_LEVELS
+#if CONFIG_ARM64_PGTABLE_LEVELS > 2
#define pud_none(pud) (!pud_val(pud))
#define pud_bad(pud) (!(pud_val(pud) & 2))
@@ -353,9 +353,9 @@ static inline pmd_t *pud_page_vaddr(pud_t pud)
return __va(pud_val(pud) & PHYS_MASK & (s32)PAGE_MASK);
}
-#endif /* CONFIG_ARM64_2_LEVELS */
+#endif /* CONFIG_ARM64_PGTABLE_LEVELS > 2 */
-#ifdef CONFIG_ARM64_4_LEVELS
+#if CONFIG_ARM64_PGTABLE_LEVELS > 3
#define pgd_none(pgd) (!pgd_val(pgd))
#define pgd_bad(pgd) (!(pgd_val(pgd) & 2))
@@ -377,7 +377,7 @@ static inline pud_t *pgd_page_vaddr(pgd_t pgd)
return __va(pgd_val(pgd) & PHYS_MASK & (s32)PAGE_MASK);
}
-#endif /* CONFIG_ARM64_4_LEVELS */
+#endif /* CONFIG_ARM64_PGTABLE_LEVELS > 3 */
/* to find an entry in a page-table-directory */
#define pgd_index(addr) (((addr) >> PGDIR_SHIFT) & (PTRS_PER_PGD - 1))
@@ -387,7 +387,7 @@ static inline pud_t *pgd_page_vaddr(pgd_t pgd)
/* to find an entry in a kernel page-table-directory */
#define pgd_offset_k(addr) pgd_offset(&init_mm, addr)
-#ifdef CONFIG_ARM64_4_LEVELS
+#if CONFIG_ARM64_PGTABLE_LEVELS > 3
#define pud_index(addr) (((addr) >> PUD_SHIFT) & (PTRS_PER_PUD - 1))
static inline pud_t *pud_offset(pgd_t *pgd, unsigned long addr)
{
@@ -396,7 +396,7 @@ static inline pud_t *pud_offset(pgd_t *pgd, unsigned long addr)
#endif
/* Find an entry in the second-level page table.. */
-#ifndef CONFIG_ARM64_2_LEVELS
+#if CONFIG_ARM64_PGTABLE_LEVELS > 2
#define pmd_index(addr) (((addr) >> PMD_SHIFT) & (PTRS_PER_PMD - 1))
static inline pmd_t *pmd_offset(pud_t *pud, unsigned long addr)
{
diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h
index 49dc8f03362f..62731ef9749a 100644
--- a/arch/arm64/include/asm/tlb.h
+++ b/arch/arm64/include/asm/tlb.h
@@ -91,7 +91,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte,
tlb_remove_page(tlb, pte);
}
-#ifndef CONFIG_ARM64_2_LEVELS
+#if CONFIG_ARM64_PGTABLE_LEVELS > 2
static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp,
unsigned long addr)
{
@@ -100,7 +100,7 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp,
}
#endif
-#ifdef CONFIG_ARM64_4_LEVELS
+#if CONFIG_ARM64_PGTABLE_LEVELS > 3
static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp,
unsigned long addr)
{
diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index 847b99daad79..019f81d9f1d5 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -484,7 +484,7 @@ ENDPROC(__calc_phys_offset)
* Returns: pud
*/
.macro create_pud_entry, pgd, tbl, virt, pud, tmp1, tmp2
-#ifdef CONFIG_ARM64_4_LEVELS
+#if CONFIG_ARM64_PGTABLE_LEVELS == 4
add \tbl, \tbl, #PAGE_SIZE // bump tbl 1 page up.
// to make room for pud
add \pud, \pgd, #PAGE_SIZE // pgd points to pud which
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH v7 09/11] arm64: Remove asm/pgtable-*level-hwdef.h files
2014-07-16 19:09 [PATCH v7 00/11] arm64: Support 4 levels of translation tables Catalin Marinas
` (7 preceding siblings ...)
2014-07-16 19:09 ` [PATCH v7 08/11] arm64: Convert bool ARM64_x_LEVELS to int ARM64_PGTABLE_LEVELS Catalin Marinas
@ 2014-07-16 19:09 ` Catalin Marinas
2014-07-16 19:09 ` [PATCH v7 10/11] arm64: Clean up the initial page table creation in head.S Catalin Marinas
` (4 subsequent siblings)
13 siblings, 0 replies; 18+ messages in thread
From: Catalin Marinas @ 2014-07-16 19:09 UTC (permalink / raw)
To: linux-arm-kernel
The macros in these files can easily be computed based on PAGE_SHIFT and
VA_BITS, so just remove them and add the corresponding macros to
asm/pgtable-hwdef.h
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
---
arch/arm64/include/asm/pgtable-2level-hwdef.h | 43 -----------------------
arch/arm64/include/asm/pgtable-3level-hwdef.h | 50 ---------------------------
arch/arm64/include/asm/pgtable-4level-hwdef.h | 50 ---------------------------
arch/arm64/include/asm/pgtable-hwdef.h | 42 ++++++++++++++++++----
4 files changed, 36 insertions(+), 149 deletions(-)
delete mode 100644 arch/arm64/include/asm/pgtable-2level-hwdef.h
delete mode 100644 arch/arm64/include/asm/pgtable-3level-hwdef.h
delete mode 100644 arch/arm64/include/asm/pgtable-4level-hwdef.h
diff --git a/arch/arm64/include/asm/pgtable-2level-hwdef.h b/arch/arm64/include/asm/pgtable-2level-hwdef.h
deleted file mode 100644
index 2593b490c56a..000000000000
--- a/arch/arm64/include/asm/pgtable-2level-hwdef.h
+++ /dev/null
@@ -1,43 +0,0 @@
-/*
- * Copyright (C) 2012 ARM Ltd.
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program. If not, see <http://www.gnu.org/licenses/>.
- */
-#ifndef __ASM_PGTABLE_2LEVEL_HWDEF_H
-#define __ASM_PGTABLE_2LEVEL_HWDEF_H
-
-/*
- * With LPAE and 64KB pages, there are 2 levels of page tables. Each level has
- * 8192 entries of 8 bytes each, occupying a 64KB page. Levels 0 and 1 are not
- * used. The 2nd level table (PGD for Linux) can cover a range of 4TB, each
- * entry representing 512MB. The user and kernel address spaces are limited to
- * 4TB in the 64KB page configuration.
- */
-#define PTRS_PER_PTE 8192
-#define PTRS_PER_PGD 8192
-
-/*
- * PGDIR_SHIFT determines the size a top-level page table entry can map.
- */
-#define PGDIR_SHIFT 29
-#define PGDIR_SIZE (_AC(1, UL) << PGDIR_SHIFT)
-#define PGDIR_MASK (~(PGDIR_SIZE-1))
-
-/*
- * section address mask and size definitions.
- */
-#define SECTION_SHIFT 29
-#define SECTION_SIZE (_AC(1, UL) << SECTION_SHIFT)
-#define SECTION_MASK (~(SECTION_SIZE-1))
-
-#endif
diff --git a/arch/arm64/include/asm/pgtable-3level-hwdef.h b/arch/arm64/include/asm/pgtable-3level-hwdef.h
deleted file mode 100644
index 3dbf941d7767..000000000000
--- a/arch/arm64/include/asm/pgtable-3level-hwdef.h
+++ /dev/null
@@ -1,50 +0,0 @@
-/*
- * Copyright (C) 2012 ARM Ltd.
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program. If not, see <http://www.gnu.org/licenses/>.
- */
-#ifndef __ASM_PGTABLE_3LEVEL_HWDEF_H
-#define __ASM_PGTABLE_3LEVEL_HWDEF_H
-
-/*
- * With LPAE and 4KB pages, there are 3 levels of page tables. Each level has
- * 512 entries of 8 bytes each, occupying a 4K page. The first level table
- * covers a range of 512GB, each entry representing 1GB. The user and kernel
- * address spaces are limited to 512GB each.
- */
-#define PTRS_PER_PTE 512
-#define PTRS_PER_PMD 512
-#define PTRS_PER_PGD 512
-
-/*
- * PGDIR_SHIFT determines the size a top-level page table entry can map.
- */
-#define PGDIR_SHIFT 30
-#define PGDIR_SIZE (_AC(1, UL) << PGDIR_SHIFT)
-#define PGDIR_MASK (~(PGDIR_SIZE-1))
-
-/*
- * PMD_SHIFT determines the size a middle-level page table entry can map.
- */
-#define PMD_SHIFT 21
-#define PMD_SIZE (_AC(1, UL) << PMD_SHIFT)
-#define PMD_MASK (~(PMD_SIZE-1))
-
-/*
- * section address mask and size definitions.
- */
-#define SECTION_SHIFT 21
-#define SECTION_SIZE (_AC(1, UL) << SECTION_SHIFT)
-#define SECTION_MASK (~(SECTION_SIZE-1))
-
-#endif
diff --git a/arch/arm64/include/asm/pgtable-4level-hwdef.h b/arch/arm64/include/asm/pgtable-4level-hwdef.h
deleted file mode 100644
index 0ec84e2274a3..000000000000
--- a/arch/arm64/include/asm/pgtable-4level-hwdef.h
+++ /dev/null
@@ -1,50 +0,0 @@
-/*
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program. If not, see <http://www.gnu.org/licenses/>.
- */
-#ifndef __ASM_PGTABLE_4LEVEL_HWDEF_H
-#define __ASM_PGTABLE_4LEVEL_HWDEF_H
-
-#define PTRS_PER_PTE 512
-#define PTRS_PER_PMD 512
-#define PTRS_PER_PUD 512
-#define PTRS_PER_PGD 512
-
-/*
- * PGDIR_SHIFT determines the size a top-level page table entry can map.
- */
-#define PGDIR_SHIFT 39
-#define PGDIR_SIZE (_AC(1, UL) << PGDIR_SHIFT)
-#define PGDIR_MASK (~(PGDIR_SIZE-1))
-
-/*
- * PUD_SHIFT determines the size the second level page table entry can map.
- */
-#define PUD_SHIFT 30
-#define PUD_SIZE (_AC(1, UL) << PUD_SHIFT)
-#define PUD_MASK (~(PUD_SIZE-1))
-
-/*
- * PMD_SHIFT determines the size the third level page table entry can map.
- */
-#define PMD_SHIFT 21
-#define PMD_SIZE (_AC(1, UL) << PMD_SHIFT)
-#define PMD_MASK (~(PMD_SIZE-1))
-
-/*
- * section address mask and size definitions.
- */
-#define SECTION_SHIFT 21
-#define SECTION_SIZE (_AC(1, UL) << SECTION_SHIFT)
-#define SECTION_MASK (~(SECTION_SIZE-1))
-
-#endif
diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h
index d453e8bfef06..88174e0bfafe 100644
--- a/arch/arm64/include/asm/pgtable-hwdef.h
+++ b/arch/arm64/include/asm/pgtable-hwdef.h
@@ -16,15 +16,45 @@
#ifndef __ASM_PGTABLE_HWDEF_H
#define __ASM_PGTABLE_HWDEF_H
-#if CONFIG_ARM64_PGTABLE_LEVELS == 2
-#include <asm/pgtable-2level-hwdef.h>
-#elif CONFIG_ARM64_PGTABLE_LEVELS == 3
-#include <asm/pgtable-3level-hwdef.h>
-#else
-#include <asm/pgtable-4level-hwdef.h>
+#define PTRS_PER_PTE (1 << (PAGE_SHIFT - 3))
+
+/*
+ * PMD_SHIFT determines the size a level 2 page table entry can map.
+ */
+#if CONFIG_ARM64_PGTABLE_LEVELS > 2
+#define PMD_SHIFT ((PAGE_SHIFT - 3) * 2 + 3)
+#define PMD_SIZE (_AC(1, UL) << PMD_SHIFT)
+#define PMD_MASK (~(PMD_SIZE-1))
+#define PTRS_PER_PMD PTRS_PER_PTE
+#endif
+
+/*
+ * PUD_SHIFT determines the size a level 1 page table entry can map.
+ */
+#if CONFIG_ARM64_PGTABLE_LEVELS > 3
+#define PUD_SHIFT ((PAGE_SHIFT - 3) * 3 + 3)
+#define PUD_SIZE (_AC(1, UL) << PUD_SHIFT)
+#define PUD_MASK (~(PUD_SIZE-1))
+#define PTRS_PER_PUD PTRS_PER_PTE
#endif
/*
+ * PGDIR_SHIFT determines the size a top-level page table entry can map
+ * (depending on the configuration, this level can be 0, 1 or 2).
+ */
+#define PGDIR_SHIFT ((PAGE_SHIFT - 3) * CONFIG_ARM64_PGTABLE_LEVELS + 3)
+#define PGDIR_SIZE (_AC(1, UL) << PGDIR_SHIFT)
+#define PGDIR_MASK (~(PGDIR_SIZE-1))
+#define PTRS_PER_PGD (1 << (VA_BITS - PGDIR_SHIFT))
+
+/*
+ * Section address mask and size definitions.
+ */
+#define SECTION_SHIFT PMD_SHIFT
+#define SECTION_SIZE (_AC(1, UL) << SECTION_SHIFT)
+#define SECTION_MASK (~(SECTION_SIZE-1))
+
+/*
* Hardware page table definitions.
*
* Level 1 descriptor (PUD).
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH v7 10/11] arm64: Clean up the initial page table creation in head.S
2014-07-16 19:09 [PATCH v7 00/11] arm64: Support 4 levels of translation tables Catalin Marinas
` (8 preceding siblings ...)
2014-07-16 19:09 ` [PATCH v7 09/11] arm64: Remove asm/pgtable-*level-hwdef.h files Catalin Marinas
@ 2014-07-16 19:09 ` Catalin Marinas
2014-07-16 19:09 ` [PATCH v7 11/11] arm64: Determine the vmalloc/vmemmap space at build time based on VA_BITS Catalin Marinas
` (3 subsequent siblings)
13 siblings, 0 replies; 18+ messages in thread
From: Catalin Marinas @ 2014-07-16 19:09 UTC (permalink / raw)
To: linux-arm-kernel
This patch adds a create_table_entry macro which is used to populate pgd
and pud entries, also reducing the number of arguments for
create_pgd_entry.
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
---
arch/arm64/kernel/head.S | 59 ++++++++++++++++++++++--------------------------
1 file changed, 27 insertions(+), 32 deletions(-)
diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index 019f81d9f1d5..a6db505411bc 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -476,43 +476,38 @@ ENDPROC(__calc_phys_offset)
.quad PAGE_OFFSET
/*
- * Macro to populate the PUD for the corresponding block entry in the next
- * level (tbl) for the given virtual address in case of 4 levels.
+ * Macro to create a table entry to the next page.
*
- * Preserves: pgd, virt
- * Corrupts: tbl, tmp1, tmp2
- * Returns: pud
+ * tbl: page table address
+ * virt: virtual address
+ * shift: #imm page table shift
+ * ptrs: #imm pointers per table page
+ *
+ * Preserves: virt
+ * Corrupts: tmp1, tmp2
+ * Returns: tbl -> next level table page address
*/
- .macro create_pud_entry, pgd, tbl, virt, pud, tmp1, tmp2
-#if CONFIG_ARM64_PGTABLE_LEVELS == 4
- add \tbl, \tbl, #PAGE_SIZE // bump tbl 1 page up.
- // to make room for pud
- add \pud, \pgd, #PAGE_SIZE // pgd points to pud which
- // follows pgd
- lsr \tmp1, \virt, #PUD_SHIFT
- and \tmp1, \tmp1, #PTRS_PER_PUD - 1 // PUD index
- orr \tmp2, \tbl, #3 // PUD entry table type
- str \tmp2, [\pud, \tmp1, lsl #3]
-#else
- mov \pud, \tbl
-#endif
+ .macro create_table_entry, tbl, virt, shift, ptrs, tmp1, tmp2
+ lsr \tmp1, \virt, #\shift
+ and \tmp1, \tmp1, #\ptrs - 1 // table index
+ add \tmp2, \tbl, #PAGE_SIZE
+ orr \tmp2, \tmp2, #PMD_TYPE_TABLE // address of next table and entry type
+ str \tmp2, [\tbl, \tmp1, lsl #3]
+ add \tbl, \tbl, #PAGE_SIZE // next level table page
.endm
/*
* Macro to populate the PGD (and possibily PUD) for the corresponding
* block entry in the next level (tbl) for the given virtual address.
*
- * Preserves: pgd, virt
- * Corrupts: tmp1, tmp2, tmp3
- * Returns: tbl -> page where block mappings can be placed
- * (changed to make room for pud with 4 levels, preserved otherwise)
+ * Preserves: tbl, next, virt
+ * Corrupts: tmp1, tmp2
*/
- .macro create_pgd_entry, pgd, tbl, virt, tmp1, tmp2, tmp3
- create_pud_entry \pgd, \tbl, \virt, \tmp3, \tmp1, \tmp2
- lsr \tmp1, \virt, #PGDIR_SHIFT
- and \tmp1, \tmp1, #PTRS_PER_PGD - 1 // PGD index
- orr \tmp2, \tmp3, #3 // PGD entry table type
- str \tmp2, [\pgd, \tmp1, lsl #3]
+ .macro create_pgd_entry, tbl, virt, tmp1, tmp2
+ create_table_entry \tbl, \virt, PGDIR_SHIFT, PTRS_PER_PGD, \tmp1, \tmp2
+#if CONFIG_ARM64_PGTABLE_LEVELS == 4
+ create_table_entry \tbl, \virt, PUD_SHIFT, PTRS_PER_PUD, \tmp1, \tmp2
+#endif
.endm
/*
@@ -573,10 +568,10 @@ __create_page_tables:
/*
* Create the identity mapping.
*/
- add x0, x25, #PAGE_SIZE // section table address
+ mov x0, x25 // idmap_pg_dir
ldr x3, =KERNEL_START
add x3, x3, x28 // __pa(KERNEL_START)
- create_pgd_entry x25, x0, x3, x1, x5, x6
+ create_pgd_entry x0, x3, x5, x6
ldr x6, =KERNEL_END
mov x5, x3 // __pa(KERNEL_START)
add x6, x6, x28 // __pa(KERNEL_END)
@@ -585,9 +580,9 @@ __create_page_tables:
/*
* Map the kernel image (starting with PHYS_OFFSET).
*/
- add x0, x26, #PAGE_SIZE // section table address
+ mov x0, x26 // swapper_pg_dir
mov x5, #PAGE_OFFSET
- create_pgd_entry x26, x0, x5, x1, x3, x6
+ create_pgd_entry x0, x5, x3, x6
ldr x6, =KERNEL_END
mov x3, x24 // phys offset
create_block_map x0, x7, x3, x5, x6
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH v7 11/11] arm64: Determine the vmalloc/vmemmap space at build time based on VA_BITS
2014-07-16 19:09 [PATCH v7 00/11] arm64: Support 4 levels of translation tables Catalin Marinas
` (9 preceding siblings ...)
2014-07-16 19:09 ` [PATCH v7 10/11] arm64: Clean up the initial page table creation in head.S Catalin Marinas
@ 2014-07-16 19:09 ` Catalin Marinas
2014-07-17 10:15 ` [PATCH v7 00/11] arm64: Support 4 levels of translation tables Will Deacon
` (2 subsequent siblings)
13 siblings, 0 replies; 18+ messages in thread
From: Catalin Marinas @ 2014-07-16 19:09 UTC (permalink / raw)
To: linux-arm-kernel
Rather than guessing what the maximum vmmemap space should be, this
patch allows the calculation based on the VA_BITS and sizeof(struct
page). The vmalloc space extends to the beginning of the vmemmap space.
Since the virtual kernel memory layout now depends on the build
configuration, this patch removes the detailed description in
Documentation/arm64/memory.txt in favour of information printed during
kernel booting.
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
---
Documentation/arm64/memory.txt | 98 ++++++----------------------------------
arch/arm64/include/asm/pgtable.h | 13 ++++--
arch/arm64/mm/init.c | 22 ++++++---
3 files changed, 38 insertions(+), 95 deletions(-)
diff --git a/Documentation/arm64/memory.txt b/Documentation/arm64/memory.txt
index 4c720d698e8e..8845d0847a66 100644
--- a/Documentation/arm64/memory.txt
+++ b/Documentation/arm64/memory.txt
@@ -2,19 +2,18 @@
==============================
Author: Catalin Marinas <catalin.marinas@arm.com>
-Date : 20 February 2012
This document describes the virtual memory layout used by the AArch64
Linux kernel. The architecture allows up to 4 levels of translation
tables with a 4KB page size and up to 3 levels with a 64KB page size.
-AArch64 Linux uses either 3 levels or 4 levels of translation tables with
-the 4KB page configuration, allowing 39-bit (512GB) or 48-bit (256TB)
-virtual addresses, respectively, for both user and kernel. With 64KB
-pages, only 2 levels of translation tables, allowing 42-bit (4TB)
+AArch64 Linux uses either 3 levels or 4 levels of translation tables
+with the 4KB page configuration, allowing 39-bit (512GB) or 48-bit
+(256TB) virtual addresses, respectively, for both user and kernel. With
+64KB pages, only 2 levels of translation tables, allowing 42-bit (4TB)
virtual address, are used but the memory layout is the same.
-User addresses have bits 63:39 set to 0 while the kernel addresses have
+User addresses have bits 63:48 set to 0 while the kernel addresses have
the same bits set to 1. TTBRx selection is given by bit 63 of the
virtual address. The swapper_pg_dir contains only kernel (global)
mappings while the user pgd contains only user (non-global) mappings.
@@ -27,26 +26,7 @@ AArch64 Linux memory layout with 4KB pages + 3 levels:
Start End Size Use
-----------------------------------------------------------------------
0000000000000000 0000007fffffffff 512GB user
-
-ffffff8000000000 ffffffbbfffeffff ~240GB vmalloc
-
-ffffffbbffff0000 ffffffbbffffffff 64KB [guard page]
-
-ffffffbc00000000 ffffffbdffffffff 8GB vmemmap
-
-ffffffbe00000000 ffffffbffbbfffff ~8GB [guard, future vmmemap]
-
-ffffffbffa000000 ffffffbffaffffff 16MB PCI I/O space
-
-ffffffbffb000000 ffffffbffbbfffff 12MB [guard]
-
-ffffffbffbc00000 ffffffbffbdfffff 2MB fixed mappings
-
-ffffffbffbe00000 ffffffbffbffffff 2MB [guard]
-
-ffffffbffc000000 ffffffbfffffffff 64MB modules
-
-ffffffc000000000 ffffffffffffffff 256GB kernel logical memory map
+ffffff8000000000 ffffffffffffffff 512GB kernel
AArch64 Linux memory layout with 4KB pages + 4 levels:
@@ -54,26 +34,7 @@ AArch64 Linux memory layout with 4KB pages + 4 levels:
Start End Size Use
-----------------------------------------------------------------------
0000000000000000 0000ffffffffffff 256TB user
-
-ffff000000000000 ffff7bfffffeffff ~124TB vmalloc
-
-ffff7bffffff0000 ffff7bffffffffff 64KB [guard page]
-
-ffff7c0000000000 ffff7dffffffffff 2TB vmemmap
-
-ffff7e0000000000 ffff7ffffbbfffff ~2TB [guard, future vmmemap]
-
-ffff7ffffa000000 ffff7ffffaffffff 16MB PCI I/O space
-
-ffff7ffffb000000 ffff7ffffbbfffff 12MB [guard]
-
-ffff7ffffbc00000 ffff7ffffbdfffff 2MB fixed mappings
-
-ffff7ffffbe00000 ffff7ffffbffffff 2MB [guard]
-
-ffff7ffffc000000 ffff7fffffffffff 64MB modules
-
-ffff800000000000 ffffffffffffffff 128TB kernel logical memory map
+ffff000000000000 ffffffffffffffff 256TB kernel
AArch64 Linux memory layout with 64KB pages + 2 levels:
@@ -81,44 +42,14 @@ AArch64 Linux memory layout with 64KB pages + 2 levels:
Start End Size Use
-----------------------------------------------------------------------
0000000000000000 000003ffffffffff 4TB user
+fffffc0000000000 ffffffffffffffff 4TB kernel
-fffffc0000000000 fffffdfbfffeffff ~2TB vmalloc
-
-fffffdfbffff0000 fffffdfbffffffff 64KB [guard page]
-fffffdfc00000000 fffffdfdffffffff 8GB vmemmap
+For details of the virtual kernel memory layout please see the kernel
+booting log.
-fffffdfe00000000 fffffdfffbbfffff ~8GB [guard, future vmmemap]
-fffffdfffa000000 fffffdfffaffffff 16MB PCI I/O space
-
-fffffdfffb000000 fffffdfffbbfffff 12MB [guard]
-
-fffffdfffbc00000 fffffdfffbdfffff 2MB fixed mappings
-
-fffffdfffbe00000 fffffdfffbffffff 2MB [guard]
-
-fffffdfffc000000 fffffdffffffffff 64MB modules
-
-fffffe0000000000 ffffffffffffffff 2TB kernel logical memory map
-
-
-Translation table lookup with 4KB pages + 3 levels:
-
-+--------+--------+--------+--------+--------+--------+--------+--------+
-|63 56|55 48|47 40|39 32|31 24|23 16|15 8|7 0|
-+--------+--------+--------+--------+--------+--------+--------+--------+
- | | | | | |
- | | | | | v
- | | | | | [11:0] in-page offset
- | | | | +-> [20:12] L3 index
- | | | +-----------> [29:21] L2 index
- | | +---------------------> [38:30] L1 index
- | +-------------------------------> [47:39] L0 index (not used)
- +-------------------------------------------------> [63] TTBR0/1
-
-
-Translation table lookup with 4KB pages + 4 levels:
+Translation table lookup with 4KB pages:
+--------+--------+--------+--------+--------+--------+--------+--------+
|63 56|55 48|47 40|39 32|31 24|23 16|15 8|7 0|
@@ -133,7 +64,7 @@ Translation table lookup with 4KB pages + 4 levels:
+-------------------------------------------------> [63] TTBR0/1
-Translation table lookup with 64KB pages + 2 levels:
+Translation table lookup with 64KB pages:
+--------+--------+--------+--------+--------+--------+--------+--------+
|63 56|55 48|47 40|39 32|31 24|23 16|15 8|7 0|
@@ -142,10 +73,11 @@ Translation table lookup with 64KB pages + 2 levels:
| | | | v
| | | | [15:0] in-page offset
| | | +----------> [28:16] L3 index
- | | +--------------------------> [41:29] L2 index (only 38:29 used)
- | +-------------------------------> [47:42] L1 index (not used)
+ | | +--------------------------> [41:29] L2 index
+ | +-------------------------------> [47:42] L1 index
+-------------------------------------------------> [63] TTBR0/1
+
When using KVM, the hypervisor maps kernel pages in EL2, at a fixed
offset from the kernel VA (top 24bits of the kernel VA set to zero):
diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index 9f862e6e9286..ec82789d03c3 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -33,13 +33,16 @@
/*
* VMALLOC and SPARSEMEM_VMEMMAP ranges.
+ *
+ * VMEMAP_SIZE: allows the whole VA space to be covered by a struct page array
+ * (rounded up to PUD_SIZE).
+ * VMALLOC_START: beginning of the kernel VA space
+ * VMALLOC_END: extends to the available space below vmmemmap, PCI I/O space,
+ * fixed mappings and modules
*/
+#define VMEMMAP_SIZE ALIGN((1UL << (VA_BITS - PAGE_SHIFT)) * sizeof(struct page), PUD_SIZE)
#define VMALLOC_START (UL(0xffffffffffffffff) << VA_BITS)
-#if CONFIG_ARM64_PGTABLE_LEVELS != 4
-#define VMALLOC_END (PAGE_OFFSET - UL(0x400000000) - SZ_64K)
-#else
-#define VMALLOC_END (PAGE_OFFSET - UL(0x40000000000) - SZ_64K)
-#endif
+#define VMALLOC_END (PAGE_OFFSET - PUD_SIZE - VMEMMAP_SIZE - SZ_64K)
#define vmemmap ((struct page *)(VMALLOC_END + SZ_64K))
diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
index 7f68804814a1..0b32504e280f 100644
--- a/arch/arm64/mm/init.c
+++ b/arch/arm64/mm/init.c
@@ -33,6 +33,7 @@
#include <linux/dma-mapping.h>
#include <linux/dma-contiguous.h>
+#include <asm/fixmap.h>
#include <asm/sections.h>
#include <asm/setup.h>
#include <asm/sizes.h>
@@ -256,26 +257,33 @@ void __init mem_init(void)
#define MLK(b, t) b, t, ((t) - (b)) >> 10
#define MLM(b, t) b, t, ((t) - (b)) >> 20
+#define MLG(b, t) b, t, ((t) - (b)) >> 30
#define MLK_ROUNDUP(b, t) b, t, DIV_ROUND_UP(((t) - (b)), SZ_1K)
pr_notice("Virtual kernel memory layout:\n"
- " vmalloc : 0x%16lx - 0x%16lx (%6ld MB)\n"
+ " vmalloc : 0x%16lx - 0x%16lx (%6ld GB)\n"
#ifdef CONFIG_SPARSEMEM_VMEMMAP
- " vmemmap : 0x%16lx - 0x%16lx (%6ld MB)\n"
+ " vmemmap : 0x%16lx - 0x%16lx (%6ld GB maximum)\n"
+ " 0x%16lx - 0x%16lx (%6ld MB actual)\n"
#endif
+ " PCI I/O : 0x%16lx - 0x%16lx (%6ld MB)\n"
+ " fixed : 0x%16lx - 0x%16lx (%6ld KB)\n"
" modules : 0x%16lx - 0x%16lx (%6ld MB)\n"
" memory : 0x%16lx - 0x%16lx (%6ld MB)\n"
- " .init : 0x%p" " - 0x%p" " (%6ld kB)\n"
- " .text : 0x%p" " - 0x%p" " (%6ld kB)\n"
- " .data : 0x%p" " - 0x%p" " (%6ld kB)\n",
- MLM(VMALLOC_START, VMALLOC_END),
+ " .init : 0x%p" " - 0x%p" " (%6ld KB)\n"
+ " .text : 0x%p" " - 0x%p" " (%6ld KB)\n"
+ " .data : 0x%p" " - 0x%p" " (%6ld KB)\n",
+ MLG(VMALLOC_START, VMALLOC_END),
#ifdef CONFIG_SPARSEMEM_VMEMMAP
+ MLG((unsigned long)vmemmap,
+ (unsigned long)vmemmap + VMEMMAP_SIZE),
MLM((unsigned long)virt_to_page(PAGE_OFFSET),
(unsigned long)virt_to_page(high_memory)),
#endif
+ MLM((unsigned long)PCI_IOBASE, (unsigned long)PCI_IOBASE + SZ_16M),
+ MLK(FIXADDR_START, FIXADDR_TOP),
MLM(MODULES_VADDR, MODULES_END),
MLM(PAGE_OFFSET, (unsigned long)high_memory),
-
MLK_ROUNDUP(__init_begin, __init_end),
MLK_ROUNDUP(_text, _etext),
MLK_ROUNDUP(_sdata, _edata));
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH v7 03/11] arm64: Do not initialise the fixmap page tables in head.S
2014-07-16 19:09 ` [PATCH v7 03/11] arm64: Do not initialise the fixmap page tables in head.S Catalin Marinas
@ 2014-07-16 23:14 ` Geoff Levand
0 siblings, 0 replies; 18+ messages in thread
From: Geoff Levand @ 2014-07-16 23:14 UTC (permalink / raw)
To: linux-arm-kernel
On Wed, 2014-07-16 at 20:09 +0100, Catalin Marinas wrote:
> diff --git a/arch/arm64/include/asm/page.h b/arch/arm64/include/asm/page.h
> + * image. Both require a pgd and a next level table to (section) map the
> + * kernel. The the swapper also maaps the FDT (see __create_page_tables for
^^^^^ maps
-Geoff
^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH v7 00/11] arm64: Support 4 levels of translation tables
2014-07-16 19:09 [PATCH v7 00/11] arm64: Support 4 levels of translation tables Catalin Marinas
` (10 preceding siblings ...)
2014-07-16 19:09 ` [PATCH v7 11/11] arm64: Determine the vmalloc/vmemmap space at build time based on VA_BITS Catalin Marinas
@ 2014-07-17 10:15 ` Will Deacon
2014-07-18 17:12 ` [PATCH 12/11] arm64: Remove asm/pgtable-*level-types.h files Catalin Marinas
2014-07-21 15:09 ` [PATCH 13/11] arm64: Add support for 48-bit VA space with 64KB page configuration Catalin Marinas
13 siblings, 0 replies; 18+ messages in thread
From: Will Deacon @ 2014-07-17 10:15 UTC (permalink / raw)
To: linux-arm-kernel
On Wed, Jul 16, 2014 at 08:09:41PM +0100, Catalin Marinas wrote:
> Hi,
>
> I've taken over Jungseok's 4-level page table series here:
>
> https://lkml.kernel.org/g/000001cf6dc6$03ded0c0$0b9c7240$@samsung.com
>
> I kept the same series numbering, so this is normally version 7. Changes
> since v6:
>
> - Additional fixes for duplicate (SWAPPER|IDMAP)_DIR_SIZE definitions
> - Removed the fixmap pmd from swapper_pg_dir populated in head.S
> - Several clean-ups in Jungseok's patches (annotated above my
> signed-off-by line)
> - Removal of the pgtable-*level-hwdef.h files
> - Converting levels config options to int ARM64_PGTABLE_LEVELS
>
> Important change: I dropped the KVM support for 4 levels temporarily. I
> plan to revive them but didn't have for v7.
While you're looking at KVM, can you check if this breaks the SMMU driver
too, please? We re-use the CPU page-table functions in there and, whilst it
does have PUD code, a quick eyeball/test wouldn't hurt.
Will
^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH v7 02/11] arm64: Remove duplicate (SWAPPER|IDMAP)_DIR_SIZE definitions
2014-07-16 19:09 ` [PATCH v7 02/11] arm64: Remove duplicate (SWAPPER|IDMAP)_DIR_SIZE definitions Catalin Marinas
@ 2014-07-17 10:49 ` Mark Rutland
0 siblings, 0 replies; 18+ messages in thread
From: Mark Rutland @ 2014-07-17 10:49 UTC (permalink / raw)
To: linux-arm-kernel
On Wed, Jul 16, 2014 at 08:09:43PM +0100, Catalin Marinas wrote:
> Just keep the asm/page.h definition as this is included in vmlinux.lds.S
> as well.
My bad, I should've spotted that when I moved idmap and swapper next to
the BSS. Thanks for cleaning this up.
I guess we got lucky with nothing including both pgtable.h and page.h? I
don't recall seeing any duplicate definition warnings.
> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Acked-by: Mark Rutland <mark.rutland@arm.com>
Thanks,
Mark.
> ---
> arch/arm64/include/asm/pgtable.h | 3 ---
> 1 file changed, 3 deletions(-)
>
> diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
> index e0ccceb317d9..d7455fa83bc7 100644
> --- a/arch/arm64/include/asm/pgtable.h
> +++ b/arch/arm64/include/asm/pgtable.h
> @@ -383,9 +383,6 @@ static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot)
> extern pgd_t swapper_pg_dir[PTRS_PER_PGD];
> extern pgd_t idmap_pg_dir[PTRS_PER_PGD];
>
> -#define SWAPPER_DIR_SIZE (3 * PAGE_SIZE)
> -#define IDMAP_DIR_SIZE (2 * PAGE_SIZE)
> -
> /*
> * Encode and decode a swap entry:
> * bits 0-1: present (must be zero)
>
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>
^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH 12/11] arm64: Remove asm/pgtable-*level-types.h files
2014-07-16 19:09 [PATCH v7 00/11] arm64: Support 4 levels of translation tables Catalin Marinas
` (11 preceding siblings ...)
2014-07-17 10:15 ` [PATCH v7 00/11] arm64: Support 4 levels of translation tables Will Deacon
@ 2014-07-18 17:12 ` Catalin Marinas
2014-07-21 15:09 ` [PATCH 13/11] arm64: Add support for 48-bit VA space with 64KB page configuration Catalin Marinas
13 siblings, 0 replies; 18+ messages in thread
From: Catalin Marinas @ 2014-07-18 17:12 UTC (permalink / raw)
To: linux-arm-kernel
The macros and typedefs in these files are already duplicated, so just
use a single pgtable-types.h file with the corresponding #ifdefs.
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
---
Some more clean-up similar to the removal of pgtable-*level-hwdef.h
arch/arm64/include/asm/page.h | 8 +--
arch/arm64/include/asm/pgtable-2level-types.h | 62 -------------------
arch/arm64/include/asm/pgtable-3level-types.h | 68 --------------------
.../{pgtable-4level-types.h => pgtable-types.h} | 72 ++++++++++++++--------
4 files changed, 49 insertions(+), 161 deletions(-)
delete mode 100644 arch/arm64/include/asm/pgtable-2level-types.h
delete mode 100644 arch/arm64/include/asm/pgtable-3level-types.h
rename arch/arm64/include/asm/{pgtable-4level-types.h => pgtable-types.h} (73%)
diff --git a/arch/arm64/include/asm/page.h b/arch/arm64/include/asm/page.h
index a998ff478777..2502754d1921 100644
--- a/arch/arm64/include/asm/page.h
+++ b/arch/arm64/include/asm/page.h
@@ -47,13 +47,7 @@
#ifndef __ASSEMBLY__
-#if CONFIG_ARM64_PGTABLE_LEVELS == 2
-#include <asm/pgtable-2level-types.h>
-#elif CONFIG_ARM64_PGTABLE_LEVELS == 3
-#include <asm/pgtable-3level-types.h>
-#else
-#include <asm/pgtable-4level-types.h>
-#endif
+#include <asm/pgtable-types.h>
extern void __cpu_clear_user_page(void *p, unsigned long user);
extern void __cpu_copy_user_page(void *to, const void *from,
diff --git a/arch/arm64/include/asm/pgtable-2level-types.h b/arch/arm64/include/asm/pgtable-2level-types.h
deleted file mode 100644
index 5f101e63dfc1..000000000000
--- a/arch/arm64/include/asm/pgtable-2level-types.h
+++ /dev/null
@@ -1,62 +0,0 @@
-/*
- * Copyright (C) 2012 ARM Ltd.
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program. If not, see <http://www.gnu.org/licenses/>.
- */
-#ifndef __ASM_PGTABLE_2LEVEL_TYPES_H
-#define __ASM_PGTABLE_2LEVEL_TYPES_H
-
-#include <asm/types.h>
-
-typedef u64 pteval_t;
-typedef u64 pgdval_t;
-typedef pgdval_t pmdval_t;
-
-#undef STRICT_MM_TYPECHECKS
-
-#ifdef STRICT_MM_TYPECHECKS
-
-/*
- * These are used to make use of C type-checking..
- */
-typedef struct { pteval_t pte; } pte_t;
-typedef struct { pgdval_t pgd; } pgd_t;
-typedef struct { pteval_t pgprot; } pgprot_t;
-
-#define pte_val(x) ((x).pte)
-#define pgd_val(x) ((x).pgd)
-#define pgprot_val(x) ((x).pgprot)
-
-#define __pte(x) ((pte_t) { (x) } )
-#define __pgd(x) ((pgd_t) { (x) } )
-#define __pgprot(x) ((pgprot_t) { (x) } )
-
-#else /* !STRICT_MM_TYPECHECKS */
-
-typedef pteval_t pte_t;
-typedef pgdval_t pgd_t;
-typedef pteval_t pgprot_t;
-
-#define pte_val(x) (x)
-#define pgd_val(x) (x)
-#define pgprot_val(x) (x)
-
-#define __pte(x) (x)
-#define __pgd(x) (x)
-#define __pgprot(x) (x)
-
-#endif /* STRICT_MM_TYPECHECKS */
-
-#include <asm-generic/pgtable-nopmd.h>
-
-#endif /* __ASM_PGTABLE_2LEVEL_TYPES_H */
diff --git a/arch/arm64/include/asm/pgtable-3level-types.h b/arch/arm64/include/asm/pgtable-3level-types.h
deleted file mode 100644
index 4e94424938a4..000000000000
--- a/arch/arm64/include/asm/pgtable-3level-types.h
+++ /dev/null
@@ -1,68 +0,0 @@
-/*
- * Copyright (C) 2012 ARM Ltd.
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program. If not, see <http://www.gnu.org/licenses/>.
- */
-#ifndef __ASM_PGTABLE_3LEVEL_TYPES_H
-#define __ASM_PGTABLE_3LEVEL_TYPES_H
-
-#include <asm/types.h>
-
-typedef u64 pteval_t;
-typedef u64 pmdval_t;
-typedef u64 pgdval_t;
-
-#undef STRICT_MM_TYPECHECKS
-
-#ifdef STRICT_MM_TYPECHECKS
-
-/*
- * These are used to make use of C type-checking..
- */
-typedef struct { pteval_t pte; } pte_t;
-typedef struct { pmdval_t pmd; } pmd_t;
-typedef struct { pgdval_t pgd; } pgd_t;
-typedef struct { pteval_t pgprot; } pgprot_t;
-
-#define pte_val(x) ((x).pte)
-#define pmd_val(x) ((x).pmd)
-#define pgd_val(x) ((x).pgd)
-#define pgprot_val(x) ((x).pgprot)
-
-#define __pte(x) ((pte_t) { (x) } )
-#define __pmd(x) ((pmd_t) { (x) } )
-#define __pgd(x) ((pgd_t) { (x) } )
-#define __pgprot(x) ((pgprot_t) { (x) } )
-
-#else /* !STRICT_MM_TYPECHECKS */
-
-typedef pteval_t pte_t;
-typedef pmdval_t pmd_t;
-typedef pgdval_t pgd_t;
-typedef pteval_t pgprot_t;
-
-#define pte_val(x) (x)
-#define pmd_val(x) (x)
-#define pgd_val(x) (x)
-#define pgprot_val(x) (x)
-
-#define __pte(x) (x)
-#define __pmd(x) (x)
-#define __pgd(x) (x)
-#define __pgprot(x) (x)
-
-#endif /* STRICT_MM_TYPECHECKS */
-
-#include <asm-generic/pgtable-nopud.h>
-
-#endif /* __ASM_PGTABLE_3LEVEL_TYPES_H */
diff --git a/arch/arm64/include/asm/pgtable-4level-types.h b/arch/arm64/include/asm/pgtable-types.h
similarity index 73%
rename from arch/arm64/include/asm/pgtable-4level-types.h
rename to arch/arm64/include/asm/pgtable-types.h
index 7ad8dd257ea1..ca9df80af896 100644
--- a/arch/arm64/include/asm/pgtable-4level-types.h
+++ b/arch/arm64/include/asm/pgtable-types.h
@@ -1,5 +1,10 @@
/*
- * This program is free software; you can redistribute it and/or modify
+ * Page table types definitions.
+ *
+ * Copyright (C) 2014 ARM Ltd.
+ * Author: Catalin Marinas <catalin.marinas@arm.com>
+ *
+ * This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
*
@@ -11,8 +16,9 @@
* You should have received a copy of the GNU General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
-#ifndef __ASM_PGTABLE_4LEVEL_TYPES_H
-#define __ASM_PGTABLE_4LEVEL_TYPES_H
+
+#ifndef __ASM_PGTABLE_TYPES_H
+#define __ASM_PGTABLE_TYPES_H
#include <asm/types.h>
@@ -29,43 +35,61 @@ typedef u64 pgdval_t;
* These are used to make use of C type-checking..
*/
typedef struct { pteval_t pte; } pte_t;
-typedef struct { pmdval_t pmd; } pmd_t;
-typedef struct { pudval_t pud; } pud_t;
-typedef struct { pgdval_t pgd; } pgd_t;
-typedef struct { pteval_t pgprot; } pgprot_t;
-
#define pte_val(x) ((x).pte)
-#define pmd_val(x) ((x).pmd)
-#define pud_val(x) ((x).pud)
-#define pgd_val(x) ((x).pgd)
-#define pgprot_val(x) ((x).pgprot)
-
#define __pte(x) ((pte_t) { (x) } )
+
+#if CONFIG_ARM64_PGTABLE_LEVELS > 2
+typedef struct { pmdval_t pmd; } pmd_t;
+#define pmd_val(x) ((x).pmd)
#define __pmd(x) ((pmd_t) { (x) } )
+#endif
+
+#if CONFIG_ARM64_PGTABLE_LEVELS > 3
+typedef struct { pudval_t pud; } pud_t;
+#define pud_val(x) ((x).pud)
#define __pud(x) ((pud_t) { (x) } )
+#endif
+
+typedef struct { pgdval_t pgd; } pgd_t;
+#define pgd_val(x) ((x).pgd)
#define __pgd(x) ((pgd_t) { (x) } )
+
+typedef struct { pteval_t pgprot; } pgprot_t;
+#define pgprot_val(x) ((x).pgprot)
#define __pgprot(x) ((pgprot_t) { (x) } )
#else /* !STRICT_MM_TYPECHECKS */
typedef pteval_t pte_t;
-typedef pmdval_t pmd_t;
-typedef pudval_t pud_t;
-typedef pgdval_t pgd_t;
-typedef pteval_t pgprot_t;
-
#define pte_val(x) (x)
-#define pmd_val(x) (x)
-#define pud_val(x) (x)
-#define pgd_val(x) (x)
-#define pgprot_val(x) (x)
-
#define __pte(x) (x)
+
+#if CONFIG_ARM64_PGTABLE_LEVELS > 2
+typedef pmdval_t pmd_t;
+#define pmd_val(x) (x)
#define __pmd(x) (x)
+#endif
+
+#if CONFIG_ARM64_PGTABLE_LEVELS > 3
+typedef pudval_t pud_t;
+#define pud_val(x) (x)
#define __pud(x) (x)
+#endif
+
+typedef pgdval_t pgd_t;
+#define pgd_val(x) (x)
#define __pgd(x) (x)
+
+typedef pteval_t pgprot_t;
+#define pgprot_val(x) (x)
#define __pgprot(x) (x)
#endif /* STRICT_MM_TYPECHECKS */
-#endif /* __ASM_PGTABLE_4LEVEL_TYPES_H */
+#if CONFIG_ARM64_PGTABLE_LEVELS == 2
+#include <asm-generic/pgtable-nopmd.h>
+#elif CONFIG_ARM64_PGTABLE_LEVELS == 3
+#include <asm-generic/pgtable-nopud.h>
+#endif
+
+#endif /* __ASM_PGTABLE_TYPES_H */
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH 13/11] arm64: Add support for 48-bit VA space with 64KB page configuration
2014-07-16 19:09 [PATCH v7 00/11] arm64: Support 4 levels of translation tables Catalin Marinas
` (12 preceding siblings ...)
2014-07-18 17:12 ` [PATCH 12/11] arm64: Remove asm/pgtable-*level-types.h files Catalin Marinas
@ 2014-07-21 15:09 ` Catalin Marinas
13 siblings, 0 replies; 18+ messages in thread
From: Catalin Marinas @ 2014-07-21 15:09 UTC (permalink / raw)
To: linux-arm-kernel
This patch allows support for 3 levels of page tables with 64KB page
configuration allowing 48-bit VA space. The pgd is no longer a full
PAGE_SIZE (PTRS_PER_PGD is 64) and (swapper|idmap)_pg_dir are not fully
populated (pgd_alloc falls back to kzalloc).
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
---
Some minor clean-up and 64K page configuration can use 48-bit VA space.
Documentation/arm64/memory.txt | 8 ++++++++
arch/arm64/Kconfig | 2 +-
arch/arm64/include/asm/page.h | 16 +++++++++-------
arch/arm64/kernel/head.S | 6 ++++--
4 files changed, 22 insertions(+), 10 deletions(-)
diff --git a/Documentation/arm64/memory.txt b/Documentation/arm64/memory.txt
index 8845d0847a66..344e85cc7323 100644
--- a/Documentation/arm64/memory.txt
+++ b/Documentation/arm64/memory.txt
@@ -45,6 +45,14 @@ Start End Size Use
fffffc0000000000 ffffffffffffffff 4TB kernel
+AArch64 Linux memory layout with 64KB pages + 3 levels:
+
+Start End Size Use
+-----------------------------------------------------------------------
+0000000000000000 0000ffffffffffff 256TB user
+ffff000000000000 ffffffffffffffff 256TB kernel
+
+
For details of the virtual kernel memory layout please see the kernel
booting log.
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 9fe62025776b..2c83f6f3019f 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -199,7 +199,6 @@ config ARM64_VA_BITS_42
config ARM64_VA_BITS_48
bool "48-bit"
depends on !KVM
- depends on ARM64_4K_PAGES
endchoice
@@ -212,6 +211,7 @@ config ARM64_VA_BITS
config ARM64_PGTABLE_LEVELS
int
default 2 if ARM64_64K_PAGES && ARM64_VA_BITS_42
+ default 3 if ARM64_64K_PAGES && ARM64_VA_BITS_48
default 3 if ARM64_4K_PAGES && ARM64_VA_BITS_39
default 4 if ARM64_4K_PAGES && ARM64_VA_BITS_48
diff --git a/arch/arm64/include/asm/page.h b/arch/arm64/include/asm/page.h
index 2502754d1921..7a3f462133b0 100644
--- a/arch/arm64/include/asm/page.h
+++ b/arch/arm64/include/asm/page.h
@@ -34,17 +34,19 @@
/*
* The idmap and swapper page tables need some space reserved in the kernel
* image. Both require pgd, pud (4 levels only) and pmd tables to (section)
- * map the kernel. The swapper also maps the FDT (see __create_page_tables for
- * more information).
+ * map the kernel. With the 64K page configuration, swapper and idmap need to
+ * map to pte level. The swapper also maps the FDT (see __create_page_tables
+ * for more information).
*/
-#if CONFIG_ARM64_PGTABLE_LEVELS == 4
-#define SWAPPER_DIR_SIZE (3 * PAGE_SIZE)
-#define IDMAP_DIR_SIZE (3 * PAGE_SIZE)
+#ifdef CONFIG_ARM64_64K_PAGES
+#define SWAPPER_PGTABLE_LEVELS (CONFIG_ARM64_PGTABLE_LEVELS)
#else
-#define SWAPPER_DIR_SIZE (2 * PAGE_SIZE)
-#define IDMAP_DIR_SIZE (2 * PAGE_SIZE)
+#define SWAPPER_PGTABLE_LEVELS (CONFIG_ARM64_PGTABLE_LEVELS - 1)
#endif
+#define SWAPPER_DIR_SIZE (SWAPPER_PGTABLE_LEVELS * PAGE_SIZE)
+#define IDMAP_DIR_SIZE (SWAPPER_DIR_SIZE)
+
#ifndef __ASSEMBLY__
#include <asm/pgtable-types.h>
diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index a6db505411bc..0bce493495e9 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -55,9 +55,11 @@
#ifdef CONFIG_ARM64_64K_PAGES
#define BLOCK_SHIFT PAGE_SHIFT
#define BLOCK_SIZE PAGE_SIZE
+#define TABLE_SHIFT PMD_SHIFT
#else
#define BLOCK_SHIFT SECTION_SHIFT
#define BLOCK_SIZE SECTION_SIZE
+#define TABLE_SHIFT PUD_SHIFT
#endif
#define KERNEL_START KERNEL_RAM_VADDR
@@ -505,8 +507,8 @@ ENDPROC(__calc_phys_offset)
*/
.macro create_pgd_entry, tbl, virt, tmp1, tmp2
create_table_entry \tbl, \virt, PGDIR_SHIFT, PTRS_PER_PGD, \tmp1, \tmp2
-#if CONFIG_ARM64_PGTABLE_LEVELS == 4
- create_table_entry \tbl, \virt, PUD_SHIFT, PTRS_PER_PUD, \tmp1, \tmp2
+#if SWAPPER_PGTABLE_LEVELS == 3
+ create_table_entry \tbl, \virt, TABLE_SHIFT, PTRS_PER_PTE, \tmp1, \tmp2
#endif
.endm
^ permalink raw reply related [flat|nested] 18+ messages in thread
* [PATCH v7 07/11] arm64: mm: Implement 4 levels of translation tables
2014-07-16 19:09 ` [PATCH v7 07/11] arm64: mm: Implement 4 levels of translation tables Catalin Marinas
@ 2014-07-28 15:40 ` Joel Schopp
0 siblings, 0 replies; 18+ messages in thread
From: Joel Schopp @ 2014-07-28 15:40 UTC (permalink / raw)
To: linux-arm-kernel
On 07/16/2014 02:09 PM, Catalin Marinas wrote:
> From: Jungseok Lee <jays.lee@samsung.com>
>
> This patch implements 4 levels of translation tables since 3 levels
> of page tables with 4KB pages cannot support 40-bit physical address
> space described in [1] due to the following issue.
>
> It is a restriction that kernel logical memory map with 4KB + 3 levels
> (0xffffffc000000000-0xffffffffffffffff) cannot cover RAM region from
> 544GB to 1024GB in [1]. Specifically, ARM64 kernel fails to create
> mapping for this region in map_mem function since __phys_to_virt for
> this region reaches to address overflow.
>
> If SoC design follows the document, [1], over 32GB RAM would be placed
> from 544GB. Even 64GB system is supposed to use the region from 544GB
> to 576GB for only 32GB RAM. Naturally, it would reach to enable 4 levels
> of page tables to avoid hacking __virt_to_phys and __phys_to_virt.
>
> However, it is recommended 4 levels of page table should be only enabled
> if memory map is too sparse or there is about 512GB RAM.
>
> References
> ----------
> [1]: Principles of ARM Memory Maps, White Paper, Issue C
>
> Signed-off-by: Jungseok Lee <jays.lee@samsung.com>
> Reviewed-by: Sungjinn Chung <sungjinn.chung@samsung.com>
> Acked-by: Kukjin Kim <kgene.kim@samsung.com>
> Reviewed-by: Christoffer Dall <christoffer.dall@linaro.org>
> Reviewed-by: Steve Capper <steve.capper@linaro.org>
> [catalin.marinas at arm.com: MEMBLOCK_INITIAL_LIMIT removed, same as PUD_SIZE]
> [catalin.marinas at arm.com: early_ioremap_init() updated for 4 levels]
> [catalin.marinas at arm.com: 4 page tables levels only if !KVM]
> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
> ---
> arch/arm64/Kconfig | 9 ++++++++
> arch/arm64/include/asm/page.h | 13 ++++++++---
> arch/arm64/include/asm/pgalloc.h | 20 ++++++++++++++++
> arch/arm64/include/asm/pgtable-hwdef.h | 6 +++--
> arch/arm64/include/asm/pgtable.h | 40 ++++++++++++++++++++++++++++++++
> arch/arm64/include/asm/tlb.h | 9 ++++++++
> arch/arm64/kernel/head.S | 42 +++++++++++++++++++++++++++-------
> arch/arm64/kernel/traps.c | 5 ++++
> arch/arm64/mm/fault.c | 1 +
> arch/arm64/mm/ioremap.c | 6 ++++-
> arch/arm64/mm/mmu.c | 14 +++++++++---
> 11 files changed, 148 insertions(+), 17 deletions(-)
>
> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
> index 4daf11f5b403..24cbe72c0da9 100644
> --- a/arch/arm64/Kconfig
> +++ b/arch/arm64/Kconfig
> @@ -196,12 +196,18 @@ config ARM64_VA_BITS_42
> bool "42-bit"
> depends on ARM64_64K_PAGES
>
> +config ARM64_VA_BITS_48
> + bool "48-bit"
> + depends on !KVM
> + depends on ARM64_4K_PAGES
> +
> endchoice
Shouldn't we be able to support 48 bit VA with 3 level 64K pages? If so
why the dependency on ARM64_4K_PAGES?
More generally it seems like a problem to tie the equate the VA_BITS the
page table could address with the VA_BITS the hardware could address.
Even with 4 level 4K page tables that can address 48 bits the hardware
may only support say 42 bit address space.
>
> config ARM64_VA_BITS
> int
> default 39 if ARM64_VA_BITS_39
> default 42 if ARM64_VA_BITS_42
> + default 48 if ARM64_VA_BITS_48
>
> config ARM64_2_LEVELS
> def_bool y if ARM64_64K_PAGES && ARM64_VA_BITS_42
> @@ -209,6 +215,9 @@ config ARM64_2_LEVELS
> config ARM64_3_LEVELS
> def_bool y if ARM64_4K_PAGES && ARM64_VA_BITS_39
>
> +config ARM64_4_LEVELS
> + def_bool y if ARM64_4K_PAGES && ARM64_VA_BITS_48
> +
It seems like we should also do ARM64_4K_PAGES and ARM64_VA_BITS_42 as a
valid combination for ARM64_4_LEVELS. At least if we are assuming the
VA_BITS correspond to hardware.
> config CPU_BIG_ENDIAN
> bool "Build big-endian kernel"
> help
> diff --git a/arch/arm64/include/asm/page.h b/arch/arm64/include/asm/page.h
> index 6bf139188792..cf9afa0366b6 100644
> --- a/arch/arm64/include/asm/page.h
> +++ b/arch/arm64/include/asm/page.h
> @@ -33,19 +33,26 @@
>
> /*
> * The idmap and swapper page tables need some space reserved in the kernel
> - * image. Both require a pgd and a next level table to (section) map the
> - * kernel. The the swapper also maaps the FDT (see __create_page_tables for
> + * image. Both require pgd, pud (4 levels only) and pmd tables to (section)
> + * map the kernel. The swapper also maps the FDT (see __create_page_tables for
> * more information).
> */
> +#ifdef CONFIG_ARM64_4_LEVELS
> +#define SWAPPER_DIR_SIZE (3 * PAGE_SIZE)
> +#define IDMAP_DIR_SIZE (3 * PAGE_SIZE)
> +#else
> #define SWAPPER_DIR_SIZE (2 * PAGE_SIZE)
> #define IDMAP_DIR_SIZE (2 * PAGE_SIZE)
> +#endif
>
> #ifndef __ASSEMBLY__
>
> #ifdef CONFIG_ARM64_2_LEVELS
> #include <asm/pgtable-2level-types.h>
> -#else
> +#elif defined(CONFIG_ARM64_3_LEVELS)
> #include <asm/pgtable-3level-types.h>
> +#else
> +#include <asm/pgtable-4level-types.h>
> #endif
>
> extern void __cpu_clear_user_page(void *p, unsigned long user);
> diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h
> index 48298376e46a..8d745fae4c2d 100644
> --- a/arch/arm64/include/asm/pgalloc.h
> +++ b/arch/arm64/include/asm/pgalloc.h
> @@ -26,6 +26,26 @@
>
> #define check_pgt_cache() do { } while (0)
>
> +#ifdef CONFIG_ARM64_4_LEVELS
> +
> +static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long addr)
> +{
> + return (pud_t *)get_zeroed_page(GFP_KERNEL | __GFP_REPEAT);
> +}
> +
> +static inline void pud_free(struct mm_struct *mm, pud_t *pud)
> +{
> + BUG_ON((unsigned long)pud & (PAGE_SIZE-1));
> + free_page((unsigned long)pud);
> +}
> +
> +static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgd, pud_t *pud)
> +{
> + set_pgd(pgd, __pgd(__pa(pud) | PUD_TYPE_TABLE));
> +}
> +
> +#endif /* CONFIG_ARM64_4_LEVELS */
> +
> #ifndef CONFIG_ARM64_2_LEVELS
>
> static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long addr)
> diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h
> index c7c603b489b8..fddcc3efa569 100644
> --- a/arch/arm64/include/asm/pgtable-hwdef.h
> +++ b/arch/arm64/include/asm/pgtable-hwdef.h
> @@ -18,8 +18,10 @@
>
> #ifdef CONFIG_ARM64_2_LEVELS
> #include <asm/pgtable-2level-hwdef.h>
> -#else
> +#elif defined(CONFIG_ARM64_3_LEVELS)
> #include <asm/pgtable-3level-hwdef.h>
> +#else
> +#include <asm/pgtable-4level-hwdef.h>
> #endif
>
> /*
> @@ -27,7 +29,7 @@
> *
> * Level 1 descriptor (PUD).
> */
> -
> +#define PUD_TYPE_TABLE (_AT(pudval_t, 3) << 0)
> #define PUD_TABLE_BIT (_AT(pgdval_t, 1) << 1)
> #define PUD_TYPE_MASK (_AT(pgdval_t, 3) << 0)
> #define PUD_TYPE_SECT (_AT(pgdval_t, 1) << 0)
> diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
> index 6d5854972a77..d9b23efdaded 100644
> --- a/arch/arm64/include/asm/pgtable.h
> +++ b/arch/arm64/include/asm/pgtable.h
> @@ -35,7 +35,11 @@
> * VMALLOC and SPARSEMEM_VMEMMAP ranges.
> */
> #define VMALLOC_START (UL(0xffffffffffffffff) << VA_BITS)
Here's a good example of where we run into trouble equating page table
addressable bits with hardware addressable bits. If VA_BITS is 48 due
to 4K 4 level page tables but is running on a 42 bit system this will
end up being out of range.
> +#ifndef CONFIG_ARM64_4_LEVELS
> #define VMALLOC_END (PAGE_OFFSET - UL(0x400000000) - SZ_64K)
> +#else
> +#define VMALLOC_END (PAGE_OFFSET - UL(0x40000000000) - SZ_64K)
> +#endif
>
> #define vmemmap ((struct page *)(VMALLOC_END + SZ_64K))
>
> @@ -44,12 +48,16 @@
> #ifndef __ASSEMBLY__
> extern void __pte_error(const char *file, int line, unsigned long val);
> extern void __pmd_error(const char *file, int line, unsigned long val);
> +extern void __pud_error(const char *file, int line, unsigned long val);
> extern void __pgd_error(const char *file, int line, unsigned long val);
>
> #define pte_ERROR(pte) __pte_error(__FILE__, __LINE__, pte_val(pte))
> #ifndef CONFIG_ARM64_2_LEVELS
> #define pmd_ERROR(pmd) __pmd_error(__FILE__, __LINE__, pmd_val(pmd))
> #endif
> +#ifdef CONFIG_ARM64_4_LEVELS
> +#define pud_ERROR(pud) __pud_error(__FILE__, __LINE__, pud_val(pud))
> +#endif
> #define pgd_ERROR(pgd) __pgd_error(__FILE__, __LINE__, pgd_val(pgd))
>
> #ifdef CONFIG_SMP
> @@ -347,6 +355,30 @@ static inline pmd_t *pud_page_vaddr(pud_t pud)
>
> #endif /* CONFIG_ARM64_2_LEVELS */
>
> +#ifdef CONFIG_ARM64_4_LEVELS
> +
> +#define pgd_none(pgd) (!pgd_val(pgd))
> +#define pgd_bad(pgd) (!(pgd_val(pgd) & 2))
> +#define pgd_present(pgd) (pgd_val(pgd))
> +
> +static inline void set_pgd(pgd_t *pgdp, pgd_t pgd)
> +{
> + *pgdp = pgd;
> + dsb(ishst);
> +}
> +
> +static inline void pgd_clear(pgd_t *pgdp)
> +{
> + set_pgd(pgdp, __pgd(0));
> +}
> +
> +static inline pud_t *pgd_page_vaddr(pgd_t pgd)
> +{
> + return __va(pgd_val(pgd) & PHYS_MASK & (s32)PAGE_MASK);
> +}
> +
> +#endif /* CONFIG_ARM64_4_LEVELS */
> +
> /* to find an entry in a page-table-directory */
> #define pgd_index(addr) (((addr) >> PGDIR_SHIFT) & (PTRS_PER_PGD - 1))
>
> @@ -355,6 +387,14 @@ static inline pmd_t *pud_page_vaddr(pud_t pud)
> /* to find an entry in a kernel page-table-directory */
> #define pgd_offset_k(addr) pgd_offset(&init_mm, addr)
>
> +#ifdef CONFIG_ARM64_4_LEVELS
> +#define pud_index(addr) (((addr) >> PUD_SHIFT) & (PTRS_PER_PUD - 1))
> +static inline pud_t *pud_offset(pgd_t *pgd, unsigned long addr)
> +{
> + return (pud_t *)pgd_page_vaddr(*pgd) + pud_index(addr);
> +}
> +#endif
> +
> /* Find an entry in the second-level page table.. */
> #ifndef CONFIG_ARM64_2_LEVELS
> #define pmd_index(addr) (((addr) >> PMD_SHIFT) & (PTRS_PER_PMD - 1))
> diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h
> index bc19101edaeb..49dc8f03362f 100644
> --- a/arch/arm64/include/asm/tlb.h
> +++ b/arch/arm64/include/asm/tlb.h
> @@ -100,6 +100,15 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp,
> }
> #endif
>
> +#ifdef CONFIG_ARM64_4_LEVELS
> +static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp,
> + unsigned long addr)
> +{
> + tlb_add_flush(tlb, addr);
> + tlb_remove_page(tlb, virt_to_page(pudp));
> +}
> +#endif
> +
> static inline void __tlb_remove_pmd_tlb_entry(struct mmu_gather *tlb, pmd_t *pmdp,
> unsigned long address)
> {
> diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
> index fa3b7fb8a77a..847b99daad79 100644
> --- a/arch/arm64/kernel/head.S
> +++ b/arch/arm64/kernel/head.S
> @@ -476,16 +476,42 @@ ENDPROC(__calc_phys_offset)
> .quad PAGE_OFFSET
>
> /*
> - * Macro to populate the PGD for the corresponding block entry in the next
> - * level (tbl) for the given virtual address.
> + * Macro to populate the PUD for the corresponding block entry in the next
> + * level (tbl) for the given virtual address in case of 4 levels.
> *
> - * Preserves: pgd, tbl, virt
> - * Corrupts: tmp1, tmp2
> + * Preserves: pgd, virt
> + * Corrupts: tbl, tmp1, tmp2
> + * Returns: pud
> */
> - .macro create_pgd_entry, pgd, tbl, virt, tmp1, tmp2
> + .macro create_pud_entry, pgd, tbl, virt, pud, tmp1, tmp2
> +#ifdef CONFIG_ARM64_4_LEVELS
> + add \tbl, \tbl, #PAGE_SIZE // bump tbl 1 page up.
> + // to make room for pud
> + add \pud, \pgd, #PAGE_SIZE // pgd points to pud which
> + // follows pgd
> + lsr \tmp1, \virt, #PUD_SHIFT
> + and \tmp1, \tmp1, #PTRS_PER_PUD - 1 // PUD index
> + orr \tmp2, \tbl, #3 // PUD entry table type
> + str \tmp2, [\pud, \tmp1, lsl #3]
> +#else
> + mov \pud, \tbl
> +#endif
> + .endm
> +
> +/*
> + * Macro to populate the PGD (and possibily PUD) for the corresponding
> + * block entry in the next level (tbl) for the given virtual address.
> + *
> + * Preserves: pgd, virt
> + * Corrupts: tmp1, tmp2, tmp3
> + * Returns: tbl -> page where block mappings can be placed
> + * (changed to make room for pud with 4 levels, preserved otherwise)
> + */
> + .macro create_pgd_entry, pgd, tbl, virt, tmp1, tmp2, tmp3
> + create_pud_entry \pgd, \tbl, \virt, \tmp3, \tmp1, \tmp2
> lsr \tmp1, \virt, #PGDIR_SHIFT
> and \tmp1, \tmp1, #PTRS_PER_PGD - 1 // PGD index
> - orr \tmp2, \tbl, #3 // PGD entry table type
> + orr \tmp2, \tmp3, #3 // PGD entry table type
> str \tmp2, [\pgd, \tmp1, lsl #3]
> .endm
>
> @@ -550,7 +576,7 @@ __create_page_tables:
> add x0, x25, #PAGE_SIZE // section table address
> ldr x3, =KERNEL_START
> add x3, x3, x28 // __pa(KERNEL_START)
> - create_pgd_entry x25, x0, x3, x5, x6
> + create_pgd_entry x25, x0, x3, x1, x5, x6
> ldr x6, =KERNEL_END
> mov x5, x3 // __pa(KERNEL_START)
> add x6, x6, x28 // __pa(KERNEL_END)
> @@ -561,7 +587,7 @@ __create_page_tables:
> */
> add x0, x26, #PAGE_SIZE // section table address
> mov x5, #PAGE_OFFSET
> - create_pgd_entry x26, x0, x5, x3, x6
> + create_pgd_entry x26, x0, x5, x1, x3, x6
> ldr x6, =KERNEL_END
> mov x3, x24 // phys offset
> create_block_map x0, x7, x3, x5, x6
> diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c
> index 506f7814e305..02cd3f023e9a 100644
> --- a/arch/arm64/kernel/traps.c
> +++ b/arch/arm64/kernel/traps.c
> @@ -339,6 +339,11 @@ void __pmd_error(const char *file, int line, unsigned long val)
> pr_crit("%s:%d: bad pmd %016lx.\n", file, line, val);
> }
>
> +void __pud_error(const char *file, int line, unsigned long val)
> +{
> + pr_crit("%s:%d: bad pud %016lx.\n", file, line, val);
> +}
> +
> void __pgd_error(const char *file, int line, unsigned long val)
> {
> pr_crit("%s:%d: bad pgd %016lx.\n", file, line, val);
> diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
> index bcc965e2cce1..41cb6d3d6075 100644
> --- a/arch/arm64/mm/fault.c
> +++ b/arch/arm64/mm/fault.c
> @@ -62,6 +62,7 @@ void show_pte(struct mm_struct *mm, unsigned long addr)
> break;
>
> pud = pud_offset(pgd, addr);
> + printk(", *pud=%016llx", pud_val(*pud));
> if (pud_none(*pud) || pud_bad(*pud))
> break;
>
> diff --git a/arch/arm64/mm/ioremap.c b/arch/arm64/mm/ioremap.c
> index 69000efa015e..fa324bd5a5c4 100644
> --- a/arch/arm64/mm/ioremap.c
> +++ b/arch/arm64/mm/ioremap.c
> @@ -104,9 +104,12 @@ void __iomem *ioremap_cache(phys_addr_t phys_addr, size_t size)
> EXPORT_SYMBOL(ioremap_cache);
>
> static pte_t bm_pte[PTRS_PER_PTE] __page_aligned_bss;
> -#ifndef CONFIG_ARM64_64K_PAGES
> +#if CONFIG_ARM64_PGTABLE_LEVELS > 2
> static pte_t bm_pmd[PTRS_PER_PMD] __page_aligned_bss;
> #endif
> +#if CONFIG_ARM64_PGTABLE_LEVELS > 3
> +static pte_t bm_pud[PTRS_PER_PUD] __page_aligned_bss;
> +#endif
>
> static inline pud_t * __init early_ioremap_pud(unsigned long addr)
> {
> @@ -144,6 +147,7 @@ void __init early_ioremap_init(void)
> unsigned long addr = fix_to_virt(FIX_BTMAP_BEGIN);
>
> pgd = pgd_offset_k(addr);
> + pgd_populate(&init_mm, pgd, bm_pud);
> pud = pud_offset(pgd, addr);
> pud_populate(&init_mm, pud, bm_pmd);
> pmd = pmd_offset(pud, addr);
> diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
> index c43f1dd19489..c55567283cde 100644
> --- a/arch/arm64/mm/mmu.c
> +++ b/arch/arm64/mm/mmu.c
> @@ -32,6 +32,7 @@
> #include <asm/setup.h>
> #include <asm/sizes.h>
> #include <asm/tlb.h>
> +#include <asm/memblock.h>
> #include <asm/mmu_context.h>
>
> #include "mm.h"
> @@ -204,9 +205,16 @@ static void __init alloc_init_pud(pgd_t *pgd, unsigned long addr,
> unsigned long end, unsigned long phys,
> int map_io)
> {
> - pud_t *pud = pud_offset(pgd, addr);
> + pud_t *pud;
> unsigned long next;
>
> + if (pgd_none(*pgd)) {
> + pud = early_alloc(PTRS_PER_PUD * sizeof(pud_t));
> + pgd_populate(&init_mm, pgd, pud);
> + }
> + BUG_ON(pgd_bad(*pgd));
> +
> + pud = pud_offset(pgd, addr);
> do {
> next = pud_addr_end(addr, end);
>
> @@ -290,10 +298,10 @@ static void __init map_mem(void)
> * memory addressable from the initial direct kernel mapping.
> *
> * The initial direct kernel mapping, located at swapper_pg_dir,
> - * gives us PGDIR_SIZE memory starting from PHYS_OFFSET (which must be
> + * gives us PUD_SIZE memory starting from PHYS_OFFSET (which must be
> * aligned to 2MB as per Documentation/arm64/booting.txt).
> */
> - limit = PHYS_OFFSET + PGDIR_SIZE;
> + limit = PHYS_OFFSET + PUD_SIZE;
> memblock_set_current_limit(limit);
>
> /* map all the memory banks */
>
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 18+ messages in thread
end of thread, other threads:[~2014-07-28 15:40 UTC | newest]
Thread overview: 18+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-07-16 19:09 [PATCH v7 00/11] arm64: Support 4 levels of translation tables Catalin Marinas
2014-07-16 19:09 ` [PATCH v7 01/11] arm64: Use pr_* instead of printk Catalin Marinas
2014-07-16 19:09 ` [PATCH v7 02/11] arm64: Remove duplicate (SWAPPER|IDMAP)_DIR_SIZE definitions Catalin Marinas
2014-07-17 10:49 ` Mark Rutland
2014-07-16 19:09 ` [PATCH v7 03/11] arm64: Do not initialise the fixmap page tables in head.S Catalin Marinas
2014-07-16 23:14 ` Geoff Levand
2014-07-16 19:09 ` [PATCH v7 04/11] arm64: Introduce VA_BITS and translation level options Catalin Marinas
2014-07-16 19:09 ` [PATCH v7 05/11] arm64: Add a description on 48-bit address space with 4KB pages Catalin Marinas
2014-07-16 19:09 ` [PATCH v7 06/11] arm64: Add 4 levels of page tables definition " Catalin Marinas
2014-07-16 19:09 ` [PATCH v7 07/11] arm64: mm: Implement 4 levels of translation tables Catalin Marinas
2014-07-28 15:40 ` Joel Schopp
2014-07-16 19:09 ` [PATCH v7 08/11] arm64: Convert bool ARM64_x_LEVELS to int ARM64_PGTABLE_LEVELS Catalin Marinas
2014-07-16 19:09 ` [PATCH v7 09/11] arm64: Remove asm/pgtable-*level-hwdef.h files Catalin Marinas
2014-07-16 19:09 ` [PATCH v7 10/11] arm64: Clean up the initial page table creation in head.S Catalin Marinas
2014-07-16 19:09 ` [PATCH v7 11/11] arm64: Determine the vmalloc/vmemmap space at build time based on VA_BITS Catalin Marinas
2014-07-17 10:15 ` [PATCH v7 00/11] arm64: Support 4 levels of translation tables Will Deacon
2014-07-18 17:12 ` [PATCH 12/11] arm64: Remove asm/pgtable-*level-types.h files Catalin Marinas
2014-07-21 15:09 ` [PATCH 13/11] arm64: Add support for 48-bit VA space with 64KB page configuration Catalin Marinas
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.