* [PATCH 0/4] Add support for fast mremap
@ 2018-10-13 1:31 Joel Fernandes (Google)
2018-10-13 1:31 ` Joel Fernandes (Google)
` (4 more replies)
0 siblings, 5 replies; 52+ messages in thread
From: Joel Fernandes (Google) @ 2018-10-13 1:31 UTC (permalink / raw)
To: linux-riscv
Hi,
Here is the latest "fast mremap" series. The main change in this submission is
to enable the fast mremap optimization on a per-architecture basis to prevent
possible issues with architectures that may not behave well with such change.
x86: select HAVE_MOVE_PMD for faster mremap (v1)
arm64: select HAVE_MOVE_PMD for faster mremap (v1)
mm: speed up mremap by 500x on large regions (v2)
v1->v2: Added support for per-arch enablement (Kirill Shutemov)
treewide: remove unused address argument from pte_alloc functions (v2)
v1->v2: fix arch/um/ prototype which was missed in v1 (Anton Ivanov)
update changelog with manual fixups for m68k and microblaze.
Joel Fernandes (Google) (4):
treewide: remove unused address argument from pte_alloc functions (v2)
mm: speed up mremap by 500x on large regions (v2)
arm64: select HAVE_MOVE_PMD for faster mremap (v1)
x86: select HAVE_MOVE_PMD for faster mremap (v1)
arch/Kconfig | 5 ++
arch/alpha/include/asm/pgalloc.h | 6 +-
arch/arc/include/asm/pgalloc.h | 5 +-
arch/arm/include/asm/pgalloc.h | 4 +-
arch/arm64/Kconfig | 1 +
arch/arm64/include/asm/pgalloc.h | 4 +-
arch/hexagon/include/asm/pgalloc.h | 6 +-
arch/ia64/include/asm/pgalloc.h | 5 +-
arch/m68k/include/asm/mcf_pgalloc.h | 8 +--
arch/m68k/include/asm/motorola_pgalloc.h | 4 +-
arch/m68k/include/asm/sun3_pgalloc.h | 6 +-
arch/microblaze/include/asm/pgalloc.h | 19 +-----
arch/microblaze/mm/pgtable.c | 3 +-
arch/mips/include/asm/pgalloc.h | 6 +-
arch/nds32/include/asm/pgalloc.h | 5 +-
arch/nios2/include/asm/pgalloc.h | 6 +-
arch/openrisc/include/asm/pgalloc.h | 5 +-
arch/openrisc/mm/ioremap.c | 3 +-
arch/parisc/include/asm/pgalloc.h | 4 +-
arch/powerpc/include/asm/book3s/32/pgalloc.h | 4 +-
arch/powerpc/include/asm/book3s/64/pgalloc.h | 12 ++--
arch/powerpc/include/asm/nohash/32/pgalloc.h | 4 +-
arch/powerpc/include/asm/nohash/64/pgalloc.h | 6 +-
arch/powerpc/mm/pgtable-book3s64.c | 2 +-
arch/powerpc/mm/pgtable_32.c | 4 +-
arch/riscv/include/asm/pgalloc.h | 6 +-
arch/s390/include/asm/pgalloc.h | 4 +-
arch/sh/include/asm/pgalloc.h | 6 +-
arch/sparc/include/asm/pgalloc_32.h | 5 +-
arch/sparc/include/asm/pgalloc_64.h | 6 +-
arch/sparc/mm/init_64.c | 6 +-
arch/sparc/mm/srmmu.c | 4 +-
arch/um/include/asm/pgalloc.h | 4 +-
arch/um/kernel/mem.c | 4 +-
arch/unicore32/include/asm/pgalloc.h | 4 +-
arch/x86/Kconfig | 1 +
arch/x86/include/asm/pgalloc.h | 4 +-
arch/x86/mm/pgtable.c | 4 +-
arch/xtensa/include/asm/pgalloc.h | 8 +--
include/linux/mm.h | 13 ++--
mm/huge_memory.c | 8 +--
mm/kasan/kasan_init.c | 2 +-
mm/memory.c | 17 +++--
mm/migrate.c | 2 +-
mm/mremap.c | 67 +++++++++++++++++++-
mm/userfaultfd.c | 2 +-
virt/kvm/arm/mmu.c | 2 +-
47 files changed, 169 insertions(+), 147 deletions(-)
--
2.19.0.605.g01d371f741-goog
^ permalink raw reply [flat|nested] 52+ messages in thread* [PATCH 0/4] Add support for fast mremap 2018-10-13 1:31 [PATCH 0/4] Add support for fast mremap Joel Fernandes (Google) @ 2018-10-13 1:31 ` Joel Fernandes (Google) 2018-10-13 1:31 ` [PATCH 1/4] treewide: remove unused address argument from pte_alloc functions (v2) Joel Fernandes (Google) ` (3 subsequent siblings) 4 siblings, 0 replies; 52+ messages in thread From: Joel Fernandes (Google) @ 2018-10-13 1:31 UTC (permalink / raw) To: linux-kernel Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Peter Zijlstra, Catalin Marinas, Dave Hansen, Will Deacon, mhocko, linux-mm, lokeshgidra, Joel Fernandes (Google), linux-riscv, elfring, Jonas Bonn, kvmarm, dancol, Yoshinori Sato, sparclinux, linux-xtensa, linux-hexagon, Helge Deller, maintainer:X86 ARCHITECTURE 32-BIT AND 64-BIT, hughd, James E.J. Bottomley, kasan-dev, anton.ivanov, Ingo Molnar, Geert Uytterhoeven, Andrey Ryabinin, linux-snps-arc, kernel-team, Sam Creasey, Fenghua Yu, linux-s390, Jeff Dike, linux-um, Stefan Kristiansson, Julia Lawall, linux-m68k, Borislav Petkov, Andy Lutomirski, nios2-dev, Kirill A. Shutemov, Stafford Horne, Guan Xuetao, Chris Zankel, Tony Luck, Richard Weinberger, linux-parisc, Max Filippov, pantin, minchan, Thomas Gleixner, linux-alpha, Ley Foon Tan, akpm, linuxppc-dev, David S. Miller Hi, Here is the latest "fast mremap" series. The main change in this submission is to enable the fast mremap optimization on a per-architecture basis to prevent possible issues with architectures that may not behave well with such change. x86: select HAVE_MOVE_PMD for faster mremap (v1) arm64: select HAVE_MOVE_PMD for faster mremap (v1) mm: speed up mremap by 500x on large regions (v2) v1->v2: Added support for per-arch enablement (Kirill Shutemov) treewide: remove unused address argument from pte_alloc functions (v2) v1->v2: fix arch/um/ prototype which was missed in v1 (Anton Ivanov) update changelog with manual fixups for m68k and microblaze. Joel Fernandes (Google) (4): treewide: remove unused address argument from pte_alloc functions (v2) mm: speed up mremap by 500x on large regions (v2) arm64: select HAVE_MOVE_PMD for faster mremap (v1) x86: select HAVE_MOVE_PMD for faster mremap (v1) arch/Kconfig | 5 ++ arch/alpha/include/asm/pgalloc.h | 6 +- arch/arc/include/asm/pgalloc.h | 5 +- arch/arm/include/asm/pgalloc.h | 4 +- arch/arm64/Kconfig | 1 + arch/arm64/include/asm/pgalloc.h | 4 +- arch/hexagon/include/asm/pgalloc.h | 6 +- arch/ia64/include/asm/pgalloc.h | 5 +- arch/m68k/include/asm/mcf_pgalloc.h | 8 +-- arch/m68k/include/asm/motorola_pgalloc.h | 4 +- arch/m68k/include/asm/sun3_pgalloc.h | 6 +- arch/microblaze/include/asm/pgalloc.h | 19 +----- arch/microblaze/mm/pgtable.c | 3 +- arch/mips/include/asm/pgalloc.h | 6 +- arch/nds32/include/asm/pgalloc.h | 5 +- arch/nios2/include/asm/pgalloc.h | 6 +- arch/openrisc/include/asm/pgalloc.h | 5 +- arch/openrisc/mm/ioremap.c | 3 +- arch/parisc/include/asm/pgalloc.h | 4 +- arch/powerpc/include/asm/book3s/32/pgalloc.h | 4 +- arch/powerpc/include/asm/book3s/64/pgalloc.h | 12 ++-- arch/powerpc/include/asm/nohash/32/pgalloc.h | 4 +- arch/powerpc/include/asm/nohash/64/pgalloc.h | 6 +- arch/powerpc/mm/pgtable-book3s64.c | 2 +- arch/powerpc/mm/pgtable_32.c | 4 +- arch/riscv/include/asm/pgalloc.h | 6 +- arch/s390/include/asm/pgalloc.h | 4 +- arch/sh/include/asm/pgalloc.h | 6 +- arch/sparc/include/asm/pgalloc_32.h | 5 +- arch/sparc/include/asm/pgalloc_64.h | 6 +- arch/sparc/mm/init_64.c | 6 +- arch/sparc/mm/srmmu.c | 4 +- arch/um/include/asm/pgalloc.h | 4 +- arch/um/kernel/mem.c | 4 +- arch/unicore32/include/asm/pgalloc.h | 4 +- arch/x86/Kconfig | 1 + arch/x86/include/asm/pgalloc.h | 4 +- arch/x86/mm/pgtable.c | 4 +- arch/xtensa/include/asm/pgalloc.h | 8 +-- include/linux/mm.h | 13 ++-- mm/huge_memory.c | 8 +-- mm/kasan/kasan_init.c | 2 +- mm/memory.c | 17 +++-- mm/migrate.c | 2 +- mm/mremap.c | 67 +++++++++++++++++++- mm/userfaultfd.c | 2 +- virt/kvm/arm/mmu.c | 2 +- 47 files changed, 169 insertions(+), 147 deletions(-) -- 2.19.0.605.g01d371f741-goog _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 52+ messages in thread
* [PATCH 1/4] treewide: remove unused address argument from pte_alloc functions (v2) 2018-10-13 1:31 [PATCH 0/4] Add support for fast mremap Joel Fernandes (Google) 2018-10-13 1:31 ` Joel Fernandes (Google) @ 2018-10-13 1:31 ` Joel Fernandes (Google) 2018-10-13 1:31 ` Joel Fernandes (Google) 2018-10-24 8:37 ` Peter Zijlstra 2018-10-13 1:31 ` [PATCH 2/4] mm: speed up mremap by 500x on large regions (v2) Joel Fernandes (Google) ` (2 subsequent siblings) 4 siblings, 2 replies; 52+ messages in thread From: Joel Fernandes (Google) @ 2018-10-13 1:31 UTC (permalink / raw) To: linux-riscv This series speeds up mremap(2) syscall by copying page tables at the PMD level even for non-THP systems. There is concern that the extra 'address' argument that mremap passes to pte_alloc may do something subtle architecture related in the future that may make the scheme not work. Also we find that there is no point in passing the 'address' to pte_alloc since its unused. So this patch therefore removes this argument tree-wide resulting in a nice negative diff as well. Also ensuring along the way that the enabled architectures do not do anything funky with 'address' argument that goes unnoticed by the optimization. Build and boot tested on x86-64. Build tested on arm64. The changes were obtained by applying the following Coccinelle script. (thanks Julia for answering all Coccinelle questions!). Following fix ups were done manually: * Removal of address argument from pte_fragment_alloc * Removal of pte_alloc_one_fast definitions from m68k and microblaze. // Options: --include-headers --no-includes // Note: I split the 'identifier fn' line, so if you are manually // running it, please unsplit it so it runs for you. virtual patch @pte_alloc_func_def depends on patch exists@ identifier E2; identifier fn =~ "^(__pte_alloc|pte_alloc_one|pte_alloc|__pte_alloc_kernel|pte_alloc_one_kernel)$"; type T2; @@ fn(... - , T2 E2 ) { ... } @pte_alloc_func_proto_noarg depends on patch exists@ type T1, T2, T3, T4; identifier fn =~ "^(__pte_alloc|pte_alloc_one|pte_alloc|__pte_alloc_kernel|pte_alloc_one_kernel)$"; @@ ( - T3 fn(T1, T2); + T3 fn(T1); | - T3 fn(T1, T2, T4); + T3 fn(T1, T2); ) @pte_alloc_func_proto depends on patch exists@ identifier E1, E2, E4; type T1, T2, T3, T4; identifier fn =~ "^(__pte_alloc|pte_alloc_one|pte_alloc|__pte_alloc_kernel|pte_alloc_one_kernel)$"; @@ ( - T3 fn(T1 E1, T2 E2); + T3 fn(T1 E1); | - T3 fn(T1 E1, T2 E2, T4 E4); + T3 fn(T1 E1, T2 E2); ) @pte_alloc_func_call depends on patch exists@ expression E2; identifier fn =~ "^(__pte_alloc|pte_alloc_one|pte_alloc|__pte_alloc_kernel|pte_alloc_one_kernel)$"; @@ fn(... -, E2 ) @pte_alloc_macro depends on patch exists@ identifier fn =~ "^(__pte_alloc|pte_alloc_one|pte_alloc|__pte_alloc_kernel|pte_alloc_one_kernel)$"; identifier a, b, c; expression e; position p; @@ ( - #define fn(a, b, c) e + #define fn(a, b) e | - #define fn(a, b) e + #define fn(a) e ) Suggested-by: Kirill A. Shutemov <kirill@shutemov.name> Cc: Kirill A. Shutemov <kirill@shutemov.name> Cc: Michal Hocko <mhocko@kernel.org> Cc: Julia Lawall <Julia.Lawall@lip6.fr> Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org> --- arch/alpha/include/asm/pgalloc.h | 6 +++--- arch/arc/include/asm/pgalloc.h | 5 ++--- arch/arm/include/asm/pgalloc.h | 4 ++-- arch/arm64/include/asm/pgalloc.h | 4 ++-- arch/hexagon/include/asm/pgalloc.h | 6 ++---- arch/ia64/include/asm/pgalloc.h | 5 ++--- arch/m68k/include/asm/mcf_pgalloc.h | 8 ++------ arch/m68k/include/asm/motorola_pgalloc.h | 4 ++-- arch/m68k/include/asm/sun3_pgalloc.h | 6 ++---- arch/microblaze/include/asm/pgalloc.h | 19 ++----------------- arch/microblaze/mm/pgtable.c | 3 +-- arch/mips/include/asm/pgalloc.h | 6 ++---- arch/nds32/include/asm/pgalloc.h | 5 ++--- arch/nios2/include/asm/pgalloc.h | 6 ++---- arch/openrisc/include/asm/pgalloc.h | 5 ++--- arch/openrisc/mm/ioremap.c | 3 +-- arch/parisc/include/asm/pgalloc.h | 4 ++-- arch/powerpc/include/asm/book3s/32/pgalloc.h | 4 ++-- arch/powerpc/include/asm/book3s/64/pgalloc.h | 12 +++++------- arch/powerpc/include/asm/nohash/32/pgalloc.h | 4 ++-- arch/powerpc/include/asm/nohash/64/pgalloc.h | 6 ++---- arch/powerpc/mm/pgtable-book3s64.c | 2 +- arch/powerpc/mm/pgtable_32.c | 4 ++-- arch/riscv/include/asm/pgalloc.h | 6 ++---- arch/s390/include/asm/pgalloc.h | 4 ++-- arch/sh/include/asm/pgalloc.h | 6 ++---- arch/sparc/include/asm/pgalloc_32.h | 5 ++--- arch/sparc/include/asm/pgalloc_64.h | 6 ++---- arch/sparc/mm/init_64.c | 6 ++---- arch/sparc/mm/srmmu.c | 4 ++-- arch/um/include/asm/pgalloc.h | 4 ++-- arch/um/kernel/mem.c | 4 ++-- arch/unicore32/include/asm/pgalloc.h | 4 ++-- arch/x86/include/asm/pgalloc.h | 4 ++-- arch/x86/mm/pgtable.c | 4 ++-- arch/xtensa/include/asm/pgalloc.h | 8 +++----- include/linux/mm.h | 13 ++++++------- mm/huge_memory.c | 8 ++++---- mm/kasan/kasan_init.c | 2 +- mm/memory.c | 17 ++++++++--------- mm/migrate.c | 2 +- mm/mremap.c | 2 +- mm/userfaultfd.c | 2 +- virt/kvm/arm/mmu.c | 2 +- 44 files changed, 97 insertions(+), 147 deletions(-) diff --git a/arch/alpha/include/asm/pgalloc.h b/arch/alpha/include/asm/pgalloc.h index ab3e3a8638fb..02f9f91bb4f0 100644 --- a/arch/alpha/include/asm/pgalloc.h +++ b/arch/alpha/include/asm/pgalloc.h @@ -52,7 +52,7 @@ pmd_free(struct mm_struct *mm, pmd_t *pmd) } static inline pte_t * -pte_alloc_one_kernel(struct mm_struct *mm, unsigned long address) +pte_alloc_one_kernel(struct mm_struct *mm) { pte_t *pte = (pte_t *)__get_free_page(GFP_KERNEL|__GFP_ZERO); return pte; @@ -65,9 +65,9 @@ pte_free_kernel(struct mm_struct *mm, pte_t *pte) } static inline pgtable_t -pte_alloc_one(struct mm_struct *mm, unsigned long address) +pte_alloc_one(struct mm_struct *mm) { - pte_t *pte = pte_alloc_one_kernel(mm, address); + pte_t *pte = pte_alloc_one_kernel(mm); struct page *page; if (!pte) diff --git a/arch/arc/include/asm/pgalloc.h b/arch/arc/include/asm/pgalloc.h index 3749234b7419..9c9b5a5ebf2e 100644 --- a/arch/arc/include/asm/pgalloc.h +++ b/arch/arc/include/asm/pgalloc.h @@ -90,8 +90,7 @@ static inline int __get_order_pte(void) return get_order(PTRS_PER_PTE * sizeof(pte_t)); } -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, - unsigned long address) +static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { pte_t *pte; @@ -102,7 +101,7 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, } static inline pgtable_t -pte_alloc_one(struct mm_struct *mm, unsigned long address) +pte_alloc_one(struct mm_struct *mm) { pgtable_t pte_pg; struct page *page; diff --git a/arch/arm/include/asm/pgalloc.h b/arch/arm/include/asm/pgalloc.h index 2d7344f0e208..17ab72f0cc4e 100644 --- a/arch/arm/include/asm/pgalloc.h +++ b/arch/arm/include/asm/pgalloc.h @@ -81,7 +81,7 @@ static inline void clean_pte_table(pte_t *pte) * +------------+ */ static inline pte_t * -pte_alloc_one_kernel(struct mm_struct *mm, unsigned long addr) +pte_alloc_one_kernel(struct mm_struct *mm) { pte_t *pte; @@ -93,7 +93,7 @@ pte_alloc_one_kernel(struct mm_struct *mm, unsigned long addr) } static inline pgtable_t -pte_alloc_one(struct mm_struct *mm, unsigned long addr) +pte_alloc_one(struct mm_struct *mm) { struct page *pte; diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index 2e05bcd944c8..52fa47c73bf0 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -91,13 +91,13 @@ extern pgd_t *pgd_alloc(struct mm_struct *mm); extern void pgd_free(struct mm_struct *mm, pgd_t *pgdp); static inline pte_t * -pte_alloc_one_kernel(struct mm_struct *mm, unsigned long addr) +pte_alloc_one_kernel(struct mm_struct *mm) { return (pte_t *)__get_free_page(PGALLOC_GFP); } static inline pgtable_t -pte_alloc_one(struct mm_struct *mm, unsigned long addr) +pte_alloc_one(struct mm_struct *mm) { struct page *pte; diff --git a/arch/hexagon/include/asm/pgalloc.h b/arch/hexagon/include/asm/pgalloc.h index eeebf862c46c..d36183887b60 100644 --- a/arch/hexagon/include/asm/pgalloc.h +++ b/arch/hexagon/include/asm/pgalloc.h @@ -59,8 +59,7 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) free_page((unsigned long) pgd); } -static inline struct page *pte_alloc_one(struct mm_struct *mm, - unsigned long address) +static inline struct page *pte_alloc_one(struct mm_struct *mm) { struct page *pte; @@ -75,8 +74,7 @@ static inline struct page *pte_alloc_one(struct mm_struct *mm, } /* _kernel variant gets to use a different allocator */ -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, - unsigned long address) +static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { gfp_t flags = GFP_KERNEL | __GFP_ZERO; return (pte_t *) __get_free_page(flags); diff --git a/arch/ia64/include/asm/pgalloc.h b/arch/ia64/include/asm/pgalloc.h index 3ee5362f2661..c9e481023c25 100644 --- a/arch/ia64/include/asm/pgalloc.h +++ b/arch/ia64/include/asm/pgalloc.h @@ -83,7 +83,7 @@ pmd_populate_kernel(struct mm_struct *mm, pmd_t * pmd_entry, pte_t * pte) pmd_val(*pmd_entry) = __pa(pte); } -static inline pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long addr) +static inline pgtable_t pte_alloc_one(struct mm_struct *mm) { struct page *page; void *pg; @@ -99,8 +99,7 @@ static inline pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long addr) return page; } -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, - unsigned long addr) +static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { return quicklist_alloc(0, GFP_KERNEL, NULL); } diff --git a/arch/m68k/include/asm/mcf_pgalloc.h b/arch/m68k/include/asm/mcf_pgalloc.h index 12fe700632f4..4399d712f6db 100644 --- a/arch/m68k/include/asm/mcf_pgalloc.h +++ b/arch/m68k/include/asm/mcf_pgalloc.h @@ -12,8 +12,7 @@ extern inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte) extern const char bad_pmd_string[]; -extern inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, - unsigned long address) +extern inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { unsigned long page = __get_free_page(GFP_DMA); @@ -32,8 +31,6 @@ extern inline pmd_t *pmd_alloc_kernel(pgd_t *pgd, unsigned long address) #define pmd_alloc_one_fast(mm, address) ({ BUG(); ((pmd_t *)1); }) #define pmd_alloc_one(mm, address) ({ BUG(); ((pmd_t *)2); }) -#define pte_alloc_one_fast(mm, addr) pte_alloc_one(mm, addr) - #define pmd_populate(mm, pmd, page) (pmd_val(*pmd) = \ (unsigned long)(page_address(page))) @@ -50,8 +47,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t page, #define __pmd_free_tlb(tlb, pmd, address) do { } while (0) -static inline struct page *pte_alloc_one(struct mm_struct *mm, - unsigned long address) +static inline struct page *pte_alloc_one(struct mm_struct *mm) { struct page *page = alloc_pages(GFP_DMA, 0); pte_t *pte; diff --git a/arch/m68k/include/asm/motorola_pgalloc.h b/arch/m68k/include/asm/motorola_pgalloc.h index 7859a86319cf..d04d9ba9b976 100644 --- a/arch/m68k/include/asm/motorola_pgalloc.h +++ b/arch/m68k/include/asm/motorola_pgalloc.h @@ -8,7 +8,7 @@ extern pmd_t *get_pointer_table(void); extern int free_pointer_table(pmd_t *); -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long address) +static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { pte_t *pte; @@ -28,7 +28,7 @@ static inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte) free_page((unsigned long) pte); } -static inline pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long address) +static inline pgtable_t pte_alloc_one(struct mm_struct *mm) { struct page *page; pte_t *pte; diff --git a/arch/m68k/include/asm/sun3_pgalloc.h b/arch/m68k/include/asm/sun3_pgalloc.h index 11485d38de4e..1456c5eecbd9 100644 --- a/arch/m68k/include/asm/sun3_pgalloc.h +++ b/arch/m68k/include/asm/sun3_pgalloc.h @@ -35,8 +35,7 @@ do { \ tlb_remove_page((tlb), pte); \ } while (0) -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, - unsigned long address) +static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { unsigned long page = __get_free_page(GFP_KERNEL); @@ -47,8 +46,7 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, return (pte_t *) (page); } -static inline pgtable_t pte_alloc_one(struct mm_struct *mm, - unsigned long address) +static inline pgtable_t pte_alloc_one(struct mm_struct *mm) { struct page *page = alloc_pages(GFP_KERNEL, 0); diff --git a/arch/microblaze/include/asm/pgalloc.h b/arch/microblaze/include/asm/pgalloc.h index 7c89390c0c13..f4cc9ffc449e 100644 --- a/arch/microblaze/include/asm/pgalloc.h +++ b/arch/microblaze/include/asm/pgalloc.h @@ -108,10 +108,9 @@ static inline void free_pgd_slow(pgd_t *pgd) #define pmd_alloc_one_fast(mm, address) ({ BUG(); ((pmd_t *)1); }) #define pmd_alloc_one(mm, address) ({ BUG(); ((pmd_t *)2); }) -extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long addr); +extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm); -static inline struct page *pte_alloc_one(struct mm_struct *mm, - unsigned long address) +static inline struct page *pte_alloc_one(struct mm_struct *mm) { struct page *ptepage; @@ -132,20 +131,6 @@ static inline struct page *pte_alloc_one(struct mm_struct *mm, return ptepage; } -static inline pte_t *pte_alloc_one_fast(struct mm_struct *mm, - unsigned long address) -{ - unsigned long *ret; - - ret = pte_quicklist; - if (ret != NULL) { - pte_quicklist = (unsigned long *)(*ret); - ret[0] = 0; - pgtable_cache_size--; - } - return (pte_t *)ret; -} - static inline void pte_free_fast(pte_t *pte) { *(unsigned long **)pte = pte_quicklist; diff --git a/arch/microblaze/mm/pgtable.c b/arch/microblaze/mm/pgtable.c index 7f525962cdfa..c2ce1e42b888 100644 --- a/arch/microblaze/mm/pgtable.c +++ b/arch/microblaze/mm/pgtable.c @@ -235,8 +235,7 @@ unsigned long iopa(unsigned long addr) return pa; } -__ref pte_t *pte_alloc_one_kernel(struct mm_struct *mm, - unsigned long address) +__ref pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { pte_t *pte; if (mem_init_done) { diff --git a/arch/mips/include/asm/pgalloc.h b/arch/mips/include/asm/pgalloc.h index 39b9f311c4ef..27808d9461f4 100644 --- a/arch/mips/include/asm/pgalloc.h +++ b/arch/mips/include/asm/pgalloc.h @@ -50,14 +50,12 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) free_pages((unsigned long)pgd, PGD_ORDER); } -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, - unsigned long address) +static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { return (pte_t *)__get_free_pages(GFP_KERNEL | __GFP_ZERO, PTE_ORDER); } -static inline struct page *pte_alloc_one(struct mm_struct *mm, - unsigned long address) +static inline struct page *pte_alloc_one(struct mm_struct *mm) { struct page *pte; diff --git a/arch/nds32/include/asm/pgalloc.h b/arch/nds32/include/asm/pgalloc.h index 27448869131a..3c5fee5b5759 100644 --- a/arch/nds32/include/asm/pgalloc.h +++ b/arch/nds32/include/asm/pgalloc.h @@ -22,8 +22,7 @@ extern void pgd_free(struct mm_struct *mm, pgd_t * pgd); #define check_pgt_cache() do { } while (0) -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, - unsigned long addr) +static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { pte_t *pte; @@ -34,7 +33,7 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, return pte; } -static inline pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long addr) +static inline pgtable_t pte_alloc_one(struct mm_struct *mm) { pgtable_t pte; diff --git a/arch/nios2/include/asm/pgalloc.h b/arch/nios2/include/asm/pgalloc.h index bb47d08c8ef7..3a149ead1207 100644 --- a/arch/nios2/include/asm/pgalloc.h +++ b/arch/nios2/include/asm/pgalloc.h @@ -37,8 +37,7 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) free_pages((unsigned long)pgd, PGD_ORDER); } -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, - unsigned long address) +static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { pte_t *pte; @@ -47,8 +46,7 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, return pte; } -static inline pgtable_t pte_alloc_one(struct mm_struct *mm, - unsigned long address) +static inline pgtable_t pte_alloc_one(struct mm_struct *mm) { struct page *pte; diff --git a/arch/openrisc/include/asm/pgalloc.h b/arch/openrisc/include/asm/pgalloc.h index 8999b9226512..149c82ee4b8b 100644 --- a/arch/openrisc/include/asm/pgalloc.h +++ b/arch/openrisc/include/asm/pgalloc.h @@ -70,10 +70,9 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) free_page((unsigned long)pgd); } -extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long address); +extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm); -static inline struct page *pte_alloc_one(struct mm_struct *mm, - unsigned long address) +static inline struct page *pte_alloc_one(struct mm_struct *mm) { struct page *pte; pte = alloc_pages(GFP_KERNEL, 0); diff --git a/arch/openrisc/mm/ioremap.c b/arch/openrisc/mm/ioremap.c index 2175e4bfd9fc..24fb1021c75a 100644 --- a/arch/openrisc/mm/ioremap.c +++ b/arch/openrisc/mm/ioremap.c @@ -118,8 +118,7 @@ EXPORT_SYMBOL(iounmap); * the memblock infrastructure. */ -pte_t __ref *pte_alloc_one_kernel(struct mm_struct *mm, - unsigned long address) +pte_t __ref *pte_alloc_one_kernel(struct mm_struct *mm) { pte_t *pte; diff --git a/arch/parisc/include/asm/pgalloc.h b/arch/parisc/include/asm/pgalloc.h index cf13275f7c6d..d05c678c77c4 100644 --- a/arch/parisc/include/asm/pgalloc.h +++ b/arch/parisc/include/asm/pgalloc.h @@ -122,7 +122,7 @@ pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, pte_t *pte) #define pmd_pgtable(pmd) pmd_page(pmd) static inline pgtable_t -pte_alloc_one(struct mm_struct *mm, unsigned long address) +pte_alloc_one(struct mm_struct *mm) { struct page *page = alloc_page(GFP_KERNEL|__GFP_ZERO); if (!page) @@ -135,7 +135,7 @@ pte_alloc_one(struct mm_struct *mm, unsigned long address) } static inline pte_t * -pte_alloc_one_kernel(struct mm_struct *mm, unsigned long addr) +pte_alloc_one_kernel(struct mm_struct *mm) { pte_t *pte = (pte_t *)__get_free_page(GFP_KERNEL|__GFP_ZERO); return pte; diff --git a/arch/powerpc/include/asm/book3s/32/pgalloc.h b/arch/powerpc/include/asm/book3s/32/pgalloc.h index 82e44b1a00ae..af9e13555d95 100644 --- a/arch/powerpc/include/asm/book3s/32/pgalloc.h +++ b/arch/powerpc/include/asm/book3s/32/pgalloc.h @@ -82,8 +82,8 @@ static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmdp, #define pmd_pgtable(pmd) pmd_page(pmd) #endif -extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long addr); -extern pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long addr); +extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm); +extern pgtable_t pte_alloc_one(struct mm_struct *mm); static inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte) { diff --git a/arch/powerpc/include/asm/book3s/64/pgalloc.h b/arch/powerpc/include/asm/book3s/64/pgalloc.h index 391ed2c3b697..8f1d92e99fe5 100644 --- a/arch/powerpc/include/asm/book3s/64/pgalloc.h +++ b/arch/powerpc/include/asm/book3s/64/pgalloc.h @@ -42,7 +42,7 @@ extern struct kmem_cache *pgtable_cache[]; pgtable_cache[(shift) - 1]; \ }) -extern pte_t *pte_fragment_alloc(struct mm_struct *, unsigned long, int); +extern pte_t *pte_fragment_alloc(struct mm_struct *, int); extern pmd_t *pmd_fragment_alloc(struct mm_struct *, unsigned long); extern void pte_fragment_free(unsigned long *, int); extern void pmd_fragment_free(unsigned long *); @@ -192,16 +192,14 @@ static inline pgtable_t pmd_pgtable(pmd_t pmd) return (pgtable_t)pmd_page_vaddr(pmd); } -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, - unsigned long address) +static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { - return (pte_t *)pte_fragment_alloc(mm, address, 1); + return (pte_t *)pte_fragment_alloc(mm, 1); } -static inline pgtable_t pte_alloc_one(struct mm_struct *mm, - unsigned long address) +static inline pgtable_t pte_alloc_one(struct mm_struct *mm) { - return (pgtable_t)pte_fragment_alloc(mm, address, 0); + return (pgtable_t)pte_fragment_alloc(mm, 0); } static inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte) diff --git a/arch/powerpc/include/asm/nohash/32/pgalloc.h b/arch/powerpc/include/asm/nohash/32/pgalloc.h index 8825953c225b..16623f53f0d4 100644 --- a/arch/powerpc/include/asm/nohash/32/pgalloc.h +++ b/arch/powerpc/include/asm/nohash/32/pgalloc.h @@ -83,8 +83,8 @@ static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmdp, #define pmd_pgtable(pmd) pmd_page(pmd) #endif -extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long addr); -extern pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long addr); +extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm); +extern pgtable_t pte_alloc_one(struct mm_struct *mm); static inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte) { diff --git a/arch/powerpc/include/asm/nohash/64/pgalloc.h b/arch/powerpc/include/asm/nohash/64/pgalloc.h index e2d62d033708..2e7e0230edf4 100644 --- a/arch/powerpc/include/asm/nohash/64/pgalloc.h +++ b/arch/powerpc/include/asm/nohash/64/pgalloc.h @@ -96,14 +96,12 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) } -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, - unsigned long address) +static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { return (pte_t *)__get_free_page(GFP_KERNEL | __GFP_ZERO); } -static inline pgtable_t pte_alloc_one(struct mm_struct *mm, - unsigned long address) +static inline pgtable_t pte_alloc_one(struct mm_struct *mm) { struct page *page; pte_t *pte; diff --git a/arch/powerpc/mm/pgtable-book3s64.c b/arch/powerpc/mm/pgtable-book3s64.c index 01d7c0f7c4f0..cff1d426ca6a 100644 --- a/arch/powerpc/mm/pgtable-book3s64.c +++ b/arch/powerpc/mm/pgtable-book3s64.c @@ -379,7 +379,7 @@ static pte_t *__alloc_for_ptecache(struct mm_struct *mm, int kernel) return (pte_t *)ret; } -pte_t *pte_fragment_alloc(struct mm_struct *mm, unsigned long vmaddr, int kernel) +pte_t *pte_fragment_alloc(struct mm_struct *mm, int kernel) { pte_t *pte; diff --git a/arch/powerpc/mm/pgtable_32.c b/arch/powerpc/mm/pgtable_32.c index 120a49bfb9c6..b99a89cdcc5e 100644 --- a/arch/powerpc/mm/pgtable_32.c +++ b/arch/powerpc/mm/pgtable_32.c @@ -43,7 +43,7 @@ EXPORT_SYMBOL(ioremap_bot); /* aka VMALLOC_END */ extern char etext[], _stext[], _sinittext[], _einittext[]; -__ref pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long address) +__ref pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { pte_t *pte; @@ -57,7 +57,7 @@ __ref pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long address) return pte; } -pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long address) +pgtable_t pte_alloc_one(struct mm_struct *mm) { struct page *ptepage; diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index a79ed5faff3a..94043cf83c90 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -82,15 +82,13 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) #endif /* __PAGETABLE_PMD_FOLDED */ -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, - unsigned long address) +static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { return (pte_t *)__get_free_page( GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_ZERO); } -static inline struct page *pte_alloc_one(struct mm_struct *mm, - unsigned long address) +static inline struct page *pte_alloc_one(struct mm_struct *mm) { struct page *pte; diff --git a/arch/s390/include/asm/pgalloc.h b/arch/s390/include/asm/pgalloc.h index f0f9bcf94c03..ce2ca8cbd2ec 100644 --- a/arch/s390/include/asm/pgalloc.h +++ b/arch/s390/include/asm/pgalloc.h @@ -139,8 +139,8 @@ static inline void pmd_populate(struct mm_struct *mm, /* * page table entry allocation/free routines. */ -#define pte_alloc_one_kernel(mm, vmaddr) ((pte_t *) page_table_alloc(mm)) -#define pte_alloc_one(mm, vmaddr) ((pte_t *) page_table_alloc(mm)) +#define pte_alloc_one_kernel(mm) ((pte_t *)page_table_alloc(mm)) +#define pte_alloc_one(mm) ((pte_t *)page_table_alloc(mm)) #define pte_free_kernel(mm, pte) page_table_free(mm, (unsigned long *) pte) #define pte_free(mm, pte) page_table_free(mm, (unsigned long *) pte) diff --git a/arch/sh/include/asm/pgalloc.h b/arch/sh/include/asm/pgalloc.h index ed053a359ab7..8ad73cb31121 100644 --- a/arch/sh/include/asm/pgalloc.h +++ b/arch/sh/include/asm/pgalloc.h @@ -32,14 +32,12 @@ static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd, /* * Allocate and free page tables. */ -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, - unsigned long address) +static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { return quicklist_alloc(QUICK_PT, GFP_KERNEL, NULL); } -static inline pgtable_t pte_alloc_one(struct mm_struct *mm, - unsigned long address) +static inline pgtable_t pte_alloc_one(struct mm_struct *mm) { struct page *page; void *pg; diff --git a/arch/sparc/include/asm/pgalloc_32.h b/arch/sparc/include/asm/pgalloc_32.h index 90459481c6c7..282be50a4adf 100644 --- a/arch/sparc/include/asm/pgalloc_32.h +++ b/arch/sparc/include/asm/pgalloc_32.h @@ -58,10 +58,9 @@ void pmd_populate(struct mm_struct *mm, pmd_t *pmdp, struct page *ptep); void pmd_set(pmd_t *pmdp, pte_t *ptep); #define pmd_populate_kernel(MM, PMD, PTE) pmd_set(PMD, PTE) -pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long address); +pgtable_t pte_alloc_one(struct mm_struct *mm); -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, - unsigned long address) +static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { return srmmu_get_nocache(PTE_SIZE, PTE_SIZE); } diff --git a/arch/sparc/include/asm/pgalloc_64.h b/arch/sparc/include/asm/pgalloc_64.h index 874632f34f62..48abccba4991 100644 --- a/arch/sparc/include/asm/pgalloc_64.h +++ b/arch/sparc/include/asm/pgalloc_64.h @@ -60,10 +60,8 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) kmem_cache_free(pgtable_cache, pmd); } -pte_t *pte_alloc_one_kernel(struct mm_struct *mm, - unsigned long address); -pgtable_t pte_alloc_one(struct mm_struct *mm, - unsigned long address); +pte_t *pte_alloc_one_kernel(struct mm_struct *mm); +pgtable_t pte_alloc_one(struct mm_struct *mm); void pte_free_kernel(struct mm_struct *mm, pte_t *pte); void pte_free(struct mm_struct *mm, pgtable_t ptepage); diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c index f396048a0d68..6133f21811e9 100644 --- a/arch/sparc/mm/init_64.c +++ b/arch/sparc/mm/init_64.c @@ -2921,8 +2921,7 @@ void __flush_tlb_all(void) : : "r" (pstate)); } -pte_t *pte_alloc_one_kernel(struct mm_struct *mm, - unsigned long address) +pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { struct page *page = alloc_page(GFP_KERNEL | __GFP_ZERO); pte_t *pte = NULL; @@ -2933,8 +2932,7 @@ pte_t *pte_alloc_one_kernel(struct mm_struct *mm, return pte; } -pgtable_t pte_alloc_one(struct mm_struct *mm, - unsigned long address) +pgtable_t pte_alloc_one(struct mm_struct *mm) { struct page *page = alloc_page(GFP_KERNEL | __GFP_ZERO); if (!page) diff --git a/arch/sparc/mm/srmmu.c b/arch/sparc/mm/srmmu.c index be9cb0065179..ce67a96e70c3 100644 --- a/arch/sparc/mm/srmmu.c +++ b/arch/sparc/mm/srmmu.c @@ -364,12 +364,12 @@ pgd_t *get_pgd_fast(void) * Alignments up to the page size are the same for physical and virtual * addresses of the nocache area. */ -pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long address) +pgtable_t pte_alloc_one(struct mm_struct *mm) { unsigned long pte; struct page *page; - if ((pte = (unsigned long)pte_alloc_one_kernel(mm, address)) == 0) + if ((pte = (unsigned long)pte_alloc_one_kernel(mm)) == 0) return NULL; page = pfn_to_page(__nocache_pa(pte) >> PAGE_SHIFT); if (!pgtable_page_ctor(page)) { diff --git a/arch/um/include/asm/pgalloc.h b/arch/um/include/asm/pgalloc.h index bf90b2aa2002..99eb5682792a 100644 --- a/arch/um/include/asm/pgalloc.h +++ b/arch/um/include/asm/pgalloc.h @@ -25,8 +25,8 @@ extern pgd_t *pgd_alloc(struct mm_struct *); extern void pgd_free(struct mm_struct *mm, pgd_t *pgd); -extern pte_t *pte_alloc_one_kernel(struct mm_struct *, unsigned long); -extern pgtable_t pte_alloc_one(struct mm_struct *, unsigned long); +extern pte_t *pte_alloc_one_kernel(struct mm_struct *); +extern pgtable_t pte_alloc_one(struct mm_struct *); static inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte) { diff --git a/arch/um/kernel/mem.c b/arch/um/kernel/mem.c index 3c0e470ea646..1f277191fbf3 100644 --- a/arch/um/kernel/mem.c +++ b/arch/um/kernel/mem.c @@ -197,7 +197,7 @@ void pgd_free(struct mm_struct *mm, pgd_t *pgd) free_page((unsigned long) pgd); } -pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long address) +pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { pte_t *pte; @@ -205,7 +205,7 @@ pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long address) return pte; } -pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long address) +pgtable_t pte_alloc_one(struct mm_struct *mm) { struct page *pte; diff --git a/arch/unicore32/include/asm/pgalloc.h b/arch/unicore32/include/asm/pgalloc.h index f0fdb268f8f2..7cceabecf4e3 100644 --- a/arch/unicore32/include/asm/pgalloc.h +++ b/arch/unicore32/include/asm/pgalloc.h @@ -34,7 +34,7 @@ extern void free_pgd_slow(struct mm_struct *mm, pgd_t *pgd); * Allocate one PTE table. */ static inline pte_t * -pte_alloc_one_kernel(struct mm_struct *mm, unsigned long addr) +pte_alloc_one_kernel(struct mm_struct *mm) { pte_t *pte; @@ -46,7 +46,7 @@ pte_alloc_one_kernel(struct mm_struct *mm, unsigned long addr) } static inline pgtable_t -pte_alloc_one(struct mm_struct *mm, unsigned long addr) +pte_alloc_one(struct mm_struct *mm) { struct page *pte; diff --git a/arch/x86/include/asm/pgalloc.h b/arch/x86/include/asm/pgalloc.h index fbd578daa66e..5068e85165b2 100644 --- a/arch/x86/include/asm/pgalloc.h +++ b/arch/x86/include/asm/pgalloc.h @@ -47,8 +47,8 @@ extern gfp_t __userpte_alloc_gfp; extern pgd_t *pgd_alloc(struct mm_struct *); extern void pgd_free(struct mm_struct *mm, pgd_t *pgd); -extern pte_t *pte_alloc_one_kernel(struct mm_struct *, unsigned long); -extern pgtable_t pte_alloc_one(struct mm_struct *, unsigned long); +extern pte_t *pte_alloc_one_kernel(struct mm_struct *); +extern pgtable_t pte_alloc_one(struct mm_struct *); /* Should really implement gc for free page table pages. This could be done with a reference count in struct page. */ diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 089e78c4effd..a2eff247377b 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -23,12 +23,12 @@ EXPORT_SYMBOL(physical_mask); gfp_t __userpte_alloc_gfp = PGALLOC_GFP | PGALLOC_USER_GFP; -pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long address) +pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { return (pte_t *)__get_free_page(PGALLOC_GFP & ~__GFP_ACCOUNT); } -pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long address) +pgtable_t pte_alloc_one(struct mm_struct *mm) { struct page *pte; diff --git a/arch/xtensa/include/asm/pgalloc.h b/arch/xtensa/include/asm/pgalloc.h index 1065bc8bcae5..b3b388ff2f01 100644 --- a/arch/xtensa/include/asm/pgalloc.h +++ b/arch/xtensa/include/asm/pgalloc.h @@ -38,8 +38,7 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) free_page((unsigned long)pgd); } -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, - unsigned long address) +static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { pte_t *ptep; int i; @@ -52,13 +51,12 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, return ptep; } -static inline pgtable_t pte_alloc_one(struct mm_struct *mm, - unsigned long addr) +static inline pgtable_t pte_alloc_one(struct mm_struct *mm) { pte_t *pte; struct page *page; - pte = pte_alloc_one_kernel(mm, addr); + pte = pte_alloc_one_kernel(mm); if (!pte) return NULL; page = virt_to_page(pte); diff --git a/include/linux/mm.h b/include/linux/mm.h index 0416a7204be3..43ce50edc499 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1789,8 +1789,8 @@ static inline void mm_inc_nr_ptes(struct mm_struct *mm) {} static inline void mm_dec_nr_ptes(struct mm_struct *mm) {} #endif -int __pte_alloc(struct mm_struct *mm, pmd_t *pmd, unsigned long address); -int __pte_alloc_kernel(pmd_t *pmd, unsigned long address); +int __pte_alloc(struct mm_struct *mm, pmd_t *pmd); +int __pte_alloc_kernel(pmd_t *pmd); /* * The following ifdef needed to get the 4level-fixup.h header to work. @@ -1928,18 +1928,17 @@ static inline void pgtable_page_dtor(struct page *page) pte_unmap(pte); \ } while (0) -#define pte_alloc(mm, pmd, address) \ - (unlikely(pmd_none(*(pmd))) && __pte_alloc(mm, pmd, address)) +#define pte_alloc(mm, pmd) (unlikely(pmd_none(*(pmd))) && __pte_alloc(mm, pmd)) #define pte_alloc_map(mm, pmd, address) \ - (pte_alloc(mm, pmd, address) ? NULL : pte_offset_map(pmd, address)) + (pte_alloc(mm, pmd) ? NULL : pte_offset_map(pmd, address)) #define pte_alloc_map_lock(mm, pmd, address, ptlp) \ - (pte_alloc(mm, pmd, address) ? \ + (pte_alloc(mm, pmd) ? \ NULL : pte_offset_map_lock(mm, pmd, address, ptlp)) #define pte_alloc_kernel(pmd, address) \ - ((unlikely(pmd_none(*(pmd))) && __pte_alloc_kernel(pmd, address))? \ + ((unlikely(pmd_none(*(pmd))) && __pte_alloc_kernel(pmd))? \ NULL: pte_offset_kernel(pmd, address)) #if USE_SPLIT_PMD_PTLOCKS diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 00704060b7f7..fd7e8714e5a1 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -558,7 +558,7 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf, return VM_FAULT_FALLBACK; } - pgtable = pte_alloc_one(vma->vm_mm, haddr); + pgtable = pte_alloc_one(vma->vm_mm); if (unlikely(!pgtable)) { ret = VM_FAULT_OOM; goto release; @@ -683,7 +683,7 @@ vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf) struct page *zero_page; bool set; vm_fault_t ret; - pgtable = pte_alloc_one(vma->vm_mm, haddr); + pgtable = pte_alloc_one(vma->vm_mm); if (unlikely(!pgtable)) return VM_FAULT_OOM; zero_page = mm_get_huge_zero_page(vma->vm_mm); @@ -772,7 +772,7 @@ vm_fault_t vmf_insert_pfn_pmd(struct vm_area_struct *vma, unsigned long addr, return VM_FAULT_SIGBUS; if (arch_needs_pgtable_deposit()) { - pgtable = pte_alloc_one(vma->vm_mm, addr); + pgtable = pte_alloc_one(vma->vm_mm); if (!pgtable) return VM_FAULT_OOM; } @@ -910,7 +910,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, if (!vma_is_anonymous(vma)) return 0; - pgtable = pte_alloc_one(dst_mm, addr); + pgtable = pte_alloc_one(dst_mm); if (unlikely(!pgtable)) goto out; diff --git a/mm/kasan/kasan_init.c b/mm/kasan/kasan_init.c index 7a2a2f13f86f..272849cd2007 100644 --- a/mm/kasan/kasan_init.c +++ b/mm/kasan/kasan_init.c @@ -121,7 +121,7 @@ static int __ref zero_pmd_populate(pud_t *pud, unsigned long addr, pte_t *p; if (slab_is_available()) - p = pte_alloc_one_kernel(&init_mm, addr); + p = pte_alloc_one_kernel(&init_mm); else p = early_alloc(PAGE_SIZE, NUMA_NO_NODE); if (!p) diff --git a/mm/memory.c b/mm/memory.c index c467102a5cbc..3afdcf38993d 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -647,10 +647,10 @@ void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *vma, } } -int __pte_alloc(struct mm_struct *mm, pmd_t *pmd, unsigned long address) +int __pte_alloc(struct mm_struct *mm, pmd_t *pmd) { spinlock_t *ptl; - pgtable_t new = pte_alloc_one(mm, address); + pgtable_t new = pte_alloc_one(mm); if (!new) return -ENOMEM; @@ -681,9 +681,9 @@ int __pte_alloc(struct mm_struct *mm, pmd_t *pmd, unsigned long address) return 0; } -int __pte_alloc_kernel(pmd_t *pmd, unsigned long address) +int __pte_alloc_kernel(pmd_t *pmd) { - pte_t *new = pte_alloc_one_kernel(&init_mm, address); + pte_t *new = pte_alloc_one_kernel(&init_mm); if (!new) return -ENOMEM; @@ -3139,7 +3139,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) * * Here we only have down_read(mmap_sem). */ - if (pte_alloc(vma->vm_mm, vmf->pmd, vmf->address)) + if (pte_alloc(vma->vm_mm, vmf->pmd)) return VM_FAULT_OOM; /* See the comment in pte_alloc_one_map() */ @@ -3286,7 +3286,7 @@ static vm_fault_t pte_alloc_one_map(struct vm_fault *vmf) pmd_populate(vma->vm_mm, vmf->pmd, vmf->prealloc_pte); spin_unlock(vmf->ptl); vmf->prealloc_pte = NULL; - } else if (unlikely(pte_alloc(vma->vm_mm, vmf->pmd, vmf->address))) { + } else if (unlikely(pte_alloc(vma->vm_mm, vmf->pmd))) { return VM_FAULT_OOM; } map_pte: @@ -3365,7 +3365,7 @@ static vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page) * related to pte entry. Use the preallocated table for that. */ if (arch_needs_pgtable_deposit() && !vmf->prealloc_pte) { - vmf->prealloc_pte = pte_alloc_one(vma->vm_mm, vmf->address); + vmf->prealloc_pte = pte_alloc_one(vma->vm_mm); if (!vmf->prealloc_pte) return VM_FAULT_OOM; smp_wmb(); /* See comment in __pte_alloc() */ @@ -3603,8 +3603,7 @@ static vm_fault_t do_fault_around(struct vm_fault *vmf) start_pgoff + nr_pages - 1); if (pmd_none(*vmf->pmd)) { - vmf->prealloc_pte = pte_alloc_one(vmf->vma->vm_mm, - vmf->address); + vmf->prealloc_pte = pte_alloc_one(vmf->vma->vm_mm); if (!vmf->prealloc_pte) goto out; smp_wmb(); /* See comment in __pte_alloc() */ diff --git a/mm/migrate.c b/mm/migrate.c index 84381b55b2bd..3080b0626026 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -2605,7 +2605,7 @@ static void migrate_vma_insert_page(struct migrate_vma *migrate, * * Here we only have down_read(mmap_sem). */ - if (pte_alloc(mm, pmdp, addr)) + if (pte_alloc(mm, pmdp)) goto abort; /* See the comment in pte_alloc_one_map() */ diff --git a/mm/mremap.c b/mm/mremap.c index 5c2e18505f75..9e68a02a52b1 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -240,7 +240,7 @@ unsigned long move_page_tables(struct vm_area_struct *vma, if (pmd_trans_unstable(old_pmd)) continue; } - if (pte_alloc(new_vma->vm_mm, new_pmd, new_addr)) + if (pte_alloc(new_vma->vm_mm, new_pmd)) break; next = (new_addr + PMD_SIZE) & PMD_MASK; if (extent > next - new_addr) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 5029f241908f..f05c8bc38ca5 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -513,7 +513,7 @@ static __always_inline ssize_t __mcopy_atomic(struct mm_struct *dst_mm, break; } if (unlikely(pmd_none(dst_pmdval)) && - unlikely(__pte_alloc(dst_mm, dst_pmd, dst_addr))) { + unlikely(__pte_alloc(dst_mm, dst_pmd))) { err = -ENOMEM; break; } diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c index ed162a6c57c5..3f8180414301 100644 --- a/virt/kvm/arm/mmu.c +++ b/virt/kvm/arm/mmu.c @@ -628,7 +628,7 @@ static int create_hyp_pmd_mappings(pud_t *pud, unsigned long start, BUG_ON(pmd_sect(*pmd)); if (pmd_none(*pmd)) { - pte = pte_alloc_one_kernel(NULL, addr); + pte = pte_alloc_one_kernel(NULL); if (!pte) { kvm_err("Cannot allocate Hyp pte\n"); return -ENOMEM; -- 2.19.0.605.g01d371f741-goog ^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH 1/4] treewide: remove unused address argument from pte_alloc functions (v2) 2018-10-13 1:31 ` [PATCH 1/4] treewide: remove unused address argument from pte_alloc functions (v2) Joel Fernandes (Google) @ 2018-10-13 1:31 ` Joel Fernandes (Google) 2018-10-24 8:37 ` Peter Zijlstra 1 sibling, 0 replies; 52+ messages in thread From: Joel Fernandes (Google) @ 2018-10-13 1:31 UTC (permalink / raw) To: linux-kernel Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Peter Zijlstra, Catalin Marinas, Dave Hansen, Will Deacon, Michal Hocko, linux-mm, lokeshgidra, Joel Fernandes (Google), linux-riscv, elfring, Jonas Bonn, kvmarm, dancol, Yoshinori Sato, sparclinux, linux-xtensa, linux-hexagon, Helge Deller, maintainer:X86 ARCHITECTURE 32-BIT AND 64-BIT, hughd, James E.J. Bottomley, kasan-dev, anton.ivanov, Ingo Molnar, Geert Uytterhoeven, Andrey Ryabinin, linux-snps-arc, kernel-team, Sam Creasey, Fenghua Yu, linux-s390, Jeff Dike, linux-um, Stefan Kristiansson, Julia Lawall, linux-m68k, Borislav Petkov, Andy Lutomirski, nios2-dev, Kirill A . Shutemov, Stafford Horne, Guan Xuetao, Chris Zankel, Tony Luck, Richard Weinberger, linux-parisc, Max Filippov, pantin, minchan, Thomas Gleixner, linux-alpha, Ley Foon Tan, akpm, linuxppc-dev, David S. Miller This series speeds up mremap(2) syscall by copying page tables at the PMD level even for non-THP systems. There is concern that the extra 'address' argument that mremap passes to pte_alloc may do something subtle architecture related in the future that may make the scheme not work. Also we find that there is no point in passing the 'address' to pte_alloc since its unused. So this patch therefore removes this argument tree-wide resulting in a nice negative diff as well. Also ensuring along the way that the enabled architectures do not do anything funky with 'address' argument that goes unnoticed by the optimization. Build and boot tested on x86-64. Build tested on arm64. The changes were obtained by applying the following Coccinelle script. (thanks Julia for answering all Coccinelle questions!). Following fix ups were done manually: * Removal of address argument from pte_fragment_alloc * Removal of pte_alloc_one_fast definitions from m68k and microblaze. // Options: --include-headers --no-includes // Note: I split the 'identifier fn' line, so if you are manually // running it, please unsplit it so it runs for you. virtual patch @pte_alloc_func_def depends on patch exists@ identifier E2; identifier fn =~ "^(__pte_alloc|pte_alloc_one|pte_alloc|__pte_alloc_kernel|pte_alloc_one_kernel)$"; type T2; @@ fn(... - , T2 E2 ) { ... } @pte_alloc_func_proto_noarg depends on patch exists@ type T1, T2, T3, T4; identifier fn =~ "^(__pte_alloc|pte_alloc_one|pte_alloc|__pte_alloc_kernel|pte_alloc_one_kernel)$"; @@ ( - T3 fn(T1, T2); + T3 fn(T1); | - T3 fn(T1, T2, T4); + T3 fn(T1, T2); ) @pte_alloc_func_proto depends on patch exists@ identifier E1, E2, E4; type T1, T2, T3, T4; identifier fn =~ "^(__pte_alloc|pte_alloc_one|pte_alloc|__pte_alloc_kernel|pte_alloc_one_kernel)$"; @@ ( - T3 fn(T1 E1, T2 E2); + T3 fn(T1 E1); | - T3 fn(T1 E1, T2 E2, T4 E4); + T3 fn(T1 E1, T2 E2); ) @pte_alloc_func_call depends on patch exists@ expression E2; identifier fn =~ "^(__pte_alloc|pte_alloc_one|pte_alloc|__pte_alloc_kernel|pte_alloc_one_kernel)$"; @@ fn(... -, E2 ) @pte_alloc_macro depends on patch exists@ identifier fn =~ "^(__pte_alloc|pte_alloc_one|pte_alloc|__pte_alloc_kernel|pte_alloc_one_kernel)$"; identifier a, b, c; expression e; position p; @@ ( - #define fn(a, b, c) e + #define fn(a, b) e | - #define fn(a, b) e + #define fn(a) e ) Suggested-by: Kirill A. Shutemov <kirill@shutemov.name> Cc: Kirill A. Shutemov <kirill@shutemov.name> Cc: Michal Hocko <mhocko@kernel.org> Cc: Julia Lawall <Julia.Lawall@lip6.fr> Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org> --- arch/alpha/include/asm/pgalloc.h | 6 +++--- arch/arc/include/asm/pgalloc.h | 5 ++--- arch/arm/include/asm/pgalloc.h | 4 ++-- arch/arm64/include/asm/pgalloc.h | 4 ++-- arch/hexagon/include/asm/pgalloc.h | 6 ++---- arch/ia64/include/asm/pgalloc.h | 5 ++--- arch/m68k/include/asm/mcf_pgalloc.h | 8 ++------ arch/m68k/include/asm/motorola_pgalloc.h | 4 ++-- arch/m68k/include/asm/sun3_pgalloc.h | 6 ++---- arch/microblaze/include/asm/pgalloc.h | 19 ++----------------- arch/microblaze/mm/pgtable.c | 3 +-- arch/mips/include/asm/pgalloc.h | 6 ++---- arch/nds32/include/asm/pgalloc.h | 5 ++--- arch/nios2/include/asm/pgalloc.h | 6 ++---- arch/openrisc/include/asm/pgalloc.h | 5 ++--- arch/openrisc/mm/ioremap.c | 3 +-- arch/parisc/include/asm/pgalloc.h | 4 ++-- arch/powerpc/include/asm/book3s/32/pgalloc.h | 4 ++-- arch/powerpc/include/asm/book3s/64/pgalloc.h | 12 +++++------- arch/powerpc/include/asm/nohash/32/pgalloc.h | 4 ++-- arch/powerpc/include/asm/nohash/64/pgalloc.h | 6 ++---- arch/powerpc/mm/pgtable-book3s64.c | 2 +- arch/powerpc/mm/pgtable_32.c | 4 ++-- arch/riscv/include/asm/pgalloc.h | 6 ++---- arch/s390/include/asm/pgalloc.h | 4 ++-- arch/sh/include/asm/pgalloc.h | 6 ++---- arch/sparc/include/asm/pgalloc_32.h | 5 ++--- arch/sparc/include/asm/pgalloc_64.h | 6 ++---- arch/sparc/mm/init_64.c | 6 ++---- arch/sparc/mm/srmmu.c | 4 ++-- arch/um/include/asm/pgalloc.h | 4 ++-- arch/um/kernel/mem.c | 4 ++-- arch/unicore32/include/asm/pgalloc.h | 4 ++-- arch/x86/include/asm/pgalloc.h | 4 ++-- arch/x86/mm/pgtable.c | 4 ++-- arch/xtensa/include/asm/pgalloc.h | 8 +++----- include/linux/mm.h | 13 ++++++------- mm/huge_memory.c | 8 ++++---- mm/kasan/kasan_init.c | 2 +- mm/memory.c | 17 ++++++++--------- mm/migrate.c | 2 +- mm/mremap.c | 2 +- mm/userfaultfd.c | 2 +- virt/kvm/arm/mmu.c | 2 +- 44 files changed, 97 insertions(+), 147 deletions(-) diff --git a/arch/alpha/include/asm/pgalloc.h b/arch/alpha/include/asm/pgalloc.h index ab3e3a8638fb..02f9f91bb4f0 100644 --- a/arch/alpha/include/asm/pgalloc.h +++ b/arch/alpha/include/asm/pgalloc.h @@ -52,7 +52,7 @@ pmd_free(struct mm_struct *mm, pmd_t *pmd) } static inline pte_t * -pte_alloc_one_kernel(struct mm_struct *mm, unsigned long address) +pte_alloc_one_kernel(struct mm_struct *mm) { pte_t *pte = (pte_t *)__get_free_page(GFP_KERNEL|__GFP_ZERO); return pte; @@ -65,9 +65,9 @@ pte_free_kernel(struct mm_struct *mm, pte_t *pte) } static inline pgtable_t -pte_alloc_one(struct mm_struct *mm, unsigned long address) +pte_alloc_one(struct mm_struct *mm) { - pte_t *pte = pte_alloc_one_kernel(mm, address); + pte_t *pte = pte_alloc_one_kernel(mm); struct page *page; if (!pte) diff --git a/arch/arc/include/asm/pgalloc.h b/arch/arc/include/asm/pgalloc.h index 3749234b7419..9c9b5a5ebf2e 100644 --- a/arch/arc/include/asm/pgalloc.h +++ b/arch/arc/include/asm/pgalloc.h @@ -90,8 +90,7 @@ static inline int __get_order_pte(void) return get_order(PTRS_PER_PTE * sizeof(pte_t)); } -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, - unsigned long address) +static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { pte_t *pte; @@ -102,7 +101,7 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, } static inline pgtable_t -pte_alloc_one(struct mm_struct *mm, unsigned long address) +pte_alloc_one(struct mm_struct *mm) { pgtable_t pte_pg; struct page *page; diff --git a/arch/arm/include/asm/pgalloc.h b/arch/arm/include/asm/pgalloc.h index 2d7344f0e208..17ab72f0cc4e 100644 --- a/arch/arm/include/asm/pgalloc.h +++ b/arch/arm/include/asm/pgalloc.h @@ -81,7 +81,7 @@ static inline void clean_pte_table(pte_t *pte) * +------------+ */ static inline pte_t * -pte_alloc_one_kernel(struct mm_struct *mm, unsigned long addr) +pte_alloc_one_kernel(struct mm_struct *mm) { pte_t *pte; @@ -93,7 +93,7 @@ pte_alloc_one_kernel(struct mm_struct *mm, unsigned long addr) } static inline pgtable_t -pte_alloc_one(struct mm_struct *mm, unsigned long addr) +pte_alloc_one(struct mm_struct *mm) { struct page *pte; diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index 2e05bcd944c8..52fa47c73bf0 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -91,13 +91,13 @@ extern pgd_t *pgd_alloc(struct mm_struct *mm); extern void pgd_free(struct mm_struct *mm, pgd_t *pgdp); static inline pte_t * -pte_alloc_one_kernel(struct mm_struct *mm, unsigned long addr) +pte_alloc_one_kernel(struct mm_struct *mm) { return (pte_t *)__get_free_page(PGALLOC_GFP); } static inline pgtable_t -pte_alloc_one(struct mm_struct *mm, unsigned long addr) +pte_alloc_one(struct mm_struct *mm) { struct page *pte; diff --git a/arch/hexagon/include/asm/pgalloc.h b/arch/hexagon/include/asm/pgalloc.h index eeebf862c46c..d36183887b60 100644 --- a/arch/hexagon/include/asm/pgalloc.h +++ b/arch/hexagon/include/asm/pgalloc.h @@ -59,8 +59,7 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) free_page((unsigned long) pgd); } -static inline struct page *pte_alloc_one(struct mm_struct *mm, - unsigned long address) +static inline struct page *pte_alloc_one(struct mm_struct *mm) { struct page *pte; @@ -75,8 +74,7 @@ static inline struct page *pte_alloc_one(struct mm_struct *mm, } /* _kernel variant gets to use a different allocator */ -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, - unsigned long address) +static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { gfp_t flags = GFP_KERNEL | __GFP_ZERO; return (pte_t *) __get_free_page(flags); diff --git a/arch/ia64/include/asm/pgalloc.h b/arch/ia64/include/asm/pgalloc.h index 3ee5362f2661..c9e481023c25 100644 --- a/arch/ia64/include/asm/pgalloc.h +++ b/arch/ia64/include/asm/pgalloc.h @@ -83,7 +83,7 @@ pmd_populate_kernel(struct mm_struct *mm, pmd_t * pmd_entry, pte_t * pte) pmd_val(*pmd_entry) = __pa(pte); } -static inline pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long addr) +static inline pgtable_t pte_alloc_one(struct mm_struct *mm) { struct page *page; void *pg; @@ -99,8 +99,7 @@ static inline pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long addr) return page; } -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, - unsigned long addr) +static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { return quicklist_alloc(0, GFP_KERNEL, NULL); } diff --git a/arch/m68k/include/asm/mcf_pgalloc.h b/arch/m68k/include/asm/mcf_pgalloc.h index 12fe700632f4..4399d712f6db 100644 --- a/arch/m68k/include/asm/mcf_pgalloc.h +++ b/arch/m68k/include/asm/mcf_pgalloc.h @@ -12,8 +12,7 @@ extern inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte) extern const char bad_pmd_string[]; -extern inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, - unsigned long address) +extern inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { unsigned long page = __get_free_page(GFP_DMA); @@ -32,8 +31,6 @@ extern inline pmd_t *pmd_alloc_kernel(pgd_t *pgd, unsigned long address) #define pmd_alloc_one_fast(mm, address) ({ BUG(); ((pmd_t *)1); }) #define pmd_alloc_one(mm, address) ({ BUG(); ((pmd_t *)2); }) -#define pte_alloc_one_fast(mm, addr) pte_alloc_one(mm, addr) - #define pmd_populate(mm, pmd, page) (pmd_val(*pmd) = \ (unsigned long)(page_address(page))) @@ -50,8 +47,7 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t page, #define __pmd_free_tlb(tlb, pmd, address) do { } while (0) -static inline struct page *pte_alloc_one(struct mm_struct *mm, - unsigned long address) +static inline struct page *pte_alloc_one(struct mm_struct *mm) { struct page *page = alloc_pages(GFP_DMA, 0); pte_t *pte; diff --git a/arch/m68k/include/asm/motorola_pgalloc.h b/arch/m68k/include/asm/motorola_pgalloc.h index 7859a86319cf..d04d9ba9b976 100644 --- a/arch/m68k/include/asm/motorola_pgalloc.h +++ b/arch/m68k/include/asm/motorola_pgalloc.h @@ -8,7 +8,7 @@ extern pmd_t *get_pointer_table(void); extern int free_pointer_table(pmd_t *); -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long address) +static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { pte_t *pte; @@ -28,7 +28,7 @@ static inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte) free_page((unsigned long) pte); } -static inline pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long address) +static inline pgtable_t pte_alloc_one(struct mm_struct *mm) { struct page *page; pte_t *pte; diff --git a/arch/m68k/include/asm/sun3_pgalloc.h b/arch/m68k/include/asm/sun3_pgalloc.h index 11485d38de4e..1456c5eecbd9 100644 --- a/arch/m68k/include/asm/sun3_pgalloc.h +++ b/arch/m68k/include/asm/sun3_pgalloc.h @@ -35,8 +35,7 @@ do { \ tlb_remove_page((tlb), pte); \ } while (0) -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, - unsigned long address) +static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { unsigned long page = __get_free_page(GFP_KERNEL); @@ -47,8 +46,7 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, return (pte_t *) (page); } -static inline pgtable_t pte_alloc_one(struct mm_struct *mm, - unsigned long address) +static inline pgtable_t pte_alloc_one(struct mm_struct *mm) { struct page *page = alloc_pages(GFP_KERNEL, 0); diff --git a/arch/microblaze/include/asm/pgalloc.h b/arch/microblaze/include/asm/pgalloc.h index 7c89390c0c13..f4cc9ffc449e 100644 --- a/arch/microblaze/include/asm/pgalloc.h +++ b/arch/microblaze/include/asm/pgalloc.h @@ -108,10 +108,9 @@ static inline void free_pgd_slow(pgd_t *pgd) #define pmd_alloc_one_fast(mm, address) ({ BUG(); ((pmd_t *)1); }) #define pmd_alloc_one(mm, address) ({ BUG(); ((pmd_t *)2); }) -extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long addr); +extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm); -static inline struct page *pte_alloc_one(struct mm_struct *mm, - unsigned long address) +static inline struct page *pte_alloc_one(struct mm_struct *mm) { struct page *ptepage; @@ -132,20 +131,6 @@ static inline struct page *pte_alloc_one(struct mm_struct *mm, return ptepage; } -static inline pte_t *pte_alloc_one_fast(struct mm_struct *mm, - unsigned long address) -{ - unsigned long *ret; - - ret = pte_quicklist; - if (ret != NULL) { - pte_quicklist = (unsigned long *)(*ret); - ret[0] = 0; - pgtable_cache_size--; - } - return (pte_t *)ret; -} - static inline void pte_free_fast(pte_t *pte) { *(unsigned long **)pte = pte_quicklist; diff --git a/arch/microblaze/mm/pgtable.c b/arch/microblaze/mm/pgtable.c index 7f525962cdfa..c2ce1e42b888 100644 --- a/arch/microblaze/mm/pgtable.c +++ b/arch/microblaze/mm/pgtable.c @@ -235,8 +235,7 @@ unsigned long iopa(unsigned long addr) return pa; } -__ref pte_t *pte_alloc_one_kernel(struct mm_struct *mm, - unsigned long address) +__ref pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { pte_t *pte; if (mem_init_done) { diff --git a/arch/mips/include/asm/pgalloc.h b/arch/mips/include/asm/pgalloc.h index 39b9f311c4ef..27808d9461f4 100644 --- a/arch/mips/include/asm/pgalloc.h +++ b/arch/mips/include/asm/pgalloc.h @@ -50,14 +50,12 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) free_pages((unsigned long)pgd, PGD_ORDER); } -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, - unsigned long address) +static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { return (pte_t *)__get_free_pages(GFP_KERNEL | __GFP_ZERO, PTE_ORDER); } -static inline struct page *pte_alloc_one(struct mm_struct *mm, - unsigned long address) +static inline struct page *pte_alloc_one(struct mm_struct *mm) { struct page *pte; diff --git a/arch/nds32/include/asm/pgalloc.h b/arch/nds32/include/asm/pgalloc.h index 27448869131a..3c5fee5b5759 100644 --- a/arch/nds32/include/asm/pgalloc.h +++ b/arch/nds32/include/asm/pgalloc.h @@ -22,8 +22,7 @@ extern void pgd_free(struct mm_struct *mm, pgd_t * pgd); #define check_pgt_cache() do { } while (0) -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, - unsigned long addr) +static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { pte_t *pte; @@ -34,7 +33,7 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, return pte; } -static inline pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long addr) +static inline pgtable_t pte_alloc_one(struct mm_struct *mm) { pgtable_t pte; diff --git a/arch/nios2/include/asm/pgalloc.h b/arch/nios2/include/asm/pgalloc.h index bb47d08c8ef7..3a149ead1207 100644 --- a/arch/nios2/include/asm/pgalloc.h +++ b/arch/nios2/include/asm/pgalloc.h @@ -37,8 +37,7 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) free_pages((unsigned long)pgd, PGD_ORDER); } -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, - unsigned long address) +static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { pte_t *pte; @@ -47,8 +46,7 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, return pte; } -static inline pgtable_t pte_alloc_one(struct mm_struct *mm, - unsigned long address) +static inline pgtable_t pte_alloc_one(struct mm_struct *mm) { struct page *pte; diff --git a/arch/openrisc/include/asm/pgalloc.h b/arch/openrisc/include/asm/pgalloc.h index 8999b9226512..149c82ee4b8b 100644 --- a/arch/openrisc/include/asm/pgalloc.h +++ b/arch/openrisc/include/asm/pgalloc.h @@ -70,10 +70,9 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) free_page((unsigned long)pgd); } -extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long address); +extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm); -static inline struct page *pte_alloc_one(struct mm_struct *mm, - unsigned long address) +static inline struct page *pte_alloc_one(struct mm_struct *mm) { struct page *pte; pte = alloc_pages(GFP_KERNEL, 0); diff --git a/arch/openrisc/mm/ioremap.c b/arch/openrisc/mm/ioremap.c index 2175e4bfd9fc..24fb1021c75a 100644 --- a/arch/openrisc/mm/ioremap.c +++ b/arch/openrisc/mm/ioremap.c @@ -118,8 +118,7 @@ EXPORT_SYMBOL(iounmap); * the memblock infrastructure. */ -pte_t __ref *pte_alloc_one_kernel(struct mm_struct *mm, - unsigned long address) +pte_t __ref *pte_alloc_one_kernel(struct mm_struct *mm) { pte_t *pte; diff --git a/arch/parisc/include/asm/pgalloc.h b/arch/parisc/include/asm/pgalloc.h index cf13275f7c6d..d05c678c77c4 100644 --- a/arch/parisc/include/asm/pgalloc.h +++ b/arch/parisc/include/asm/pgalloc.h @@ -122,7 +122,7 @@ pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmd, pte_t *pte) #define pmd_pgtable(pmd) pmd_page(pmd) static inline pgtable_t -pte_alloc_one(struct mm_struct *mm, unsigned long address) +pte_alloc_one(struct mm_struct *mm) { struct page *page = alloc_page(GFP_KERNEL|__GFP_ZERO); if (!page) @@ -135,7 +135,7 @@ pte_alloc_one(struct mm_struct *mm, unsigned long address) } static inline pte_t * -pte_alloc_one_kernel(struct mm_struct *mm, unsigned long addr) +pte_alloc_one_kernel(struct mm_struct *mm) { pte_t *pte = (pte_t *)__get_free_page(GFP_KERNEL|__GFP_ZERO); return pte; diff --git a/arch/powerpc/include/asm/book3s/32/pgalloc.h b/arch/powerpc/include/asm/book3s/32/pgalloc.h index 82e44b1a00ae..af9e13555d95 100644 --- a/arch/powerpc/include/asm/book3s/32/pgalloc.h +++ b/arch/powerpc/include/asm/book3s/32/pgalloc.h @@ -82,8 +82,8 @@ static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmdp, #define pmd_pgtable(pmd) pmd_page(pmd) #endif -extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long addr); -extern pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long addr); +extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm); +extern pgtable_t pte_alloc_one(struct mm_struct *mm); static inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte) { diff --git a/arch/powerpc/include/asm/book3s/64/pgalloc.h b/arch/powerpc/include/asm/book3s/64/pgalloc.h index 391ed2c3b697..8f1d92e99fe5 100644 --- a/arch/powerpc/include/asm/book3s/64/pgalloc.h +++ b/arch/powerpc/include/asm/book3s/64/pgalloc.h @@ -42,7 +42,7 @@ extern struct kmem_cache *pgtable_cache[]; pgtable_cache[(shift) - 1]; \ }) -extern pte_t *pte_fragment_alloc(struct mm_struct *, unsigned long, int); +extern pte_t *pte_fragment_alloc(struct mm_struct *, int); extern pmd_t *pmd_fragment_alloc(struct mm_struct *, unsigned long); extern void pte_fragment_free(unsigned long *, int); extern void pmd_fragment_free(unsigned long *); @@ -192,16 +192,14 @@ static inline pgtable_t pmd_pgtable(pmd_t pmd) return (pgtable_t)pmd_page_vaddr(pmd); } -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, - unsigned long address) +static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { - return (pte_t *)pte_fragment_alloc(mm, address, 1); + return (pte_t *)pte_fragment_alloc(mm, 1); } -static inline pgtable_t pte_alloc_one(struct mm_struct *mm, - unsigned long address) +static inline pgtable_t pte_alloc_one(struct mm_struct *mm) { - return (pgtable_t)pte_fragment_alloc(mm, address, 0); + return (pgtable_t)pte_fragment_alloc(mm, 0); } static inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte) diff --git a/arch/powerpc/include/asm/nohash/32/pgalloc.h b/arch/powerpc/include/asm/nohash/32/pgalloc.h index 8825953c225b..16623f53f0d4 100644 --- a/arch/powerpc/include/asm/nohash/32/pgalloc.h +++ b/arch/powerpc/include/asm/nohash/32/pgalloc.h @@ -83,8 +83,8 @@ static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmdp, #define pmd_pgtable(pmd) pmd_page(pmd) #endif -extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long addr); -extern pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long addr); +extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm); +extern pgtable_t pte_alloc_one(struct mm_struct *mm); static inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte) { diff --git a/arch/powerpc/include/asm/nohash/64/pgalloc.h b/arch/powerpc/include/asm/nohash/64/pgalloc.h index e2d62d033708..2e7e0230edf4 100644 --- a/arch/powerpc/include/asm/nohash/64/pgalloc.h +++ b/arch/powerpc/include/asm/nohash/64/pgalloc.h @@ -96,14 +96,12 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) } -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, - unsigned long address) +static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { return (pte_t *)__get_free_page(GFP_KERNEL | __GFP_ZERO); } -static inline pgtable_t pte_alloc_one(struct mm_struct *mm, - unsigned long address) +static inline pgtable_t pte_alloc_one(struct mm_struct *mm) { struct page *page; pte_t *pte; diff --git a/arch/powerpc/mm/pgtable-book3s64.c b/arch/powerpc/mm/pgtable-book3s64.c index 01d7c0f7c4f0..cff1d426ca6a 100644 --- a/arch/powerpc/mm/pgtable-book3s64.c +++ b/arch/powerpc/mm/pgtable-book3s64.c @@ -379,7 +379,7 @@ static pte_t *__alloc_for_ptecache(struct mm_struct *mm, int kernel) return (pte_t *)ret; } -pte_t *pte_fragment_alloc(struct mm_struct *mm, unsigned long vmaddr, int kernel) +pte_t *pte_fragment_alloc(struct mm_struct *mm, int kernel) { pte_t *pte; diff --git a/arch/powerpc/mm/pgtable_32.c b/arch/powerpc/mm/pgtable_32.c index 120a49bfb9c6..b99a89cdcc5e 100644 --- a/arch/powerpc/mm/pgtable_32.c +++ b/arch/powerpc/mm/pgtable_32.c @@ -43,7 +43,7 @@ EXPORT_SYMBOL(ioremap_bot); /* aka VMALLOC_END */ extern char etext[], _stext[], _sinittext[], _einittext[]; -__ref pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long address) +__ref pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { pte_t *pte; @@ -57,7 +57,7 @@ __ref pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long address) return pte; } -pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long address) +pgtable_t pte_alloc_one(struct mm_struct *mm) { struct page *ptepage; diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index a79ed5faff3a..94043cf83c90 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -82,15 +82,13 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) #endif /* __PAGETABLE_PMD_FOLDED */ -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, - unsigned long address) +static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { return (pte_t *)__get_free_page( GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_ZERO); } -static inline struct page *pte_alloc_one(struct mm_struct *mm, - unsigned long address) +static inline struct page *pte_alloc_one(struct mm_struct *mm) { struct page *pte; diff --git a/arch/s390/include/asm/pgalloc.h b/arch/s390/include/asm/pgalloc.h index f0f9bcf94c03..ce2ca8cbd2ec 100644 --- a/arch/s390/include/asm/pgalloc.h +++ b/arch/s390/include/asm/pgalloc.h @@ -139,8 +139,8 @@ static inline void pmd_populate(struct mm_struct *mm, /* * page table entry allocation/free routines. */ -#define pte_alloc_one_kernel(mm, vmaddr) ((pte_t *) page_table_alloc(mm)) -#define pte_alloc_one(mm, vmaddr) ((pte_t *) page_table_alloc(mm)) +#define pte_alloc_one_kernel(mm) ((pte_t *)page_table_alloc(mm)) +#define pte_alloc_one(mm) ((pte_t *)page_table_alloc(mm)) #define pte_free_kernel(mm, pte) page_table_free(mm, (unsigned long *) pte) #define pte_free(mm, pte) page_table_free(mm, (unsigned long *) pte) diff --git a/arch/sh/include/asm/pgalloc.h b/arch/sh/include/asm/pgalloc.h index ed053a359ab7..8ad73cb31121 100644 --- a/arch/sh/include/asm/pgalloc.h +++ b/arch/sh/include/asm/pgalloc.h @@ -32,14 +32,12 @@ static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd, /* * Allocate and free page tables. */ -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, - unsigned long address) +static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { return quicklist_alloc(QUICK_PT, GFP_KERNEL, NULL); } -static inline pgtable_t pte_alloc_one(struct mm_struct *mm, - unsigned long address) +static inline pgtable_t pte_alloc_one(struct mm_struct *mm) { struct page *page; void *pg; diff --git a/arch/sparc/include/asm/pgalloc_32.h b/arch/sparc/include/asm/pgalloc_32.h index 90459481c6c7..282be50a4adf 100644 --- a/arch/sparc/include/asm/pgalloc_32.h +++ b/arch/sparc/include/asm/pgalloc_32.h @@ -58,10 +58,9 @@ void pmd_populate(struct mm_struct *mm, pmd_t *pmdp, struct page *ptep); void pmd_set(pmd_t *pmdp, pte_t *ptep); #define pmd_populate_kernel(MM, PMD, PTE) pmd_set(PMD, PTE) -pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long address); +pgtable_t pte_alloc_one(struct mm_struct *mm); -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, - unsigned long address) +static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { return srmmu_get_nocache(PTE_SIZE, PTE_SIZE); } diff --git a/arch/sparc/include/asm/pgalloc_64.h b/arch/sparc/include/asm/pgalloc_64.h index 874632f34f62..48abccba4991 100644 --- a/arch/sparc/include/asm/pgalloc_64.h +++ b/arch/sparc/include/asm/pgalloc_64.h @@ -60,10 +60,8 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) kmem_cache_free(pgtable_cache, pmd); } -pte_t *pte_alloc_one_kernel(struct mm_struct *mm, - unsigned long address); -pgtable_t pte_alloc_one(struct mm_struct *mm, - unsigned long address); +pte_t *pte_alloc_one_kernel(struct mm_struct *mm); +pgtable_t pte_alloc_one(struct mm_struct *mm); void pte_free_kernel(struct mm_struct *mm, pte_t *pte); void pte_free(struct mm_struct *mm, pgtable_t ptepage); diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c index f396048a0d68..6133f21811e9 100644 --- a/arch/sparc/mm/init_64.c +++ b/arch/sparc/mm/init_64.c @@ -2921,8 +2921,7 @@ void __flush_tlb_all(void) : : "r" (pstate)); } -pte_t *pte_alloc_one_kernel(struct mm_struct *mm, - unsigned long address) +pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { struct page *page = alloc_page(GFP_KERNEL | __GFP_ZERO); pte_t *pte = NULL; @@ -2933,8 +2932,7 @@ pte_t *pte_alloc_one_kernel(struct mm_struct *mm, return pte; } -pgtable_t pte_alloc_one(struct mm_struct *mm, - unsigned long address) +pgtable_t pte_alloc_one(struct mm_struct *mm) { struct page *page = alloc_page(GFP_KERNEL | __GFP_ZERO); if (!page) diff --git a/arch/sparc/mm/srmmu.c b/arch/sparc/mm/srmmu.c index be9cb0065179..ce67a96e70c3 100644 --- a/arch/sparc/mm/srmmu.c +++ b/arch/sparc/mm/srmmu.c @@ -364,12 +364,12 @@ pgd_t *get_pgd_fast(void) * Alignments up to the page size are the same for physical and virtual * addresses of the nocache area. */ -pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long address) +pgtable_t pte_alloc_one(struct mm_struct *mm) { unsigned long pte; struct page *page; - if ((pte = (unsigned long)pte_alloc_one_kernel(mm, address)) == 0) + if ((pte = (unsigned long)pte_alloc_one_kernel(mm)) == 0) return NULL; page = pfn_to_page(__nocache_pa(pte) >> PAGE_SHIFT); if (!pgtable_page_ctor(page)) { diff --git a/arch/um/include/asm/pgalloc.h b/arch/um/include/asm/pgalloc.h index bf90b2aa2002..99eb5682792a 100644 --- a/arch/um/include/asm/pgalloc.h +++ b/arch/um/include/asm/pgalloc.h @@ -25,8 +25,8 @@ extern pgd_t *pgd_alloc(struct mm_struct *); extern void pgd_free(struct mm_struct *mm, pgd_t *pgd); -extern pte_t *pte_alloc_one_kernel(struct mm_struct *, unsigned long); -extern pgtable_t pte_alloc_one(struct mm_struct *, unsigned long); +extern pte_t *pte_alloc_one_kernel(struct mm_struct *); +extern pgtable_t pte_alloc_one(struct mm_struct *); static inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte) { diff --git a/arch/um/kernel/mem.c b/arch/um/kernel/mem.c index 3c0e470ea646..1f277191fbf3 100644 --- a/arch/um/kernel/mem.c +++ b/arch/um/kernel/mem.c @@ -197,7 +197,7 @@ void pgd_free(struct mm_struct *mm, pgd_t *pgd) free_page((unsigned long) pgd); } -pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long address) +pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { pte_t *pte; @@ -205,7 +205,7 @@ pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long address) return pte; } -pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long address) +pgtable_t pte_alloc_one(struct mm_struct *mm) { struct page *pte; diff --git a/arch/unicore32/include/asm/pgalloc.h b/arch/unicore32/include/asm/pgalloc.h index f0fdb268f8f2..7cceabecf4e3 100644 --- a/arch/unicore32/include/asm/pgalloc.h +++ b/arch/unicore32/include/asm/pgalloc.h @@ -34,7 +34,7 @@ extern void free_pgd_slow(struct mm_struct *mm, pgd_t *pgd); * Allocate one PTE table. */ static inline pte_t * -pte_alloc_one_kernel(struct mm_struct *mm, unsigned long addr) +pte_alloc_one_kernel(struct mm_struct *mm) { pte_t *pte; @@ -46,7 +46,7 @@ pte_alloc_one_kernel(struct mm_struct *mm, unsigned long addr) } static inline pgtable_t -pte_alloc_one(struct mm_struct *mm, unsigned long addr) +pte_alloc_one(struct mm_struct *mm) { struct page *pte; diff --git a/arch/x86/include/asm/pgalloc.h b/arch/x86/include/asm/pgalloc.h index fbd578daa66e..5068e85165b2 100644 --- a/arch/x86/include/asm/pgalloc.h +++ b/arch/x86/include/asm/pgalloc.h @@ -47,8 +47,8 @@ extern gfp_t __userpte_alloc_gfp; extern pgd_t *pgd_alloc(struct mm_struct *); extern void pgd_free(struct mm_struct *mm, pgd_t *pgd); -extern pte_t *pte_alloc_one_kernel(struct mm_struct *, unsigned long); -extern pgtable_t pte_alloc_one(struct mm_struct *, unsigned long); +extern pte_t *pte_alloc_one_kernel(struct mm_struct *); +extern pgtable_t pte_alloc_one(struct mm_struct *); /* Should really implement gc for free page table pages. This could be done with a reference count in struct page. */ diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 089e78c4effd..a2eff247377b 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -23,12 +23,12 @@ EXPORT_SYMBOL(physical_mask); gfp_t __userpte_alloc_gfp = PGALLOC_GFP | PGALLOC_USER_GFP; -pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long address) +pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { return (pte_t *)__get_free_page(PGALLOC_GFP & ~__GFP_ACCOUNT); } -pgtable_t pte_alloc_one(struct mm_struct *mm, unsigned long address) +pgtable_t pte_alloc_one(struct mm_struct *mm) { struct page *pte; diff --git a/arch/xtensa/include/asm/pgalloc.h b/arch/xtensa/include/asm/pgalloc.h index 1065bc8bcae5..b3b388ff2f01 100644 --- a/arch/xtensa/include/asm/pgalloc.h +++ b/arch/xtensa/include/asm/pgalloc.h @@ -38,8 +38,7 @@ static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) free_page((unsigned long)pgd); } -static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, - unsigned long address) +static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { pte_t *ptep; int i; @@ -52,13 +51,12 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, return ptep; } -static inline pgtable_t pte_alloc_one(struct mm_struct *mm, - unsigned long addr) +static inline pgtable_t pte_alloc_one(struct mm_struct *mm) { pte_t *pte; struct page *page; - pte = pte_alloc_one_kernel(mm, addr); + pte = pte_alloc_one_kernel(mm); if (!pte) return NULL; page = virt_to_page(pte); diff --git a/include/linux/mm.h b/include/linux/mm.h index 0416a7204be3..43ce50edc499 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1789,8 +1789,8 @@ static inline void mm_inc_nr_ptes(struct mm_struct *mm) {} static inline void mm_dec_nr_ptes(struct mm_struct *mm) {} #endif -int __pte_alloc(struct mm_struct *mm, pmd_t *pmd, unsigned long address); -int __pte_alloc_kernel(pmd_t *pmd, unsigned long address); +int __pte_alloc(struct mm_struct *mm, pmd_t *pmd); +int __pte_alloc_kernel(pmd_t *pmd); /* * The following ifdef needed to get the 4level-fixup.h header to work. @@ -1928,18 +1928,17 @@ static inline void pgtable_page_dtor(struct page *page) pte_unmap(pte); \ } while (0) -#define pte_alloc(mm, pmd, address) \ - (unlikely(pmd_none(*(pmd))) && __pte_alloc(mm, pmd, address)) +#define pte_alloc(mm, pmd) (unlikely(pmd_none(*(pmd))) && __pte_alloc(mm, pmd)) #define pte_alloc_map(mm, pmd, address) \ - (pte_alloc(mm, pmd, address) ? NULL : pte_offset_map(pmd, address)) + (pte_alloc(mm, pmd) ? NULL : pte_offset_map(pmd, address)) #define pte_alloc_map_lock(mm, pmd, address, ptlp) \ - (pte_alloc(mm, pmd, address) ? \ + (pte_alloc(mm, pmd) ? \ NULL : pte_offset_map_lock(mm, pmd, address, ptlp)) #define pte_alloc_kernel(pmd, address) \ - ((unlikely(pmd_none(*(pmd))) && __pte_alloc_kernel(pmd, address))? \ + ((unlikely(pmd_none(*(pmd))) && __pte_alloc_kernel(pmd))? \ NULL: pte_offset_kernel(pmd, address)) #if USE_SPLIT_PMD_PTLOCKS diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 00704060b7f7..fd7e8714e5a1 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -558,7 +558,7 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf, return VM_FAULT_FALLBACK; } - pgtable = pte_alloc_one(vma->vm_mm, haddr); + pgtable = pte_alloc_one(vma->vm_mm); if (unlikely(!pgtable)) { ret = VM_FAULT_OOM; goto release; @@ -683,7 +683,7 @@ vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf) struct page *zero_page; bool set; vm_fault_t ret; - pgtable = pte_alloc_one(vma->vm_mm, haddr); + pgtable = pte_alloc_one(vma->vm_mm); if (unlikely(!pgtable)) return VM_FAULT_OOM; zero_page = mm_get_huge_zero_page(vma->vm_mm); @@ -772,7 +772,7 @@ vm_fault_t vmf_insert_pfn_pmd(struct vm_area_struct *vma, unsigned long addr, return VM_FAULT_SIGBUS; if (arch_needs_pgtable_deposit()) { - pgtable = pte_alloc_one(vma->vm_mm, addr); + pgtable = pte_alloc_one(vma->vm_mm); if (!pgtable) return VM_FAULT_OOM; } @@ -910,7 +910,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, if (!vma_is_anonymous(vma)) return 0; - pgtable = pte_alloc_one(dst_mm, addr); + pgtable = pte_alloc_one(dst_mm); if (unlikely(!pgtable)) goto out; diff --git a/mm/kasan/kasan_init.c b/mm/kasan/kasan_init.c index 7a2a2f13f86f..272849cd2007 100644 --- a/mm/kasan/kasan_init.c +++ b/mm/kasan/kasan_init.c @@ -121,7 +121,7 @@ static int __ref zero_pmd_populate(pud_t *pud, unsigned long addr, pte_t *p; if (slab_is_available()) - p = pte_alloc_one_kernel(&init_mm, addr); + p = pte_alloc_one_kernel(&init_mm); else p = early_alloc(PAGE_SIZE, NUMA_NO_NODE); if (!p) diff --git a/mm/memory.c b/mm/memory.c index c467102a5cbc..3afdcf38993d 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -647,10 +647,10 @@ void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *vma, } } -int __pte_alloc(struct mm_struct *mm, pmd_t *pmd, unsigned long address) +int __pte_alloc(struct mm_struct *mm, pmd_t *pmd) { spinlock_t *ptl; - pgtable_t new = pte_alloc_one(mm, address); + pgtable_t new = pte_alloc_one(mm); if (!new) return -ENOMEM; @@ -681,9 +681,9 @@ int __pte_alloc(struct mm_struct *mm, pmd_t *pmd, unsigned long address) return 0; } -int __pte_alloc_kernel(pmd_t *pmd, unsigned long address) +int __pte_alloc_kernel(pmd_t *pmd) { - pte_t *new = pte_alloc_one_kernel(&init_mm, address); + pte_t *new = pte_alloc_one_kernel(&init_mm); if (!new) return -ENOMEM; @@ -3139,7 +3139,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) * * Here we only have down_read(mmap_sem). */ - if (pte_alloc(vma->vm_mm, vmf->pmd, vmf->address)) + if (pte_alloc(vma->vm_mm, vmf->pmd)) return VM_FAULT_OOM; /* See the comment in pte_alloc_one_map() */ @@ -3286,7 +3286,7 @@ static vm_fault_t pte_alloc_one_map(struct vm_fault *vmf) pmd_populate(vma->vm_mm, vmf->pmd, vmf->prealloc_pte); spin_unlock(vmf->ptl); vmf->prealloc_pte = NULL; - } else if (unlikely(pte_alloc(vma->vm_mm, vmf->pmd, vmf->address))) { + } else if (unlikely(pte_alloc(vma->vm_mm, vmf->pmd))) { return VM_FAULT_OOM; } map_pte: @@ -3365,7 +3365,7 @@ static vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page) * related to pte entry. Use the preallocated table for that. */ if (arch_needs_pgtable_deposit() && !vmf->prealloc_pte) { - vmf->prealloc_pte = pte_alloc_one(vma->vm_mm, vmf->address); + vmf->prealloc_pte = pte_alloc_one(vma->vm_mm); if (!vmf->prealloc_pte) return VM_FAULT_OOM; smp_wmb(); /* See comment in __pte_alloc() */ @@ -3603,8 +3603,7 @@ static vm_fault_t do_fault_around(struct vm_fault *vmf) start_pgoff + nr_pages - 1); if (pmd_none(*vmf->pmd)) { - vmf->prealloc_pte = pte_alloc_one(vmf->vma->vm_mm, - vmf->address); + vmf->prealloc_pte = pte_alloc_one(vmf->vma->vm_mm); if (!vmf->prealloc_pte) goto out; smp_wmb(); /* See comment in __pte_alloc() */ diff --git a/mm/migrate.c b/mm/migrate.c index 84381b55b2bd..3080b0626026 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -2605,7 +2605,7 @@ static void migrate_vma_insert_page(struct migrate_vma *migrate, * * Here we only have down_read(mmap_sem). */ - if (pte_alloc(mm, pmdp, addr)) + if (pte_alloc(mm, pmdp)) goto abort; /* See the comment in pte_alloc_one_map() */ diff --git a/mm/mremap.c b/mm/mremap.c index 5c2e18505f75..9e68a02a52b1 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -240,7 +240,7 @@ unsigned long move_page_tables(struct vm_area_struct *vma, if (pmd_trans_unstable(old_pmd)) continue; } - if (pte_alloc(new_vma->vm_mm, new_pmd, new_addr)) + if (pte_alloc(new_vma->vm_mm, new_pmd)) break; next = (new_addr + PMD_SIZE) & PMD_MASK; if (extent > next - new_addr) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 5029f241908f..f05c8bc38ca5 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -513,7 +513,7 @@ static __always_inline ssize_t __mcopy_atomic(struct mm_struct *dst_mm, break; } if (unlikely(pmd_none(dst_pmdval)) && - unlikely(__pte_alloc(dst_mm, dst_pmd, dst_addr))) { + unlikely(__pte_alloc(dst_mm, dst_pmd))) { err = -ENOMEM; break; } diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c index ed162a6c57c5..3f8180414301 100644 --- a/virt/kvm/arm/mmu.c +++ b/virt/kvm/arm/mmu.c @@ -628,7 +628,7 @@ static int create_hyp_pmd_mappings(pud_t *pud, unsigned long start, BUG_ON(pmd_sect(*pmd)); if (pmd_none(*pmd)) { - pte = pte_alloc_one_kernel(NULL, addr); + pte = pte_alloc_one_kernel(NULL); if (!pte) { kvm_err("Cannot allocate Hyp pte\n"); return -ENOMEM; -- 2.19.0.605.g01d371f741-goog _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH 1/4] treewide: remove unused address argument from pte_alloc functions (v2) 2018-10-13 1:31 ` [PATCH 1/4] treewide: remove unused address argument from pte_alloc functions (v2) Joel Fernandes (Google) 2018-10-13 1:31 ` Joel Fernandes (Google) @ 2018-10-24 8:37 ` Peter Zijlstra 2018-10-24 8:37 ` Peter Zijlstra ` (2 more replies) 1 sibling, 3 replies; 52+ messages in thread From: Peter Zijlstra @ 2018-10-24 8:37 UTC (permalink / raw) To: linux-riscv On Fri, Oct 12, 2018 at 06:31:57PM -0700, Joel Fernandes (Google) wrote: > This series speeds up mremap(2) syscall by copying page tables at the > PMD level even for non-THP systems. There is concern that the extra > 'address' argument that mremap passes to pte_alloc may do something > subtle architecture related in the future that may make the scheme not > work. Also we find that there is no point in passing the 'address' to > pte_alloc since its unused. So this patch therefore removes this > argument tree-wide resulting in a nice negative diff as well. Also > ensuring along the way that the enabled architectures do not do anything > funky with 'address' argument that goes unnoticed by the optimization. Did you happen to look at the history of where that address argument came from? -- just being curious here. ISTR something vague about architectures having different paging structure for different memory ranges. ^ permalink raw reply [flat|nested] 52+ messages in thread
* Re: [PATCH 1/4] treewide: remove unused address argument from pte_alloc functions (v2) 2018-10-24 8:37 ` Peter Zijlstra @ 2018-10-24 8:37 ` Peter Zijlstra 2018-10-25 2:21 ` Joel Fernandes 2018-10-25 10:47 ` Kirill A. Shutemov 2 siblings, 0 replies; 52+ messages in thread From: Peter Zijlstra @ 2018-10-24 8:37 UTC (permalink / raw) To: Joel Fernandes (Google) Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Catalin Marinas, Dave Hansen, Will Deacon, Michal Hocko, linux-mm, lokeshgidra, sparclinux, linux-riscv, elfring, Jonas Bonn, kvmarm, dancol, Yoshinori Sato, linux-xtensa, linux-hexagon, Helge Deller, maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT), hughd, James E.J. Bottomley, kasan-dev, anton.ivanov, Ingo Molnar, Geert Uytterhoeven, Andrey Ryabinin, linux-snps-arc, kernel-team, Sam Creasey, Fenghua Yu, linux-s390, Jeff Dike, linux-um, Stefan Kristiansson, Julia Lawall, linux-m68k, Borislav Petkov, Andy Lutomirski, nios2-dev, Kirill A . Shutemov, Stafford Horne, Guan Xuetao, Chris Zankel, Tony Luck, Richard Weinberger, linux-parisc, Max Filippov, pantin, linux-kernel, minchan, Thomas Gleixner, linux-alpha, Ley Foon Tan, akpm, linuxppc-dev, David S. Miller On Fri, Oct 12, 2018 at 06:31:57PM -0700, Joel Fernandes (Google) wrote: > This series speeds up mremap(2) syscall by copying page tables at the > PMD level even for non-THP systems. There is concern that the extra > 'address' argument that mremap passes to pte_alloc may do something > subtle architecture related in the future that may make the scheme not > work. Also we find that there is no point in passing the 'address' to > pte_alloc since its unused. So this patch therefore removes this > argument tree-wide resulting in a nice negative diff as well. Also > ensuring along the way that the enabled architectures do not do anything > funky with 'address' argument that goes unnoticed by the optimization. Did you happen to look at the history of where that address argument came from? -- just being curious here. ISTR something vague about architectures having different paging structure for different memory ranges. _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 52+ messages in thread
* [PATCH 1/4] treewide: remove unused address argument from pte_alloc functions (v2) 2018-10-24 8:37 ` Peter Zijlstra 2018-10-24 8:37 ` Peter Zijlstra @ 2018-10-25 2:21 ` Joel Fernandes 2018-10-25 2:21 ` Joel Fernandes 2018-10-26 8:52 ` Peter Zijlstra 2018-10-25 10:47 ` Kirill A. Shutemov 2 siblings, 2 replies; 52+ messages in thread From: Joel Fernandes @ 2018-10-25 2:21 UTC (permalink / raw) To: linux-riscv On Wed, Oct 24, 2018 at 10:37:16AM +0200, Peter Zijlstra wrote: > On Fri, Oct 12, 2018 at 06:31:57PM -0700, Joel Fernandes (Google) wrote: > > This series speeds up mremap(2) syscall by copying page tables at the > > PMD level even for non-THP systems. There is concern that the extra > > 'address' argument that mremap passes to pte_alloc may do something > > subtle architecture related in the future that may make the scheme not > > work. Also we find that there is no point in passing the 'address' to > > pte_alloc since its unused. So this patch therefore removes this > > argument tree-wide resulting in a nice negative diff as well. Also > > ensuring along the way that the enabled architectures do not do anything > > funky with 'address' argument that goes unnoticed by the optimization. > > Did you happen to look at the history of where that address argument > came from? -- just being curious here. ISTR something vague about > architectures having different paging structure for different memory > ranges. I didn't happen to do that analysis but from code analysis, no architecutre is using it. Since its unused in the kernel, may be such architectures don't exist or were removed, so we don't need to bother? Could you share more about your concern with the removal of this argument? thanks, - Joel ^ permalink raw reply [flat|nested] 52+ messages in thread
* Re: [PATCH 1/4] treewide: remove unused address argument from pte_alloc functions (v2) 2018-10-25 2:21 ` Joel Fernandes @ 2018-10-25 2:21 ` Joel Fernandes 2018-10-26 8:52 ` Peter Zijlstra 1 sibling, 0 replies; 52+ messages in thread From: Joel Fernandes @ 2018-10-25 2:21 UTC (permalink / raw) To: Peter Zijlstra Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Catalin Marinas, Dave Hansen, Will Deacon, Michal Hocko, linux-mm, lokeshgidra, sparclinux, linux-riscv, elfring, Jonas Bonn, kvmarm, dancol, Yoshinori Sato, linux-xtensa, linux-hexagon, Helge Deller, maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT), hughd, James E.J. Bottomley, kasan-dev, anton.ivanov, Ingo Molnar, Geert Uytterhoeven, Andrey Ryabinin, linux-snps-arc, kernel-team, Sam Creasey, Fenghua Yu, linux-s390, Jeff Dike, linux-um, Stefan Kristiansson, Julia Lawall, linux-m68k, Borislav Petkov, Andy Lutomirski, nios2-dev, Kirill A . Shutemov, Stafford Horne, Guan Xuetao, Chris Zankel, Tony Luck, Richard Weinberger, linux-parisc, Max Filippov, pantin, linux-kernel, minchan, Thomas Gleixner, linux-alpha, Ley Foon Tan, akpm, linuxppc-dev, David S. Miller On Wed, Oct 24, 2018 at 10:37:16AM +0200, Peter Zijlstra wrote: > On Fri, Oct 12, 2018 at 06:31:57PM -0700, Joel Fernandes (Google) wrote: > > This series speeds up mremap(2) syscall by copying page tables at the > > PMD level even for non-THP systems. There is concern that the extra > > 'address' argument that mremap passes to pte_alloc may do something > > subtle architecture related in the future that may make the scheme not > > work. Also we find that there is no point in passing the 'address' to > > pte_alloc since its unused. So this patch therefore removes this > > argument tree-wide resulting in a nice negative diff as well. Also > > ensuring along the way that the enabled architectures do not do anything > > funky with 'address' argument that goes unnoticed by the optimization. > > Did you happen to look at the history of where that address argument > came from? -- just being curious here. ISTR something vague about > architectures having different paging structure for different memory > ranges. I didn't happen to do that analysis but from code analysis, no architecutre is using it. Since its unused in the kernel, may be such architectures don't exist or were removed, so we don't need to bother? Could you share more about your concern with the removal of this argument? thanks, - Joel _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 52+ messages in thread
* [PATCH 1/4] treewide: remove unused address argument from pte_alloc functions (v2) 2018-10-25 2:21 ` Joel Fernandes 2018-10-25 2:21 ` Joel Fernandes @ 2018-10-26 8:52 ` Peter Zijlstra 2018-10-26 8:52 ` Peter Zijlstra 1 sibling, 1 reply; 52+ messages in thread From: Peter Zijlstra @ 2018-10-26 8:52 UTC (permalink / raw) To: linux-riscv On Wed, Oct 24, 2018 at 07:21:19PM -0700, Joel Fernandes wrote: > On Wed, Oct 24, 2018 at 10:37:16AM +0200, Peter Zijlstra wrote: > > On Fri, Oct 12, 2018 at 06:31:57PM -0700, Joel Fernandes (Google) wrote: > > > This series speeds up mremap(2) syscall by copying page tables at the > > > PMD level even for non-THP systems. There is concern that the extra > > > 'address' argument that mremap passes to pte_alloc may do something > > > subtle architecture related in the future that may make the scheme not > > > work. Also we find that there is no point in passing the 'address' to > > > pte_alloc since its unused. So this patch therefore removes this > > > argument tree-wide resulting in a nice negative diff as well. Also > > > ensuring along the way that the enabled architectures do not do anything > > > funky with 'address' argument that goes unnoticed by the optimization. > > > > Did you happen to look at the history of where that address argument > > came from? -- just being curious here. ISTR something vague about > > architectures having different paging structure for different memory > > ranges. > > I didn't happen to do that analysis but from code analysis, no architecutre > is using it. Since its unused in the kernel, may be such architectures don't > exist or were removed, so we don't need to bother? Could you share more about > your concern with the removal of this argument? No concerns at all with removing it; I was purely curious as to the origin of the unused argument. Kirill provided that answer. ^ permalink raw reply [flat|nested] 52+ messages in thread
* Re: [PATCH 1/4] treewide: remove unused address argument from pte_alloc functions (v2) 2018-10-26 8:52 ` Peter Zijlstra @ 2018-10-26 8:52 ` Peter Zijlstra 0 siblings, 0 replies; 52+ messages in thread From: Peter Zijlstra @ 2018-10-26 8:52 UTC (permalink / raw) To: Joel Fernandes Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Catalin Marinas, Dave Hansen, Will Deacon, Michal Hocko, linux-mm, lokeshgidra, sparclinux, linux-riscv, elfring, Jonas Bonn, kvmarm, dancol, Yoshinori Sato, linux-xtensa, linux-hexagon, Helge Deller, maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT), hughd, James E.J. Bottomley, kasan-dev, anton.ivanov, Ingo Molnar, Geert Uytterhoeven, Andrey Ryabinin, linux-snps-arc, kernel-team, Sam Creasey, Fenghua Yu, linux-s390, Jeff Dike, linux-um, Stefan Kristiansson, Julia Lawall, linux-m68k, Borislav Petkov, Andy Lutomirski, nios2-dev, Kirill A . Shutemov, Stafford Horne, Guan Xuetao, Chris Zankel, Tony Luck, Richard Weinberger, linux-parisc, Max Filippov, pantin, linux-kernel, minchan, Thomas Gleixner, linux-alpha, Ley Foon Tan, akpm, linuxppc-dev, David S. Miller On Wed, Oct 24, 2018 at 07:21:19PM -0700, Joel Fernandes wrote: > On Wed, Oct 24, 2018 at 10:37:16AM +0200, Peter Zijlstra wrote: > > On Fri, Oct 12, 2018 at 06:31:57PM -0700, Joel Fernandes (Google) wrote: > > > This series speeds up mremap(2) syscall by copying page tables at the > > > PMD level even for non-THP systems. There is concern that the extra > > > 'address' argument that mremap passes to pte_alloc may do something > > > subtle architecture related in the future that may make the scheme not > > > work. Also we find that there is no point in passing the 'address' to > > > pte_alloc since its unused. So this patch therefore removes this > > > argument tree-wide resulting in a nice negative diff as well. Also > > > ensuring along the way that the enabled architectures do not do anything > > > funky with 'address' argument that goes unnoticed by the optimization. > > > > Did you happen to look at the history of where that address argument > > came from? -- just being curious here. ISTR something vague about > > architectures having different paging structure for different memory > > ranges. > > I didn't happen to do that analysis but from code analysis, no architecutre > is using it. Since its unused in the kernel, may be such architectures don't > exist or were removed, so we don't need to bother? Could you share more about > your concern with the removal of this argument? No concerns at all with removing it; I was purely curious as to the origin of the unused argument. Kirill provided that answer. _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 52+ messages in thread
* [PATCH 1/4] treewide: remove unused address argument from pte_alloc functions (v2) 2018-10-24 8:37 ` Peter Zijlstra 2018-10-24 8:37 ` Peter Zijlstra 2018-10-25 2:21 ` Joel Fernandes @ 2018-10-25 10:47 ` Kirill A. Shutemov 2018-10-25 10:47 ` Kirill A. Shutemov 2018-10-26 8:50 ` Peter Zijlstra 2 siblings, 2 replies; 52+ messages in thread From: Kirill A. Shutemov @ 2018-10-25 10:47 UTC (permalink / raw) To: linux-riscv On Wed, Oct 24, 2018 at 10:37:16AM +0200, Peter Zijlstra wrote: > On Fri, Oct 12, 2018 at 06:31:57PM -0700, Joel Fernandes (Google) wrote: > > This series speeds up mremap(2) syscall by copying page tables at the > > PMD level even for non-THP systems. There is concern that the extra > > 'address' argument that mremap passes to pte_alloc may do something > > subtle architecture related in the future that may make the scheme not > > work. Also we find that there is no point in passing the 'address' to > > pte_alloc since its unused. So this patch therefore removes this > > argument tree-wide resulting in a nice negative diff as well. Also > > ensuring along the way that the enabled architectures do not do anything > > funky with 'address' argument that goes unnoticed by the optimization. > > Did you happen to look at the history of where that address argument > came from? -- just being curious here. ISTR something vague about > architectures having different paging structure for different memory > ranges. I see some archicetures (i.e. sparc and, I believe power) used the address for coloring. It's not needed anymore. Page allocator and SL?B are good enough now. See 3c936465249f ("[SPARC64]: Kill pgtable quicklists and use SLAB.") -- Kirill A. Shutemov ^ permalink raw reply [flat|nested] 52+ messages in thread
* Re: [PATCH 1/4] treewide: remove unused address argument from pte_alloc functions (v2) 2018-10-25 10:47 ` Kirill A. Shutemov @ 2018-10-25 10:47 ` Kirill A. Shutemov 2018-10-26 8:50 ` Peter Zijlstra 1 sibling, 0 replies; 52+ messages in thread From: Kirill A. Shutemov @ 2018-10-25 10:47 UTC (permalink / raw) To: Peter Zijlstra Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Catalin Marinas, Dave Hansen, Will Deacon, Michal Hocko, linux-mm, lokeshgidra, Joel Fernandes (Google), linux-riscv, elfring, Jonas Bonn, kvmarm, dancol, Yoshinori Sato, sparclinux, linux-xtensa, linux-hexagon, Helge Deller, maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT), hughd, James E.J. Bottomley, kasan-dev, anton.ivanov, Ingo Molnar, Geert Uytterhoeven, Andrey Ryabinin, linux-snps-arc, kernel-team, Sam Creasey, Fenghua Yu, linux-s390, Jeff Dike, linux-um, Stefan Kristiansson, Julia Lawall, linux-m68k, Borislav Petkov, Andy Lutomirski, nios2-dev, Stafford Horne, Guan Xuetao, Chris Zankel, Tony Luck, Richard Weinberger, linux-parisc, Max Filippov, pantin, linux-kernel, minchan, Thomas Gleixner, linux-alpha, Ley Foon Tan, akpm, linuxppc-dev, David S. Miller On Wed, Oct 24, 2018 at 10:37:16AM +0200, Peter Zijlstra wrote: > On Fri, Oct 12, 2018 at 06:31:57PM -0700, Joel Fernandes (Google) wrote: > > This series speeds up mremap(2) syscall by copying page tables at the > > PMD level even for non-THP systems. There is concern that the extra > > 'address' argument that mremap passes to pte_alloc may do something > > subtle architecture related in the future that may make the scheme not > > work. Also we find that there is no point in passing the 'address' to > > pte_alloc since its unused. So this patch therefore removes this > > argument tree-wide resulting in a nice negative diff as well. Also > > ensuring along the way that the enabled architectures do not do anything > > funky with 'address' argument that goes unnoticed by the optimization. > > Did you happen to look at the history of where that address argument > came from? -- just being curious here. ISTR something vague about > architectures having different paging structure for different memory > ranges. I see some archicetures (i.e. sparc and, I believe power) used the address for coloring. It's not needed anymore. Page allocator and SL?B are good enough now. See 3c936465249f ("[SPARC64]: Kill pgtable quicklists and use SLAB.") -- Kirill A. Shutemov _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 52+ messages in thread
* [PATCH 1/4] treewide: remove unused address argument from pte_alloc functions (v2) 2018-10-25 10:47 ` Kirill A. Shutemov 2018-10-25 10:47 ` Kirill A. Shutemov @ 2018-10-26 8:50 ` Peter Zijlstra 2018-10-26 8:50 ` Peter Zijlstra 1 sibling, 1 reply; 52+ messages in thread From: Peter Zijlstra @ 2018-10-26 8:50 UTC (permalink / raw) To: linux-riscv On Thu, Oct 25, 2018 at 01:47:03PM +0300, Kirill A. Shutemov wrote: > On Wed, Oct 24, 2018 at 10:37:16AM +0200, Peter Zijlstra wrote: > > On Fri, Oct 12, 2018 at 06:31:57PM -0700, Joel Fernandes (Google) wrote: > > > This series speeds up mremap(2) syscall by copying page tables at the > > > PMD level even for non-THP systems. There is concern that the extra > > > 'address' argument that mremap passes to pte_alloc may do something > > > subtle architecture related in the future that may make the scheme not > > > work. Also we find that there is no point in passing the 'address' to > > > pte_alloc since its unused. So this patch therefore removes this > > > argument tree-wide resulting in a nice negative diff as well. Also > > > ensuring along the way that the enabled architectures do not do anything > > > funky with 'address' argument that goes unnoticed by the optimization. > > > > Did you happen to look at the history of where that address argument > > came from? -- just being curious here. ISTR something vague about > > architectures having different paging structure for different memory > > ranges. > > I see some archicetures (i.e. sparc and, I believe power) used the address > for coloring. It's not needed anymore. Page allocator and SL?B are good > enough now. > > See 3c936465249f ("[SPARC64]: Kill pgtable quicklists and use SLAB.") Ah, shiny. Thanks. ^ permalink raw reply [flat|nested] 52+ messages in thread
* Re: [PATCH 1/4] treewide: remove unused address argument from pte_alloc functions (v2) 2018-10-26 8:50 ` Peter Zijlstra @ 2018-10-26 8:50 ` Peter Zijlstra 0 siblings, 0 replies; 52+ messages in thread From: Peter Zijlstra @ 2018-10-26 8:50 UTC (permalink / raw) To: Kirill A. Shutemov Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Catalin Marinas, Dave Hansen, Will Deacon, Michal Hocko, linux-mm, lokeshgidra, Joel Fernandes (Google), linux-riscv, elfring, Jonas Bonn, kvmarm, dancol, Yoshinori Sato, sparclinux, linux-xtensa, linux-hexagon, Helge Deller, maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT), hughd, James E.J. Bottomley, kasan-dev, anton.ivanov, Ingo Molnar, Geert Uytterhoeven, Andrey Ryabinin, linux-snps-arc, kernel-team, Sam Creasey, Fenghua Yu, linux-s390, Jeff Dike, linux-um, Stefan Kristiansson, Julia Lawall, linux-m68k, Borislav Petkov, Andy Lutomirski, nios2-dev, Stafford Horne, Guan Xuetao, Chris Zankel, Tony Luck, Richard Weinberger, linux-parisc, Max Filippov, pantin, linux-kernel, minchan, Thomas Gleixner, linux-alpha, Ley Foon Tan, akpm, linuxppc-dev, David S. Miller On Thu, Oct 25, 2018 at 01:47:03PM +0300, Kirill A. Shutemov wrote: > On Wed, Oct 24, 2018 at 10:37:16AM +0200, Peter Zijlstra wrote: > > On Fri, Oct 12, 2018 at 06:31:57PM -0700, Joel Fernandes (Google) wrote: > > > This series speeds up mremap(2) syscall by copying page tables at the > > > PMD level even for non-THP systems. There is concern that the extra > > > 'address' argument that mremap passes to pte_alloc may do something > > > subtle architecture related in the future that may make the scheme not > > > work. Also we find that there is no point in passing the 'address' to > > > pte_alloc since its unused. So this patch therefore removes this > > > argument tree-wide resulting in a nice negative diff as well. Also > > > ensuring along the way that the enabled architectures do not do anything > > > funky with 'address' argument that goes unnoticed by the optimization. > > > > Did you happen to look at the history of where that address argument > > came from? -- just being curious here. ISTR something vague about > > architectures having different paging structure for different memory > > ranges. > > I see some archicetures (i.e. sparc and, I believe power) used the address > for coloring. It's not needed anymore. Page allocator and SL?B are good > enough now. > > See 3c936465249f ("[SPARC64]: Kill pgtable quicklists and use SLAB.") Ah, shiny. Thanks. _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 52+ messages in thread
* [PATCH 2/4] mm: speed up mremap by 500x on large regions (v2) 2018-10-13 1:31 [PATCH 0/4] Add support for fast mremap Joel Fernandes (Google) 2018-10-13 1:31 ` Joel Fernandes (Google) 2018-10-13 1:31 ` [PATCH 1/4] treewide: remove unused address argument from pte_alloc functions (v2) Joel Fernandes (Google) @ 2018-10-13 1:31 ` Joel Fernandes (Google) 2018-10-13 1:31 ` Joel Fernandes (Google) ` (2 more replies) 2018-10-13 1:31 ` [PATCH 3/4] arm64: select HAVE_MOVE_PMD for faster mremap (v1) Joel Fernandes (Google) 2018-10-13 1:32 ` [PATCH 4/4] x86: " Joel Fernandes (Google) 4 siblings, 3 replies; 52+ messages in thread From: Joel Fernandes (Google) @ 2018-10-13 1:31 UTC (permalink / raw) To: linux-riscv Android needs to mremap large regions of memory during memory management related operations. The mremap system call can be really slow if THP is not enabled. The bottleneck is move_page_tables, which is copying each pte at a time, and can be really slow across a large map. Turning on THP may not be a viable option, and is not for us. This patch speeds up the performance for non-THP system by copying at the PMD level when possible. The speed up is three orders of magnitude. On a 1GB mremap, the mremap completion times drops from 160-250 millesconds to 380-400 microseconds. Before: Total mremap time for 1GB data: 242321014 nanoseconds. Total mremap time for 1GB data: 196842467 nanoseconds. Total mremap time for 1GB data: 167051162 nanoseconds. After: Total mremap time for 1GB data: 385781 nanoseconds. Total mremap time for 1GB data: 388959 nanoseconds. Total mremap time for 1GB data: 402813 nanoseconds. Incase THP is enabled, the optimization is skipped. I also flush the tlb every time we do this optimization since I couldn't find a way to determine if the low-level PTEs are dirty. It is seen that the cost of doing so is not much compared the improvement, on both x86-64 and arm64. Cc: minchan at kernel.org Cc: pantin at google.com Cc: hughd at google.com Cc: lokeshgidra at google.com Cc: dancol at google.com Cc: mhocko at kernel.org Cc: kirill at shutemov.name Cc: akpm at linux-foundation.org Cc: kernel-team at android.com Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org> --- arch/Kconfig | 5 ++++ mm/mremap.c | 65 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 70 insertions(+) diff --git a/arch/Kconfig b/arch/Kconfig index 6801123932a5..9724fe39884f 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -518,6 +518,11 @@ config HAVE_IRQ_TIME_ACCOUNTING Archs need to ensure they use a high enough resolution clock to support irq time accounting and then call enable_sched_clock_irqtime(). +config HAVE_MOVE_PMD + bool + help + Archs that select this are able to move page tables at the PMD level. + config HAVE_ARCH_TRANSPARENT_HUGEPAGE bool diff --git a/mm/mremap.c b/mm/mremap.c index 9e68a02a52b1..2fd163cff406 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -191,6 +191,54 @@ static void move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, drop_rmap_locks(vma); } +static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr, + unsigned long new_addr, unsigned long old_end, + pmd_t *old_pmd, pmd_t *new_pmd, bool *need_flush) +{ + spinlock_t *old_ptl, *new_ptl; + struct mm_struct *mm = vma->vm_mm; + + if ((old_addr & ~PMD_MASK) || (new_addr & ~PMD_MASK) + || old_end - old_addr < PMD_SIZE) + return false; + + /* + * The destination pmd shouldn't be established, free_pgtables() + * should have release it. + */ + if (WARN_ON(!pmd_none(*new_pmd))) + return false; + + /* + * We don't have to worry about the ordering of src and dst + * ptlocks because exclusive mmap_sem prevents deadlock. + */ + old_ptl = pmd_lock(vma->vm_mm, old_pmd); + if (old_ptl) { + pmd_t pmd; + + new_ptl = pmd_lockptr(mm, new_pmd); + if (new_ptl != old_ptl) + spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING); + + /* Clear the pmd */ + pmd = *old_pmd; + pmd_clear(old_pmd); + + VM_BUG_ON(!pmd_none(*new_pmd)); + + /* Set the new pmd */ + set_pmd_at(mm, new_addr, new_pmd, pmd); + if (new_ptl != old_ptl) + spin_unlock(new_ptl); + spin_unlock(old_ptl); + + *need_flush = true; + return true; + } + return false; +} + unsigned long move_page_tables(struct vm_area_struct *vma, unsigned long old_addr, struct vm_area_struct *new_vma, unsigned long new_addr, unsigned long len, @@ -239,7 +287,24 @@ unsigned long move_page_tables(struct vm_area_struct *vma, split_huge_pmd(vma, old_pmd, old_addr); if (pmd_trans_unstable(old_pmd)) continue; + } else if (extent == PMD_SIZE && IS_ENABLED(CONFIG_HAVE_MOVE_PMD)) { + /* + * If the extent is PMD-sized, try to speed the move by + * moving at the PMD level if possible. + */ + bool moved; + + if (need_rmap_locks) + take_rmap_locks(vma); + moved = move_normal_pmd(vma, old_addr, new_addr, + old_end, old_pmd, new_pmd, + &need_flush); + if (need_rmap_locks) + drop_rmap_locks(vma); + if (moved) + continue; } + if (pte_alloc(new_vma->vm_mm, new_pmd)) break; next = (new_addr + PMD_SIZE) & PMD_MASK; -- 2.19.0.605.g01d371f741-goog ^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH 2/4] mm: speed up mremap by 500x on large regions (v2) 2018-10-13 1:31 ` [PATCH 2/4] mm: speed up mremap by 500x on large regions (v2) Joel Fernandes (Google) @ 2018-10-13 1:31 ` Joel Fernandes (Google) 2018-10-15 9:42 ` Christoph Hellwig 2018-10-24 10:12 ` Kirill A. Shutemov 2 siblings, 0 replies; 52+ messages in thread From: Joel Fernandes (Google) @ 2018-10-13 1:31 UTC (permalink / raw) To: linux-kernel Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Peter Zijlstra, Catalin Marinas, Dave Hansen, Will Deacon, mhocko, linux-mm, lokeshgidra, Joel Fernandes (Google), linux-riscv, elfring, Jonas Bonn, kvmarm, dancol, Yoshinori Sato, sparclinux, linux-xtensa, linux-hexagon, Helge Deller, maintainer:X86 ARCHITECTURE 32-BIT AND 64-BIT, hughd, James E.J. Bottomley, kasan-dev, anton.ivanov, Ingo Molnar, Geert Uytterhoeven, Andrey Ryabinin, linux-snps-arc, kernel-team, Sam Creasey, Fenghua Yu, linux-s390, Jeff Dike, linux-um, Stefan Kristiansson, Julia Lawall, linux-m68k, Borislav Petkov, Andy Lutomirski, nios2-dev, kirill, Stafford Horne, Guan Xuetao, Chris Zankel, Tony Luck, Richard Weinberger, linux-parisc, pantin, Max Filippov, minchan, Thomas Gleixner, linux-alpha, Ley Foon Tan, akpm, linuxppc-dev, David S. Miller Android needs to mremap large regions of memory during memory management related operations. The mremap system call can be really slow if THP is not enabled. The bottleneck is move_page_tables, which is copying each pte at a time, and can be really slow across a large map. Turning on THP may not be a viable option, and is not for us. This patch speeds up the performance for non-THP system by copying at the PMD level when possible. The speed up is three orders of magnitude. On a 1GB mremap, the mremap completion times drops from 160-250 millesconds to 380-400 microseconds. Before: Total mremap time for 1GB data: 242321014 nanoseconds. Total mremap time for 1GB data: 196842467 nanoseconds. Total mremap time for 1GB data: 167051162 nanoseconds. After: Total mremap time for 1GB data: 385781 nanoseconds. Total mremap time for 1GB data: 388959 nanoseconds. Total mremap time for 1GB data: 402813 nanoseconds. Incase THP is enabled, the optimization is skipped. I also flush the tlb every time we do this optimization since I couldn't find a way to determine if the low-level PTEs are dirty. It is seen that the cost of doing so is not much compared the improvement, on both x86-64 and arm64. Cc: minchan@kernel.org Cc: pantin@google.com Cc: hughd@google.com Cc: lokeshgidra@google.com Cc: dancol@google.com Cc: mhocko@kernel.org Cc: kirill@shutemov.name Cc: akpm@linux-foundation.org Cc: kernel-team@android.com Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org> --- arch/Kconfig | 5 ++++ mm/mremap.c | 65 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 70 insertions(+) diff --git a/arch/Kconfig b/arch/Kconfig index 6801123932a5..9724fe39884f 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -518,6 +518,11 @@ config HAVE_IRQ_TIME_ACCOUNTING Archs need to ensure they use a high enough resolution clock to support irq time accounting and then call enable_sched_clock_irqtime(). +config HAVE_MOVE_PMD + bool + help + Archs that select this are able to move page tables at the PMD level. + config HAVE_ARCH_TRANSPARENT_HUGEPAGE bool diff --git a/mm/mremap.c b/mm/mremap.c index 9e68a02a52b1..2fd163cff406 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -191,6 +191,54 @@ static void move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, drop_rmap_locks(vma); } +static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr, + unsigned long new_addr, unsigned long old_end, + pmd_t *old_pmd, pmd_t *new_pmd, bool *need_flush) +{ + spinlock_t *old_ptl, *new_ptl; + struct mm_struct *mm = vma->vm_mm; + + if ((old_addr & ~PMD_MASK) || (new_addr & ~PMD_MASK) + || old_end - old_addr < PMD_SIZE) + return false; + + /* + * The destination pmd shouldn't be established, free_pgtables() + * should have release it. + */ + if (WARN_ON(!pmd_none(*new_pmd))) + return false; + + /* + * We don't have to worry about the ordering of src and dst + * ptlocks because exclusive mmap_sem prevents deadlock. + */ + old_ptl = pmd_lock(vma->vm_mm, old_pmd); + if (old_ptl) { + pmd_t pmd; + + new_ptl = pmd_lockptr(mm, new_pmd); + if (new_ptl != old_ptl) + spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING); + + /* Clear the pmd */ + pmd = *old_pmd; + pmd_clear(old_pmd); + + VM_BUG_ON(!pmd_none(*new_pmd)); + + /* Set the new pmd */ + set_pmd_at(mm, new_addr, new_pmd, pmd); + if (new_ptl != old_ptl) + spin_unlock(new_ptl); + spin_unlock(old_ptl); + + *need_flush = true; + return true; + } + return false; +} + unsigned long move_page_tables(struct vm_area_struct *vma, unsigned long old_addr, struct vm_area_struct *new_vma, unsigned long new_addr, unsigned long len, @@ -239,7 +287,24 @@ unsigned long move_page_tables(struct vm_area_struct *vma, split_huge_pmd(vma, old_pmd, old_addr); if (pmd_trans_unstable(old_pmd)) continue; + } else if (extent == PMD_SIZE && IS_ENABLED(CONFIG_HAVE_MOVE_PMD)) { + /* + * If the extent is PMD-sized, try to speed the move by + * moving at the PMD level if possible. + */ + bool moved; + + if (need_rmap_locks) + take_rmap_locks(vma); + moved = move_normal_pmd(vma, old_addr, new_addr, + old_end, old_pmd, new_pmd, + &need_flush); + if (need_rmap_locks) + drop_rmap_locks(vma); + if (moved) + continue; } + if (pte_alloc(new_vma->vm_mm, new_pmd)) break; next = (new_addr + PMD_SIZE) & PMD_MASK; -- 2.19.0.605.g01d371f741-goog _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH 2/4] mm: speed up mremap by 500x on large regions (v2) 2018-10-13 1:31 ` [PATCH 2/4] mm: speed up mremap by 500x on large regions (v2) Joel Fernandes (Google) 2018-10-13 1:31 ` Joel Fernandes (Google) @ 2018-10-15 9:42 ` Christoph Hellwig 2018-10-15 9:42 ` Christoph Hellwig 2018-10-15 22:33 ` Joel Fernandes 2018-10-24 10:12 ` Kirill A. Shutemov 2 siblings, 2 replies; 52+ messages in thread From: Christoph Hellwig @ 2018-10-15 9:42 UTC (permalink / raw) To: linux-riscv On Fri, Oct 12, 2018 at 06:31:58PM -0700, Joel Fernandes (Google) wrote: > Android needs to mremap large regions of memory during memory management > related operations. Just curious: why? > + if ((old_addr & ~PMD_MASK) || (new_addr & ~PMD_MASK) > + || old_end - old_addr < PMD_SIZE) The || goes on the first line. > + } else if (extent == PMD_SIZE && IS_ENABLED(CONFIG_HAVE_MOVE_PMD)) { Overly long line. ^ permalink raw reply [flat|nested] 52+ messages in thread
* Re: [PATCH 2/4] mm: speed up mremap by 500x on large regions (v2) 2018-10-15 9:42 ` Christoph Hellwig @ 2018-10-15 9:42 ` Christoph Hellwig 2018-10-15 22:33 ` Joel Fernandes 1 sibling, 0 replies; 52+ messages in thread From: Christoph Hellwig @ 2018-10-15 9:42 UTC (permalink / raw) To: Joel Fernandes (Google) Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Peter Zijlstra, Catalin Marinas, Dave Hansen, Will Deacon, mhocko, linux-mm, lokeshgidra, sparclinux, linux-riscv, kvmarm, Jonas Bonn, linux-s390, dancol, Yoshinori Sato, Max Filippov, linux-hexagon, Helge Deller, maintainer:X86 ARCHITECTURE 32-BIT AND 64-BIT, hughd, James E.J. Bottomley, kasan-dev, elfring, Ingo Molnar, Geert Uytterhoeven, Andrey Ryabinin, linux-snps-arc, kernel-team, Sam Creasey, linux-xtensa, Jeff Dike, linux-alpha, linux-um, Stefan Kristiansson, Julia Lawall, linux-m68k, Borislav Petkov, Andy Lutomirski, Ley Foon Tan, kirill, Stafford Horne, Guan Xuetao, Chris Zankel, Tony Luck, linux-parisc, pantin, linux-kernel, Fenghua Yu, minchan, Thomas Gleixner, Richard Weinberger, anton.ivanov, nios2-dev, akpm, linuxppc-dev, David S. Miller On Fri, Oct 12, 2018 at 06:31:58PM -0700, Joel Fernandes (Google) wrote: > Android needs to mremap large regions of memory during memory management > related operations. Just curious: why? > + if ((old_addr & ~PMD_MASK) || (new_addr & ~PMD_MASK) > + || old_end - old_addr < PMD_SIZE) The || goes on the first line. > + } else if (extent == PMD_SIZE && IS_ENABLED(CONFIG_HAVE_MOVE_PMD)) { Overly long line. _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 52+ messages in thread
* [PATCH 2/4] mm: speed up mremap by 500x on large regions (v2) 2018-10-15 9:42 ` Christoph Hellwig 2018-10-15 9:42 ` Christoph Hellwig @ 2018-10-15 22:33 ` Joel Fernandes 2018-10-15 22:33 ` Joel Fernandes 2018-10-16 11:29 ` Vlastimil Babka 1 sibling, 2 replies; 52+ messages in thread From: Joel Fernandes @ 2018-10-15 22:33 UTC (permalink / raw) To: linux-riscv On Mon, Oct 15, 2018 at 02:42:09AM -0700, Christoph Hellwig wrote: > On Fri, Oct 12, 2018 at 06:31:58PM -0700, Joel Fernandes (Google) wrote: > > Android needs to mremap large regions of memory during memory management > > related operations. > > Just curious: why? In Android we have a requirement of moving a large (up to a GB now, but may grow bigger in future) memory range from one location to another. This move operation has to happen when the application threads are paused for this operation. Therefore, an inefficient move like it is now (for example 250ms on arm64) will cause response time issues for applications, which is not acceptable. Huge pages cannot be used in such memory ranges to avoid this inefficiency as (when the application threads are running) our fault handlers are designed to process 4KB pages at a time, to keep response times low. So using huge pages in this context can, again, cause response time issues. Also, the mremap syscall waiting for quarter of a second for a large mremap is quite weird and we ought to improve it where possible. > > + if ((old_addr & ~PMD_MASK) || (new_addr & ~PMD_MASK) > > + || old_end - old_addr < PMD_SIZE) > > The || goes on the first line. Ok, fixed. > > + } else if (extent == PMD_SIZE && IS_ENABLED(CONFIG_HAVE_MOVE_PMD)) { > > Overly long line. Ok, fixed. Preview of updated patch is below. thanks, - Joel ------8<--- From: "Joel Fernandes (Google)" <joel@joelfernandes.org> Subject: [PATCH 2/4] mm: speed up mremap by 500x on large regions (v3) Android needs to mremap large regions of memory during memory management related operations. The mremap system call can be really slow if THP is not enabled. The bottleneck is move_page_tables, which is copying each pte at a time, and can be really slow across a large map. Turning on THP may not be a viable option, and is not for us. This patch speeds up the performance for non-THP system by copying at the PMD level when possible. The speed up is three orders of magnitude. On a 1GB mremap, the mremap completion times drops from 160-250 millesconds to 380-400 microseconds. Before: Total mremap time for 1GB data: 242321014 nanoseconds. Total mremap time for 1GB data: 196842467 nanoseconds. Total mremap time for 1GB data: 167051162 nanoseconds. After: Total mremap time for 1GB data: 385781 nanoseconds. Total mremap time for 1GB data: 388959 nanoseconds. Total mremap time for 1GB data: 402813 nanoseconds. Incase THP is enabled, the optimization is mostly skipped except in certain situations. I also flush the tlb every time we do this optimization since I couldn't find a way to determine if the low-level PTEs are dirty. It is seen that the cost of doing so is not much compared the improvement, on both x86-64 and arm64. Cc: minchan at kernel.org Cc: pantin at google.com Cc: hughd at google.com Cc: lokeshgidra at google.com Cc: dancol at google.com Cc: mhocko at kernel.org Cc: kirill at shutemov.name Cc: akpm at linux-foundation.org Cc: kernel-team at android.com Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org> --- arch/Kconfig | 5 ++++ mm/mremap.c | 66 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 71 insertions(+) diff --git a/arch/Kconfig b/arch/Kconfig index 6801123932a5..9724fe39884f 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -518,6 +518,11 @@ config HAVE_IRQ_TIME_ACCOUNTING Archs need to ensure they use a high enough resolution clock to support irq time accounting and then call enable_sched_clock_irqtime(). +config HAVE_MOVE_PMD + bool + help + Archs that select this are able to move page tables at the PMD level. + config HAVE_ARCH_TRANSPARENT_HUGEPAGE bool diff --git a/mm/mremap.c b/mm/mremap.c index 9e68a02a52b1..a8dd98a59975 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -191,6 +191,54 @@ static void move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, drop_rmap_locks(vma); } +static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr, + unsigned long new_addr, unsigned long old_end, + pmd_t *old_pmd, pmd_t *new_pmd, bool *need_flush) +{ + spinlock_t *old_ptl, *new_ptl; + struct mm_struct *mm = vma->vm_mm; + + if ((old_addr & ~PMD_MASK) || (new_addr & ~PMD_MASK) || + old_end - old_addr < PMD_SIZE) + return false; + + /* + * The destination pmd shouldn't be established, free_pgtables() + * should have release it. + */ + if (WARN_ON(!pmd_none(*new_pmd))) + return false; + + /* + * We don't have to worry about the ordering of src and dst + * ptlocks because exclusive mmap_sem prevents deadlock. + */ + old_ptl = pmd_lock(vma->vm_mm, old_pmd); + if (old_ptl) { + pmd_t pmd; + + new_ptl = pmd_lockptr(mm, new_pmd); + if (new_ptl != old_ptl) + spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING); + + /* Clear the pmd */ + pmd = *old_pmd; + pmd_clear(old_pmd); + + VM_BUG_ON(!pmd_none(*new_pmd)); + + /* Set the new pmd */ + set_pmd_at(mm, new_addr, new_pmd, pmd); + if (new_ptl != old_ptl) + spin_unlock(new_ptl); + spin_unlock(old_ptl); + + *need_flush = true; + return true; + } + return false; +} + unsigned long move_page_tables(struct vm_area_struct *vma, unsigned long old_addr, struct vm_area_struct *new_vma, unsigned long new_addr, unsigned long len, @@ -239,7 +287,25 @@ unsigned long move_page_tables(struct vm_area_struct *vma, split_huge_pmd(vma, old_pmd, old_addr); if (pmd_trans_unstable(old_pmd)) continue; + } else if (extent == PMD_SIZE && + IS_ENABLED(CONFIG_HAVE_MOVE_PMD)) { + /* + * If the extent is PMD-sized, try to speed the move by + * moving at the PMD level if possible. + */ + bool moved; + + if (need_rmap_locks) + take_rmap_locks(vma); + moved = move_normal_pmd(vma, old_addr, new_addr, + old_end, old_pmd, new_pmd, + &need_flush); + if (need_rmap_locks) + drop_rmap_locks(vma); + if (moved) + continue; } + if (pte_alloc(new_vma->vm_mm, new_pmd)) break; next = (new_addr + PMD_SIZE) & PMD_MASK; -- 2.19.1.331.ge82ca0e54c-goog ^ permalink raw reply related [flat|nested] 52+ messages in thread
* Re: [PATCH 2/4] mm: speed up mremap by 500x on large regions (v2) 2018-10-15 22:33 ` Joel Fernandes @ 2018-10-15 22:33 ` Joel Fernandes 2018-10-16 11:29 ` Vlastimil Babka 1 sibling, 0 replies; 52+ messages in thread From: Joel Fernandes @ 2018-10-15 22:33 UTC (permalink / raw) To: Christoph Hellwig Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Peter Zijlstra, Catalin Marinas, Dave Hansen, Will Deacon, mhocko, linux-mm, lokeshgidra, sparclinux, linux-riscv, kvmarm, Jonas Bonn, linux-s390, dancol, Yoshinori Sato, Max Filippov, linux-hexagon, Helge Deller, maintainer:X86 ARCHITECTURE 32-BIT AND 64-BIT, hughd, James E.J. Bottomley, kasan-dev, elfring, Ingo Molnar, Geert Uytterhoeven, Andrey Ryabinin, linux-snps-arc, kernel-team, Sam Creasey, linux-xtensa, Jeff Dike, linux-alpha, linux-um, Stefan Kristiansson, Julia Lawall, linux-m68k, Borislav Petkov, Andy Lutomirski, Ley Foon Tan, kirill, Stafford Horne, Guan Xuetao, Chris Zankel, Tony Luck, linux-parisc, pantin, linux-kernel, Fenghua Yu, minchan, Thomas Gleixner, Richard Weinberger, anton.ivanov, nios2-dev, akpm, linuxppc-dev, David S. Miller On Mon, Oct 15, 2018 at 02:42:09AM -0700, Christoph Hellwig wrote: > On Fri, Oct 12, 2018 at 06:31:58PM -0700, Joel Fernandes (Google) wrote: > > Android needs to mremap large regions of memory during memory management > > related operations. > > Just curious: why? In Android we have a requirement of moving a large (up to a GB now, but may grow bigger in future) memory range from one location to another. This move operation has to happen when the application threads are paused for this operation. Therefore, an inefficient move like it is now (for example 250ms on arm64) will cause response time issues for applications, which is not acceptable. Huge pages cannot be used in such memory ranges to avoid this inefficiency as (when the application threads are running) our fault handlers are designed to process 4KB pages at a time, to keep response times low. So using huge pages in this context can, again, cause response time issues. Also, the mremap syscall waiting for quarter of a second for a large mremap is quite weird and we ought to improve it where possible. > > + if ((old_addr & ~PMD_MASK) || (new_addr & ~PMD_MASK) > > + || old_end - old_addr < PMD_SIZE) > > The || goes on the first line. Ok, fixed. > > + } else if (extent == PMD_SIZE && IS_ENABLED(CONFIG_HAVE_MOVE_PMD)) { > > Overly long line. Ok, fixed. Preview of updated patch is below. thanks, - Joel ------8<--- From: "Joel Fernandes (Google)" <joel@joelfernandes.org> Subject: [PATCH 2/4] mm: speed up mremap by 500x on large regions (v3) Android needs to mremap large regions of memory during memory management related operations. The mremap system call can be really slow if THP is not enabled. The bottleneck is move_page_tables, which is copying each pte at a time, and can be really slow across a large map. Turning on THP may not be a viable option, and is not for us. This patch speeds up the performance for non-THP system by copying at the PMD level when possible. The speed up is three orders of magnitude. On a 1GB mremap, the mremap completion times drops from 160-250 millesconds to 380-400 microseconds. Before: Total mremap time for 1GB data: 242321014 nanoseconds. Total mremap time for 1GB data: 196842467 nanoseconds. Total mremap time for 1GB data: 167051162 nanoseconds. After: Total mremap time for 1GB data: 385781 nanoseconds. Total mremap time for 1GB data: 388959 nanoseconds. Total mremap time for 1GB data: 402813 nanoseconds. Incase THP is enabled, the optimization is mostly skipped except in certain situations. I also flush the tlb every time we do this optimization since I couldn't find a way to determine if the low-level PTEs are dirty. It is seen that the cost of doing so is not much compared the improvement, on both x86-64 and arm64. Cc: minchan@kernel.org Cc: pantin@google.com Cc: hughd@google.com Cc: lokeshgidra@google.com Cc: dancol@google.com Cc: mhocko@kernel.org Cc: kirill@shutemov.name Cc: akpm@linux-foundation.org Cc: kernel-team@android.com Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org> --- arch/Kconfig | 5 ++++ mm/mremap.c | 66 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 71 insertions(+) diff --git a/arch/Kconfig b/arch/Kconfig index 6801123932a5..9724fe39884f 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -518,6 +518,11 @@ config HAVE_IRQ_TIME_ACCOUNTING Archs need to ensure they use a high enough resolution clock to support irq time accounting and then call enable_sched_clock_irqtime(). +config HAVE_MOVE_PMD + bool + help + Archs that select this are able to move page tables at the PMD level. + config HAVE_ARCH_TRANSPARENT_HUGEPAGE bool diff --git a/mm/mremap.c b/mm/mremap.c index 9e68a02a52b1..a8dd98a59975 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -191,6 +191,54 @@ static void move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, drop_rmap_locks(vma); } +static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr, + unsigned long new_addr, unsigned long old_end, + pmd_t *old_pmd, pmd_t *new_pmd, bool *need_flush) +{ + spinlock_t *old_ptl, *new_ptl; + struct mm_struct *mm = vma->vm_mm; + + if ((old_addr & ~PMD_MASK) || (new_addr & ~PMD_MASK) || + old_end - old_addr < PMD_SIZE) + return false; + + /* + * The destination pmd shouldn't be established, free_pgtables() + * should have release it. + */ + if (WARN_ON(!pmd_none(*new_pmd))) + return false; + + /* + * We don't have to worry about the ordering of src and dst + * ptlocks because exclusive mmap_sem prevents deadlock. + */ + old_ptl = pmd_lock(vma->vm_mm, old_pmd); + if (old_ptl) { + pmd_t pmd; + + new_ptl = pmd_lockptr(mm, new_pmd); + if (new_ptl != old_ptl) + spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING); + + /* Clear the pmd */ + pmd = *old_pmd; + pmd_clear(old_pmd); + + VM_BUG_ON(!pmd_none(*new_pmd)); + + /* Set the new pmd */ + set_pmd_at(mm, new_addr, new_pmd, pmd); + if (new_ptl != old_ptl) + spin_unlock(new_ptl); + spin_unlock(old_ptl); + + *need_flush = true; + return true; + } + return false; +} + unsigned long move_page_tables(struct vm_area_struct *vma, unsigned long old_addr, struct vm_area_struct *new_vma, unsigned long new_addr, unsigned long len, @@ -239,7 +287,25 @@ unsigned long move_page_tables(struct vm_area_struct *vma, split_huge_pmd(vma, old_pmd, old_addr); if (pmd_trans_unstable(old_pmd)) continue; + } else if (extent == PMD_SIZE && + IS_ENABLED(CONFIG_HAVE_MOVE_PMD)) { + /* + * If the extent is PMD-sized, try to speed the move by + * moving at the PMD level if possible. + */ + bool moved; + + if (need_rmap_locks) + take_rmap_locks(vma); + moved = move_normal_pmd(vma, old_addr, new_addr, + old_end, old_pmd, new_pmd, + &need_flush); + if (need_rmap_locks) + drop_rmap_locks(vma); + if (moved) + continue; } + if (pte_alloc(new_vma->vm_mm, new_pmd)) break; next = (new_addr + PMD_SIZE) & PMD_MASK; -- 2.19.1.331.ge82ca0e54c-goog _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH 2/4] mm: speed up mremap by 500x on large regions (v2) 2018-10-15 22:33 ` Joel Fernandes 2018-10-15 22:33 ` Joel Fernandes @ 2018-10-16 11:29 ` Vlastimil Babka 2018-10-16 11:29 ` Vlastimil Babka 2018-10-16 19:43 ` Joel Fernandes 1 sibling, 2 replies; 52+ messages in thread From: Vlastimil Babka @ 2018-10-16 11:29 UTC (permalink / raw) To: linux-riscv On 10/16/18 12:33 AM, Joel Fernandes wrote: > On Mon, Oct 15, 2018 at 02:42:09AM -0700, Christoph Hellwig wrote: >> On Fri, Oct 12, 2018 at 06:31:58PM -0700, Joel Fernandes (Google) wrote: >>> Android needs to mremap large regions of memory during memory management >>> related operations. >> >> Just curious: why? > > In Android we have a requirement of moving a large (up to a GB now, but may > grow bigger in future) memory range from one location to another. I think Christoph's "why?" was about the requirement, not why it hurts applications. I admit I'm now also curious :) > This move > operation has to happen when the application threads are paused for this > operation. Therefore, an inefficient move like it is now (for example 250ms > on arm64) will cause response time issues for applications, which is not > acceptable. Huge pages cannot be used in such memory ranges to avoid this > inefficiency as (when the application threads are running) our fault handlers > are designed to process 4KB pages at a time, to keep response times low. So > using huge pages in this context can, again, cause response time issues. > > Also, the mremap syscall waiting for quarter of a second for a large mremap > is quite weird and we ought to improve it where possible. ^ permalink raw reply [flat|nested] 52+ messages in thread
* Re: [PATCH 2/4] mm: speed up mremap by 500x on large regions (v2) 2018-10-16 11:29 ` Vlastimil Babka @ 2018-10-16 11:29 ` Vlastimil Babka 2018-10-16 19:43 ` Joel Fernandes 1 sibling, 0 replies; 52+ messages in thread From: Vlastimil Babka @ 2018-10-16 11:29 UTC (permalink / raw) To: Joel Fernandes, Christoph Hellwig Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Peter Zijlstra, Catalin Marinas, Dave Hansen, Will Deacon, mhocko, linux-mm, lokeshgidra, sparclinux, linux-riscv, kvmarm, Jonas Bonn, linux-s390, dancol, Yoshinori Sato, Max Filippov, linux-hexagon, Helge Deller, maintainer:X86 ARCHITECTURE 32-BIT AND 64-BIT, hughd, James E.J. Bottomley, kasan-dev, elfring, Ingo Molnar, Geert Uytterhoeven, Andrey Ryabinin, linux-snps-arc, kernel-team, Sam Creasey, linux-xtensa, Jeff Dike, linux-alpha, linux-um, Stefan Kristiansson, Julia Lawall, linux-m68k, Borislav Petkov, Andy Lutomirski, Ley Foon Tan, kirill, Stafford Horne, Guan Xuetao, Chris Zankel, Tony Luck, linux-parisc, pantin, linux-kernel, Fenghua Yu, minchan, Thomas Gleixner, Richard Weinberger, anton.ivanov, nios2-dev, akpm, linuxppc-dev, David S. Miller On 10/16/18 12:33 AM, Joel Fernandes wrote: > On Mon, Oct 15, 2018 at 02:42:09AM -0700, Christoph Hellwig wrote: >> On Fri, Oct 12, 2018 at 06:31:58PM -0700, Joel Fernandes (Google) wrote: >>> Android needs to mremap large regions of memory during memory management >>> related operations. >> >> Just curious: why? > > In Android we have a requirement of moving a large (up to a GB now, but may > grow bigger in future) memory range from one location to another. I think Christoph's "why?" was about the requirement, not why it hurts applications. I admit I'm now also curious :) > This move > operation has to happen when the application threads are paused for this > operation. Therefore, an inefficient move like it is now (for example 250ms > on arm64) will cause response time issues for applications, which is not > acceptable. Huge pages cannot be used in such memory ranges to avoid this > inefficiency as (when the application threads are running) our fault handlers > are designed to process 4KB pages at a time, to keep response times low. So > using huge pages in this context can, again, cause response time issues. > > Also, the mremap syscall waiting for quarter of a second for a large mremap > is quite weird and we ought to improve it where possible. _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 52+ messages in thread
* [PATCH 2/4] mm: speed up mremap by 500x on large regions (v2) 2018-10-16 11:29 ` Vlastimil Babka 2018-10-16 11:29 ` Vlastimil Babka @ 2018-10-16 19:43 ` Joel Fernandes 2018-10-16 19:43 ` Joel Fernandes 2018-10-17 7:38 ` Vlastimil Babka 1 sibling, 2 replies; 52+ messages in thread From: Joel Fernandes @ 2018-10-16 19:43 UTC (permalink / raw) To: linux-riscv On Tue, Oct 16, 2018 at 01:29:52PM +0200, Vlastimil Babka wrote: > On 10/16/18 12:33 AM, Joel Fernandes wrote: > > On Mon, Oct 15, 2018 at 02:42:09AM -0700, Christoph Hellwig wrote: > >> On Fri, Oct 12, 2018 at 06:31:58PM -0700, Joel Fernandes (Google) wrote: > >>> Android needs to mremap large regions of memory during memory management > >>> related operations. > >> > >> Just curious: why? > > > > In Android we have a requirement of moving a large (up to a GB now, but may > > grow bigger in future) memory range from one location to another. > > I think Christoph's "why?" was about the requirement, not why it hurts > applications. I admit I'm now also curious :) This issue was discovered when we wanted to be able to move the physical pages of a memory range to another location quickly so that, after the application threads are resumed, UFFDIO_REGISTER_MODE_MISSING userfaultfd faults can be received on the original memory range. The actual operations performed on the memory range are beyond the scope of this discussion. The user threads continue to refer to the old address which will now fault. The reason we want retain the old memory range and receives faults there is to avoid the need to fix the addresses all over the address space of the threads after we finish with performing operations on them in the fault handlers, so we mremap it and receive faults at the old addresses. Does that answer your question? thanks, - Joel ^ permalink raw reply [flat|nested] 52+ messages in thread
* Re: [PATCH 2/4] mm: speed up mremap by 500x on large regions (v2) 2018-10-16 19:43 ` Joel Fernandes @ 2018-10-16 19:43 ` Joel Fernandes 2018-10-17 7:38 ` Vlastimil Babka 1 sibling, 0 replies; 52+ messages in thread From: Joel Fernandes @ 2018-10-16 19:43 UTC (permalink / raw) To: Vlastimil Babka Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Peter Zijlstra, Catalin Marinas, Dave Hansen, Will Deacon, mhocko, linux-mm, lokeshgidra, sparclinux, linux-riscv, kvmarm, Jonas Bonn, linux-s390, dancol, Yoshinori Sato, Max Filippov, linux-hexagon, Helge Deller, maintainer:X86 ARCHITECTURE 32-BIT AND 64-BIT, hughd, James E.J. Bottomley, kasan-dev, elfring, Christoph Hellwig, Ingo Molnar, Geert Uytterhoeven, Andrey Ryabinin, linux-snps-arc, kernel-team, Sam Creasey, linux-xtensa, Jeff Dike, linux-alpha, linux-um, Stefan Kristiansson, Julia Lawall, linux-m68k, Borislav Petkov, Andy Lutomirski, Ley Foon Tan, kirill, Stafford Horne, Guan Xuetao, Chris Zankel, Tony Luck, linux-parisc, pantin, linux-kernel, Fenghua Yu, minchan, Thomas Gleixner, Richard Weinberger, anton.ivanov, nios2-dev, akpm, linuxppc-dev, David S. Miller On Tue, Oct 16, 2018 at 01:29:52PM +0200, Vlastimil Babka wrote: > On 10/16/18 12:33 AM, Joel Fernandes wrote: > > On Mon, Oct 15, 2018 at 02:42:09AM -0700, Christoph Hellwig wrote: > >> On Fri, Oct 12, 2018 at 06:31:58PM -0700, Joel Fernandes (Google) wrote: > >>> Android needs to mremap large regions of memory during memory management > >>> related operations. > >> > >> Just curious: why? > > > > In Android we have a requirement of moving a large (up to a GB now, but may > > grow bigger in future) memory range from one location to another. > > I think Christoph's "why?" was about the requirement, not why it hurts > applications. I admit I'm now also curious :) This issue was discovered when we wanted to be able to move the physical pages of a memory range to another location quickly so that, after the application threads are resumed, UFFDIO_REGISTER_MODE_MISSING userfaultfd faults can be received on the original memory range. The actual operations performed on the memory range are beyond the scope of this discussion. The user threads continue to refer to the old address which will now fault. The reason we want retain the old memory range and receives faults there is to avoid the need to fix the addresses all over the address space of the threads after we finish with performing operations on them in the fault handlers, so we mremap it and receive faults at the old addresses. Does that answer your question? thanks, - Joel _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 52+ messages in thread
* [PATCH 2/4] mm: speed up mremap by 500x on large regions (v2) 2018-10-16 19:43 ` Joel Fernandes 2018-10-16 19:43 ` Joel Fernandes @ 2018-10-17 7:38 ` Vlastimil Babka 2018-10-17 7:38 ` Vlastimil Babka 1 sibling, 1 reply; 52+ messages in thread From: Vlastimil Babka @ 2018-10-17 7:38 UTC (permalink / raw) To: linux-riscv On 10/16/18 9:43 PM, Joel Fernandes wrote: > On Tue, Oct 16, 2018 at 01:29:52PM +0200, Vlastimil Babka wrote: >> On 10/16/18 12:33 AM, Joel Fernandes wrote: >>> On Mon, Oct 15, 2018 at 02:42:09AM -0700, Christoph Hellwig wrote: >>>> On Fri, Oct 12, 2018 at 06:31:58PM -0700, Joel Fernandes (Google) wrote: >>>>> Android needs to mremap large regions of memory during memory management >>>>> related operations. >>>> >>>> Just curious: why? >>> >>> In Android we have a requirement of moving a large (up to a GB now, but may >>> grow bigger in future) memory range from one location to another. >> >> I think Christoph's "why?" was about the requirement, not why it hurts >> applications. I admit I'm now also curious :) > > This issue was discovered when we wanted to be able to move the physical > pages of a memory range to another location quickly so that, after the > application threads are resumed, UFFDIO_REGISTER_MODE_MISSING userfaultfd > faults can be received on the original memory range. The actual operations > performed on the memory range are beyond the scope of this discussion. The > user threads continue to refer to the old address which will now fault. The > reason we want retain the old memory range and receives faults there is to > avoid the need to fix the addresses all over the address space of the threads > after we finish with performing operations on them in the fault handlers, so > we mremap it and receive faults at the old addresses. > > Does that answer your question? Yes, interesting, thanks! Vlastimil > thanks, > > - Joel > ^ permalink raw reply [flat|nested] 52+ messages in thread
* Re: [PATCH 2/4] mm: speed up mremap by 500x on large regions (v2) 2018-10-17 7:38 ` Vlastimil Babka @ 2018-10-17 7:38 ` Vlastimil Babka 0 siblings, 0 replies; 52+ messages in thread From: Vlastimil Babka @ 2018-10-17 7:38 UTC (permalink / raw) To: Joel Fernandes Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Peter Zijlstra, Catalin Marinas, Dave Hansen, Will Deacon, mhocko, linux-mm, lokeshgidra, sparclinux, linux-riscv, kvmarm, Jonas Bonn, linux-s390, dancol, Yoshinori Sato, Max Filippov, linux-hexagon, Helge Deller, maintainer:X86 ARCHITECTURE 32-BIT AND 64-BIT, hughd, James E.J. Bottomley, kasan-dev, elfring, Christoph Hellwig, Ingo Molnar, Geert Uytterhoeven, Andrey Ryabinin, linux-snps-arc, kernel-team, Sam Creasey, linux-xtensa, Jeff Dike, linux-alpha, linux-um, Stefan Kristiansson, Julia Lawall, linux-m68k, Borislav Petkov, Andy Lutomirski, Ley Foon Tan, kirill, Stafford Horne, Guan Xuetao, Chris Zankel, Tony Luck, linux-parisc, pantin, linux-kernel, Fenghua Yu, minchan, Thomas Gleixner, Richard Weinberger, anton.ivanov, nios2-dev, akpm, linuxppc-dev, David S. Miller On 10/16/18 9:43 PM, Joel Fernandes wrote: > On Tue, Oct 16, 2018 at 01:29:52PM +0200, Vlastimil Babka wrote: >> On 10/16/18 12:33 AM, Joel Fernandes wrote: >>> On Mon, Oct 15, 2018 at 02:42:09AM -0700, Christoph Hellwig wrote: >>>> On Fri, Oct 12, 2018 at 06:31:58PM -0700, Joel Fernandes (Google) wrote: >>>>> Android needs to mremap large regions of memory during memory management >>>>> related operations. >>>> >>>> Just curious: why? >>> >>> In Android we have a requirement of moving a large (up to a GB now, but may >>> grow bigger in future) memory range from one location to another. >> >> I think Christoph's "why?" was about the requirement, not why it hurts >> applications. I admit I'm now also curious :) > > This issue was discovered when we wanted to be able to move the physical > pages of a memory range to another location quickly so that, after the > application threads are resumed, UFFDIO_REGISTER_MODE_MISSING userfaultfd > faults can be received on the original memory range. The actual operations > performed on the memory range are beyond the scope of this discussion. The > user threads continue to refer to the old address which will now fault. The > reason we want retain the old memory range and receives faults there is to > avoid the need to fix the addresses all over the address space of the threads > after we finish with performing operations on them in the fault handlers, so > we mremap it and receive faults at the old addresses. > > Does that answer your question? Yes, interesting, thanks! Vlastimil > thanks, > > - Joel > _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 52+ messages in thread
* [PATCH 2/4] mm: speed up mremap by 500x on large regions (v2) 2018-10-13 1:31 ` [PATCH 2/4] mm: speed up mremap by 500x on large regions (v2) Joel Fernandes (Google) 2018-10-13 1:31 ` Joel Fernandes (Google) 2018-10-15 9:42 ` Christoph Hellwig @ 2018-10-24 10:12 ` Kirill A. Shutemov 2018-10-24 10:12 ` Kirill A. Shutemov 2018-10-24 11:57 ` Balbir Singh 2 siblings, 2 replies; 52+ messages in thread From: Kirill A. Shutemov @ 2018-10-24 10:12 UTC (permalink / raw) To: linux-riscv On Fri, Oct 12, 2018 at 06:31:58PM -0700, Joel Fernandes (Google) wrote: > diff --git a/mm/mremap.c b/mm/mremap.c > index 9e68a02a52b1..2fd163cff406 100644 > --- a/mm/mremap.c > +++ b/mm/mremap.c > @@ -191,6 +191,54 @@ static void move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, > drop_rmap_locks(vma); > } > > +static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr, > + unsigned long new_addr, unsigned long old_end, > + pmd_t *old_pmd, pmd_t *new_pmd, bool *need_flush) > +{ > + spinlock_t *old_ptl, *new_ptl; > + struct mm_struct *mm = vma->vm_mm; > + > + if ((old_addr & ~PMD_MASK) || (new_addr & ~PMD_MASK) > + || old_end - old_addr < PMD_SIZE) > + return false; > + > + /* > + * The destination pmd shouldn't be established, free_pgtables() > + * should have release it. > + */ > + if (WARN_ON(!pmd_none(*new_pmd))) > + return false; > + > + /* > + * We don't have to worry about the ordering of src and dst > + * ptlocks because exclusive mmap_sem prevents deadlock. > + */ > + old_ptl = pmd_lock(vma->vm_mm, old_pmd); > + if (old_ptl) { How can it ever be false? > + pmd_t pmd; > + > + new_ptl = pmd_lockptr(mm, new_pmd); > + if (new_ptl != old_ptl) > + spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING); > + > + /* Clear the pmd */ > + pmd = *old_pmd; > + pmd_clear(old_pmd); > + > + VM_BUG_ON(!pmd_none(*new_pmd)); > + > + /* Set the new pmd */ > + set_pmd_at(mm, new_addr, new_pmd, pmd); > + if (new_ptl != old_ptl) > + spin_unlock(new_ptl); > + spin_unlock(old_ptl); > + > + *need_flush = true; > + return true; > + } > + return false; > +} > + -- Kirill A. Shutemov ^ permalink raw reply [flat|nested] 52+ messages in thread
* Re: [PATCH 2/4] mm: speed up mremap by 500x on large regions (v2) 2018-10-24 10:12 ` Kirill A. Shutemov @ 2018-10-24 10:12 ` Kirill A. Shutemov 2018-10-24 11:57 ` Balbir Singh 1 sibling, 0 replies; 52+ messages in thread From: Kirill A. Shutemov @ 2018-10-24 10:12 UTC (permalink / raw) To: Joel Fernandes (Google) Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Peter Zijlstra, Catalin Marinas, Dave Hansen, Will Deacon, mhocko, linux-mm, lokeshgidra, sparclinux, linux-riscv, elfring, Jonas Bonn, kvmarm, dancol, Yoshinori Sato, linux-xtensa, linux-hexagon, Helge Deller, maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT), hughd, James E.J. Bottomley, kasan-dev, anton.ivanov, Ingo Molnar, Geert Uytterhoeven, Andrey Ryabinin, linux-snps-arc, kernel-team, Sam Creasey, Fenghua Yu, linux-s390, Jeff Dike, linux-um, Stefan Kristiansson, Julia Lawall, linux-m68k, Borislav Petkov, Andy Lutomirski, nios2-dev, Stafford Horne, Guan Xuetao, Chris Zankel, Tony Luck, Richard Weinberger, linux-parisc, pantin, Max Filippov, linux-kernel, minchan, Thomas Gleixner, linux-alpha, Ley Foon Tan, akpm, linuxppc-dev, David S. Miller On Fri, Oct 12, 2018 at 06:31:58PM -0700, Joel Fernandes (Google) wrote: > diff --git a/mm/mremap.c b/mm/mremap.c > index 9e68a02a52b1..2fd163cff406 100644 > --- a/mm/mremap.c > +++ b/mm/mremap.c > @@ -191,6 +191,54 @@ static void move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, > drop_rmap_locks(vma); > } > > +static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr, > + unsigned long new_addr, unsigned long old_end, > + pmd_t *old_pmd, pmd_t *new_pmd, bool *need_flush) > +{ > + spinlock_t *old_ptl, *new_ptl; > + struct mm_struct *mm = vma->vm_mm; > + > + if ((old_addr & ~PMD_MASK) || (new_addr & ~PMD_MASK) > + || old_end - old_addr < PMD_SIZE) > + return false; > + > + /* > + * The destination pmd shouldn't be established, free_pgtables() > + * should have release it. > + */ > + if (WARN_ON(!pmd_none(*new_pmd))) > + return false; > + > + /* > + * We don't have to worry about the ordering of src and dst > + * ptlocks because exclusive mmap_sem prevents deadlock. > + */ > + old_ptl = pmd_lock(vma->vm_mm, old_pmd); > + if (old_ptl) { How can it ever be false? > + pmd_t pmd; > + > + new_ptl = pmd_lockptr(mm, new_pmd); > + if (new_ptl != old_ptl) > + spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING); > + > + /* Clear the pmd */ > + pmd = *old_pmd; > + pmd_clear(old_pmd); > + > + VM_BUG_ON(!pmd_none(*new_pmd)); > + > + /* Set the new pmd */ > + set_pmd_at(mm, new_addr, new_pmd, pmd); > + if (new_ptl != old_ptl) > + spin_unlock(new_ptl); > + spin_unlock(old_ptl); > + > + *need_flush = true; > + return true; > + } > + return false; > +} > + -- Kirill A. Shutemov _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 52+ messages in thread
* [PATCH 2/4] mm: speed up mremap by 500x on large regions (v2) 2018-10-24 10:12 ` Kirill A. Shutemov 2018-10-24 10:12 ` Kirill A. Shutemov @ 2018-10-24 11:57 ` Balbir Singh 2018-10-24 11:57 ` Balbir Singh ` (2 more replies) 1 sibling, 3 replies; 52+ messages in thread From: Balbir Singh @ 2018-10-24 11:57 UTC (permalink / raw) To: linux-riscv On Wed, Oct 24, 2018 at 01:12:56PM +0300, Kirill A. Shutemov wrote: > On Fri, Oct 12, 2018 at 06:31:58PM -0700, Joel Fernandes (Google) wrote: > > diff --git a/mm/mremap.c b/mm/mremap.c > > index 9e68a02a52b1..2fd163cff406 100644 > > --- a/mm/mremap.c > > +++ b/mm/mremap.c > > @@ -191,6 +191,54 @@ static void move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, > > drop_rmap_locks(vma); > > } > > > > +static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr, > > + unsigned long new_addr, unsigned long old_end, > > + pmd_t *old_pmd, pmd_t *new_pmd, bool *need_flush) > > +{ > > + spinlock_t *old_ptl, *new_ptl; > > + struct mm_struct *mm = vma->vm_mm; > > + > > + if ((old_addr & ~PMD_MASK) || (new_addr & ~PMD_MASK) > > + || old_end - old_addr < PMD_SIZE) > > + return false; > > + > > + /* > > + * The destination pmd shouldn't be established, free_pgtables() > > + * should have release it. > > + */ > > + if (WARN_ON(!pmd_none(*new_pmd))) > > + return false; > > + > > + /* > > + * We don't have to worry about the ordering of src and dst > > + * ptlocks because exclusive mmap_sem prevents deadlock. > > + */ > > + old_ptl = pmd_lock(vma->vm_mm, old_pmd); > > + if (old_ptl) { > > How can it ever be false? > > > + pmd_t pmd; > > + > > + new_ptl = pmd_lockptr(mm, new_pmd); Looks like this is largely inspired by move_huge_pmd(), I guess a lot of the code applies, why not just reuse as much as possible? The same comments w.r.t mmap_sem helping protect against lock order issues applies as well. > > + if (new_ptl != old_ptl) > > + spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING); > > + > > + /* Clear the pmd */ > > + pmd = *old_pmd; > > + pmd_clear(old_pmd); > > + > > + VM_BUG_ON(!pmd_none(*new_pmd)); > > + > > + /* Set the new pmd */ > > + set_pmd_at(mm, new_addr, new_pmd, pmd); > > + if (new_ptl != old_ptl) > > + spin_unlock(new_ptl); > > + spin_unlock(old_ptl); > > + > > + *need_flush = true; > > + return true; > > + } > > + return false; > > +} > > + > -- > Kirill A. Shutemov > ^ permalink raw reply [flat|nested] 52+ messages in thread
* Re: [PATCH 2/4] mm: speed up mremap by 500x on large regions (v2) 2018-10-24 11:57 ` Balbir Singh @ 2018-10-24 11:57 ` Balbir Singh 2018-10-24 12:57 ` Kirill A. Shutemov 2018-10-25 2:13 ` Joel Fernandes 2 siblings, 0 replies; 52+ messages in thread From: Balbir Singh @ 2018-10-24 11:57 UTC (permalink / raw) To: Kirill A. Shutemov Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Peter Zijlstra, Catalin Marinas, Dave Hansen, Will Deacon, mhocko, linux-mm, lokeshgidra, Joel Fernandes (Google), linux-riscv, elfring, Jonas Bonn, kvmarm, dancol, Yoshinori Sato, sparclinux, linux-xtensa, linux-hexagon, Helge Deller, maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT), hughd, James E.J. Bottomley, kasan-dev, anton.ivanov, Ingo Molnar, Geert Uytterhoeven, Andrey Ryabinin, linux-snps-arc, kernel-team, Sam Creasey, Fenghua Yu, linux-s390, Jeff Dike, linux-um, Stefan Kristiansson, Julia Lawall, linux-m68k, Borislav Petkov, Andy Lutomirski, nios2-dev, Stafford Horne, Guan Xuetao, Chris Zankel, Tony Luck, Richard Weinberger, linux-parisc, pantin, Max Filippov, linux-kernel, minchan, Thomas Gleixner, linux-alpha, Ley Foon Tan, akpm, linuxppc-dev, David S. Miller On Wed, Oct 24, 2018 at 01:12:56PM +0300, Kirill A. Shutemov wrote: > On Fri, Oct 12, 2018 at 06:31:58PM -0700, Joel Fernandes (Google) wrote: > > diff --git a/mm/mremap.c b/mm/mremap.c > > index 9e68a02a52b1..2fd163cff406 100644 > > --- a/mm/mremap.c > > +++ b/mm/mremap.c > > @@ -191,6 +191,54 @@ static void move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, > > drop_rmap_locks(vma); > > } > > > > +static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr, > > + unsigned long new_addr, unsigned long old_end, > > + pmd_t *old_pmd, pmd_t *new_pmd, bool *need_flush) > > +{ > > + spinlock_t *old_ptl, *new_ptl; > > + struct mm_struct *mm = vma->vm_mm; > > + > > + if ((old_addr & ~PMD_MASK) || (new_addr & ~PMD_MASK) > > + || old_end - old_addr < PMD_SIZE) > > + return false; > > + > > + /* > > + * The destination pmd shouldn't be established, free_pgtables() > > + * should have release it. > > + */ > > + if (WARN_ON(!pmd_none(*new_pmd))) > > + return false; > > + > > + /* > > + * We don't have to worry about the ordering of src and dst > > + * ptlocks because exclusive mmap_sem prevents deadlock. > > + */ > > + old_ptl = pmd_lock(vma->vm_mm, old_pmd); > > + if (old_ptl) { > > How can it ever be false? > > > + pmd_t pmd; > > + > > + new_ptl = pmd_lockptr(mm, new_pmd); Looks like this is largely inspired by move_huge_pmd(), I guess a lot of the code applies, why not just reuse as much as possible? The same comments w.r.t mmap_sem helping protect against lock order issues applies as well. > > + if (new_ptl != old_ptl) > > + spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING); > > + > > + /* Clear the pmd */ > > + pmd = *old_pmd; > > + pmd_clear(old_pmd); > > + > > + VM_BUG_ON(!pmd_none(*new_pmd)); > > + > > + /* Set the new pmd */ > > + set_pmd_at(mm, new_addr, new_pmd, pmd); > > + if (new_ptl != old_ptl) > > + spin_unlock(new_ptl); > > + spin_unlock(old_ptl); > > + > > + *need_flush = true; > > + return true; > > + } > > + return false; > > +} > > + > -- > Kirill A. Shutemov > _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 52+ messages in thread
* [PATCH 2/4] mm: speed up mremap by 500x on large regions (v2) 2018-10-24 11:57 ` Balbir Singh 2018-10-24 11:57 ` Balbir Singh @ 2018-10-24 12:57 ` Kirill A. Shutemov 2018-10-24 12:57 ` Kirill A. Shutemov 2018-10-25 2:09 ` Joel Fernandes 2018-10-25 2:13 ` Joel Fernandes 2 siblings, 2 replies; 52+ messages in thread From: Kirill A. Shutemov @ 2018-10-24 12:57 UTC (permalink / raw) To: linux-riscv On Wed, Oct 24, 2018 at 10:57:33PM +1100, Balbir Singh wrote: > On Wed, Oct 24, 2018 at 01:12:56PM +0300, Kirill A. Shutemov wrote: > > On Fri, Oct 12, 2018 at 06:31:58PM -0700, Joel Fernandes (Google) wrote: > > > diff --git a/mm/mremap.c b/mm/mremap.c > > > index 9e68a02a52b1..2fd163cff406 100644 > > > --- a/mm/mremap.c > > > +++ b/mm/mremap.c > > > @@ -191,6 +191,54 @@ static void move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, > > > drop_rmap_locks(vma); > > > } > > > > > > +static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr, > > > + unsigned long new_addr, unsigned long old_end, > > > + pmd_t *old_pmd, pmd_t *new_pmd, bool *need_flush) > > > +{ > > > + spinlock_t *old_ptl, *new_ptl; > > > + struct mm_struct *mm = vma->vm_mm; > > > + > > > + if ((old_addr & ~PMD_MASK) || (new_addr & ~PMD_MASK) > > > + || old_end - old_addr < PMD_SIZE) > > > + return false; > > > + > > > + /* > > > + * The destination pmd shouldn't be established, free_pgtables() > > > + * should have release it. > > > + */ > > > + if (WARN_ON(!pmd_none(*new_pmd))) > > > + return false; > > > + > > > + /* > > > + * We don't have to worry about the ordering of src and dst > > > + * ptlocks because exclusive mmap_sem prevents deadlock. > > > + */ > > > + old_ptl = pmd_lock(vma->vm_mm, old_pmd); > > > + if (old_ptl) { > > > > How can it ever be false? > > > > > + pmd_t pmd; > > > + > > > + new_ptl = pmd_lockptr(mm, new_pmd); > > > Looks like this is largely inspired by move_huge_pmd(), I guess a lot of > the code applies, why not just reuse as much as possible? The same comments > w.r.t mmap_sem helping protect against lock order issues applies as well. pmd_lock() cannot fail, but __pmd_trans_huge_lock() can. We should not copy the code blindly. -- Kirill A. Shutemov ^ permalink raw reply [flat|nested] 52+ messages in thread
* Re: [PATCH 2/4] mm: speed up mremap by 500x on large regions (v2) 2018-10-24 12:57 ` Kirill A. Shutemov @ 2018-10-24 12:57 ` Kirill A. Shutemov 2018-10-25 2:09 ` Joel Fernandes 1 sibling, 0 replies; 52+ messages in thread From: Kirill A. Shutemov @ 2018-10-24 12:57 UTC (permalink / raw) To: Balbir Singh Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Peter Zijlstra, Catalin Marinas, Dave Hansen, Will Deacon, mhocko, linux-mm, lokeshgidra, Joel Fernandes (Google), linux-riscv, elfring, Jonas Bonn, kvmarm, dancol, Yoshinori Sato, sparclinux, linux-xtensa, linux-hexagon, Helge Deller, maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT), hughd, James E.J. Bottomley, kasan-dev, anton.ivanov, Ingo Molnar, Geert Uytterhoeven, Andrey Ryabinin, linux-snps-arc, kernel-team, Sam Creasey, Fenghua Yu, linux-s390, Jeff Dike, linux-um, Stefan Kristiansson, Julia Lawall, linux-m68k, Borislav Petkov, Andy Lutomirski, nios2-dev, Stafford Horne, Guan Xuetao, Chris Zankel, Tony Luck, Richard Weinberger, linux-parisc, pantin, Max Filippov, linux-kernel, minchan, Thomas Gleixner, linux-alpha, Ley Foon Tan, akpm, linuxppc-dev, David S. Miller On Wed, Oct 24, 2018 at 10:57:33PM +1100, Balbir Singh wrote: > On Wed, Oct 24, 2018 at 01:12:56PM +0300, Kirill A. Shutemov wrote: > > On Fri, Oct 12, 2018 at 06:31:58PM -0700, Joel Fernandes (Google) wrote: > > > diff --git a/mm/mremap.c b/mm/mremap.c > > > index 9e68a02a52b1..2fd163cff406 100644 > > > --- a/mm/mremap.c > > > +++ b/mm/mremap.c > > > @@ -191,6 +191,54 @@ static void move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, > > > drop_rmap_locks(vma); > > > } > > > > > > +static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr, > > > + unsigned long new_addr, unsigned long old_end, > > > + pmd_t *old_pmd, pmd_t *new_pmd, bool *need_flush) > > > +{ > > > + spinlock_t *old_ptl, *new_ptl; > > > + struct mm_struct *mm = vma->vm_mm; > > > + > > > + if ((old_addr & ~PMD_MASK) || (new_addr & ~PMD_MASK) > > > + || old_end - old_addr < PMD_SIZE) > > > + return false; > > > + > > > + /* > > > + * The destination pmd shouldn't be established, free_pgtables() > > > + * should have release it. > > > + */ > > > + if (WARN_ON(!pmd_none(*new_pmd))) > > > + return false; > > > + > > > + /* > > > + * We don't have to worry about the ordering of src and dst > > > + * ptlocks because exclusive mmap_sem prevents deadlock. > > > + */ > > > + old_ptl = pmd_lock(vma->vm_mm, old_pmd); > > > + if (old_ptl) { > > > > How can it ever be false? > > > > > + pmd_t pmd; > > > + > > > + new_ptl = pmd_lockptr(mm, new_pmd); > > > Looks like this is largely inspired by move_huge_pmd(), I guess a lot of > the code applies, why not just reuse as much as possible? The same comments > w.r.t mmap_sem helping protect against lock order issues applies as well. pmd_lock() cannot fail, but __pmd_trans_huge_lock() can. We should not copy the code blindly. -- Kirill A. Shutemov _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 52+ messages in thread
* [PATCH 2/4] mm: speed up mremap by 500x on large regions (v2) 2018-10-24 12:57 ` Kirill A. Shutemov 2018-10-24 12:57 ` Kirill A. Shutemov @ 2018-10-25 2:09 ` Joel Fernandes 2018-10-25 2:09 ` Joel Fernandes 2018-10-25 10:19 ` Kirill A. Shutemov 1 sibling, 2 replies; 52+ messages in thread From: Joel Fernandes @ 2018-10-25 2:09 UTC (permalink / raw) To: linux-riscv On Wed, Oct 24, 2018 at 03:57:24PM +0300, Kirill A. Shutemov wrote: > On Wed, Oct 24, 2018 at 10:57:33PM +1100, Balbir Singh wrote: > > On Wed, Oct 24, 2018 at 01:12:56PM +0300, Kirill A. Shutemov wrote: > > > On Fri, Oct 12, 2018 at 06:31:58PM -0700, Joel Fernandes (Google) wrote: > > > > diff --git a/mm/mremap.c b/mm/mremap.c > > > > index 9e68a02a52b1..2fd163cff406 100644 > > > > --- a/mm/mremap.c > > > > +++ b/mm/mremap.c > > > > @@ -191,6 +191,54 @@ static void move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, > > > > drop_rmap_locks(vma); > > > > } > > > > > > > > +static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr, > > > > + unsigned long new_addr, unsigned long old_end, > > > > + pmd_t *old_pmd, pmd_t *new_pmd, bool *need_flush) > > > > +{ > > > > + spinlock_t *old_ptl, *new_ptl; > > > > + struct mm_struct *mm = vma->vm_mm; > > > > + > > > > + if ((old_addr & ~PMD_MASK) || (new_addr & ~PMD_MASK) > > > > + || old_end - old_addr < PMD_SIZE) > > > > + return false; > > > > + > > > > + /* > > > > + * The destination pmd shouldn't be established, free_pgtables() > > > > + * should have release it. > > > > + */ > > > > + if (WARN_ON(!pmd_none(*new_pmd))) > > > > + return false; > > > > + > > > > + /* > > > > + * We don't have to worry about the ordering of src and dst > > > > + * ptlocks because exclusive mmap_sem prevents deadlock. > > > > + */ > > > > + old_ptl = pmd_lock(vma->vm_mm, old_pmd); > > > > + if (old_ptl) { > > > > > > How can it ever be false? Kirill, It cannot, you are right. I'll remove the test. By the way, there are new changes upstream by Linus which flush the TLB before releasing the ptlock instead of after. I'm guessing that patch came about because of reviews of this patch and someone spotted an issue in the existing code :) Anyway the patch in concern is: eb66ae030829 ("mremap: properly flush TLB before releasing the page") I need to rebase on top of that with appropriate modifications, but I worry that this patch will slow down performance since we have to flush at every PMD/PTE move before releasing the ptlock. Where as with my patch, the intention is to flush only at once in the end of move_page_tables. When I tried to flush TLB on every PMD move, it was quite slow on my arm64 device [2]. Further observation [1] is, it seems like the move_huge_pmds and move_ptes code is a bit sub optimal in the sense, we are acquiring and releasing the same ptlock for a bunch of PMDs if the said PMDs are on the same page-table page right? Instead we can do better by acquiring and release the ptlock less often. I think this observation [1] and the frequent TLB flush issue [2] can be solved by acquiring the ptlock once for a bunch of PMDs, move them all, then flush the tlb and then release the ptlock, and then proceed to doing the same thing for the PMDs in the next page-table page. What do you think? - Joel ^ permalink raw reply [flat|nested] 52+ messages in thread
* Re: [PATCH 2/4] mm: speed up mremap by 500x on large regions (v2) 2018-10-25 2:09 ` Joel Fernandes @ 2018-10-25 2:09 ` Joel Fernandes 2018-10-25 10:19 ` Kirill A. Shutemov 1 sibling, 0 replies; 52+ messages in thread From: Joel Fernandes @ 2018-10-25 2:09 UTC (permalink / raw) To: Kirill A. Shutemov Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Peter Zijlstra, Catalin Marinas, Balbir Singh, Dave Hansen, Will Deacon, mhocko, linux-mm, lokeshgidra, sparclinux, linux-riscv, elfring, Jonas Bonn, kvmarm, dancol, Yoshinori Sato, linux-xtensa, linux-hexagon, Helge Deller, maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT), hughd, James E.J. Bottomley, kasan-dev, anton.ivanov, Ingo Molnar, Geert Uytterhoeven, Andrey Ryabinin, linux-snps-arc, kernel-team, Sam Creasey, Fenghua Yu, linux-s390, Jeff Dike, linux-um, Stefan Kristiansson, Julia Lawall, linux-m68k, Borislav Petkov, Andy Lutomirski, nios2-dev, Stafford Horne, Guan Xuetao, Chris Zankel, Tony Luck, Richard Weinberger, linux-parisc, pantin, Max Filippov, linux-kernel, minchan, Thomas Gleixner, linux-alpha, Ley Foon Tan, akpm, linuxppc-dev, David S. Miller On Wed, Oct 24, 2018 at 03:57:24PM +0300, Kirill A. Shutemov wrote: > On Wed, Oct 24, 2018 at 10:57:33PM +1100, Balbir Singh wrote: > > On Wed, Oct 24, 2018 at 01:12:56PM +0300, Kirill A. Shutemov wrote: > > > On Fri, Oct 12, 2018 at 06:31:58PM -0700, Joel Fernandes (Google) wrote: > > > > diff --git a/mm/mremap.c b/mm/mremap.c > > > > index 9e68a02a52b1..2fd163cff406 100644 > > > > --- a/mm/mremap.c > > > > +++ b/mm/mremap.c > > > > @@ -191,6 +191,54 @@ static void move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, > > > > drop_rmap_locks(vma); > > > > } > > > > > > > > +static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr, > > > > + unsigned long new_addr, unsigned long old_end, > > > > + pmd_t *old_pmd, pmd_t *new_pmd, bool *need_flush) > > > > +{ > > > > + spinlock_t *old_ptl, *new_ptl; > > > > + struct mm_struct *mm = vma->vm_mm; > > > > + > > > > + if ((old_addr & ~PMD_MASK) || (new_addr & ~PMD_MASK) > > > > + || old_end - old_addr < PMD_SIZE) > > > > + return false; > > > > + > > > > + /* > > > > + * The destination pmd shouldn't be established, free_pgtables() > > > > + * should have release it. > > > > + */ > > > > + if (WARN_ON(!pmd_none(*new_pmd))) > > > > + return false; > > > > + > > > > + /* > > > > + * We don't have to worry about the ordering of src and dst > > > > + * ptlocks because exclusive mmap_sem prevents deadlock. > > > > + */ > > > > + old_ptl = pmd_lock(vma->vm_mm, old_pmd); > > > > + if (old_ptl) { > > > > > > How can it ever be false? Kirill, It cannot, you are right. I'll remove the test. By the way, there are new changes upstream by Linus which flush the TLB before releasing the ptlock instead of after. I'm guessing that patch came about because of reviews of this patch and someone spotted an issue in the existing code :) Anyway the patch in concern is: eb66ae030829 ("mremap: properly flush TLB before releasing the page") I need to rebase on top of that with appropriate modifications, but I worry that this patch will slow down performance since we have to flush at every PMD/PTE move before releasing the ptlock. Where as with my patch, the intention is to flush only at once in the end of move_page_tables. When I tried to flush TLB on every PMD move, it was quite slow on my arm64 device [2]. Further observation [1] is, it seems like the move_huge_pmds and move_ptes code is a bit sub optimal in the sense, we are acquiring and releasing the same ptlock for a bunch of PMDs if the said PMDs are on the same page-table page right? Instead we can do better by acquiring and release the ptlock less often. I think this observation [1] and the frequent TLB flush issue [2] can be solved by acquiring the ptlock once for a bunch of PMDs, move them all, then flush the tlb and then release the ptlock, and then proceed to doing the same thing for the PMDs in the next page-table page. What do you think? - Joel _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 52+ messages in thread
* [PATCH 2/4] mm: speed up mremap by 500x on large regions (v2) 2018-10-25 2:09 ` Joel Fernandes 2018-10-25 2:09 ` Joel Fernandes @ 2018-10-25 10:19 ` Kirill A. Shutemov 2018-10-25 10:19 ` Kirill A. Shutemov 2018-10-26 21:11 ` Joel Fernandes 1 sibling, 2 replies; 52+ messages in thread From: Kirill A. Shutemov @ 2018-10-25 10:19 UTC (permalink / raw) To: linux-riscv On Wed, Oct 24, 2018 at 07:09:07PM -0700, Joel Fernandes wrote: > On Wed, Oct 24, 2018 at 03:57:24PM +0300, Kirill A. Shutemov wrote: > > On Wed, Oct 24, 2018 at 10:57:33PM +1100, Balbir Singh wrote: > > > On Wed, Oct 24, 2018 at 01:12:56PM +0300, Kirill A. Shutemov wrote: > > > > On Fri, Oct 12, 2018 at 06:31:58PM -0700, Joel Fernandes (Google) wrote: > > > > > diff --git a/mm/mremap.c b/mm/mremap.c > > > > > index 9e68a02a52b1..2fd163cff406 100644 > > > > > --- a/mm/mremap.c > > > > > +++ b/mm/mremap.c > > > > > @@ -191,6 +191,54 @@ static void move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, > > > > > drop_rmap_locks(vma); > > > > > } > > > > > > > > > > +static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr, > > > > > + unsigned long new_addr, unsigned long old_end, > > > > > + pmd_t *old_pmd, pmd_t *new_pmd, bool *need_flush) > > > > > +{ > > > > > + spinlock_t *old_ptl, *new_ptl; > > > > > + struct mm_struct *mm = vma->vm_mm; > > > > > + > > > > > + if ((old_addr & ~PMD_MASK) || (new_addr & ~PMD_MASK) > > > > > + || old_end - old_addr < PMD_SIZE) > > > > > + return false; > > > > > + > > > > > + /* > > > > > + * The destination pmd shouldn't be established, free_pgtables() > > > > > + * should have release it. > > > > > + */ > > > > > + if (WARN_ON(!pmd_none(*new_pmd))) > > > > > + return false; > > > > > + > > > > > + /* > > > > > + * We don't have to worry about the ordering of src and dst > > > > > + * ptlocks because exclusive mmap_sem prevents deadlock. > > > > > + */ > > > > > + old_ptl = pmd_lock(vma->vm_mm, old_pmd); > > > > > + if (old_ptl) { > > > > > > > > How can it ever be false? > > Kirill, > It cannot, you are right. I'll remove the test. > > By the way, there are new changes upstream by Linus which flush the TLB > before releasing the ptlock instead of after. I'm guessing that patch came > about because of reviews of this patch and someone spotted an issue in the > existing code :) > > Anyway the patch in concern is: > eb66ae030829 ("mremap: properly flush TLB before releasing the page") > > I need to rebase on top of that with appropriate modifications, but I worry > that this patch will slow down performance since we have to flush at every > PMD/PTE move before releasing the ptlock. Where as with my patch, the > intention is to flush only at once in the end of move_page_tables. When I > tried to flush TLB on every PMD move, it was quite slow on my arm64 device [2]. > > Further observation [1] is, it seems like the move_huge_pmds and move_ptes code > is a bit sub optimal in the sense, we are acquiring and releasing the same > ptlock for a bunch of PMDs if the said PMDs are on the same page-table page > right? Instead we can do better by acquiring and release the ptlock less > often. > > I think this observation [1] and the frequent TLB flush issue [2] can be solved > by acquiring the ptlock once for a bunch of PMDs, move them all, then flush > the tlb and then release the ptlock, and then proceed to doing the same thing > for the PMDs in the next page-table page. What do you think? Yeah, that's viable optimization. The tricky part is that one PMD page table can have PMD entires of different types: THP, page table that you can move as whole and the one that you cannot (for any reason). If we cannot move the PMD entry as a whole and must go to PTE page table we would need to drop PMD ptl and take PTE ptl (it might be the same lock in some configuations). Also we don't want to take PMD lock unless it's required. I expect it to be not very trivial to get everything right. But take a shot :) -- Kirill A. Shutemov ^ permalink raw reply [flat|nested] 52+ messages in thread
* Re: [PATCH 2/4] mm: speed up mremap by 500x on large regions (v2) 2018-10-25 10:19 ` Kirill A. Shutemov @ 2018-10-25 10:19 ` Kirill A. Shutemov 2018-10-26 21:11 ` Joel Fernandes 1 sibling, 0 replies; 52+ messages in thread From: Kirill A. Shutemov @ 2018-10-25 10:19 UTC (permalink / raw) To: Joel Fernandes Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Peter Zijlstra, Catalin Marinas, Balbir Singh, Dave Hansen, Will Deacon, mhocko, linux-mm, lokeshgidra, sparclinux, linux-riscv, elfring, Jonas Bonn, kvmarm, dancol, Yoshinori Sato, linux-xtensa, linux-hexagon, Helge Deller, maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT), hughd, James E.J. Bottomley, kasan-dev, anton.ivanov, Ingo Molnar, Geert Uytterhoeven, Andrey Ryabinin, linux-snps-arc, kernel-team, Sam Creasey, Fenghua Yu, linux-s390, Jeff Dike, linux-um, Stefan Kristiansson, Julia Lawall, linux-m68k, Borislav Petkov, Andy Lutomirski, nios2-dev, Stafford Horne, Guan Xuetao, Chris Zankel, Tony Luck, Richard Weinberger, linux-parisc, pantin, Max Filippov, linux-kernel, minchan, Thomas Gleixner, linux-alpha, Ley Foon Tan, akpm, linuxppc-dev, David S. Miller On Wed, Oct 24, 2018 at 07:09:07PM -0700, Joel Fernandes wrote: > On Wed, Oct 24, 2018 at 03:57:24PM +0300, Kirill A. Shutemov wrote: > > On Wed, Oct 24, 2018 at 10:57:33PM +1100, Balbir Singh wrote: > > > On Wed, Oct 24, 2018 at 01:12:56PM +0300, Kirill A. Shutemov wrote: > > > > On Fri, Oct 12, 2018 at 06:31:58PM -0700, Joel Fernandes (Google) wrote: > > > > > diff --git a/mm/mremap.c b/mm/mremap.c > > > > > index 9e68a02a52b1..2fd163cff406 100644 > > > > > --- a/mm/mremap.c > > > > > +++ b/mm/mremap.c > > > > > @@ -191,6 +191,54 @@ static void move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, > > > > > drop_rmap_locks(vma); > > > > > } > > > > > > > > > > +static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr, > > > > > + unsigned long new_addr, unsigned long old_end, > > > > > + pmd_t *old_pmd, pmd_t *new_pmd, bool *need_flush) > > > > > +{ > > > > > + spinlock_t *old_ptl, *new_ptl; > > > > > + struct mm_struct *mm = vma->vm_mm; > > > > > + > > > > > + if ((old_addr & ~PMD_MASK) || (new_addr & ~PMD_MASK) > > > > > + || old_end - old_addr < PMD_SIZE) > > > > > + return false; > > > > > + > > > > > + /* > > > > > + * The destination pmd shouldn't be established, free_pgtables() > > > > > + * should have release it. > > > > > + */ > > > > > + if (WARN_ON(!pmd_none(*new_pmd))) > > > > > + return false; > > > > > + > > > > > + /* > > > > > + * We don't have to worry about the ordering of src and dst > > > > > + * ptlocks because exclusive mmap_sem prevents deadlock. > > > > > + */ > > > > > + old_ptl = pmd_lock(vma->vm_mm, old_pmd); > > > > > + if (old_ptl) { > > > > > > > > How can it ever be false? > > Kirill, > It cannot, you are right. I'll remove the test. > > By the way, there are new changes upstream by Linus which flush the TLB > before releasing the ptlock instead of after. I'm guessing that patch came > about because of reviews of this patch and someone spotted an issue in the > existing code :) > > Anyway the patch in concern is: > eb66ae030829 ("mremap: properly flush TLB before releasing the page") > > I need to rebase on top of that with appropriate modifications, but I worry > that this patch will slow down performance since we have to flush at every > PMD/PTE move before releasing the ptlock. Where as with my patch, the > intention is to flush only at once in the end of move_page_tables. When I > tried to flush TLB on every PMD move, it was quite slow on my arm64 device [2]. > > Further observation [1] is, it seems like the move_huge_pmds and move_ptes code > is a bit sub optimal in the sense, we are acquiring and releasing the same > ptlock for a bunch of PMDs if the said PMDs are on the same page-table page > right? Instead we can do better by acquiring and release the ptlock less > often. > > I think this observation [1] and the frequent TLB flush issue [2] can be solved > by acquiring the ptlock once for a bunch of PMDs, move them all, then flush > the tlb and then release the ptlock, and then proceed to doing the same thing > for the PMDs in the next page-table page. What do you think? Yeah, that's viable optimization. The tricky part is that one PMD page table can have PMD entires of different types: THP, page table that you can move as whole and the one that you cannot (for any reason). If we cannot move the PMD entry as a whole and must go to PTE page table we would need to drop PMD ptl and take PTE ptl (it might be the same lock in some configuations). Also we don't want to take PMD lock unless it's required. I expect it to be not very trivial to get everything right. But take a shot :) -- Kirill A. Shutemov _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 52+ messages in thread
* [PATCH 2/4] mm: speed up mremap by 500x on large regions (v2) 2018-10-25 10:19 ` Kirill A. Shutemov 2018-10-25 10:19 ` Kirill A. Shutemov @ 2018-10-26 21:11 ` Joel Fernandes 2018-10-26 21:11 ` Joel Fernandes 2018-10-29 10:28 ` Will Deacon 1 sibling, 2 replies; 52+ messages in thread From: Joel Fernandes @ 2018-10-26 21:11 UTC (permalink / raw) To: linux-riscv On Thu, Oct 25, 2018 at 01:19:00PM +0300, Kirill A. Shutemov wrote: > On Wed, Oct 24, 2018 at 07:09:07PM -0700, Joel Fernandes wrote: > > On Wed, Oct 24, 2018 at 03:57:24PM +0300, Kirill A. Shutemov wrote: > > > On Wed, Oct 24, 2018 at 10:57:33PM +1100, Balbir Singh wrote: > > > > On Wed, Oct 24, 2018 at 01:12:56PM +0300, Kirill A. Shutemov wrote: > > > > > On Fri, Oct 12, 2018 at 06:31:58PM -0700, Joel Fernandes (Google) wrote: > > > > > > diff --git a/mm/mremap.c b/mm/mremap.c > > > > > > index 9e68a02a52b1..2fd163cff406 100644 > > > > > > --- a/mm/mremap.c > > > > > > +++ b/mm/mremap.c > > > > > > @@ -191,6 +191,54 @@ static void move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, > > > > > > drop_rmap_locks(vma); > > > > > > } > > > > > > > > > > > > +static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr, > > > > > > + unsigned long new_addr, unsigned long old_end, > > > > > > + pmd_t *old_pmd, pmd_t *new_pmd, bool *need_flush) > > > > > > +{ > > > > > > + spinlock_t *old_ptl, *new_ptl; > > > > > > + struct mm_struct *mm = vma->vm_mm; > > > > > > + > > > > > > + if ((old_addr & ~PMD_MASK) || (new_addr & ~PMD_MASK) > > > > > > + || old_end - old_addr < PMD_SIZE) > > > > > > + return false; > > > > > > + > > > > > > + /* > > > > > > + * The destination pmd shouldn't be established, free_pgtables() > > > > > > + * should have release it. > > > > > > + */ > > > > > > + if (WARN_ON(!pmd_none(*new_pmd))) > > > > > > + return false; > > > > > > + > > > > > > + /* > > > > > > + * We don't have to worry about the ordering of src and dst > > > > > > + * ptlocks because exclusive mmap_sem prevents deadlock. > > > > > > + */ > > > > > > + old_ptl = pmd_lock(vma->vm_mm, old_pmd); > > > > > > + if (old_ptl) { > > > > > > > > > > How can it ever be false? > > > > Kirill, > > It cannot, you are right. I'll remove the test. > > > > By the way, there are new changes upstream by Linus which flush the TLB > > before releasing the ptlock instead of after. I'm guessing that patch came > > about because of reviews of this patch and someone spotted an issue in the > > existing code :) > > > > Anyway the patch in concern is: > > eb66ae030829 ("mremap: properly flush TLB before releasing the page") > > > > I need to rebase on top of that with appropriate modifications, but I worry > > that this patch will slow down performance since we have to flush at every > > PMD/PTE move before releasing the ptlock. Where as with my patch, the > > intention is to flush only at once in the end of move_page_tables. When I > > tried to flush TLB on every PMD move, it was quite slow on my arm64 device [2]. > > > > Further observation [1] is, it seems like the move_huge_pmds and move_ptes code > > is a bit sub optimal in the sense, we are acquiring and releasing the same > > ptlock for a bunch of PMDs if the said PMDs are on the same page-table page > > right? Instead we can do better by acquiring and release the ptlock less > > often. > > > > I think this observation [1] and the frequent TLB flush issue [2] can be solved > > by acquiring the ptlock once for a bunch of PMDs, move them all, then flush > > the tlb and then release the ptlock, and then proceed to doing the same thing > > for the PMDs in the next page-table page. What do you think? > > Yeah, that's viable optimization. > > The tricky part is that one PMD page table can have PMD entires of > different types: THP, page table that you can move as whole and the one > that you cannot (for any reason). > > If we cannot move the PMD entry as a whole and must go to PTE page table > we would need to drop PMD ptl and take PTE ptl (it might be the same lock > in some configuations). > Also we don't want to take PMD lock unless it's required. > > I expect it to be not very trivial to get everything right. But take a > shot :) Yes, that is exactly the issue I hit when I attempted it. :) The locks need to be release if we do something different on the next loop iteration. It complicates the code and not sure if it is worth it in the long run. On x86 atleast, I don't see any perf issues with the TLB-flush per-PMD move, so the patch is Ok there. On arm64, it negates the performance benefit even though its not any worse than what we are doing currently at the PTE level. My thinking is to take it slow and get the patch in in its current state, since it improves x86. Then as a next step, look into why the arm64 tlb flushes are that expensive and look into optimizing that. On arm64 I am testing on a 4.9 kernel so I'm wondering there are any optimizations since 4.9 that can help speed it up there. After that, if all else fails about speeding up arm64, then I look into developing the cleanest possible solution where we can keep the lock held for longer and flush lesser. thanks, - Joel ^ permalink raw reply [flat|nested] 52+ messages in thread
* Re: [PATCH 2/4] mm: speed up mremap by 500x on large regions (v2) 2018-10-26 21:11 ` Joel Fernandes @ 2018-10-26 21:11 ` Joel Fernandes 2018-10-29 10:28 ` Will Deacon 1 sibling, 0 replies; 52+ messages in thread From: Joel Fernandes @ 2018-10-26 21:11 UTC (permalink / raw) To: Kirill A. Shutemov Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Peter Zijlstra, Catalin Marinas, Balbir Singh, Dave Hansen, Will Deacon, mhocko, linux-mm, lokeshgidra, sparclinux, linux-riscv, elfring, Jonas Bonn, kvmarm, dancol, Yoshinori Sato, linux-xtensa, linux-hexagon, Helge Deller, maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT), hughd, James E.J. Bottomley, kasan-dev, anton.ivanov, Ingo Molnar, Geert Uytterhoeven, Andrey Ryabinin, linux-snps-arc, kernel-team, Sam Creasey, Fenghua Yu, linux-s390, Jeff Dike, linux-um, Stefan Kristiansson, Julia Lawall, linux-m68k, Borislav Petkov, Andy Lutomirski, nios2-dev, Stafford Horne, Guan Xuetao, Chris Zankel, Tony Luck, Richard Weinberger, linux-parisc, pantin, Max Filippov, linux-kernel, minchan, Thomas Gleixner, linux-alpha, Ley Foon Tan, akpm, linuxppc-dev, David S. Miller On Thu, Oct 25, 2018 at 01:19:00PM +0300, Kirill A. Shutemov wrote: > On Wed, Oct 24, 2018 at 07:09:07PM -0700, Joel Fernandes wrote: > > On Wed, Oct 24, 2018 at 03:57:24PM +0300, Kirill A. Shutemov wrote: > > > On Wed, Oct 24, 2018 at 10:57:33PM +1100, Balbir Singh wrote: > > > > On Wed, Oct 24, 2018 at 01:12:56PM +0300, Kirill A. Shutemov wrote: > > > > > On Fri, Oct 12, 2018 at 06:31:58PM -0700, Joel Fernandes (Google) wrote: > > > > > > diff --git a/mm/mremap.c b/mm/mremap.c > > > > > > index 9e68a02a52b1..2fd163cff406 100644 > > > > > > --- a/mm/mremap.c > > > > > > +++ b/mm/mremap.c > > > > > > @@ -191,6 +191,54 @@ static void move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, > > > > > > drop_rmap_locks(vma); > > > > > > } > > > > > > > > > > > > +static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr, > > > > > > + unsigned long new_addr, unsigned long old_end, > > > > > > + pmd_t *old_pmd, pmd_t *new_pmd, bool *need_flush) > > > > > > +{ > > > > > > + spinlock_t *old_ptl, *new_ptl; > > > > > > + struct mm_struct *mm = vma->vm_mm; > > > > > > + > > > > > > + if ((old_addr & ~PMD_MASK) || (new_addr & ~PMD_MASK) > > > > > > + || old_end - old_addr < PMD_SIZE) > > > > > > + return false; > > > > > > + > > > > > > + /* > > > > > > + * The destination pmd shouldn't be established, free_pgtables() > > > > > > + * should have release it. > > > > > > + */ > > > > > > + if (WARN_ON(!pmd_none(*new_pmd))) > > > > > > + return false; > > > > > > + > > > > > > + /* > > > > > > + * We don't have to worry about the ordering of src and dst > > > > > > + * ptlocks because exclusive mmap_sem prevents deadlock. > > > > > > + */ > > > > > > + old_ptl = pmd_lock(vma->vm_mm, old_pmd); > > > > > > + if (old_ptl) { > > > > > > > > > > How can it ever be false? > > > > Kirill, > > It cannot, you are right. I'll remove the test. > > > > By the way, there are new changes upstream by Linus which flush the TLB > > before releasing the ptlock instead of after. I'm guessing that patch came > > about because of reviews of this patch and someone spotted an issue in the > > existing code :) > > > > Anyway the patch in concern is: > > eb66ae030829 ("mremap: properly flush TLB before releasing the page") > > > > I need to rebase on top of that with appropriate modifications, but I worry > > that this patch will slow down performance since we have to flush at every > > PMD/PTE move before releasing the ptlock. Where as with my patch, the > > intention is to flush only at once in the end of move_page_tables. When I > > tried to flush TLB on every PMD move, it was quite slow on my arm64 device [2]. > > > > Further observation [1] is, it seems like the move_huge_pmds and move_ptes code > > is a bit sub optimal in the sense, we are acquiring and releasing the same > > ptlock for a bunch of PMDs if the said PMDs are on the same page-table page > > right? Instead we can do better by acquiring and release the ptlock less > > often. > > > > I think this observation [1] and the frequent TLB flush issue [2] can be solved > > by acquiring the ptlock once for a bunch of PMDs, move them all, then flush > > the tlb and then release the ptlock, and then proceed to doing the same thing > > for the PMDs in the next page-table page. What do you think? > > Yeah, that's viable optimization. > > The tricky part is that one PMD page table can have PMD entires of > different types: THP, page table that you can move as whole and the one > that you cannot (for any reason). > > If we cannot move the PMD entry as a whole and must go to PTE page table > we would need to drop PMD ptl and take PTE ptl (it might be the same lock > in some configuations). > Also we don't want to take PMD lock unless it's required. > > I expect it to be not very trivial to get everything right. But take a > shot :) Yes, that is exactly the issue I hit when I attempted it. :) The locks need to be release if we do something different on the next loop iteration. It complicates the code and not sure if it is worth it in the long run. On x86 atleast, I don't see any perf issues with the TLB-flush per-PMD move, so the patch is Ok there. On arm64, it negates the performance benefit even though its not any worse than what we are doing currently at the PTE level. My thinking is to take it slow and get the patch in in its current state, since it improves x86. Then as a next step, look into why the arm64 tlb flushes are that expensive and look into optimizing that. On arm64 I am testing on a 4.9 kernel so I'm wondering there are any optimizations since 4.9 that can help speed it up there. After that, if all else fails about speeding up arm64, then I look into developing the cleanest possible solution where we can keep the lock held for longer and flush lesser. thanks, - Joel _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 52+ messages in thread
* [PATCH 2/4] mm: speed up mremap by 500x on large regions (v2) 2018-10-26 21:11 ` Joel Fernandes 2018-10-26 21:11 ` Joel Fernandes @ 2018-10-29 10:28 ` Will Deacon 2018-10-29 10:28 ` Will Deacon 1 sibling, 1 reply; 52+ messages in thread From: Will Deacon @ 2018-10-29 10:28 UTC (permalink / raw) To: linux-riscv On Fri, Oct 26, 2018 at 02:11:48PM -0700, Joel Fernandes wrote: > My thinking is to take it slow and get the patch in in its current state, > since it improves x86. Then as a next step, look into why the arm64 tlb > flushes are that expensive and look into optimizing that. On arm64 I am > testing on a 4.9 kernel so I'm wondering there are any optimizations since > 4.9 that can help speed it up there. After that, if all else fails about > speeding up arm64, then I look into developing the cleanest possible solution > where we can keep the lock held for longer and flush lesser. We rewrote a good chunk of the arm64 TLB invalidation and core mmu_gather code this merge window, so please do have another look at -rc1! Will ^ permalink raw reply [flat|nested] 52+ messages in thread
* Re: [PATCH 2/4] mm: speed up mremap by 500x on large regions (v2) 2018-10-29 10:28 ` Will Deacon @ 2018-10-29 10:28 ` Will Deacon 0 siblings, 0 replies; 52+ messages in thread From: Will Deacon @ 2018-10-29 10:28 UTC (permalink / raw) To: Joel Fernandes Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Peter Zijlstra, Catalin Marinas, Balbir Singh, Dave Hansen, mhocko, linux-mm, lokeshgidra, sparclinux, linux-riscv, elfring, Jonas Bonn, kvmarm, dancol, Yoshinori Sato, linux-xtensa, linux-hexagon, Helge Deller, maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT), hughd, James E.J. Bottomley, kasan-dev, anton.ivanov, Ingo Molnar, Geert Uytterhoeven, Andrey Ryabinin, linux-snps-arc, kernel-team, Sam Creasey, Fenghua Yu, linux-s390, Jeff Dike, linux-um, Stefan Kristiansson, Julia Lawall, linux-m68k, Borislav Petkov, Andy Lutomirski, nios2-dev, Kirill A. Shutemov, Stafford Horne, Guan Xuetao, Chris Zankel, Tony Luck, Richard Weinberger, linux-parisc, pantin, Max Filippov, linux-kernel, minchan, Thomas Gleixner, linux-alpha, Ley Foon Tan, akpm, linuxppc-dev, David S. Miller On Fri, Oct 26, 2018 at 02:11:48PM -0700, Joel Fernandes wrote: > My thinking is to take it slow and get the patch in in its current state, > since it improves x86. Then as a next step, look into why the arm64 tlb > flushes are that expensive and look into optimizing that. On arm64 I am > testing on a 4.9 kernel so I'm wondering there are any optimizations since > 4.9 that can help speed it up there. After that, if all else fails about > speeding up arm64, then I look into developing the cleanest possible solution > where we can keep the lock held for longer and flush lesser. We rewrote a good chunk of the arm64 TLB invalidation and core mmu_gather code this merge window, so please do have another look at -rc1! Will _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 52+ messages in thread
* [PATCH 2/4] mm: speed up mremap by 500x on large regions (v2) 2018-10-24 11:57 ` Balbir Singh 2018-10-24 11:57 ` Balbir Singh 2018-10-24 12:57 ` Kirill A. Shutemov @ 2018-10-25 2:13 ` Joel Fernandes 2018-10-25 2:13 ` Joel Fernandes 2018-10-27 10:21 ` Balbir Singh 2 siblings, 2 replies; 52+ messages in thread From: Joel Fernandes @ 2018-10-25 2:13 UTC (permalink / raw) To: linux-riscv On Wed, Oct 24, 2018 at 10:57:33PM +1100, Balbir Singh wrote: [...] > > > + pmd_t pmd; > > > + > > > + new_ptl = pmd_lockptr(mm, new_pmd); > > > Looks like this is largely inspired by move_huge_pmd(), I guess a lot of > the code applies, why not just reuse as much as possible? The same comments > w.r.t mmap_sem helping protect against lock order issues applies as well. I thought about this and when I looked into it, it seemed there are subtle differences that make such sharing not worth it (or not possible). - Joel ^ permalink raw reply [flat|nested] 52+ messages in thread
* Re: [PATCH 2/4] mm: speed up mremap by 500x on large regions (v2) 2018-10-25 2:13 ` Joel Fernandes @ 2018-10-25 2:13 ` Joel Fernandes 2018-10-27 10:21 ` Balbir Singh 1 sibling, 0 replies; 52+ messages in thread From: Joel Fernandes @ 2018-10-25 2:13 UTC (permalink / raw) To: Balbir Singh Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Peter Zijlstra, Catalin Marinas, Dave Hansen, Will Deacon, mhocko, linux-mm, lokeshgidra, sparclinux, linux-riscv, elfring, Jonas Bonn, kvmarm, dancol, Yoshinori Sato, linux-xtensa, linux-hexagon, Helge Deller, maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT), hughd, James E.J. Bottomley, kasan-dev, anton.ivanov, Ingo Molnar, Geert Uytterhoeven, Andrey Ryabinin, linux-snps-arc, kernel-team, Sam Creasey, Fenghua Yu, linux-s390, Jeff Dike, linux-um, Stefan Kristiansson, Julia Lawall, linux-m68k, Borislav Petkov, Andy Lutomirski, nios2-dev, Kirill A. Shutemov, Stafford Horne, Guan Xuetao, Chris Zankel, Tony Luck, Richard Weinberger, linux-parisc, pantin, Max Filippov, linux-kernel, minchan, Thomas Gleixner, linux-alpha, Ley Foon Tan, akpm, linuxppc-dev, David S. Miller On Wed, Oct 24, 2018 at 10:57:33PM +1100, Balbir Singh wrote: [...] > > > + pmd_t pmd; > > > + > > > + new_ptl = pmd_lockptr(mm, new_pmd); > > > Looks like this is largely inspired by move_huge_pmd(), I guess a lot of > the code applies, why not just reuse as much as possible? The same comments > w.r.t mmap_sem helping protect against lock order issues applies as well. I thought about this and when I looked into it, it seemed there are subtle differences that make such sharing not worth it (or not possible). - Joel _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 52+ messages in thread
* [PATCH 2/4] mm: speed up mremap by 500x on large regions (v2) 2018-10-25 2:13 ` Joel Fernandes 2018-10-25 2:13 ` Joel Fernandes @ 2018-10-27 10:21 ` Balbir Singh 2018-10-27 10:21 ` Balbir Singh 2018-10-27 19:39 ` Joel Fernandes 1 sibling, 2 replies; 52+ messages in thread From: Balbir Singh @ 2018-10-27 10:21 UTC (permalink / raw) To: linux-riscv On Wed, Oct 24, 2018 at 07:13:50PM -0700, Joel Fernandes wrote: > On Wed, Oct 24, 2018 at 10:57:33PM +1100, Balbir Singh wrote: > [...] > > > > + pmd_t pmd; > > > > + > > > > + new_ptl = pmd_lockptr(mm, new_pmd); > > > > > > Looks like this is largely inspired by move_huge_pmd(), I guess a lot of > > the code applies, why not just reuse as much as possible? The same comments > > w.r.t mmap_sem helping protect against lock order issues applies as well. > > I thought about this and when I looked into it, it seemed there are subtle > differences that make such sharing not worth it (or not possible). > Could you elaborate on them? Thanks, Balbir Singh. ^ permalink raw reply [flat|nested] 52+ messages in thread
* Re: [PATCH 2/4] mm: speed up mremap by 500x on large regions (v2) 2018-10-27 10:21 ` Balbir Singh @ 2018-10-27 10:21 ` Balbir Singh 2018-10-27 19:39 ` Joel Fernandes 1 sibling, 0 replies; 52+ messages in thread From: Balbir Singh @ 2018-10-27 10:21 UTC (permalink / raw) To: Joel Fernandes Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Peter Zijlstra, Catalin Marinas, Dave Hansen, Will Deacon, mhocko, linux-mm, lokeshgidra, sparclinux, linux-riscv, elfring, Jonas Bonn, kvmarm, dancol, Yoshinori Sato, linux-xtensa, linux-hexagon, Helge Deller, maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT), hughd, James E.J. Bottomley, kasan-dev, anton.ivanov, Ingo Molnar, Geert Uytterhoeven, Andrey Ryabinin, linux-snps-arc, kernel-team, Sam Creasey, Fenghua Yu, linux-s390, Jeff Dike, linux-um, Stefan Kristiansson, Julia Lawall, linux-m68k, Borislav Petkov, Andy Lutomirski, nios2-dev, Kirill A. Shutemov, Stafford Horne, Guan Xuetao, Chris Zankel, Tony Luck, Richard Weinberger, linux-parisc, pantin, Max Filippov, linux-kernel, minchan, Thomas Gleixner, linux-alpha, Ley Foon Tan, akpm, linuxppc-dev, David S. Miller On Wed, Oct 24, 2018 at 07:13:50PM -0700, Joel Fernandes wrote: > On Wed, Oct 24, 2018 at 10:57:33PM +1100, Balbir Singh wrote: > [...] > > > > + pmd_t pmd; > > > > + > > > > + new_ptl = pmd_lockptr(mm, new_pmd); > > > > > > Looks like this is largely inspired by move_huge_pmd(), I guess a lot of > > the code applies, why not just reuse as much as possible? The same comments > > w.r.t mmap_sem helping protect against lock order issues applies as well. > > I thought about this and when I looked into it, it seemed there are subtle > differences that make such sharing not worth it (or not possible). > Could you elaborate on them? Thanks, Balbir Singh. _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 52+ messages in thread
* [PATCH 2/4] mm: speed up mremap by 500x on large regions (v2) 2018-10-27 10:21 ` Balbir Singh 2018-10-27 10:21 ` Balbir Singh @ 2018-10-27 19:39 ` Joel Fernandes 2018-10-27 19:39 ` Joel Fernandes 2018-10-28 22:40 ` Balbir Singh 1 sibling, 2 replies; 52+ messages in thread From: Joel Fernandes @ 2018-10-27 19:39 UTC (permalink / raw) To: linux-riscv Hi Balbir, On Sat, Oct 27, 2018 at 09:21:02PM +1100, Balbir Singh wrote: > On Wed, Oct 24, 2018 at 07:13:50PM -0700, Joel Fernandes wrote: > > On Wed, Oct 24, 2018 at 10:57:33PM +1100, Balbir Singh wrote: > > [...] > > > > > + pmd_t pmd; > > > > > + > > > > > + new_ptl = pmd_lockptr(mm, new_pmd); > > > > > > > > > Looks like this is largely inspired by move_huge_pmd(), I guess a lot of > > > the code applies, why not just reuse as much as possible? The same comments > > > w.r.t mmap_sem helping protect against lock order issues applies as well. > > > > I thought about this and when I looked into it, it seemed there are subtle > > differences that make such sharing not worth it (or not possible). > > > > Could you elaborate on them? The move_huge_page function is defined only for CONFIG_TRANSPARENT_HUGEPAGE so we cannot reuse it to begin with, since we have it disabled on our systems. I am not sure if it is a good idea to split that out and refactor it for reuse especially since our case is quite simple compared to huge pages. There are also a couple of subtle differences between the move_normal_pmd and the move_huge_pmd. Atleast 2 of them are: 1. We don't concern ourself with the PMD dirty bit, since the pages being moved are normal pages and at the soft-dirty bit accounting is at the PTE level, since we are not moving PTEs, we don't need to do that. 2. The locking is simpler as Kirill pointed, pmd_lock cannot fail however __pmd_trans_huge_lock can. I feel it is not super useful to refactor move_huge_pmd to support our case especially since move_normal_pmd is quite small, so IMHO the benefit of code reuse isn't there very much. Do let me know your thoughts and thanks for your interest in this. thanks, - Joel ^ permalink raw reply [flat|nested] 52+ messages in thread
* Re: [PATCH 2/4] mm: speed up mremap by 500x on large regions (v2) 2018-10-27 19:39 ` Joel Fernandes @ 2018-10-27 19:39 ` Joel Fernandes 2018-10-28 22:40 ` Balbir Singh 1 sibling, 0 replies; 52+ messages in thread From: Joel Fernandes @ 2018-10-27 19:39 UTC (permalink / raw) To: Balbir Singh Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Peter Zijlstra, Catalin Marinas, Dave Hansen, Will Deacon, mhocko, linux-mm, lokeshgidra, sparclinux, linux-riscv, elfring, Jonas Bonn, kvmarm, dancol, Yoshinori Sato, linux-xtensa, linux-hexagon, Helge Deller, maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT), hughd, James E.J. Bottomley, kasan-dev, anton.ivanov, Ingo Molnar, Geert Uytterhoeven, Andrey Ryabinin, linux-snps-arc, kernel-team, Sam Creasey, Fenghua Yu, linux-s390, Jeff Dike, linux-um, Stefan Kristiansson, Julia Lawall, linux-m68k, Borislav Petkov, Andy Lutomirski, nios2-dev, Kirill A. Shutemov, Stafford Horne, Guan Xuetao, Chris Zankel, Tony Luck, Richard Weinberger, linux-parisc, pantin, Max Filippov, linux-kernel, minchan, Thomas Gleixner, linux-alpha, Ley Foon Tan, akpm, linuxppc-dev, David S. Miller Hi Balbir, On Sat, Oct 27, 2018 at 09:21:02PM +1100, Balbir Singh wrote: > On Wed, Oct 24, 2018 at 07:13:50PM -0700, Joel Fernandes wrote: > > On Wed, Oct 24, 2018 at 10:57:33PM +1100, Balbir Singh wrote: > > [...] > > > > > + pmd_t pmd; > > > > > + > > > > > + new_ptl = pmd_lockptr(mm, new_pmd); > > > > > > > > > Looks like this is largely inspired by move_huge_pmd(), I guess a lot of > > > the code applies, why not just reuse as much as possible? The same comments > > > w.r.t mmap_sem helping protect against lock order issues applies as well. > > > > I thought about this and when I looked into it, it seemed there are subtle > > differences that make such sharing not worth it (or not possible). > > > > Could you elaborate on them? The move_huge_page function is defined only for CONFIG_TRANSPARENT_HUGEPAGE so we cannot reuse it to begin with, since we have it disabled on our systems. I am not sure if it is a good idea to split that out and refactor it for reuse especially since our case is quite simple compared to huge pages. There are also a couple of subtle differences between the move_normal_pmd and the move_huge_pmd. Atleast 2 of them are: 1. We don't concern ourself with the PMD dirty bit, since the pages being moved are normal pages and at the soft-dirty bit accounting is at the PTE level, since we are not moving PTEs, we don't need to do that. 2. The locking is simpler as Kirill pointed, pmd_lock cannot fail however __pmd_trans_huge_lock can. I feel it is not super useful to refactor move_huge_pmd to support our case especially since move_normal_pmd is quite small, so IMHO the benefit of code reuse isn't there very much. Do let me know your thoughts and thanks for your interest in this. thanks, - Joel _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 52+ messages in thread
* [PATCH 2/4] mm: speed up mremap by 500x on large regions (v2) 2018-10-27 19:39 ` Joel Fernandes 2018-10-27 19:39 ` Joel Fernandes @ 2018-10-28 22:40 ` Balbir Singh 2018-10-28 22:40 ` Balbir Singh 1 sibling, 1 reply; 52+ messages in thread From: Balbir Singh @ 2018-10-28 22:40 UTC (permalink / raw) To: linux-riscv On Sat, Oct 27, 2018 at 12:39:17PM -0700, Joel Fernandes wrote: > Hi Balbir, > > On Sat, Oct 27, 2018 at 09:21:02PM +1100, Balbir Singh wrote: > > On Wed, Oct 24, 2018 at 07:13:50PM -0700, Joel Fernandes wrote: > > > On Wed, Oct 24, 2018 at 10:57:33PM +1100, Balbir Singh wrote: > > > [...] > > > > > > + pmd_t pmd; > > > > > > + > > > > > > + new_ptl = pmd_lockptr(mm, new_pmd); > > > > > > > > > > > > Looks like this is largely inspired by move_huge_pmd(), I guess a lot of > > > > the code applies, why not just reuse as much as possible? The same comments > > > > w.r.t mmap_sem helping protect against lock order issues applies as well. > > > > > > I thought about this and when I looked into it, it seemed there are subtle > > > differences that make such sharing not worth it (or not possible). > > > > > > > Could you elaborate on them? > > The move_huge_page function is defined only for CONFIG_TRANSPARENT_HUGEPAGE > so we cannot reuse it to begin with, since we have it disabled on our > systems. I am not sure if it is a good idea to split that out and refactor it > for reuse especially since our case is quite simple compared to huge pages. > > There are also a couple of subtle differences between the move_normal_pmd and > the move_huge_pmd. Atleast 2 of them are: > > 1. We don't concern ourself with the PMD dirty bit, since the pages being > moved are normal pages and at the soft-dirty bit accounting is at the PTE > level, since we are not moving PTEs, we don't need to do that. > > 2. The locking is simpler as Kirill pointed, pmd_lock cannot fail however > __pmd_trans_huge_lock can. > > I feel it is not super useful to refactor move_huge_pmd to support our case > especially since move_normal_pmd is quite small, so IMHO the benefit of code > reuse isn't there very much. > My big concern is that any bug fixes will need to monitor both paths. Do you see a big overhead in checking the soft dirty bit? The locking is a little different. Having said that, I am not strictly opposed to the extra code, just concerned about missing fixes/updates as we find them. > Do let me know your thoughts and thanks for your interest in this. > > Balbir Singh. ^ permalink raw reply [flat|nested] 52+ messages in thread
* Re: [PATCH 2/4] mm: speed up mremap by 500x on large regions (v2) 2018-10-28 22:40 ` Balbir Singh @ 2018-10-28 22:40 ` Balbir Singh 0 siblings, 0 replies; 52+ messages in thread From: Balbir Singh @ 2018-10-28 22:40 UTC (permalink / raw) To: Joel Fernandes Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Peter Zijlstra, Catalin Marinas, Dave Hansen, Will Deacon, mhocko, linux-mm, lokeshgidra, sparclinux, linux-riscv, elfring, Jonas Bonn, kvmarm, dancol, Yoshinori Sato, linux-xtensa, linux-hexagon, Helge Deller, maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT), hughd, James E.J. Bottomley, kasan-dev, anton.ivanov, Ingo Molnar, Geert Uytterhoeven, Andrey Ryabinin, linux-snps-arc, kernel-team, Sam Creasey, Fenghua Yu, linux-s390, Jeff Dike, linux-um, Stefan Kristiansson, Julia Lawall, linux-m68k, Borislav Petkov, Andy Lutomirski, nios2-dev, Kirill A. Shutemov, Stafford Horne, Guan Xuetao, Chris Zankel, Tony Luck, Richard Weinberger, linux-parisc, pantin, Max Filippov, linux-kernel, minchan, Thomas Gleixner, linux-alpha, Ley Foon Tan, akpm, linuxppc-dev, David S. Miller On Sat, Oct 27, 2018 at 12:39:17PM -0700, Joel Fernandes wrote: > Hi Balbir, > > On Sat, Oct 27, 2018 at 09:21:02PM +1100, Balbir Singh wrote: > > On Wed, Oct 24, 2018 at 07:13:50PM -0700, Joel Fernandes wrote: > > > On Wed, Oct 24, 2018 at 10:57:33PM +1100, Balbir Singh wrote: > > > [...] > > > > > > + pmd_t pmd; > > > > > > + > > > > > > + new_ptl = pmd_lockptr(mm, new_pmd); > > > > > > > > > > > > Looks like this is largely inspired by move_huge_pmd(), I guess a lot of > > > > the code applies, why not just reuse as much as possible? The same comments > > > > w.r.t mmap_sem helping protect against lock order issues applies as well. > > > > > > I thought about this and when I looked into it, it seemed there are subtle > > > differences that make such sharing not worth it (or not possible). > > > > > > > Could you elaborate on them? > > The move_huge_page function is defined only for CONFIG_TRANSPARENT_HUGEPAGE > so we cannot reuse it to begin with, since we have it disabled on our > systems. I am not sure if it is a good idea to split that out and refactor it > for reuse especially since our case is quite simple compared to huge pages. > > There are also a couple of subtle differences between the move_normal_pmd and > the move_huge_pmd. Atleast 2 of them are: > > 1. We don't concern ourself with the PMD dirty bit, since the pages being > moved are normal pages and at the soft-dirty bit accounting is at the PTE > level, since we are not moving PTEs, we don't need to do that. > > 2. The locking is simpler as Kirill pointed, pmd_lock cannot fail however > __pmd_trans_huge_lock can. > > I feel it is not super useful to refactor move_huge_pmd to support our case > especially since move_normal_pmd is quite small, so IMHO the benefit of code > reuse isn't there very much. > My big concern is that any bug fixes will need to monitor both paths. Do you see a big overhead in checking the soft dirty bit? The locking is a little different. Having said that, I am not strictly opposed to the extra code, just concerned about missing fixes/updates as we find them. > Do let me know your thoughts and thanks for your interest in this. > > Balbir Singh. _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply [flat|nested] 52+ messages in thread
* [PATCH 3/4] arm64: select HAVE_MOVE_PMD for faster mremap (v1) 2018-10-13 1:31 [PATCH 0/4] Add support for fast mremap Joel Fernandes (Google) ` (2 preceding siblings ...) 2018-10-13 1:31 ` [PATCH 2/4] mm: speed up mremap by 500x on large regions (v2) Joel Fernandes (Google) @ 2018-10-13 1:31 ` Joel Fernandes (Google) 2018-10-13 1:31 ` Joel Fernandes (Google) 2018-10-13 1:32 ` [PATCH 4/4] x86: " Joel Fernandes (Google) 4 siblings, 1 reply; 52+ messages in thread From: Joel Fernandes (Google) @ 2018-10-13 1:31 UTC (permalink / raw) To: linux-riscv Moving page-tables at the PMD-level on arm64 is known to be safe. Enable this option so that we can do fast mremap when possible. Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org> --- arch/arm64/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 1b1a0e95c751..5d7c35c6f90c 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -135,6 +135,7 @@ config ARM64 select HAVE_IRQ_TIME_ACCOUNTING select HAVE_MEMBLOCK select HAVE_MEMBLOCK_NODE_MAP if NUMA + select HAVE_MOVE_PMD select HAVE_NMI select HAVE_PATA_PLATFORM select HAVE_PERF_EVENTS -- 2.19.0.605.g01d371f741-goog ^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH 3/4] arm64: select HAVE_MOVE_PMD for faster mremap (v1) 2018-10-13 1:31 ` [PATCH 3/4] arm64: select HAVE_MOVE_PMD for faster mremap (v1) Joel Fernandes (Google) @ 2018-10-13 1:31 ` Joel Fernandes (Google) 0 siblings, 0 replies; 52+ messages in thread From: Joel Fernandes (Google) @ 2018-10-13 1:31 UTC (permalink / raw) To: linux-kernel Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Peter Zijlstra, Catalin Marinas, Dave Hansen, Will Deacon, mhocko, linux-mm, lokeshgidra, Joel Fernandes (Google), linux-riscv, elfring, Jonas Bonn, kvmarm, dancol, Yoshinori Sato, sparclinux, linux-xtensa, linux-hexagon, Helge Deller, maintainer:X86 ARCHITECTURE 32-BIT AND 64-BIT, hughd, James E.J. Bottomley, kasan-dev, anton.ivanov, Ingo Molnar, Geert Uytterhoeven, Andrey Ryabinin, linux-snps-arc, kernel-team, Sam Creasey, Fenghua Yu, linux-s390, Jeff Dike, linux-um, Stefan Kristiansson, Julia Lawall, linux-m68k, Borislav Petkov, Andy Lutomirski, nios2-dev, Kirill A. Shutemov, Stafford Horne, Guan Xuetao, Chris Zankel, Tony Luck, Richard Weinberger, linux-parisc, Max Filippov, pantin, minchan, Thomas Gleixner, linux-alpha, Ley Foon Tan, akpm, linuxppc-dev, David S. Miller Moving page-tables at the PMD-level on arm64 is known to be safe. Enable this option so that we can do fast mremap when possible. Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org> --- arch/arm64/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 1b1a0e95c751..5d7c35c6f90c 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -135,6 +135,7 @@ config ARM64 select HAVE_IRQ_TIME_ACCOUNTING select HAVE_MEMBLOCK select HAVE_MEMBLOCK_NODE_MAP if NUMA + select HAVE_MOVE_PMD select HAVE_NMI select HAVE_PATA_PLATFORM select HAVE_PERF_EVENTS -- 2.19.0.605.g01d371f741-goog _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH 4/4] x86: select HAVE_MOVE_PMD for faster mremap (v1) 2018-10-13 1:31 [PATCH 0/4] Add support for fast mremap Joel Fernandes (Google) ` (3 preceding siblings ...) 2018-10-13 1:31 ` [PATCH 3/4] arm64: select HAVE_MOVE_PMD for faster mremap (v1) Joel Fernandes (Google) @ 2018-10-13 1:32 ` Joel Fernandes (Google) 2018-10-13 1:32 ` Joel Fernandes (Google) 4 siblings, 1 reply; 52+ messages in thread From: Joel Fernandes (Google) @ 2018-10-13 1:32 UTC (permalink / raw) To: linux-riscv Moving page-tables at the PMD-level on x86 is known to be safe. Enable this option so that we can do fast mremap when possible. Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org> --- arch/x86/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 1a0be022f91d..01c02a9d7825 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -171,6 +171,7 @@ config X86 select HAVE_MEMBLOCK_NODE_MAP select HAVE_MIXED_BREAKPOINTS_REGS select HAVE_MOD_ARCH_SPECIFIC + select HAVE_MOVE_PMD select HAVE_NMI select HAVE_OPROFILE select HAVE_OPTPROBES -- 2.19.0.605.g01d371f741-goog ^ permalink raw reply related [flat|nested] 52+ messages in thread
* [PATCH 4/4] x86: select HAVE_MOVE_PMD for faster mremap (v1) 2018-10-13 1:32 ` [PATCH 4/4] x86: " Joel Fernandes (Google) @ 2018-10-13 1:32 ` Joel Fernandes (Google) 0 siblings, 0 replies; 52+ messages in thread From: Joel Fernandes (Google) @ 2018-10-13 1:32 UTC (permalink / raw) To: linux-kernel Cc: linux-mips, Rich Felker, linux-ia64, linux-sh, Peter Zijlstra, Catalin Marinas, Dave Hansen, Will Deacon, mhocko, linux-mm, lokeshgidra, Joel Fernandes (Google), linux-riscv, elfring, Jonas Bonn, kvmarm, dancol, Yoshinori Sato, sparclinux, linux-xtensa, linux-hexagon, Helge Deller, maintainer:X86 ARCHITECTURE 32-BIT AND 64-BIT, hughd, James E.J. Bottomley, kasan-dev, anton.ivanov, Ingo Molnar, Geert Uytterhoeven, Andrey Ryabinin, linux-snps-arc, kernel-team, Sam Creasey, Fenghua Yu, linux-s390, Jeff Dike, linux-um, Stefan Kristiansson, Julia Lawall, linux-m68k, Borislav Petkov, Andy Lutomirski, nios2-dev, Kirill A. Shutemov, Stafford Horne, Guan Xuetao, Chris Zankel, Tony Luck, Richard Weinberger, linux-parisc, Max Filippov, pantin, minchan, Thomas Gleixner, linux-alpha, Ley Foon Tan, akpm, linuxppc-dev, David S. Miller Moving page-tables at the PMD-level on x86 is known to be safe. Enable this option so that we can do fast mremap when possible. Signed-off-by: Joel Fernandes (Google) <joel@joelfernandes.org> --- arch/x86/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 1a0be022f91d..01c02a9d7825 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -171,6 +171,7 @@ config X86 select HAVE_MEMBLOCK_NODE_MAP select HAVE_MIXED_BREAKPOINTS_REGS select HAVE_MOD_ARCH_SPECIFIC + select HAVE_MOVE_PMD select HAVE_NMI select HAVE_OPROFILE select HAVE_OPTPROBES -- 2.19.0.605.g01d371f741-goog _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv ^ permalink raw reply related [flat|nested] 52+ messages in thread
end of thread, other threads:[~2018-10-29 10:28 UTC | newest] Thread overview: 52+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-10-13 1:31 [PATCH 0/4] Add support for fast mremap Joel Fernandes (Google) 2018-10-13 1:31 ` Joel Fernandes (Google) 2018-10-13 1:31 ` [PATCH 1/4] treewide: remove unused address argument from pte_alloc functions (v2) Joel Fernandes (Google) 2018-10-13 1:31 ` Joel Fernandes (Google) 2018-10-24 8:37 ` Peter Zijlstra 2018-10-24 8:37 ` Peter Zijlstra 2018-10-25 2:21 ` Joel Fernandes 2018-10-25 2:21 ` Joel Fernandes 2018-10-26 8:52 ` Peter Zijlstra 2018-10-26 8:52 ` Peter Zijlstra 2018-10-25 10:47 ` Kirill A. Shutemov 2018-10-25 10:47 ` Kirill A. Shutemov 2018-10-26 8:50 ` Peter Zijlstra 2018-10-26 8:50 ` Peter Zijlstra 2018-10-13 1:31 ` [PATCH 2/4] mm: speed up mremap by 500x on large regions (v2) Joel Fernandes (Google) 2018-10-13 1:31 ` Joel Fernandes (Google) 2018-10-15 9:42 ` Christoph Hellwig 2018-10-15 9:42 ` Christoph Hellwig 2018-10-15 22:33 ` Joel Fernandes 2018-10-15 22:33 ` Joel Fernandes 2018-10-16 11:29 ` Vlastimil Babka 2018-10-16 11:29 ` Vlastimil Babka 2018-10-16 19:43 ` Joel Fernandes 2018-10-16 19:43 ` Joel Fernandes 2018-10-17 7:38 ` Vlastimil Babka 2018-10-17 7:38 ` Vlastimil Babka 2018-10-24 10:12 ` Kirill A. Shutemov 2018-10-24 10:12 ` Kirill A. Shutemov 2018-10-24 11:57 ` Balbir Singh 2018-10-24 11:57 ` Balbir Singh 2018-10-24 12:57 ` Kirill A. Shutemov 2018-10-24 12:57 ` Kirill A. Shutemov 2018-10-25 2:09 ` Joel Fernandes 2018-10-25 2:09 ` Joel Fernandes 2018-10-25 10:19 ` Kirill A. Shutemov 2018-10-25 10:19 ` Kirill A. Shutemov 2018-10-26 21:11 ` Joel Fernandes 2018-10-26 21:11 ` Joel Fernandes 2018-10-29 10:28 ` Will Deacon 2018-10-29 10:28 ` Will Deacon 2018-10-25 2:13 ` Joel Fernandes 2018-10-25 2:13 ` Joel Fernandes 2018-10-27 10:21 ` Balbir Singh 2018-10-27 10:21 ` Balbir Singh 2018-10-27 19:39 ` Joel Fernandes 2018-10-27 19:39 ` Joel Fernandes 2018-10-28 22:40 ` Balbir Singh 2018-10-28 22:40 ` Balbir Singh 2018-10-13 1:31 ` [PATCH 3/4] arm64: select HAVE_MOVE_PMD for faster mremap (v1) Joel Fernandes (Google) 2018-10-13 1:31 ` Joel Fernandes (Google) 2018-10-13 1:32 ` [PATCH 4/4] x86: " Joel Fernandes (Google) 2018-10-13 1:32 ` Joel Fernandes (Google)
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).