From: Nicholas Piggin <npiggin@gmail.com> To: linux-mm@kvack.org, Andrew Morton <akpm@linux-foundation.org> Cc: Nicholas Piggin <npiggin@gmail.com>, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, Jonathan Cameron <Jonathan.Cameron@Huawei.com>, Christoph Hellwig <hch@infradead.org>, Christophe Leroy <christophe.leroy@csgroup.eu>, Rick Edgecombe <rick.p.edgecombe@intel.com>, Ding Tianhong <dingtianhong@huawei.com>, linuxppc-dev@lists.ozlabs.org, Michael Ellerman <mpe@ellerman.id.au> Subject: [PATCH v13 14/14] powerpc/64s/radix: Enable huge vmalloc mappings Date: Wed, 17 Mar 2021 16:24:02 +1000 [thread overview] Message-ID: <20210317062402.533919-15-npiggin@gmail.com> (raw) In-Reply-To: <20210317062402.533919-1-npiggin@gmail.com> This reduces TLB misses by nearly 30x on a `git diff` workload on a 2-node POWER9 (59,800 -> 2,100) and reduces CPU cycles by 0.54%, due to vfs hashes being allocated with 2MB pages. Cc: linuxppc-dev@lists.ozlabs.org Acked-by: Michael Ellerman <mpe@ellerman.id.au> Signed-off-by: Nicholas Piggin <npiggin@gmail.com> --- .../admin-guide/kernel-parameters.txt | 2 ++ arch/powerpc/Kconfig | 1 + arch/powerpc/kernel/module.c | 22 +++++++++++++++---- 3 files changed, 21 insertions(+), 4 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 04545725f187..1f481f904895 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -3243,6 +3243,8 @@ nohugeiomap [KNL,X86,PPC,ARM64] Disable kernel huge I/O mappings. + nohugevmalloc [PPC] Disable kernel huge vmalloc mappings. + nosmt [KNL,S390] Disable symmetric multithreading (SMT). Equivalent to smt=1. diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index 386ae12d8523..b7cade9566da 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -181,6 +181,7 @@ config PPC select GENERIC_GETTIMEOFDAY select HAVE_ARCH_AUDITSYSCALL select HAVE_ARCH_HUGE_VMAP if PPC_BOOK3S_64 && PPC_RADIX_MMU + select HAVE_ARCH_HUGE_VMALLOC if HAVE_ARCH_HUGE_VMAP select HAVE_ARCH_JUMP_LABEL select HAVE_ARCH_KASAN if PPC32 && PPC_PAGE_SHIFT <= 14 select HAVE_ARCH_KASAN_VMALLOC if PPC32 && PPC_PAGE_SHIFT <= 14 diff --git a/arch/powerpc/kernel/module.c b/arch/powerpc/kernel/module.c index a211b0253cdb..cdb2d88c54e7 100644 --- a/arch/powerpc/kernel/module.c +++ b/arch/powerpc/kernel/module.c @@ -8,6 +8,7 @@ #include <linux/moduleloader.h> #include <linux/err.h> #include <linux/vmalloc.h> +#include <linux/mm.h> #include <linux/bug.h> #include <asm/module.h> #include <linux/uaccess.h> @@ -87,13 +88,26 @@ int module_finalize(const Elf_Ehdr *hdr, return 0; } -#ifdef MODULES_VADDR void *module_alloc(unsigned long size) { + unsigned long start = VMALLOC_START; + unsigned long end = VMALLOC_END; + +#ifdef MODULES_VADDR BUILD_BUG_ON(TASK_SIZE > MODULES_VADDR); + start = MODULES_VADDR; + end = MODULES_END; +#endif + + /* + * Don't do huge page allocations for modules yet until more testing + * is done. STRICT_MODULE_RWX may require extra work to support this + * too. + */ - return __vmalloc_node_range(size, 1, MODULES_VADDR, MODULES_END, GFP_KERNEL, - PAGE_KERNEL_EXEC, VM_FLUSH_RESET_PERMS, NUMA_NO_NODE, + return __vmalloc_node_range(size, 1, start, end, GFP_KERNEL, + PAGE_KERNEL_EXEC, + VM_NO_HUGE_VMAP | VM_FLUSH_RESET_PERMS, + NUMA_NO_NODE, __builtin_return_address(0)); } -#endif -- 2.23.0
WARNING: multiple messages have this Message-ID (diff)
From: Nicholas Piggin <npiggin@gmail.com> To: linux-mm@kvack.org, Andrew Morton <akpm@linux-foundation.org> Cc: linux-arch@vger.kernel.org, Ding Tianhong <dingtianhong@huawei.com>, linux-kernel@vger.kernel.org, Christoph Hellwig <hch@infradead.org>, Nicholas Piggin <npiggin@gmail.com>, Jonathan Cameron <Jonathan.Cameron@Huawei.com>, Rick Edgecombe <rick.p.edgecombe@intel.com>, linuxppc-dev@lists.ozlabs.org Subject: [PATCH v13 14/14] powerpc/64s/radix: Enable huge vmalloc mappings Date: Wed, 17 Mar 2021 16:24:02 +1000 [thread overview] Message-ID: <20210317062402.533919-15-npiggin@gmail.com> (raw) In-Reply-To: <20210317062402.533919-1-npiggin@gmail.com> This reduces TLB misses by nearly 30x on a `git diff` workload on a 2-node POWER9 (59,800 -> 2,100) and reduces CPU cycles by 0.54%, due to vfs hashes being allocated with 2MB pages. Cc: linuxppc-dev@lists.ozlabs.org Acked-by: Michael Ellerman <mpe@ellerman.id.au> Signed-off-by: Nicholas Piggin <npiggin@gmail.com> --- .../admin-guide/kernel-parameters.txt | 2 ++ arch/powerpc/Kconfig | 1 + arch/powerpc/kernel/module.c | 22 +++++++++++++++---- 3 files changed, 21 insertions(+), 4 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 04545725f187..1f481f904895 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -3243,6 +3243,8 @@ nohugeiomap [KNL,X86,PPC,ARM64] Disable kernel huge I/O mappings. + nohugevmalloc [PPC] Disable kernel huge vmalloc mappings. + nosmt [KNL,S390] Disable symmetric multithreading (SMT). Equivalent to smt=1. diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index 386ae12d8523..b7cade9566da 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -181,6 +181,7 @@ config PPC select GENERIC_GETTIMEOFDAY select HAVE_ARCH_AUDITSYSCALL select HAVE_ARCH_HUGE_VMAP if PPC_BOOK3S_64 && PPC_RADIX_MMU + select HAVE_ARCH_HUGE_VMALLOC if HAVE_ARCH_HUGE_VMAP select HAVE_ARCH_JUMP_LABEL select HAVE_ARCH_KASAN if PPC32 && PPC_PAGE_SHIFT <= 14 select HAVE_ARCH_KASAN_VMALLOC if PPC32 && PPC_PAGE_SHIFT <= 14 diff --git a/arch/powerpc/kernel/module.c b/arch/powerpc/kernel/module.c index a211b0253cdb..cdb2d88c54e7 100644 --- a/arch/powerpc/kernel/module.c +++ b/arch/powerpc/kernel/module.c @@ -8,6 +8,7 @@ #include <linux/moduleloader.h> #include <linux/err.h> #include <linux/vmalloc.h> +#include <linux/mm.h> #include <linux/bug.h> #include <asm/module.h> #include <linux/uaccess.h> @@ -87,13 +88,26 @@ int module_finalize(const Elf_Ehdr *hdr, return 0; } -#ifdef MODULES_VADDR void *module_alloc(unsigned long size) { + unsigned long start = VMALLOC_START; + unsigned long end = VMALLOC_END; + +#ifdef MODULES_VADDR BUILD_BUG_ON(TASK_SIZE > MODULES_VADDR); + start = MODULES_VADDR; + end = MODULES_END; +#endif + + /* + * Don't do huge page allocations for modules yet until more testing + * is done. STRICT_MODULE_RWX may require extra work to support this + * too. + */ - return __vmalloc_node_range(size, 1, MODULES_VADDR, MODULES_END, GFP_KERNEL, - PAGE_KERNEL_EXEC, VM_FLUSH_RESET_PERMS, NUMA_NO_NODE, + return __vmalloc_node_range(size, 1, start, end, GFP_KERNEL, + PAGE_KERNEL_EXEC, + VM_NO_HUGE_VMAP | VM_FLUSH_RESET_PERMS, + NUMA_NO_NODE, __builtin_return_address(0)); } -#endif -- 2.23.0
next prev parent reply other threads:[~2021-03-17 6:26 UTC|newest] Thread overview: 37+ messages / expand[flat|nested] mbox.gz Atom feed top 2021-03-17 6:23 [PATCH v13 00/14] huge vmalloc mappings Nicholas Piggin 2021-03-17 6:23 ` [PATCH v13 01/14] ARM: mm: add missing pud_page define to 2-level page tables Nicholas Piggin 2021-03-17 6:23 ` Nicholas Piggin 2021-03-17 6:23 ` [PATCH v13 02/14] mm/vmalloc: fix HUGE_VMAP regression by enabling huge pages in vmalloc_to_page Nicholas Piggin 2021-03-17 6:23 ` [PATCH v13 03/14] mm: apply_to_pte_range warn and fail if a large pte is encountered Nicholas Piggin 2021-03-17 6:23 ` [PATCH v13 04/14] mm/vmalloc: rename vmap_*_range vmap_pages_*_range Nicholas Piggin 2021-03-17 6:23 ` [PATCH v13 05/14] mm/ioremap: rename ioremap_*_range to vmap_*_range Nicholas Piggin 2021-03-17 6:23 ` [PATCH v13 06/14] mm: HUGE_VMAP arch support cleanup Nicholas Piggin 2021-03-17 6:23 ` Nicholas Piggin 2021-03-17 6:23 ` Nicholas Piggin 2021-04-28 8:32 ` Christophe Leroy 2021-04-28 8:32 ` Christophe Leroy 2021-04-28 8:32 ` Christophe Leroy 2021-04-28 8:34 ` Christophe Leroy 2021-04-28 8:34 ` Christophe Leroy 2021-04-28 8:34 ` Christophe Leroy 2021-03-17 6:23 ` [PATCH v13 07/14] powerpc: inline huge vmap supported functions Nicholas Piggin 2021-03-17 6:23 ` Nicholas Piggin 2021-03-17 6:23 ` [PATCH v13 08/14] arm64: " Nicholas Piggin 2021-03-17 6:23 ` Nicholas Piggin 2021-03-17 6:23 ` [PATCH v13 09/14] x86: " Nicholas Piggin 2021-03-17 6:23 ` [PATCH v13 10/14] mm/vmalloc: provide fallback arch huge vmap support functions Nicholas Piggin 2021-03-17 6:23 ` [PATCH v13 11/14] mm: Move vmap_range from mm/ioremap.c to mm/vmalloc.c Nicholas Piggin 2021-03-17 6:24 ` [PATCH v13 12/14] mm/vmalloc: add vmap_range_noflush variant Nicholas Piggin 2021-03-17 6:24 ` [PATCH v13 13/14] mm/vmalloc: Hugepage vmalloc mappings Nicholas Piggin 2021-03-17 6:24 ` Nicholas Piggin [this message] 2021-03-17 6:24 ` [PATCH v13 14/14] powerpc/64s/radix: Enable huge " Nicholas Piggin 2021-04-15 10:23 ` Christophe Leroy 2021-04-15 10:23 ` Christophe Leroy 2021-04-15 18:55 ` Andrew Morton 2021-04-15 18:55 ` Andrew Morton 2021-04-15 23:04 ` Stephen Rothwell 2021-04-15 23:04 ` Stephen Rothwell 2021-04-17 2:39 ` Nicholas Piggin 2021-04-17 2:39 ` Nicholas Piggin 2021-03-17 22:58 ` [PATCH v13 00/14] " Andrew Morton 2021-03-18 3:50 ` Nicholas Piggin
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20210317062402.533919-15-npiggin@gmail.com \ --to=npiggin@gmail.com \ --cc=Jonathan.Cameron@Huawei.com \ --cc=akpm@linux-foundation.org \ --cc=christophe.leroy@csgroup.eu \ --cc=dingtianhong@huawei.com \ --cc=hch@infradead.org \ --cc=linux-arch@vger.kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=linuxppc-dev@lists.ozlabs.org \ --cc=mpe@ellerman.id.au \ --cc=rick.p.edgecombe@intel.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.