From: Andrew Morton <akpm@linux-foundation.org> To: akpm@linux-foundation.org, aneesh.kumar@linux.ibm.com, linux-mm@kvack.org, mm-commits@vger.kernel.org, mpe@ellerman.id.au, peterz@infradead.org, stable@vger.kernel.org, torvalds@linux-foundation.org Subject: [patch 49/67] mm/mmu_gather: invalidate TLB correctly on batch allocation failure and flush Date: Mon, 03 Feb 2020 17:36:49 -0800 [thread overview] Message-ID: <20200204013649.Nwx3yVd8_%akpm@linux-foundation.org> (raw) In-Reply-To: <20200203173311.6269a8be06a05e5a4aa08a93@linux-foundation.org> From: Peter Zijlstra <peterz@infradead.org> Subject: mm/mmu_gather: invalidate TLB correctly on batch allocation failure and flush Architectures for which we have hardware walkers of Linux page table should flush TLB on mmu gather batch allocation failures and batch flush. Some architectures like POWER supports multiple translation modes (hash and radix) and in the case of POWER only radix translation mode needs the above TLBI. This is because for hash translation mode kernel wants to avoid this extra flush since there are no hardware walkers of linux page table. With radix translation, the hardware also walks linux page table and with that, kernel needs to make sure to TLB invalidate page walk cache before page table pages are freed. More details in commit d86564a2f085 ("mm/tlb, x86/mm: Support invalidating TLB caches for RCU_TABLE_FREE") The changes to sparc are to make sure we keep the old behavior since we are now removing HAVE_RCU_TABLE_NO_INVALIDATE. The default value for tlb_needs_table_invalidate is to always force an invalidate and sparc can avoid the table invalidate. Hence we define tlb_needs_table_invalidate to false for sparc architecture. Link: http://lkml.kernel.org/r/20200116064531.483522-3-aneesh.kumar@linux.ibm.com Fixes: a46cc7a90fd8 ("powerpc/mm/radix: Improve TLB/PWC flushes") Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com> Acked-by: Michael Ellerman <mpe@ellerman.id.au> [powerpc] Cc: <stable@vger.kernel.org> [4.14+] Signed-off-by: Andrew Morton <akpm@linux-foundation.org> --- arch/Kconfig | 3 --- arch/powerpc/Kconfig | 1 - arch/powerpc/include/asm/tlb.h | 11 +++++++++++ arch/sparc/Kconfig | 1 - arch/sparc/include/asm/tlb_64.h | 9 +++++++++ include/asm-generic/tlb.h | 22 +++++++++++++++------- mm/mmu_gather.c | 16 ++++++++-------- 7 files changed, 43 insertions(+), 20 deletions(-) --- a/arch/Kconfig~mm-mmu_gather-invalidate-tlb-correctly-on-batch-allocation-failure-and-flush +++ a/arch/Kconfig @@ -396,9 +396,6 @@ config HAVE_ARCH_JUMP_LABEL_RELATIVE config HAVE_RCU_TABLE_FREE bool -config HAVE_RCU_TABLE_NO_INVALIDATE - bool - config HAVE_MMU_GATHER_PAGE_SIZE bool --- a/arch/powerpc/include/asm/tlb.h~mm-mmu_gather-invalidate-tlb-correctly-on-batch-allocation-failure-and-flush +++ a/arch/powerpc/include/asm/tlb.h @@ -26,6 +26,17 @@ #define tlb_flush tlb_flush extern void tlb_flush(struct mmu_gather *tlb); +/* + * book3s: + * Hash does not use the linux page-tables, so we can avoid + * the TLB invalidate for page-table freeing, Radix otoh does use the + * page-tables and needs the TLBI. + * + * nohash: + * We still do TLB invalidate in the __pte_free_tlb routine before we + * add the page table pages to mmu gather table batch. + */ +#define tlb_needs_table_invalidate() radix_enabled() /* Get the generic bits... */ #include <asm-generic/tlb.h> --- a/arch/powerpc/Kconfig~mm-mmu_gather-invalidate-tlb-correctly-on-batch-allocation-failure-and-flush +++ a/arch/powerpc/Kconfig @@ -223,7 +223,6 @@ config PPC select HAVE_PERF_REGS select HAVE_PERF_USER_STACK_DUMP select HAVE_RCU_TABLE_FREE - select HAVE_RCU_TABLE_NO_INVALIDATE if HAVE_RCU_TABLE_FREE select HAVE_MMU_GATHER_PAGE_SIZE select HAVE_REGS_AND_STACK_ACCESS_API select HAVE_RELIABLE_STACKTRACE if PPC_BOOK3S_64 && CPU_LITTLE_ENDIAN --- a/arch/sparc/include/asm/tlb_64.h~mm-mmu_gather-invalidate-tlb-correctly-on-batch-allocation-failure-and-flush +++ a/arch/sparc/include/asm/tlb_64.h @@ -28,6 +28,15 @@ void flush_tlb_pending(void); #define __tlb_remove_tlb_entry(tlb, ptep, address) do { } while (0) #define tlb_flush(tlb) flush_tlb_pending() +/* + * SPARC64's hardware TLB fill does not use the Linux page-tables + * and therefore we don't need a TLBI when freeing page-table pages. + */ + +#ifdef CONFIG_HAVE_RCU_TABLE_FREE +#define tlb_needs_table_invalidate() (false) +#endif + #include <asm-generic/tlb.h> #endif /* _SPARC64_TLB_H */ --- a/arch/sparc/Kconfig~mm-mmu_gather-invalidate-tlb-correctly-on-batch-allocation-failure-and-flush +++ a/arch/sparc/Kconfig @@ -65,7 +65,6 @@ config SPARC64 select HAVE_KRETPROBES select HAVE_KPROBES select HAVE_RCU_TABLE_FREE if SMP - select HAVE_RCU_TABLE_NO_INVALIDATE if HAVE_RCU_TABLE_FREE select HAVE_MEMBLOCK_NODE_MAP select HAVE_ARCH_TRANSPARENT_HUGEPAGE select HAVE_DYNAMIC_FTRACE --- a/include/asm-generic/tlb.h~mm-mmu_gather-invalidate-tlb-correctly-on-batch-allocation-failure-and-flush +++ a/include/asm-generic/tlb.h @@ -137,13 +137,6 @@ * When used, an architecture is expected to provide __tlb_remove_table() * which does the actual freeing of these pages. * - * HAVE_RCU_TABLE_NO_INVALIDATE - * - * This makes HAVE_RCU_TABLE_FREE avoid calling tlb_flush_mmu_tlbonly() before - * freeing the page-table pages. This can be avoided if you use - * HAVE_RCU_TABLE_FREE and your architecture does _NOT_ use the Linux - * page-tables natively. - * * MMU_GATHER_NO_RANGE * * Use this if your architecture lacks an efficient flush_tlb_range(). @@ -189,8 +182,23 @@ struct mmu_table_batch { extern void tlb_remove_table(struct mmu_gather *tlb, void *table); +/* + * This allows an architecture that does not use the linux page-tables for + * hardware to skip the TLBI when freeing page tables. + */ +#ifndef tlb_needs_table_invalidate +#define tlb_needs_table_invalidate() (true) #endif +#else + +#ifdef tlb_needs_table_invalidate +#error tlb_needs_table_invalidate() requires HAVE_RCU_TABLE_FREE +#endif + +#endif /* CONFIG_HAVE_RCU_TABLE_FREE */ + + #ifndef CONFIG_HAVE_MMU_GATHER_NO_GATHER /* * If we can't allocate a page to make a big batch of page pointers --- a/mm/mmu_gather.c~mm-mmu_gather-invalidate-tlb-correctly-on-batch-allocation-failure-and-flush +++ a/mm/mmu_gather.c @@ -102,14 +102,14 @@ bool __tlb_remove_page_size(struct mmu_g */ static inline void tlb_table_invalidate(struct mmu_gather *tlb) { -#ifndef CONFIG_HAVE_RCU_TABLE_NO_INVALIDATE - /* - * Invalidate page-table caches used by hardware walkers. Then we still - * need to RCU-sched wait while freeing the pages because software - * walkers can still be in-flight. - */ - tlb_flush_mmu_tlbonly(tlb); -#endif + if (tlb_needs_table_invalidate()) { + /* + * Invalidate page-table caches used by hardware walkers. Then + * we still need to RCU-sched wait while freeing the pages + * because software walkers can still be in-flight. + */ + tlb_flush_mmu_tlbonly(tlb); + } } static void tlb_remove_table_smp_sync(void *arg) _
next prev parent reply other threads:[~2020-02-04 1:36 UTC|newest] Thread overview: 95+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-02-04 1:33 incoming Andrew Morton 2020-02-04 1:33 ` [patch 01/67] ocfs2: fix oops when writing cloned file Andrew Morton 2020-02-04 1:33 ` [patch 02/67] mm/page_alloc.c: fix uninitialized memmaps on a partially populated last section Andrew Morton 2020-02-04 1:33 ` [patch 03/67] fs/proc/page.c: allow inspection of last section and fix end detection Andrew Morton 2020-02-04 1:33 ` [patch 04/67] mm/page_alloc.c: initialize memmap of unavailable memory directly Andrew Morton 2020-02-04 1:33 ` [patch 05/67] mm/page_alloc: fix and rework pfn handling in memmap_init_zone() Andrew Morton 2020-02-04 2:45 ` Linus Torvalds 2020-02-04 1:34 ` [patch 06/67] mm: factor out next_present_section_nr() Andrew Morton 2020-02-04 3:04 ` Linus Torvalds 2020-02-04 4:29 ` Andrew Morton 2020-02-04 8:22 ` David Hildenbrand 2020-02-04 1:34 ` [patch 07/67] mm/memmap_init: update variable name in memmap_init_zone Andrew Morton 2020-02-04 1:34 ` [patch 08/67] mm/memory_hotplug: poison memmap in remove_pfn_range_from_zone() Andrew Morton 2020-02-04 1:34 ` [patch 09/67] mm/memory_hotplug: we always have a zone in find_(smallest|biggest)_section_pfn Andrew Morton 2020-02-04 1:34 ` [patch 10/67] mm/memory_hotplug: don't check for "all holes" in shrink_zone_span() Andrew Morton 2020-02-04 1:34 ` [patch 11/67] mm/memory_hotplug: drop local variables " Andrew Morton 2020-02-04 1:34 ` [patch 12/67] mm/memory_hotplug: cleanup __remove_pages() Andrew Morton 2020-02-04 1:34 ` [patch 13/67] mm/memory_hotplug: drop valid_start/valid_end from test_pages_in_a_zone() Andrew Morton 2020-02-04 1:34 ` [patch 14/67] smp_mb__{before,after}_atomic(): update Documentation Andrew Morton 2020-02-04 1:34 ` [patch 15/67] ipc/mqueue.c: remove duplicated code Andrew Morton 2020-02-04 1:34 ` [patch 16/67] ipc/mqueue.c: update/document memory barriers Andrew Morton 2020-02-04 1:34 ` [patch 17/67] ipc/msg.c: update and document " Andrew Morton 2020-02-04 1:34 ` [patch 18/67] ipc/sem.c: document and update " Andrew Morton 2020-02-04 1:34 ` [patch 19/67] ipc/msg.c: consolidate all xxxctl_down() functions Andrew Morton 2020-02-04 1:34 ` [patch 20/67] drivers/block/null_blk_main.c: fix layout Andrew Morton 2020-02-04 1:34 ` [patch 21/67] drivers/block/null_blk_main.c: fix uninitialized var warnings Andrew Morton 2020-02-04 1:34 ` [patch 22/67] pinctrl: fix pxa2xx.c build warnings Andrew Morton 2020-02-04 1:34 ` [patch 23/67] mm: remove __krealloc Andrew Morton 2020-02-04 1:35 ` [patch 24/67] mm: add generic p?d_leaf() macros Andrew Morton 2020-02-04 1:35 ` [patch 25/67] arc: mm: add p?d_leaf() definitions Andrew Morton 2020-02-04 1:35 ` [patch 26/67] arm: " Andrew Morton 2020-02-04 1:35 ` [patch 27/67] arm64: " Andrew Morton 2020-02-04 1:35 ` [patch 28/67] mips: " Andrew Morton 2020-02-04 1:35 ` [patch 29/67] powerpc: " Andrew Morton 2020-02-04 1:35 ` [patch 30/67] riscv: " Andrew Morton 2020-02-04 1:35 ` [patch 31/67] s390: " Andrew Morton 2020-02-04 1:35 ` [patch 32/67] sparc: " Andrew Morton 2020-02-04 1:35 ` [patch 33/67] x86: " Andrew Morton 2020-02-04 1:35 ` [patch 34/67] mm: pagewalk: add p4d_entry() and pgd_entry() Andrew Morton 2020-02-04 1:35 ` [patch 35/67] mm: pagewalk: allow walking without vma Andrew Morton 2020-02-04 1:35 ` [patch 36/67] mm: pagewalk: don't lock PTEs for walk_page_range_novma() Andrew Morton 2020-02-04 1:35 ` [patch 37/67] mm: pagewalk: fix termination condition in walk_pte_range() Andrew Morton 2020-02-04 1:36 ` [patch 38/67] mm: pagewalk: add 'depth' parameter to pte_hole Andrew Morton 2020-02-04 1:36 ` [patch 39/67] x86: mm: point to struct seq_file from struct pg_state Andrew Morton 2020-02-04 1:36 ` [patch 40/67] x86: mm+efi: convert ptdump_walk_pgd_level() to take a mm_struct Andrew Morton 2020-02-04 1:36 ` [patch 41/67] x86: mm: convert ptdump_walk_pgd_level_debugfs() to take an mm_struct Andrew Morton 2020-02-04 1:36 ` [patch 42/67] mm: add generic ptdump Andrew Morton 2020-02-04 1:36 ` [patch 43/67] x86: mm: convert dump_pagetables to use walk_page_range Andrew Morton 2020-02-04 1:36 ` [patch 44/67] arm64: mm: convert mm/dump.c to use walk_page_range() Andrew Morton 2020-02-04 1:36 ` [patch 45/67] arm64: mm: display non-present entries in ptdump Andrew Morton 2020-02-04 1:36 ` [patch 46/67] mm: ptdump: reduce level numbers by 1 in note_page() Andrew Morton 2020-02-04 1:36 ` [patch 47/67] x86: mm: avoid allocating struct mm_struct on the stack Andrew Morton 2020-02-04 1:36 ` [patch 48/67] powerpc/mmu_gather: enable RCU_TABLE_FREE even for !SMP case Andrew Morton 2020-02-04 1:36 ` Andrew Morton [this message] 2020-02-04 1:36 ` [patch 50/67] asm-generic/tlb: avoid potential double flush Andrew Morton 2020-02-04 1:36 ` [patch 51/67] asm-gemeric/tlb: remove stray function declarations Andrew Morton 2020-02-04 1:36 ` [patch 52/67] asm-generic/tlb: add missing CONFIG symbol Andrew Morton 2020-02-04 1:37 ` [patch 53/67] asm-generic/tlb: rename HAVE_RCU_TABLE_FREE Andrew Morton 2020-02-04 1:37 ` [patch 54/67] asm-generic/tlb: rename HAVE_MMU_GATHER_PAGE_SIZE Andrew Morton 2020-02-04 1:37 ` [patch 55/67] asm-generic/tlb: rename HAVE_MMU_GATHER_NO_GATHER Andrew Morton 2020-02-04 1:37 ` [patch 56/67] asm-generic/tlb: provide MMU_GATHER_TABLE_FREE Andrew Morton 2020-02-04 1:37 ` [patch 57/67] proc: decouple proc from VFS with "struct proc_ops" Andrew Morton 2020-02-04 1:37 ` [patch 58/67] proc: convert everything to " Andrew Morton 2020-02-04 1:37 ` [patch 59/67] lib/string: add strnchrnul() Andrew Morton 2020-02-04 1:37 ` [patch 60/67] bitops: more BITS_TO_* macros Andrew Morton 2020-02-04 1:37 ` [patch 61/67] lib: add test for bitmap_parse() Andrew Morton 2020-02-04 1:37 ` [patch 62/67] lib: make bitmap_parse_user a wrapper on bitmap_parse Andrew Morton 2020-02-04 1:37 ` [patch 63/67] lib: rework bitmap_parse() Andrew Morton 2020-02-04 1:37 ` [patch 64/67] lib: new testcases for bitmap_parse{_user} Andrew Morton 2020-02-04 1:37 ` [patch 65/67] include/linux/cpumask.h: don't calculate length of the input string Andrew Morton 2020-02-04 1:37 ` [patch 66/67] treewide: remove redundant IS_ERR() before error code check Andrew Morton 2020-02-04 1:37 ` [patch 67/67] ARM: dma-api: fix max_pfn off-by-one error in __dma_supported() Andrew Morton 2020-02-04 2:27 ` incoming Linus Torvalds 2020-02-04 2:46 ` incoming Andrew Morton 2020-02-04 3:11 ` incoming Linus Torvalds 2020-02-14 6:26 ` mmotm 2020-02-13-22-26 uploaded Andrew Morton 2020-02-14 16:29 ` mmotm 2020-02-13-22-26 uploaded (mm/hugetlb.c) Randy Dunlap 2020-02-14 17:18 ` Mike Kravetz 2020-02-14 20:51 ` Mina Almasry [not found] ` <20200214204544.231482-1-almasrymina@google.com> 2020-02-14 21:00 ` [PATCH] hugetlb: fix CONFIG_CGROUP_HUGETLB ifdefs Mina Almasry 2020-02-15 1:17 ` Randy Dunlap 2020-02-15 1:56 ` Randy Dunlap 2020-02-16 20:40 ` Mina Almasry 2020-02-16 21:03 ` Mina Almasry 2020-02-17 2:48 ` Randy Dunlap 2020-02-17 2:57 ` [PATCH] hugetlb: fix <linux/hugetlb_cgroup.h> structs Randy Dunlap 2020-02-17 3:53 ` [PATCH] hugetlb: fix CONFIG_CGROUP_HUGETLB ifdefs Stephen Rothwell 2020-02-14 16:49 ` mmotm 2020-02-13-22-26 uploaded (mm/migrate.c, hugetlb_cgroup.h) Randy Dunlap 2020-02-25 3:53 ` mmotm 2020-02-24-19-53 uploaded Andrew Morton 2020-02-25 6:16 ` mmotm 2020-02-24-19-53 uploaded (init/main.c: initrd*) Randy Dunlap 2020-02-25 6:18 ` Randy Dunlap 2020-02-25 6:21 ` Randy Dunlap 2020-02-25 16:41 ` mmotm 2020-02-24-19-53 uploaded (drivers/platform/x86/intel_pmc_core.c) Randy Dunlap 2020-02-25 17:01 ` mmotm 2020-02-24-19-53 uploaded (objtool warning) Randy Dunlap 2020-02-27 21:52 ` Josh Poimboeuf
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20200204013649.Nwx3yVd8_%akpm@linux-foundation.org \ --to=akpm@linux-foundation.org \ --cc=aneesh.kumar@linux.ibm.com \ --cc=linux-mm@kvack.org \ --cc=mm-commits@vger.kernel.org \ --cc=mpe@ellerman.id.au \ --cc=peterz@infradead.org \ --cc=stable@vger.kernel.org \ --cc=torvalds@linux-foundation.org \ --subject='Re: [patch 49/67] mm/mmu_gather: invalidate TLB correctly on batch allocation failure and flush' \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).