From: Will Deacon <will.deacon@arm.com> To: linux-kernel@vger.kernel.org Cc: peterz@infradead.org, benh@au1.ibm.com, torvalds@linux-foundation.org, npiggin@gmail.com, catalin.marinas@arm.com, linux-arm-kernel@lists.infradead.org, Will Deacon <will.deacon@arm.com> Subject: [RFC PATCH 11/11] arm64: tlb: Avoid synchronous TLBIs when freeing page tables Date: Fri, 24 Aug 2018 16:52:46 +0100 [thread overview] Message-ID: <1535125966-7666-12-git-send-email-will.deacon@arm.com> (raw) In-Reply-To: <1535125966-7666-1-git-send-email-will.deacon@arm.com> By selecting HAVE_RCU_TABLE_INVALIDATE, we can rely on tlb_flush() being called if we fail to batch table pages for freeing. This in turn allows us to postpone walk-cache invalidation until tlb_finish_mmu(), which avoids lots of unnecessary DSBs and means we can shoot down the ASID if the range is large enough. Signed-off-by: Will Deacon <will.deacon@arm.com> --- arch/arm64/Kconfig | 1 + arch/arm64/include/asm/tlb.h | 3 --- arch/arm64/include/asm/tlbflush.h | 11 ----------- 3 files changed, 1 insertion(+), 14 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 29e75b47becd..89059ee1eccc 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -142,6 +142,7 @@ config ARM64 select HAVE_PERF_USER_STACK_DUMP select HAVE_REGS_AND_STACK_ACCESS_API select HAVE_RCU_TABLE_FREE + select HAVE_RCU_TABLE_INVALIDATE select HAVE_RSEQ select HAVE_STACKPROTECTOR select HAVE_SYSCALL_TRACEPOINTS diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index baca8dff6884..4f65614b3141 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -54,7 +54,6 @@ static inline void tlb_flush(struct mmu_gather *tlb) static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) { - __flush_tlb_pgtable(tlb->mm, addr); pgtable_page_dtor(pte); tlb_remove_table(tlb, pte); } @@ -63,7 +62,6 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, unsigned long addr) { - __flush_tlb_pgtable(tlb->mm, addr); tlb_remove_table(tlb, virt_to_page(pmdp)); } #endif @@ -72,7 +70,6 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, unsigned long addr) { - __flush_tlb_pgtable(tlb->mm, addr); tlb_remove_table(tlb, virt_to_page(pudp)); } #endif diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index 1f77d08e638b..2064ba97845f 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -215,17 +215,6 @@ static inline void flush_tlb_kernel_range(unsigned long start, unsigned long end * Used to invalidate the TLB (walk caches) corresponding to intermediate page * table levels (pgd/pud/pmd). */ -static inline void __flush_tlb_pgtable(struct mm_struct *mm, - unsigned long uaddr) -{ - unsigned long addr = __TLBI_VADDR(uaddr, ASID(mm)); - - dsb(ishst); - __tlbi(vae1is, addr); - __tlbi_user(vae1is, addr); - dsb(ish); -} - static inline void __flush_tlb_kernel_pgtable(unsigned long kaddr) { unsigned long addr = __TLBI_VADDR(kaddr, 0); -- 2.1.4
WARNING: multiple messages have this Message-ID (diff)
From: will.deacon@arm.com (Will Deacon) To: linux-arm-kernel@lists.infradead.org Subject: [RFC PATCH 11/11] arm64: tlb: Avoid synchronous TLBIs when freeing page tables Date: Fri, 24 Aug 2018 16:52:46 +0100 [thread overview] Message-ID: <1535125966-7666-12-git-send-email-will.deacon@arm.com> (raw) In-Reply-To: <1535125966-7666-1-git-send-email-will.deacon@arm.com> By selecting HAVE_RCU_TABLE_INVALIDATE, we can rely on tlb_flush() being called if we fail to batch table pages for freeing. This in turn allows us to postpone walk-cache invalidation until tlb_finish_mmu(), which avoids lots of unnecessary DSBs and means we can shoot down the ASID if the range is large enough. Signed-off-by: Will Deacon <will.deacon@arm.com> --- arch/arm64/Kconfig | 1 + arch/arm64/include/asm/tlb.h | 3 --- arch/arm64/include/asm/tlbflush.h | 11 ----------- 3 files changed, 1 insertion(+), 14 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 29e75b47becd..89059ee1eccc 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -142,6 +142,7 @@ config ARM64 select HAVE_PERF_USER_STACK_DUMP select HAVE_REGS_AND_STACK_ACCESS_API select HAVE_RCU_TABLE_FREE + select HAVE_RCU_TABLE_INVALIDATE select HAVE_RSEQ select HAVE_STACKPROTECTOR select HAVE_SYSCALL_TRACEPOINTS diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index baca8dff6884..4f65614b3141 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -54,7 +54,6 @@ static inline void tlb_flush(struct mmu_gather *tlb) static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, unsigned long addr) { - __flush_tlb_pgtable(tlb->mm, addr); pgtable_page_dtor(pte); tlb_remove_table(tlb, pte); } @@ -63,7 +62,6 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, unsigned long addr) { - __flush_tlb_pgtable(tlb->mm, addr); tlb_remove_table(tlb, virt_to_page(pmdp)); } #endif @@ -72,7 +70,6 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, unsigned long addr) { - __flush_tlb_pgtable(tlb->mm, addr); tlb_remove_table(tlb, virt_to_page(pudp)); } #endif diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index 1f77d08e638b..2064ba97845f 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -215,17 +215,6 @@ static inline void flush_tlb_kernel_range(unsigned long start, unsigned long end * Used to invalidate the TLB (walk caches) corresponding to intermediate page * table levels (pgd/pud/pmd). */ -static inline void __flush_tlb_pgtable(struct mm_struct *mm, - unsigned long uaddr) -{ - unsigned long addr = __TLBI_VADDR(uaddr, ASID(mm)); - - dsb(ishst); - __tlbi(vae1is, addr); - __tlbi_user(vae1is, addr); - dsb(ish); -} - static inline void __flush_tlb_kernel_pgtable(unsigned long kaddr) { unsigned long addr = __TLBI_VADDR(kaddr, 0); -- 2.1.4
next prev parent reply other threads:[~2018-08-24 15:52 UTC|newest] Thread overview: 58+ messages / expand[flat|nested] mbox.gz Atom feed top 2018-08-24 15:52 [RFC PATCH 00/11] Avoid synchronous TLB invalidation for intermediate page-table entries on arm64 Will Deacon 2018-08-24 15:52 ` Will Deacon 2018-08-24 15:52 ` [RFC PATCH 01/11] arm64: tlb: Use last-level invalidation in flush_tlb_kernel_range() Will Deacon 2018-08-24 15:52 ` Will Deacon 2018-08-24 15:52 ` [RFC PATCH 02/11] arm64: tlb: Add DSB ISHST prior to TLBI in __flush_tlb_[kernel_]pgtable() Will Deacon 2018-08-24 15:52 ` Will Deacon 2018-08-24 17:56 ` Peter Zijlstra 2018-08-24 17:56 ` Peter Zijlstra 2018-08-28 13:03 ` Will Deacon 2018-08-28 13:03 ` Will Deacon 2018-08-24 15:52 ` [RFC PATCH 03/11] arm64: pgtable: Implement p[mu]d_valid() and check in set_p[mu]d() Will Deacon 2018-08-24 15:52 ` Will Deacon 2018-08-24 16:15 ` Linus Torvalds 2018-08-24 16:15 ` Linus Torvalds 2018-08-28 12:49 ` Will Deacon 2018-08-28 12:49 ` Will Deacon 2018-08-24 15:52 ` [RFC PATCH 04/11] arm64: tlb: Justify non-leaf invalidation in flush_tlb_range() Will Deacon 2018-08-24 15:52 ` Will Deacon 2018-08-24 15:52 ` [RFC PATCH 05/11] arm64: tlbflush: Allow stride to be specified for __flush_tlb_range() Will Deacon 2018-08-24 15:52 ` Will Deacon 2018-08-24 15:52 ` [RFC PATCH 06/11] arm64: tlb: Remove redundant !CONFIG_HAVE_RCU_TABLE_FREE code Will Deacon 2018-08-24 15:52 ` Will Deacon 2018-08-24 15:52 ` [RFC PATCH 07/11] asm-generic/tlb: Guard with #ifdef CONFIG_MMU Will Deacon 2018-08-24 15:52 ` Will Deacon 2018-08-24 15:52 ` [RFC PATCH 08/11] asm-generic/tlb: Track freeing of page-table directories in struct mmu_gather Will Deacon 2018-08-24 15:52 ` Will Deacon 2018-08-27 4:44 ` Nicholas Piggin 2018-08-27 4:44 ` Nicholas Piggin 2018-08-28 13:46 ` Peter Zijlstra 2018-08-28 13:46 ` Peter Zijlstra 2018-08-28 13:48 ` Peter Zijlstra 2018-08-28 13:48 ` Peter Zijlstra 2018-08-28 14:12 ` Nicholas Piggin 2018-08-28 14:12 ` Nicholas Piggin 2018-08-24 15:52 ` [RFC PATCH 09/11] asm-generic/tlb: Track which levels of the page tables have been cleared Will Deacon 2018-08-24 15:52 ` Will Deacon 2018-08-27 7:53 ` Peter Zijlstra 2018-08-27 7:53 ` Peter Zijlstra 2018-08-28 13:12 ` Will Deacon 2018-08-28 13:12 ` Will Deacon 2018-08-24 15:52 ` [RFC PATCH 10/11] arm64: tlb: Adjust stride and type of TLBI according to mmu_gather Will Deacon 2018-08-24 15:52 ` Will Deacon 2018-08-24 15:52 ` Will Deacon [this message] 2018-08-24 15:52 ` [RFC PATCH 11/11] arm64: tlb: Avoid synchronous TLBIs when freeing page tables Will Deacon 2018-08-24 16:20 ` [RFC PATCH 00/11] Avoid synchronous TLB invalidation for intermediate page-table entries on arm64 Linus Torvalds 2018-08-24 16:20 ` Linus Torvalds 2018-08-26 10:56 ` Peter Zijlstra 2018-08-26 10:56 ` Peter Zijlstra 2018-09-04 18:38 ` Jon Masters 2018-09-04 18:38 ` Jon Masters 2018-09-05 12:28 ` Will Deacon 2018-09-05 12:28 ` Will Deacon 2018-09-07 6:36 ` Jon Masters 2018-09-07 6:36 ` Jon Masters 2018-09-13 15:53 ` Will Deacon 2018-09-13 15:53 ` Will Deacon 2018-09-13 16:53 ` Jon Masters 2018-09-13 16:53 ` Jon Masters
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=1535125966-7666-12-git-send-email-will.deacon@arm.com \ --to=will.deacon@arm.com \ --cc=benh@au1.ibm.com \ --cc=catalin.marinas@arm.com \ --cc=linux-arm-kernel@lists.infradead.org \ --cc=linux-kernel@vger.kernel.org \ --cc=npiggin@gmail.com \ --cc=peterz@infradead.org \ --cc=torvalds@linux-foundation.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.