From: "Aneesh Kumar K.V" <aneesh.kumar@linux.vnet.ibm.com>
To: benh@kernel.crashing.org, paulus@samba.org, mpe@ellerman.id.au
Cc: linuxppc-dev@lists.ozlabs.org,
"Aneesh Kumar K.V" <aneesh.kumar@linux.vnet.ibm.com>
Subject: [PATCH V2 04/16] powerpc/mm/radix: Implement tlb mmu gather flush efficiently
Date: Wed, 8 Jun 2016 20:13:11 +0530 [thread overview]
Message-ID: <1465397003-26812-5-git-send-email-aneesh.kumar@linux.vnet.ibm.com> (raw)
In-Reply-To: <1465397003-26812-1-git-send-email-aneesh.kumar@linux.vnet.ibm.com>
Now that we track page size in mmu_gather, we can use address based
tlbie format when doing a tlb_flush(). We don't do this if we are
invalidating the full address space.
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.vnet.ibm.com>
---
.../powerpc/include/asm/book3s/64/tlbflush-radix.h | 2 +
arch/powerpc/mm/tlb-radix.c | 73 +++++++++++++++++++++-
2 files changed, 74 insertions(+), 1 deletion(-)
diff --git a/arch/powerpc/include/asm/book3s/64/tlbflush-radix.h b/arch/powerpc/include/asm/book3s/64/tlbflush-radix.h
index 3fa94fcac628..862c8fa50268 100644
--- a/arch/powerpc/include/asm/book3s/64/tlbflush-radix.h
+++ b/arch/powerpc/include/asm/book3s/64/tlbflush-radix.h
@@ -10,6 +10,8 @@ static inline int mmu_get_ap(int psize)
return mmu_psize_defs[psize].ap;
}
+extern void radix__flush_tlb_range_psize(struct mm_struct *mm, unsigned long start,
+ unsigned long end, int psize);
extern void radix__flush_tlb_range(struct vm_area_struct *vma, unsigned long start,
unsigned long end);
extern void radix__flush_tlb_kernel_range(unsigned long start, unsigned long end);
diff --git a/arch/powerpc/mm/tlb-radix.c b/arch/powerpc/mm/tlb-radix.c
index 231e3ed2e684..03e719ee6747 100644
--- a/arch/powerpc/mm/tlb-radix.c
+++ b/arch/powerpc/mm/tlb-radix.c
@@ -279,9 +279,80 @@ void radix__flush_tlb_range(struct vm_area_struct *vma, unsigned long start,
}
EXPORT_SYMBOL(radix__flush_tlb_range);
+static int radix_get_mmu_psize(int page_size)
+{
+ int psize;
+
+ if (page_size == (1UL << mmu_psize_defs[mmu_virtual_psize].shift))
+ psize = mmu_virtual_psize;
+ else if (page_size == (1UL << mmu_psize_defs[MMU_PAGE_2M].shift))
+ psize = MMU_PAGE_2M;
+ else if (page_size == (1UL << mmu_psize_defs[MMU_PAGE_1G].shift))
+ psize = MMU_PAGE_1G;
+ else
+ return -1;
+ return psize;
+}
void radix__tlb_flush(struct mmu_gather *tlb)
{
+ int psize = 0;
struct mm_struct *mm = tlb->mm;
- radix__flush_tlb_mm(mm);
+ int page_size = tlb->page_size;
+
+ psize = radix_get_mmu_psize(page_size);
+ /*
+ * if page size is not something we understand, do a full mm flush
+ */
+ if (psize != -1 && !tlb->fullmm && !tlb->need_flush_all)
+ radix__flush_tlb_range_psize(mm, tlb->start, tlb->end, psize);
+ else
+ radix__flush_tlb_mm(mm);
+}
+
+#define TLB_FLUSH_ALL -1UL
+/*
+ * Number of pages above which we will do a bcast tlbie. Just a
+ * number at this point copied from x86
+ */
+static unsigned long tlb_single_page_flush_ceiling __read_mostly = 33;
+
+void radix__flush_tlb_range_psize(struct mm_struct *mm, unsigned long start,
+ unsigned long end, int psize)
+{
+ unsigned long pid;
+ unsigned long addr;
+ int local = mm_is_core_local(mm);
+ unsigned long ap = mmu_get_ap(psize);
+ int lock_tlbie = !mmu_has_feature(MMU_FTR_LOCKLESS_TLBIE);
+ unsigned long page_size = 1UL << mmu_psize_defs[psize].shift;
+
+
+ preempt_disable();
+ pid = mm ? mm->context.id : 0;
+ if (unlikely(pid == MMU_NO_CONTEXT))
+ goto err_out;
+
+ if (end == TLB_FLUSH_ALL ||
+ (end - start) > tlb_single_page_flush_ceiling * page_size) {
+ if (local)
+ _tlbiel_pid(pid, RIC_FLUSH_TLB);
+ else
+ _tlbie_pid(pid, RIC_FLUSH_TLB);
+ goto err_out;
+ }
+ for (addr = start; addr < end; addr += page_size) {
+
+ if (local)
+ _tlbiel_va(addr, pid, ap, RIC_FLUSH_TLB);
+ else {
+ if (lock_tlbie)
+ raw_spin_lock(&native_tlbie_lock);
+ _tlbie_va(addr, pid, ap, RIC_FLUSH_TLB);
+ if (lock_tlbie)
+ raw_spin_unlock(&native_tlbie_lock);
+ }
+ }
+err_out:
+ preempt_enable();
}
--
2.7.4
next prev parent reply other threads:[~2016-06-08 14:43 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-06-08 14:43 [PATCH V2 00/16] TLB flush improvments and Segment table support Aneesh Kumar K.V
2016-06-08 14:43 ` [PATCH V2 01/16] mm/hugetlb: Simplify hugetlb unmap Aneesh Kumar K.V
2016-06-08 14:43 ` [PATCH V2 02/16] mm: Change the interface for __tlb_remove_page Aneesh Kumar K.V
2016-06-08 14:43 ` [PATCH V2 03/16] mm/mmu_gather: Track page size with mmu gather and force flush if page size change Aneesh Kumar K.V
2016-06-08 14:43 ` Aneesh Kumar K.V [this message]
2016-06-08 14:43 ` [PATCH V2 05/16] powerpc/mm: Make MMU_FTR_RADIX a MMU family feature Aneesh Kumar K.V
2016-06-08 14:43 ` [PATCH V2 06/16] powerpc/mm/hash: Add helper for finding SLBE LLP encoding Aneesh Kumar K.V
2016-06-08 14:43 ` [PATCH V2 07/16] powerpc/mm: Use hugetlb flush functions Aneesh Kumar K.V
2016-06-08 14:43 ` [PATCH V2 08/16] powerpc/mm: Drop multiple definition of mm_is_core_local Aneesh Kumar K.V
2016-06-08 14:43 ` [PATCH V2 09/16] powerpc/mm/radix: Add tlb flush of THP ptes Aneesh Kumar K.V
2016-06-08 14:43 ` [PATCH V2 10/16] powerpc/mm/radix: Rename function and drop unused arg Aneesh Kumar K.V
2016-06-08 14:43 ` [PATCH V2 11/16] powerpc/mm/radix/hugetlb: Add helper for finding page size from hstate Aneesh Kumar K.V
2016-06-08 14:43 ` [PATCH V2 12/16] powerpc/mm/hugetlb: Add flush_hugetlb_tlb_range Aneesh Kumar K.V
2016-06-08 14:43 ` [PATCH V2 13/16] powerpc/mm: remove flush_tlb_page_nohash Aneesh Kumar K.V
2016-06-08 14:43 ` [PATCH V2 14/16] powerpc/mm: Cleanup LPCR defines Aneesh Kumar K.V
2016-06-08 14:43 ` [PATCH V2 15/16] powerpc/mm: Switch user slb fault handling to translation enabled Aneesh Kumar K.V
2016-06-08 14:43 ` [PATCH V2 16/16] powerpc/mm: Support segment table for Power9 Aneesh Kumar K.V
2016-06-08 15:07 ` Aneesh Kumar K.V
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1465397003-26812-5-git-send-email-aneesh.kumar@linux.vnet.ibm.com \
--to=aneesh.kumar@linux.vnet.ibm.com \
--cc=benh@kernel.crashing.org \
--cc=linuxppc-dev@lists.ozlabs.org \
--cc=mpe@ellerman.id.au \
--cc=paulus@samba.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).