From: Atish Patra <atish.patra@wdc.com> To: linux-kernel@vger.kernel.org Cc: Atish Patra <atish.patra@wdc.com>, Albert Ou <aou@eecs.berkeley.edu>, Allison Randal <allison@lohutok.net>, Anup Patel <anup@brainfault.org>, linux-riscv@lists.infradead.org, Palmer Dabbelt <palmer@sifive.com>, Paul Walmsley <paul.walmsley@sifive.com>, Andreas Schwab <schwab@linux-m68k.org>, "hch@infradead.org" <hch@infradead.org> Subject: [v2 PATCH] RISC-V: Optimize tlb flush path. Date: Mon, 19 Aug 2019 17:47:35 -0700 [thread overview] Message-ID: <20190820004735.18518-1-atish.patra@wdc.com> (raw) In RISC-V, tlb flush happens via SBI which is expensive. If the target cpumask contains a local hartid, some cost can be saved by issuing a local tlb flush as we do that in OpenSBI anyways. There is also no need of SBI call if cpumask is empty. Do a local flush first if current cpu is present in cpumask. Invoke SBI call only if target cpumask contains any cpus other than local cpu. Signed-off-by: Atish Patra <atish.patra@wdc.com> --- arch/riscv/include/asm/tlbflush.h | 37 ++++++++++++++++++++++++++----- 1 file changed, 31 insertions(+), 6 deletions(-) diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index b5e64dc19b9e..3f9cd17b5402 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -8,6 +8,7 @@ #define _ASM_RISCV_TLBFLUSH_H #include <linux/mm_types.h> +#include <linux/sched.h> #include <asm/smp.h> /* @@ -42,20 +43,44 @@ static inline void flush_tlb_range(struct vm_area_struct *vma, #include <asm/sbi.h> -static inline void remote_sfence_vma(struct cpumask *cmask, unsigned long start, - unsigned long size) +static void __riscv_flush_tlb(struct cpumask *cmask, unsigned long start, + unsigned long size) { struct cpumask hmask; + unsigned int hartid; + unsigned int cpuid; cpumask_clear(&hmask); + + if (!cmask) { + riscv_cpuid_to_hartid_mask(cpu_online_mask, &hmask); + goto issue_sfence; + } + + cpuid = get_cpu(); + if (cpumask_test_cpu(cpuid, cmask)) { + /* Save trap cost by issuing a local tlb flush here */ + if ((start == 0 && size == -1) || (size > PAGE_SIZE)) + local_flush_tlb_all(); + else if (size == PAGE_SIZE) + local_flush_tlb_page(start); + } + if (cpumask_any_but(cmask, cpuid) >= nr_cpu_ids) + goto done; + riscv_cpuid_to_hartid_mask(cmask, &hmask); + hartid = cpuid_to_hartid_map(cpuid); + cpumask_clear_cpu(hartid, &hmask); + +issue_sfence: sbi_remote_sfence_vma(hmask.bits, start, size); +done: + put_cpu(); } -#define flush_tlb_all() sbi_remote_sfence_vma(NULL, 0, -1) - +#define flush_tlb_all() __riscv_flush_tlb(NULL, 0, -1) #define flush_tlb_range(vma, start, end) \ - remote_sfence_vma(mm_cpumask((vma)->vm_mm), start, (end) - (start)) + __riscv_flush_tlb(mm_cpumask((vma)->vm_mm), start, (end) - (start)) static inline void flush_tlb_page(struct vm_area_struct *vma, unsigned long addr) { @@ -63,7 +88,7 @@ static inline void flush_tlb_page(struct vm_area_struct *vma, } #define flush_tlb_mm(mm) \ - remote_sfence_vma(mm_cpumask(mm), 0, -1) + __riscv_flush_tlb(mm_cpumask(mm), 0, -1) #endif /* CONFIG_SMP */ -- 2.21.0
WARNING: multiple messages have this Message-ID (diff)
From: Atish Patra <atish.patra@wdc.com> To: linux-kernel@vger.kernel.org Cc: Albert Ou <aou@eecs.berkeley.edu>, Anup Patel <anup@brainfault.org>, Palmer Dabbelt <palmer@sifive.com>, "hch@infradead.org" <hch@infradead.org>, Atish Patra <atish.patra@wdc.com>, Andreas Schwab <schwab@linux-m68k.org>, Paul Walmsley <paul.walmsley@sifive.com>, linux-riscv@lists.infradead.org, Allison Randal <allison@lohutok.net> Subject: [v2 PATCH] RISC-V: Optimize tlb flush path. Date: Mon, 19 Aug 2019 17:47:35 -0700 [thread overview] Message-ID: <20190820004735.18518-1-atish.patra@wdc.com> (raw) In RISC-V, tlb flush happens via SBI which is expensive. If the target cpumask contains a local hartid, some cost can be saved by issuing a local tlb flush as we do that in OpenSBI anyways. There is also no need of SBI call if cpumask is empty. Do a local flush first if current cpu is present in cpumask. Invoke SBI call only if target cpumask contains any cpus other than local cpu. Signed-off-by: Atish Patra <atish.patra@wdc.com> --- arch/riscv/include/asm/tlbflush.h | 37 ++++++++++++++++++++++++++----- 1 file changed, 31 insertions(+), 6 deletions(-) diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index b5e64dc19b9e..3f9cd17b5402 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -8,6 +8,7 @@ #define _ASM_RISCV_TLBFLUSH_H #include <linux/mm_types.h> +#include <linux/sched.h> #include <asm/smp.h> /* @@ -42,20 +43,44 @@ static inline void flush_tlb_range(struct vm_area_struct *vma, #include <asm/sbi.h> -static inline void remote_sfence_vma(struct cpumask *cmask, unsigned long start, - unsigned long size) +static void __riscv_flush_tlb(struct cpumask *cmask, unsigned long start, + unsigned long size) { struct cpumask hmask; + unsigned int hartid; + unsigned int cpuid; cpumask_clear(&hmask); + + if (!cmask) { + riscv_cpuid_to_hartid_mask(cpu_online_mask, &hmask); + goto issue_sfence; + } + + cpuid = get_cpu(); + if (cpumask_test_cpu(cpuid, cmask)) { + /* Save trap cost by issuing a local tlb flush here */ + if ((start == 0 && size == -1) || (size > PAGE_SIZE)) + local_flush_tlb_all(); + else if (size == PAGE_SIZE) + local_flush_tlb_page(start); + } + if (cpumask_any_but(cmask, cpuid) >= nr_cpu_ids) + goto done; + riscv_cpuid_to_hartid_mask(cmask, &hmask); + hartid = cpuid_to_hartid_map(cpuid); + cpumask_clear_cpu(hartid, &hmask); + +issue_sfence: sbi_remote_sfence_vma(hmask.bits, start, size); +done: + put_cpu(); } -#define flush_tlb_all() sbi_remote_sfence_vma(NULL, 0, -1) - +#define flush_tlb_all() __riscv_flush_tlb(NULL, 0, -1) #define flush_tlb_range(vma, start, end) \ - remote_sfence_vma(mm_cpumask((vma)->vm_mm), start, (end) - (start)) + __riscv_flush_tlb(mm_cpumask((vma)->vm_mm), start, (end) - (start)) static inline void flush_tlb_page(struct vm_area_struct *vma, unsigned long addr) { @@ -63,7 +88,7 @@ static inline void flush_tlb_page(struct vm_area_struct *vma, } #define flush_tlb_mm(mm) \ - remote_sfence_vma(mm_cpumask(mm), 0, -1) + __riscv_flush_tlb(mm_cpumask(mm), 0, -1) #endif /* CONFIG_SMP */ -- 2.21.0 _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv
next reply other threads:[~2019-08-20 0:47 UTC|newest] Thread overview: 40+ messages / expand[flat|nested] mbox.gz Atom feed top 2019-08-20 0:47 Atish Patra [this message] 2019-08-20 0:47 ` [v2 PATCH] RISC-V: Optimize tlb flush path Atish Patra 2019-08-20 3:06 ` hch 2019-08-20 3:06 ` hch 2019-08-20 7:14 ` Andreas Schwab 2019-08-20 7:14 ` Andreas Schwab 2019-08-20 7:16 ` hch 2019-08-20 7:16 ` hch 2019-08-20 7:46 ` Andreas Schwab 2019-08-20 7:46 ` Andreas Schwab 2019-08-20 8:42 ` Atish Patra 2019-08-20 8:42 ` Atish Patra 2019-08-20 8:51 ` Andreas Schwab 2019-08-20 8:51 ` Andreas Schwab 2019-08-20 9:22 ` hch 2019-08-20 9:22 ` hch 2019-08-20 20:28 ` Atish Patra 2019-08-20 20:28 ` Atish Patra 2019-08-20 22:18 ` hch 2019-08-20 22:18 ` hch 2019-08-20 22:24 ` Atish Patra 2019-08-20 22:24 ` Atish Patra 2019-08-21 1:29 ` Alan Kao 2019-08-21 1:29 ` Alan Kao 2019-08-21 1:40 ` hch 2019-08-21 1:40 ` hch 2019-08-21 3:52 ` Anup Patel 2019-08-21 3:52 ` Anup Patel 2019-08-21 7:18 ` hch 2019-08-21 7:18 ` hch 2019-08-20 8:51 ` Anup Patel 2019-08-20 8:51 ` Anup Patel 2019-08-20 20:29 ` Atish Patra 2019-08-20 20:29 ` Atish Patra 2019-08-21 14:41 ` hch 2019-08-21 14:41 ` hch 2019-08-21 14:45 ` hch 2019-08-21 14:45 ` hch 2019-08-21 17:36 ` Atish Patra 2019-08-21 17:36 ` Atish Patra
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20190820004735.18518-1-atish.patra@wdc.com \ --to=atish.patra@wdc.com \ --cc=allison@lohutok.net \ --cc=anup@brainfault.org \ --cc=aou@eecs.berkeley.edu \ --cc=hch@infradead.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-riscv@lists.infradead.org \ --cc=palmer@sifive.com \ --cc=paul.walmsley@sifive.com \ --cc=schwab@linux-m68k.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.