From: guoren@kernel.org To: guoren@kernel.org, anup.patel@wdc.com, palmerdabbelt@google.com, arnd@arndb.de, wens@csie.org, maxime@cerno.tech, drew@beagleboard.org, liush@allwinnertech.com, lazyparser@gmail.com, wefu@redhat.com Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-sunxi@lists.linux.dev, Guo Ren <guoren@linux.alibaba.com>, Atish Patra <atish.patra@wdc.com> Subject: [RFC PATCH v2 08/11] riscv: cmo: Add vendor custom icache sync Date: Sun, 6 Jun 2021 09:04:06 +0000 [thread overview] Message-ID: <1622970249-50770-12-git-send-email-guoren@kernel.org> (raw) In-Reply-To: <1622970249-50770-1-git-send-email-guoren@kernel.org> From: Guo Ren <guoren@linux.alibaba.com> It's a draft version to show you how T-HEAD C9xx work with the icache sync (We use hardware broadcast mechanism, and our icache is VIPT): - icache.i(v/p)a will broadcast all harts' icache invalidtion - sync.is will broadcast all harts' pipeline flush and ensure all broadcasts finished. This patch could improve the performance of OpenJDK on JIT and reduce flush_icache_all in linux. Epecially: static inline void set_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pteval) { if (pte_present(pteval) && pte_exec(pteval)) flush_icache_pte(pteval); set_pte(ptep, pteval); } Different from sbi_dma_sync, it can't be hidden in SBI and we must set up a framework to hold all vendors' implementations in linux/arch/riscv. Signed-off-by: Guo Ren <guoren@linux.alibaba.com> Signed-off-by: Liu Shaohua <liush@allwinnertech.com> Cc: Anup Patel <anup.patel@wdc.com> Cc: Atish Patra <atish.patra@wdc.com> Cc: Palmer Dabbelt <palmerdabbelt@google.com> Cc: Chen-Yu Tsai <wens@csie.org> Cc: Drew Fustini <drew@beagleboard.org> Cc: Maxime Ripard <maxime@cerno.tech> Cc: Palmer Dabbelt <palmerdabbelt@google.com> Cc: Wei Fu <wefu@redhat.com> Cc: Wei Wu <lazyparser@gmail.com> --- arch/riscv/include/asm/cacheflush.h | 48 ++++++++++++++++++++++++++++++++++- arch/riscv/kernel/vdso/flush_icache.S | 33 +++++++++++++++++++----- arch/riscv/mm/cacheflush.c | 3 ++- 3 files changed, 76 insertions(+), 8 deletions(-) diff --git a/arch/riscv/include/asm/cacheflush.h b/arch/riscv/include/asm/cacheflush.h index 23ff703..2e2dba1 100644 --- a/arch/riscv/include/asm/cacheflush.h +++ b/arch/riscv/include/asm/cacheflush.h @@ -22,11 +22,57 @@ static inline void flush_dcache_page(struct page *page) } #define ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE 1 +#define ICACHE_IPA_X5 ".long 0x0382800b" +#define ICACHE_IVA_X5 ".long 0x0302800b" +#define SYNC_IS ".long 0x01b0000b" + +static inline void flush_icache_range(unsigned long start, unsigned long end) +{ + register unsigned long tmp asm("x5") = start & (~(L1_CACHE_BYTES-1)); + + for (; tmp < ALIGN(end, L1_CACHE_BYTES); tmp += L1_CACHE_BYTES) { + __asm__ __volatile__ ( + ICACHE_IVA_X5 + : + : "r" (tmp) + : "memory"); + } + + __asm__ __volatile__(SYNC_IS); + + return; +} + +static inline void flush_icache_range_phy(unsigned long start, unsigned long end) +{ + register unsigned long tmp asm("x5") = start & (~(L1_CACHE_BYTES-1)); + + for (; tmp < ALIGN(end, L1_CACHE_BYTES); tmp += L1_CACHE_BYTES) { + __asm__ __volatile__ ( + ICACHE_IPA_X5 + : + : "r" (tmp) + : "memory"); + } + + __asm__ __volatile__(SYNC_IS); + + return; +} + +static inline void __flush_icache_page(struct page *page) { + unsigned long start = PFN_PHYS(page_to_pfn(page)); + + flush_icache_range_phy(start, start + PAGE_SIZE); + + return; +} + /* * RISC-V doesn't have an instruction to flush parts of the instruction cache, * so instead we just flush the whole thing. */ -#define flush_icache_range(start, end) flush_icache_all() +#define flush_icache_range(start, end) flush_icache_range(start, end) #define flush_icache_user_page(vma, pg, addr, len) \ flush_icache_mm(vma->vm_mm, 0) diff --git a/arch/riscv/kernel/vdso/flush_icache.S b/arch/riscv/kernel/vdso/flush_icache.S index 82f97d6..efb2d2e 100644 --- a/arch/riscv/kernel/vdso/flush_icache.S +++ b/arch/riscv/kernel/vdso/flush_icache.S @@ -5,18 +5,39 @@ #include <linux/linkage.h> #include <asm/unistd.h> +#include <asm/cache.h> + +/* + * icache.ipa rs1 + * | 31 - 25 | 24 - 20 | 19 - 15 | 14 - 12 | 11 - 7 | 6 - 0 | + * 0000001 11000 rs1 000 00000 0001011 + * + * icache.iva rs1 + * | 31 - 25 | 24 - 20 | 19 - 15 | 14 - 12 | 11 - 7 | 6 - 0 | + * 0000001 10000 rs1 000 00000 0001011 + * + * sync.is + * | 31 - 25 | 24 - 20 | 19 - 15 | 14 - 12 | 11 - 7 | 6 - 0 | + * 0000000 11011 00000 000 00000 0001011 + */ +#define ICACHE_IPA_X5 .long 0x0382800b +#define ICACHE_IVA_X5 .long 0x0302800b +#define SYNC_IS .long 0x01b0000b .text /* int __vdso_flush_icache(void *start, void *end, unsigned long flags); */ ENTRY(__vdso_flush_icache) .cfi_startproc -#ifdef CONFIG_SMP - li a7, __NR_riscv_flush_icache - ecall -#else - fence.i + srli t0, a0, L1_CACHE_SHIFT + slli t0, t0, L1_CACHE_SHIFT + addi a1, a1, (L1_CACHE_BYTES - 1) + srli a1, a1, L1_CACHE_SHIFT + slli a1, a1, L1_CACHE_SHIFT +1: ICACHE_IVA_X5 + addi t0, t0, L1_CACHE_BYTES + bne t0, a1, 1b + SYNC_IS li a0, 0 -#endif ret .cfi_endproc ENDPROC(__vdso_flush_icache) diff --git a/arch/riscv/mm/cacheflush.c b/arch/riscv/mm/cacheflush.c index 0941186..0fb8344 100644 --- a/arch/riscv/mm/cacheflush.c +++ b/arch/riscv/mm/cacheflush.c @@ -3,6 +3,7 @@ * Copyright (C) 2017 SiFive */ +#include <linux/pfn.h> #include <asm/cacheflush.h> #ifdef CONFIG_SMP @@ -84,6 +85,6 @@ void flush_icache_pte(pte_t pte) struct page *page = pte_page(pte); if (!test_and_set_bit(PG_dcache_clean, &page->flags)) - flush_icache_all(); + __flush_icache_page(page); } #endif /* CONFIG_MMU */ -- 2.7.4
WARNING: multiple messages have this Message-ID (diff)
From: guoren@kernel.org To: guoren@kernel.org, anup.patel@wdc.com, palmerdabbelt@google.com, arnd@arndb.de, wens@csie.org, maxime@cerno.tech, drew@beagleboard.org, liush@allwinnertech.com, lazyparser@gmail.com, wefu@redhat.com Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-sunxi@lists.linux.dev, Guo Ren <guoren@linux.alibaba.com>, Atish Patra <atish.patra@wdc.com> Subject: [RFC PATCH v2 08/11] riscv: cmo: Add vendor custom icache sync Date: Sun, 6 Jun 2021 09:04:06 +0000 [thread overview] Message-ID: <1622970249-50770-12-git-send-email-guoren@kernel.org> (raw) In-Reply-To: <1622970249-50770-1-git-send-email-guoren@kernel.org> From: Guo Ren <guoren@linux.alibaba.com> It's a draft version to show you how T-HEAD C9xx work with the icache sync (We use hardware broadcast mechanism, and our icache is VIPT): - icache.i(v/p)a will broadcast all harts' icache invalidtion - sync.is will broadcast all harts' pipeline flush and ensure all broadcasts finished. This patch could improve the performance of OpenJDK on JIT and reduce flush_icache_all in linux. Epecially: static inline void set_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pteval) { if (pte_present(pteval) && pte_exec(pteval)) flush_icache_pte(pteval); set_pte(ptep, pteval); } Different from sbi_dma_sync, it can't be hidden in SBI and we must set up a framework to hold all vendors' implementations in linux/arch/riscv. Signed-off-by: Guo Ren <guoren@linux.alibaba.com> Signed-off-by: Liu Shaohua <liush@allwinnertech.com> Cc: Anup Patel <anup.patel@wdc.com> Cc: Atish Patra <atish.patra@wdc.com> Cc: Palmer Dabbelt <palmerdabbelt@google.com> Cc: Chen-Yu Tsai <wens@csie.org> Cc: Drew Fustini <drew@beagleboard.org> Cc: Maxime Ripard <maxime@cerno.tech> Cc: Palmer Dabbelt <palmerdabbelt@google.com> Cc: Wei Fu <wefu@redhat.com> Cc: Wei Wu <lazyparser@gmail.com> --- arch/riscv/include/asm/cacheflush.h | 48 ++++++++++++++++++++++++++++++++++- arch/riscv/kernel/vdso/flush_icache.S | 33 +++++++++++++++++++----- arch/riscv/mm/cacheflush.c | 3 ++- 3 files changed, 76 insertions(+), 8 deletions(-) diff --git a/arch/riscv/include/asm/cacheflush.h b/arch/riscv/include/asm/cacheflush.h index 23ff703..2e2dba1 100644 --- a/arch/riscv/include/asm/cacheflush.h +++ b/arch/riscv/include/asm/cacheflush.h @@ -22,11 +22,57 @@ static inline void flush_dcache_page(struct page *page) } #define ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE 1 +#define ICACHE_IPA_X5 ".long 0x0382800b" +#define ICACHE_IVA_X5 ".long 0x0302800b" +#define SYNC_IS ".long 0x01b0000b" + +static inline void flush_icache_range(unsigned long start, unsigned long end) +{ + register unsigned long tmp asm("x5") = start & (~(L1_CACHE_BYTES-1)); + + for (; tmp < ALIGN(end, L1_CACHE_BYTES); tmp += L1_CACHE_BYTES) { + __asm__ __volatile__ ( + ICACHE_IVA_X5 + : + : "r" (tmp) + : "memory"); + } + + __asm__ __volatile__(SYNC_IS); + + return; +} + +static inline void flush_icache_range_phy(unsigned long start, unsigned long end) +{ + register unsigned long tmp asm("x5") = start & (~(L1_CACHE_BYTES-1)); + + for (; tmp < ALIGN(end, L1_CACHE_BYTES); tmp += L1_CACHE_BYTES) { + __asm__ __volatile__ ( + ICACHE_IPA_X5 + : + : "r" (tmp) + : "memory"); + } + + __asm__ __volatile__(SYNC_IS); + + return; +} + +static inline void __flush_icache_page(struct page *page) { + unsigned long start = PFN_PHYS(page_to_pfn(page)); + + flush_icache_range_phy(start, start + PAGE_SIZE); + + return; +} + /* * RISC-V doesn't have an instruction to flush parts of the instruction cache, * so instead we just flush the whole thing. */ -#define flush_icache_range(start, end) flush_icache_all() +#define flush_icache_range(start, end) flush_icache_range(start, end) #define flush_icache_user_page(vma, pg, addr, len) \ flush_icache_mm(vma->vm_mm, 0) diff --git a/arch/riscv/kernel/vdso/flush_icache.S b/arch/riscv/kernel/vdso/flush_icache.S index 82f97d6..efb2d2e 100644 --- a/arch/riscv/kernel/vdso/flush_icache.S +++ b/arch/riscv/kernel/vdso/flush_icache.S @@ -5,18 +5,39 @@ #include <linux/linkage.h> #include <asm/unistd.h> +#include <asm/cache.h> + +/* + * icache.ipa rs1 + * | 31 - 25 | 24 - 20 | 19 - 15 | 14 - 12 | 11 - 7 | 6 - 0 | + * 0000001 11000 rs1 000 00000 0001011 + * + * icache.iva rs1 + * | 31 - 25 | 24 - 20 | 19 - 15 | 14 - 12 | 11 - 7 | 6 - 0 | + * 0000001 10000 rs1 000 00000 0001011 + * + * sync.is + * | 31 - 25 | 24 - 20 | 19 - 15 | 14 - 12 | 11 - 7 | 6 - 0 | + * 0000000 11011 00000 000 00000 0001011 + */ +#define ICACHE_IPA_X5 .long 0x0382800b +#define ICACHE_IVA_X5 .long 0x0302800b +#define SYNC_IS .long 0x01b0000b .text /* int __vdso_flush_icache(void *start, void *end, unsigned long flags); */ ENTRY(__vdso_flush_icache) .cfi_startproc -#ifdef CONFIG_SMP - li a7, __NR_riscv_flush_icache - ecall -#else - fence.i + srli t0, a0, L1_CACHE_SHIFT + slli t0, t0, L1_CACHE_SHIFT + addi a1, a1, (L1_CACHE_BYTES - 1) + srli a1, a1, L1_CACHE_SHIFT + slli a1, a1, L1_CACHE_SHIFT +1: ICACHE_IVA_X5 + addi t0, t0, L1_CACHE_BYTES + bne t0, a1, 1b + SYNC_IS li a0, 0 -#endif ret .cfi_endproc ENDPROC(__vdso_flush_icache) diff --git a/arch/riscv/mm/cacheflush.c b/arch/riscv/mm/cacheflush.c index 0941186..0fb8344 100644 --- a/arch/riscv/mm/cacheflush.c +++ b/arch/riscv/mm/cacheflush.c @@ -3,6 +3,7 @@ * Copyright (C) 2017 SiFive */ +#include <linux/pfn.h> #include <asm/cacheflush.h> #ifdef CONFIG_SMP @@ -84,6 +85,6 @@ void flush_icache_pte(pte_t pte) struct page *page = pte_page(pte); if (!test_and_set_bit(PG_dcache_clean, &page->flags)) - flush_icache_all(); + __flush_icache_page(page); } #endif /* CONFIG_MMU */ -- 2.7.4 _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv
next prev parent reply other threads:[~2021-06-06 9:05 UTC|newest] Thread overview: 107+ messages / expand[flat|nested] mbox.gz Atom feed top 2021-06-06 9:03 [RFC PATCH v2 00/11] riscv: Add DMA_COHERENT support for Allwinner D1 guoren 2021-06-06 9:03 ` guoren 2021-06-06 9:03 ` [RFC PATCH v2 01/11] riscv: asid: Use global mappings for kernel pages guoren 2021-06-06 9:03 ` guoren 2021-06-06 9:03 ` [PATCH V5 1/3] riscv: " guoren 2021-06-06 9:03 ` guoren 2021-06-06 9:03 ` [PATCH V5 2/3] riscv: Add ASID-based tlbflushing methods guoren 2021-06-06 9:03 ` guoren 2021-06-06 14:38 ` Christoph Hellwig 2021-06-06 14:38 ` Christoph Hellwig 2021-06-06 9:03 ` [RFC PATCH v2 02/11] riscv: asid: " guoren 2021-06-06 9:03 ` guoren 2021-06-06 9:04 ` [RFC PATCH v2 03/11] riscv: asid: Optimize tlbflush coding convention guoren 2021-06-06 9:04 ` guoren 2021-06-06 9:04 ` [PATCH V5 3/3] riscv: tlbflush: Optimize " guoren 2021-06-06 9:04 ` guoren 2021-06-06 9:04 ` [RFC PATCH v2 04/11] riscv: pgtable: Fixup _PAGE_CHG_MASK usage guoren 2021-06-06 9:04 ` guoren 2021-06-06 9:04 ` [RFC PATCH v2 05/11] riscv: pgtable: Add custom protection_map init guoren 2021-06-06 9:04 ` guoren 2021-06-06 9:04 ` [RFC PATCH v2 06/11] riscv: pgtable: Add DMA_COHERENT with custom PTE attributes guoren 2021-06-06 9:04 ` guoren 2021-06-06 14:39 ` Christoph Hellwig 2021-06-06 14:39 ` Christoph Hellwig 2021-06-06 15:08 ` Guo Ren 2021-06-06 15:08 ` Guo Ren 2021-06-06 17:22 ` Nick Kossifidis 2021-06-06 17:22 ` Nick Kossifidis 2021-06-07 6:19 ` Christoph Hellwig 2021-06-07 6:19 ` Christoph Hellwig 2021-06-06 9:04 ` [RFC PATCH v2 07/11] riscv: cmo: Add dma-noncoherency support guoren 2021-06-06 9:04 ` guoren 2021-10-17 9:28 ` twd2 2021-10-17 9:28 ` twd2 2021-10-20 8:11 ` Guo Ren 2021-10-20 8:11 ` Guo Ren 2021-06-06 9:04 ` guoren [this message] 2021-06-06 9:04 ` [RFC PATCH v2 08/11] riscv: cmo: Add vendor custom icache sync guoren 2021-06-06 9:04 ` [RFC PATCH v2 09/11] riscv: soc: Initial DTS for Allwinner D1 NeZha board guoren 2021-06-06 9:04 ` guoren 2021-06-06 16:26 ` Jernej Škrabec 2021-06-06 16:26 ` Jernej Škrabec 2021-06-06 17:05 ` Guo Ren 2021-06-06 17:05 ` Guo Ren 2021-06-07 3:44 ` Guo Ren 2021-06-07 3:44 ` Guo Ren 2021-06-07 7:27 ` Maxime Ripard 2021-06-07 7:27 ` Maxime Ripard 2021-06-07 7:53 ` Guo Ren 2021-06-07 7:53 ` Guo Ren 2021-06-07 7:24 ` Maxime Ripard 2021-06-07 7:24 ` Maxime Ripard 2021-06-07 8:07 ` Guo Ren 2021-06-07 8:07 ` Guo Ren 2021-06-14 15:33 ` Maxime Ripard 2021-06-14 15:33 ` Maxime Ripard 2021-06-14 16:28 ` Guo Ren 2021-06-14 16:28 ` Guo Ren 2021-06-14 16:31 ` Jernej Škrabec 2021-06-14 16:31 ` Jernej Škrabec 2021-06-06 9:04 ` [RFC PATCH v2 10/11] riscv: soc: Add Allwinner SoC kconfig option guoren 2021-06-06 9:04 ` guoren 2021-06-07 7:19 ` Maxime Ripard 2021-06-07 7:19 ` Maxime Ripard 2021-06-07 7:27 ` Arnd Bergmann 2021-06-07 7:27 ` Arnd Bergmann 2021-06-07 7:27 ` Arnd Bergmann 2021-06-07 7:45 ` Guo Ren 2021-06-07 7:45 ` Guo Ren 2021-06-07 7:43 ` Guo Ren 2021-06-07 7:43 ` Guo Ren 2021-06-07 12:12 ` Maxime Ripard 2021-06-07 12:12 ` Maxime Ripard 2021-06-07 12:39 ` Guo Ren 2021-06-07 12:39 ` Guo Ren 2021-06-06 9:04 ` [RFC PATCH v2 11/11] riscv: soc: Allwinner D1 GMAC driver only for temp use guoren 2021-06-06 9:04 ` guoren 2021-06-06 10:50 ` Andre Przywara 2021-06-06 10:50 ` Andre Przywara 2021-06-06 15:32 ` Guo Ren 2021-06-06 15:32 ` Guo Ren 2021-06-06 15:39 ` Jernej Škrabec 2021-06-06 15:39 ` Jernej Škrabec 2021-06-06 15:41 ` Guo Ren 2021-06-06 15:41 ` Guo Ren 2021-06-06 16:16 ` Arnd Bergmann 2021-06-06 16:16 ` Arnd Bergmann 2021-06-06 16:16 ` Arnd Bergmann 2021-06-06 16:32 ` Jernej Škrabec 2021-06-06 16:32 ` Jernej Škrabec 2021-06-06 16:53 ` Guo Ren 2021-06-06 16:53 ` Guo Ren 2021-06-06 16:53 ` Guo Ren 2021-06-06 16:53 ` Guo Ren 2021-06-06 16:29 ` [RFC PATCH v2 00/11] riscv: Add DMA_COHERENT support for Allwinner D1 Jernej Škrabec 2021-06-06 16:29 ` Jernej Škrabec 2021-06-06 16:54 ` Guo Ren 2021-06-06 16:54 ` Guo Ren 2021-06-06 17:14 ` Jernej Škrabec 2021-06-06 17:14 ` Jernej Škrabec 2021-06-06 23:42 ` Guo Ren 2021-06-06 23:42 ` Guo Ren 2021-06-07 3:44 ` Anup Patel 2021-06-07 3:44 ` Anup Patel 2021-06-07 4:36 ` Guo Ren 2021-06-07 4:36 ` Guo Ren 2021-06-07 4:36 ` Guo Ren
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=1622970249-50770-12-git-send-email-guoren@kernel.org \ --to=guoren@kernel.org \ --cc=anup.patel@wdc.com \ --cc=arnd@arndb.de \ --cc=atish.patra@wdc.com \ --cc=drew@beagleboard.org \ --cc=guoren@linux.alibaba.com \ --cc=lazyparser@gmail.com \ --cc=linux-arch@vger.kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-riscv@lists.infradead.org \ --cc=linux-sunxi@lists.linux.dev \ --cc=liush@allwinnertech.com \ --cc=maxime@cerno.tech \ --cc=palmerdabbelt@google.com \ --cc=wefu@redhat.com \ --cc=wens@csie.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.