From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E077770 for ; Sun, 6 Jun 2021 09:05:38 +0000 (UTC) Received: by mail.kernel.org (Postfix) with ESMTPSA id A6C7861426; Sun, 6 Jun 2021 09:05:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1622970338; bh=qph7YI4bBELR4x3cH34m49m6VFlBeHt4WLSEim7zato=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Uet+WCvoJuyWL0SFrd+1jc6Ilaj8YMQJHj1VWC39aPQPAR+SPjI1wrb7V8fb3fGh2 MEB9ca1PE+TXAUNVnNV+22vcxI8/Aup5ue6UqIY9NcIfDUHC/pYiKDcvW8Bg17F98s 2fO0OdAMZbh8VkNxMEbWm4fdnEn6yMA0IjK1oZQYrhRB73d/9G8LS1U7f+ZJxGFVKu ZqlIa8doFRapqKvNgCpGpw7NMHzRNu3cSBtDvH2WammdZLv0pYh/EudHQi7Q0bq/OM gVN6DKpvmO5B14prT9gTCYwAFoKWE3OVODc8VPoIEeHbdWqoU91glJiqloi3KAOnEf JnIQ2NHuhBKcw== From: guoren@kernel.org To: guoren@kernel.org, anup.patel@wdc.com, palmerdabbelt@google.com, arnd@arndb.de, wens@csie.org, maxime@cerno.tech, drew@beagleboard.org, liush@allwinnertech.com, lazyparser@gmail.com, wefu@redhat.com Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-sunxi@lists.linux.dev, Guo Ren , Atish Patra Subject: [RFC PATCH v2 08/11] riscv: cmo: Add vendor custom icache sync Date: Sun, 6 Jun 2021 09:04:06 +0000 Message-Id: <1622970249-50770-12-git-send-email-guoren@kernel.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1622970249-50770-1-git-send-email-guoren@kernel.org> References: <1622970249-50770-1-git-send-email-guoren@kernel.org> X-Mailing-List: linux-sunxi@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: From: Guo Ren It's a draft version to show you how T-HEAD C9xx work with the icache sync (We use hardware broadcast mechanism, and our icache is VIPT): - icache.i(v/p)a will broadcast all harts' icache invalidtion - sync.is will broadcast all harts' pipeline flush and ensure all broadcasts finished. This patch could improve the performance of OpenJDK on JIT and reduce flush_icache_all in linux. Epecially: static inline void set_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pteval) { if (pte_present(pteval) && pte_exec(pteval)) flush_icache_pte(pteval); set_pte(ptep, pteval); } Different from sbi_dma_sync, it can't be hidden in SBI and we must set up a framework to hold all vendors' implementations in linux/arch/riscv. Signed-off-by: Guo Ren Signed-off-by: Liu Shaohua Cc: Anup Patel Cc: Atish Patra Cc: Palmer Dabbelt Cc: Chen-Yu Tsai Cc: Drew Fustini Cc: Maxime Ripard Cc: Palmer Dabbelt Cc: Wei Fu Cc: Wei Wu --- arch/riscv/include/asm/cacheflush.h | 48 ++++++++++++++++++++++++++++++++++- arch/riscv/kernel/vdso/flush_icache.S | 33 +++++++++++++++++++----- arch/riscv/mm/cacheflush.c | 3 ++- 3 files changed, 76 insertions(+), 8 deletions(-) diff --git a/arch/riscv/include/asm/cacheflush.h b/arch/riscv/include/asm/cacheflush.h index 23ff703..2e2dba1 100644 --- a/arch/riscv/include/asm/cacheflush.h +++ b/arch/riscv/include/asm/cacheflush.h @@ -22,11 +22,57 @@ static inline void flush_dcache_page(struct page *page) } #define ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE 1 +#define ICACHE_IPA_X5 ".long 0x0382800b" +#define ICACHE_IVA_X5 ".long 0x0302800b" +#define SYNC_IS ".long 0x01b0000b" + +static inline void flush_icache_range(unsigned long start, unsigned long end) +{ + register unsigned long tmp asm("x5") = start & (~(L1_CACHE_BYTES-1)); + + for (; tmp < ALIGN(end, L1_CACHE_BYTES); tmp += L1_CACHE_BYTES) { + __asm__ __volatile__ ( + ICACHE_IVA_X5 + : + : "r" (tmp) + : "memory"); + } + + __asm__ __volatile__(SYNC_IS); + + return; +} + +static inline void flush_icache_range_phy(unsigned long start, unsigned long end) +{ + register unsigned long tmp asm("x5") = start & (~(L1_CACHE_BYTES-1)); + + for (; tmp < ALIGN(end, L1_CACHE_BYTES); tmp += L1_CACHE_BYTES) { + __asm__ __volatile__ ( + ICACHE_IPA_X5 + : + : "r" (tmp) + : "memory"); + } + + __asm__ __volatile__(SYNC_IS); + + return; +} + +static inline void __flush_icache_page(struct page *page) { + unsigned long start = PFN_PHYS(page_to_pfn(page)); + + flush_icache_range_phy(start, start + PAGE_SIZE); + + return; +} + /* * RISC-V doesn't have an instruction to flush parts of the instruction cache, * so instead we just flush the whole thing. */ -#define flush_icache_range(start, end) flush_icache_all() +#define flush_icache_range(start, end) flush_icache_range(start, end) #define flush_icache_user_page(vma, pg, addr, len) \ flush_icache_mm(vma->vm_mm, 0) diff --git a/arch/riscv/kernel/vdso/flush_icache.S b/arch/riscv/kernel/vdso/flush_icache.S index 82f97d6..efb2d2e 100644 --- a/arch/riscv/kernel/vdso/flush_icache.S +++ b/arch/riscv/kernel/vdso/flush_icache.S @@ -5,18 +5,39 @@ #include #include +#include + +/* + * icache.ipa rs1 + * | 31 - 25 | 24 - 20 | 19 - 15 | 14 - 12 | 11 - 7 | 6 - 0 | + * 0000001 11000 rs1 000 00000 0001011 + * + * icache.iva rs1 + * | 31 - 25 | 24 - 20 | 19 - 15 | 14 - 12 | 11 - 7 | 6 - 0 | + * 0000001 10000 rs1 000 00000 0001011 + * + * sync.is + * | 31 - 25 | 24 - 20 | 19 - 15 | 14 - 12 | 11 - 7 | 6 - 0 | + * 0000000 11011 00000 000 00000 0001011 + */ +#define ICACHE_IPA_X5 .long 0x0382800b +#define ICACHE_IVA_X5 .long 0x0302800b +#define SYNC_IS .long 0x01b0000b .text /* int __vdso_flush_icache(void *start, void *end, unsigned long flags); */ ENTRY(__vdso_flush_icache) .cfi_startproc -#ifdef CONFIG_SMP - li a7, __NR_riscv_flush_icache - ecall -#else - fence.i + srli t0, a0, L1_CACHE_SHIFT + slli t0, t0, L1_CACHE_SHIFT + addi a1, a1, (L1_CACHE_BYTES - 1) + srli a1, a1, L1_CACHE_SHIFT + slli a1, a1, L1_CACHE_SHIFT +1: ICACHE_IVA_X5 + addi t0, t0, L1_CACHE_BYTES + bne t0, a1, 1b + SYNC_IS li a0, 0 -#endif ret .cfi_endproc ENDPROC(__vdso_flush_icache) diff --git a/arch/riscv/mm/cacheflush.c b/arch/riscv/mm/cacheflush.c index 0941186..0fb8344 100644 --- a/arch/riscv/mm/cacheflush.c +++ b/arch/riscv/mm/cacheflush.c @@ -3,6 +3,7 @@ * Copyright (C) 2017 SiFive */ +#include #include #ifdef CONFIG_SMP @@ -84,6 +85,6 @@ void flush_icache_pte(pte_t pte) struct page *page = pte_page(pte); if (!test_and_set_bit(PG_dcache_clean, &page->flags)) - flush_icache_all(); + __flush_icache_page(page); } #endif /* CONFIG_MMU */ -- 2.7.4 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0E187C48BC2 for ; Sun, 6 Jun 2021 09:05:54 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C52C961420 for ; Sun, 6 Jun 2021 09:05:53 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C52C961420 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Q3xKBoVaPbD+s5+w9CtK91sozkCgScMnkCKUGYNDWBc=; b=2wuF468eSQ940o hcKnX2ITWlDaBAt+890JAI32zGX5NNCMZt7e43F1cRSIBW33OwuDUkTtrqvAVC9qjNOqSkNpdG2QO x25gWMHZl9QfNlw1t+NJuYCihMLKtzMGdjRma3Tdg46HDKlr7YpHl/j8OmS4JSJkMibbtDiFWZDZy M8FBnRUXC074tiW0TovQEisEZnjdKzvvxu4xvfEDqt436WNSFfUwqlWWhP4wlcl9mXFsLyUVrL5Fw f51mHDGZCDtfuLYxqhDPZ1yKXpZMqqmHwQ603474djqOIutnZi1z2zrq06UopS7aP0wst/vLIwjHn qjrDXUBLbZMiEMsggPmQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1lpoj2-0002rY-N5; Sun, 06 Jun 2021 09:05:44 +0000 Received: from mail.kernel.org ([198.145.29.99]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1lpoiw-0002op-Sy for linux-riscv@lists.infradead.org; Sun, 06 Jun 2021 09:05:42 +0000 Received: by mail.kernel.org (Postfix) with ESMTPSA id A6C7861426; Sun, 6 Jun 2021 09:05:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1622970338; bh=qph7YI4bBELR4x3cH34m49m6VFlBeHt4WLSEim7zato=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Uet+WCvoJuyWL0SFrd+1jc6Ilaj8YMQJHj1VWC39aPQPAR+SPjI1wrb7V8fb3fGh2 MEB9ca1PE+TXAUNVnNV+22vcxI8/Aup5ue6UqIY9NcIfDUHC/pYiKDcvW8Bg17F98s 2fO0OdAMZbh8VkNxMEbWm4fdnEn6yMA0IjK1oZQYrhRB73d/9G8LS1U7f+ZJxGFVKu ZqlIa8doFRapqKvNgCpGpw7NMHzRNu3cSBtDvH2WammdZLv0pYh/EudHQi7Q0bq/OM gVN6DKpvmO5B14prT9gTCYwAFoKWE3OVODc8VPoIEeHbdWqoU91glJiqloi3KAOnEf JnIQ2NHuhBKcw== From: guoren@kernel.org To: guoren@kernel.org, anup.patel@wdc.com, palmerdabbelt@google.com, arnd@arndb.de, wens@csie.org, maxime@cerno.tech, drew@beagleboard.org, liush@allwinnertech.com, lazyparser@gmail.com, wefu@redhat.com Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-sunxi@lists.linux.dev, Guo Ren , Atish Patra Subject: [RFC PATCH v2 08/11] riscv: cmo: Add vendor custom icache sync Date: Sun, 6 Jun 2021 09:04:06 +0000 Message-Id: <1622970249-50770-12-git-send-email-guoren@kernel.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1622970249-50770-1-git-send-email-guoren@kernel.org> References: <1622970249-50770-1-git-send-email-guoren@kernel.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210606_020538_991396_7130F5D1 X-CRM114-Status: GOOD ( 13.58 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Guo Ren It's a draft version to show you how T-HEAD C9xx work with the icache sync (We use hardware broadcast mechanism, and our icache is VIPT): - icache.i(v/p)a will broadcast all harts' icache invalidtion - sync.is will broadcast all harts' pipeline flush and ensure all broadcasts finished. This patch could improve the performance of OpenJDK on JIT and reduce flush_icache_all in linux. Epecially: static inline void set_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pteval) { if (pte_present(pteval) && pte_exec(pteval)) flush_icache_pte(pteval); set_pte(ptep, pteval); } Different from sbi_dma_sync, it can't be hidden in SBI and we must set up a framework to hold all vendors' implementations in linux/arch/riscv. Signed-off-by: Guo Ren Signed-off-by: Liu Shaohua Cc: Anup Patel Cc: Atish Patra Cc: Palmer Dabbelt Cc: Chen-Yu Tsai Cc: Drew Fustini Cc: Maxime Ripard Cc: Palmer Dabbelt Cc: Wei Fu Cc: Wei Wu --- arch/riscv/include/asm/cacheflush.h | 48 ++++++++++++++++++++++++++++++++++- arch/riscv/kernel/vdso/flush_icache.S | 33 +++++++++++++++++++----- arch/riscv/mm/cacheflush.c | 3 ++- 3 files changed, 76 insertions(+), 8 deletions(-) diff --git a/arch/riscv/include/asm/cacheflush.h b/arch/riscv/include/asm/cacheflush.h index 23ff703..2e2dba1 100644 --- a/arch/riscv/include/asm/cacheflush.h +++ b/arch/riscv/include/asm/cacheflush.h @@ -22,11 +22,57 @@ static inline void flush_dcache_page(struct page *page) } #define ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE 1 +#define ICACHE_IPA_X5 ".long 0x0382800b" +#define ICACHE_IVA_X5 ".long 0x0302800b" +#define SYNC_IS ".long 0x01b0000b" + +static inline void flush_icache_range(unsigned long start, unsigned long end) +{ + register unsigned long tmp asm("x5") = start & (~(L1_CACHE_BYTES-1)); + + for (; tmp < ALIGN(end, L1_CACHE_BYTES); tmp += L1_CACHE_BYTES) { + __asm__ __volatile__ ( + ICACHE_IVA_X5 + : + : "r" (tmp) + : "memory"); + } + + __asm__ __volatile__(SYNC_IS); + + return; +} + +static inline void flush_icache_range_phy(unsigned long start, unsigned long end) +{ + register unsigned long tmp asm("x5") = start & (~(L1_CACHE_BYTES-1)); + + for (; tmp < ALIGN(end, L1_CACHE_BYTES); tmp += L1_CACHE_BYTES) { + __asm__ __volatile__ ( + ICACHE_IPA_X5 + : + : "r" (tmp) + : "memory"); + } + + __asm__ __volatile__(SYNC_IS); + + return; +} + +static inline void __flush_icache_page(struct page *page) { + unsigned long start = PFN_PHYS(page_to_pfn(page)); + + flush_icache_range_phy(start, start + PAGE_SIZE); + + return; +} + /* * RISC-V doesn't have an instruction to flush parts of the instruction cache, * so instead we just flush the whole thing. */ -#define flush_icache_range(start, end) flush_icache_all() +#define flush_icache_range(start, end) flush_icache_range(start, end) #define flush_icache_user_page(vma, pg, addr, len) \ flush_icache_mm(vma->vm_mm, 0) diff --git a/arch/riscv/kernel/vdso/flush_icache.S b/arch/riscv/kernel/vdso/flush_icache.S index 82f97d6..efb2d2e 100644 --- a/arch/riscv/kernel/vdso/flush_icache.S +++ b/arch/riscv/kernel/vdso/flush_icache.S @@ -5,18 +5,39 @@ #include #include +#include + +/* + * icache.ipa rs1 + * | 31 - 25 | 24 - 20 | 19 - 15 | 14 - 12 | 11 - 7 | 6 - 0 | + * 0000001 11000 rs1 000 00000 0001011 + * + * icache.iva rs1 + * | 31 - 25 | 24 - 20 | 19 - 15 | 14 - 12 | 11 - 7 | 6 - 0 | + * 0000001 10000 rs1 000 00000 0001011 + * + * sync.is + * | 31 - 25 | 24 - 20 | 19 - 15 | 14 - 12 | 11 - 7 | 6 - 0 | + * 0000000 11011 00000 000 00000 0001011 + */ +#define ICACHE_IPA_X5 .long 0x0382800b +#define ICACHE_IVA_X5 .long 0x0302800b +#define SYNC_IS .long 0x01b0000b .text /* int __vdso_flush_icache(void *start, void *end, unsigned long flags); */ ENTRY(__vdso_flush_icache) .cfi_startproc -#ifdef CONFIG_SMP - li a7, __NR_riscv_flush_icache - ecall -#else - fence.i + srli t0, a0, L1_CACHE_SHIFT + slli t0, t0, L1_CACHE_SHIFT + addi a1, a1, (L1_CACHE_BYTES - 1) + srli a1, a1, L1_CACHE_SHIFT + slli a1, a1, L1_CACHE_SHIFT +1: ICACHE_IVA_X5 + addi t0, t0, L1_CACHE_BYTES + bne t0, a1, 1b + SYNC_IS li a0, 0 -#endif ret .cfi_endproc ENDPROC(__vdso_flush_icache) diff --git a/arch/riscv/mm/cacheflush.c b/arch/riscv/mm/cacheflush.c index 0941186..0fb8344 100644 --- a/arch/riscv/mm/cacheflush.c +++ b/arch/riscv/mm/cacheflush.c @@ -3,6 +3,7 @@ * Copyright (C) 2017 SiFive */ +#include #include #ifdef CONFIG_SMP @@ -84,6 +85,6 @@ void flush_icache_pte(pte_t pte) struct page *page = pte_page(pte); if (!test_and_set_bit(PG_dcache_clean, &page->flags)) - flush_icache_all(); + __flush_icache_page(page); } #endif /* CONFIG_MMU */ -- 2.7.4 _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv