From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-14.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2DF01C433B4 for ; Thu, 20 May 2021 17:03:48 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 6622261006 for ; Thu, 20 May 2021 17:03:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6622261006 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References:Message-ID: Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=cFyYZJyO0tVrhmqQQLt4cVh8rVkxmxKuIrJGVx5E6xs=; b=QToF9Fmcq7kPjmSspMY6TN22s5 jdKfvQ/h+woAVYAbMFpulBg1UmqvL9zx1yo5x/2y0GHGqclUi0XYlyOrIxW9yIUY7/7PuvsqUrpQb /BbJspchj2iMtlanGKDGT8zdndYu7vodiWY+4Fn+INjeAM8Ji9RzLFSANzo+jv/GQYsQCJNbG+LBT IyQr5TN9DRV4Mk1foDi0xlyezjoy4BokPP3jrtqFEDfQzeN8n3COimELSE4Zcjsk/p7iW7EHwRdrX WdWaT4uh94gyK7I/oFu6BGXgMvG+3JTmXC0uV5uwbLxNAfTwshxbu4q8BhQf3fctlWZul3jhv/yxQ hkMLXvgw==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1ljm3W-00228W-Fs; Thu, 20 May 2021 17:01:54 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1ljm3T-002287-5R for linux-arm-kernel@desiato.infradead.org; Thu, 20 May 2021 17:01:51 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=In-Reply-To:Content-Type:MIME-Version :References:Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=fEUrPr8Ej8az+9zvS19duxTPZoRdXFHTE/lY6Z1r2Oc=; b=HCOcWe+5DJniiQKPaBnIWHAW9E GKbGOPFcZWP6gybSb1czMHkkeCqFTM7r4yDWQK366mfaU4r1Ie4keWXxIAk1aJPvmZ2ENMdok/bbE z4XKvNrx3liqceAsEOqCdCAL98KeJWRU9xkxUFv4cvDQsH/rs/wLIa2K7A4lO1g9LQ5OAUd2iFktY fRqDH0wXJ/J/Eo+OmX7NmFbwq1voD11Qgr75zbDBX/ATkNh3jbTKcb3WHCra2K+Gmvq8P0EAuFaYK PGfztGxZPaPTfY4legFI9EeMuxz1xr4DxGcZ8MHGxMpn86RKMtYhbSnLxFnZ87oh6BI6jUUV8RpVt twfOixqg==; Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1ljm3O-00GWsr-5i for linux-arm-kernel@lists.infradead.org; Thu, 20 May 2021 17:01:49 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id A403311B3; Thu, 20 May 2021 10:01:42 -0700 (PDT) Received: from C02TD0UTHF1T.local (unknown [10.57.7.235]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id A2F4C3F719; Thu, 20 May 2021 10:01:40 -0700 (PDT) Date: Thu, 20 May 2021 18:01:38 +0100 From: Mark Rutland To: Fuad Tabba Cc: linux-arm-kernel@lists.infradead.org, will@kernel.org, catalin.marinas@arm.com, maz@kernel.org, ardb@kernel.org, james.morse@arm.com, alexandru.elisei@arm.com, suzuki.poulose@arm.com, robin.murphy@arm.com Subject: Re: [PATCH v3 18/18] arm64: Rename arm64-internal cache maintenance functions Message-ID: <20210520170138.GW17233@C02TD0UTHF1T.local> References: <20210520124406.2731873-1-tabba@google.com> <20210520124406.2731873-19-tabba@google.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20210520124406.2731873-19-tabba@google.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210520_100146_368135_A8E11685 X-CRM114-Status: GOOD ( 39.44 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Thu, May 20, 2021 at 01:44:06PM +0100, Fuad Tabba wrote: > Although naming across the codebase isn't that consistent, it > tends to follow certain patterns. Moreover, the term "flush" > isn't defined in the Arm Architecture reference manual, and might > be interpreted to mean clean, invalidate, or both for a cache. > > Rename arm64-internal functions to make the naming internally > consistent, as well as making it consistent with the Arm ARM, by > specifying whether it applies to the instruction, data, or both > caches, whether the operation is a clean, invalidate, or both. > Also specify which point the operation applies to, i.e., to the > point of unification (PoU), coherence (PoC), or persistence > (PoP). Minor nit: s/coherence/coherency/ > This commit applies the following sed transformation to all files > under arch/arm64: > > "s/\b__flush_cache_range\b/caches_clean_inval_pou_macro/g;"\ > "s/\b__flush_icache_range\b/caches_clean_inval_pou/g;"\ > "s/\binvalidate_icache_range\b/icache_inval_pou/g;"\ > "s/\b__flush_dcache_area\b/dcache_clean_inval_poc/g;"\ > "s/\b__inval_dcache_area\b/dcache_inval_poc/g;"\ > "s/__clean_dcache_area_poc\b/dcache_clean_poc/g;"\ > "s/\b__clean_dcache_area_pop\b/dcache_clean_pop/g;"\ > "s/\b__clean_dcache_area_pou\b/dcache_clean_pou/g;"\ > "s/\b__flush_cache_user_range\b/caches_clean_inval_user_pou/g;"\ > "s/\b__flush_icache_all\b/icache_inval_all_pou/g;" > > Note that __clean_dcache_area_poc is deliberately missing a word > boundary check at the beginning in order to match the efistub > symbols in image-vars.h. > > Also note that, despite its name, __flush_icache_range operates > on both instruction and data caches. The name change here > reflects that. > > No functional change intended. > > Signed-off-by: Fuad Tabba This looks great! It's especially nice to see "flush" gone, and it's now much more apparent whether a function affects the I-caches alone or also performs D-cache maintenance, so it should be much easier to avoid redundant maintenance. Acked-by: Mark Rutland I've built this and given it some light boot+userspace testing, so for the series: Tested-by: Mark Rutland Mark. > --- > arch/arm64/include/asm/arch_gicv3.h | 2 +- > arch/arm64/include/asm/cacheflush.h | 36 +++++++++--------- > arch/arm64/include/asm/efi.h | 2 +- > arch/arm64/include/asm/kvm_mmu.h | 6 +-- > arch/arm64/kernel/alternative.c | 2 +- > arch/arm64/kernel/efi-entry.S | 4 +- > arch/arm64/kernel/head.S | 8 ++-- > arch/arm64/kernel/hibernate-asm.S | 4 +- > arch/arm64/kernel/hibernate.c | 12 +++--- > arch/arm64/kernel/idreg-override.c | 2 +- > arch/arm64/kernel/image-vars.h | 2 +- > arch/arm64/kernel/insn.c | 2 +- > arch/arm64/kernel/kaslr.c | 6 +-- > arch/arm64/kernel/machine_kexec.c | 10 ++--- > arch/arm64/kernel/smp.c | 4 +- > arch/arm64/kernel/smp_spin_table.c | 4 +- > arch/arm64/kernel/sys_compat.c | 2 +- > arch/arm64/kvm/arm.c | 2 +- > arch/arm64/kvm/hyp/nvhe/cache.S | 4 +- > arch/arm64/kvm/hyp/nvhe/setup.c | 2 +- > arch/arm64/kvm/hyp/nvhe/tlb.c | 2 +- > arch/arm64/kvm/hyp/pgtable.c | 4 +- > arch/arm64/lib/uaccess_flushcache.c | 4 +- > arch/arm64/mm/cache.S | 58 ++++++++++++++--------------- > arch/arm64/mm/flush.c | 12 +++--- > 25 files changed, 98 insertions(+), 98 deletions(-) > > diff --git a/arch/arm64/include/asm/arch_gicv3.h b/arch/arm64/include/asm/arch_gicv3.h > index ed1cc9d8e6df..4ad22c3135db 100644 > --- a/arch/arm64/include/asm/arch_gicv3.h > +++ b/arch/arm64/include/asm/arch_gicv3.h > @@ -125,7 +125,7 @@ static inline u32 gic_read_rpr(void) > #define gic_write_lpir(v, c) writeq_relaxed(v, c) > > #define gic_flush_dcache_to_poc(a,l) \ > - __flush_dcache_area((unsigned long)(a), (unsigned long)(a)+(l)) > + dcache_clean_inval_poc((unsigned long)(a), (unsigned long)(a)+(l)) > > #define gits_read_baser(c) readq_relaxed(c) > #define gits_write_baser(v, c) writeq_relaxed(v, c) > diff --git a/arch/arm64/include/asm/cacheflush.h b/arch/arm64/include/asm/cacheflush.h > index 4b91d3530013..885bda37b805 100644 > --- a/arch/arm64/include/asm/cacheflush.h > +++ b/arch/arm64/include/asm/cacheflush.h > @@ -34,54 +34,54 @@ > * - start - virtual start address > * - end - virtual end address > * > - * __flush_icache_range(start, end) > + * caches_clean_inval_pou(start, end) > * > * Ensure coherency between the I-cache and the D-cache region to > * the Point of Unification. > * > - * __flush_cache_user_range(start, end) > + * caches_clean_inval_user_pou(start, end) > * > * Ensure coherency between the I-cache and the D-cache region to > * the Point of Unification. > * Use only if the region might access user memory. > * > - * invalidate_icache_range(start, end) > + * icache_inval_pou(start, end) > * > * Invalidate I-cache region to the Point of Unification. > * > - * __flush_dcache_area(start, end) > + * dcache_clean_inval_poc(start, end) > * > * Clean and invalidate D-cache region to the Point of Coherence. > * > - * __inval_dcache_area(start, end) > + * dcache_inval_poc(start, end) > * > * Invalidate D-cache region to the Point of Coherence. > * > - * __clean_dcache_area_poc(start, end) > + * dcache_clean_poc(start, end) > * > * Clean D-cache region to the Point of Coherence. > * > - * __clean_dcache_area_pop(start, end) > + * dcache_clean_pop(start, end) > * > * Clean D-cache region to the Point of Persistence. > * > - * __clean_dcache_area_pou(start, end) > + * dcache_clean_pou(start, end) > * > * Clean D-cache region to the Point of Unification. > */ > -extern void __flush_icache_range(unsigned long start, unsigned long end); > -extern void invalidate_icache_range(unsigned long start, unsigned long end); > -extern void __flush_dcache_area(unsigned long start, unsigned long end); > -extern void __inval_dcache_area(unsigned long start, unsigned long end); > -extern void __clean_dcache_area_poc(unsigned long start, unsigned long end); > -extern void __clean_dcache_area_pop(unsigned long start, unsigned long end); > -extern void __clean_dcache_area_pou(unsigned long start, unsigned long end); > -extern long __flush_cache_user_range(unsigned long start, unsigned long end); > +extern void caches_clean_inval_pou(unsigned long start, unsigned long end); > +extern void icache_inval_pou(unsigned long start, unsigned long end); > +extern void dcache_clean_inval_poc(unsigned long start, unsigned long end); > +extern void dcache_inval_poc(unsigned long start, unsigned long end); > +extern void dcache_clean_poc(unsigned long start, unsigned long end); > +extern void dcache_clean_pop(unsigned long start, unsigned long end); > +extern void dcache_clean_pou(unsigned long start, unsigned long end); > +extern long caches_clean_inval_user_pou(unsigned long start, unsigned long end); > extern void sync_icache_aliases(unsigned long start, unsigned long end); > > static inline void flush_icache_range(unsigned long start, unsigned long end) > { > - __flush_icache_range(start, end); > + caches_clean_inval_pou(start, end); > > /* > * IPI all online CPUs so that they undergo a context synchronization > @@ -135,7 +135,7 @@ extern void copy_to_user_page(struct vm_area_struct *, struct page *, > #define ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE 1 > extern void flush_dcache_page(struct page *); > > -static __always_inline void __flush_icache_all(void) > +static __always_inline void icache_inval_all_pou(void) > { > if (cpus_have_const_cap(ARM64_HAS_CACHE_DIC)) > return; > diff --git a/arch/arm64/include/asm/efi.h b/arch/arm64/include/asm/efi.h > index 0ae2397076fd..1bed37eb013a 100644 > --- a/arch/arm64/include/asm/efi.h > +++ b/arch/arm64/include/asm/efi.h > @@ -137,7 +137,7 @@ void efi_virtmap_unload(void); > > static inline void efi_capsule_flush_cache_range(void *addr, int size) > { > - __flush_dcache_area((unsigned long)addr, (unsigned long)addr + size); > + dcache_clean_inval_poc((unsigned long)addr, (unsigned long)addr + size); > } > > #endif /* _ASM_EFI_H */ > diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h > index 33293d5855af..f4cbfa9025a8 100644 > --- a/arch/arm64/include/asm/kvm_mmu.h > +++ b/arch/arm64/include/asm/kvm_mmu.h > @@ -181,7 +181,7 @@ static inline void *__kvm_vector_slot2addr(void *base, > struct kvm; > > #define kvm_flush_dcache_to_poc(a,l) \ > - __flush_dcache_area((unsigned long)(a), (unsigned long)(a)+(l)) > + dcache_clean_inval_poc((unsigned long)(a), (unsigned long)(a)+(l)) > > static inline bool vcpu_has_cache_enabled(struct kvm_vcpu *vcpu) > { > @@ -209,12 +209,12 @@ static inline void __invalidate_icache_guest_page(kvm_pfn_t pfn, > { > if (icache_is_aliasing()) { > /* any kind of VIPT cache */ > - __flush_icache_all(); > + icache_inval_all_pou(); > } else if (is_kernel_in_hyp_mode() || !icache_is_vpipt()) { > /* PIPT or VPIPT at EL2 (see comment in __kvm_tlb_flush_vmid_ipa) */ > void *va = page_address(pfn_to_page(pfn)); > > - invalidate_icache_range((unsigned long)va, > + icache_inval_pou((unsigned long)va, > (unsigned long)va + size); > } > } > diff --git a/arch/arm64/kernel/alternative.c b/arch/arm64/kernel/alternative.c > index c906d20c7b52..3fb79b76e9d9 100644 > --- a/arch/arm64/kernel/alternative.c > +++ b/arch/arm64/kernel/alternative.c > @@ -181,7 +181,7 @@ static void __nocfi __apply_alternatives(struct alt_region *region, bool is_modu > */ > if (!is_module) { > dsb(ish); > - __flush_icache_all(); > + icache_inval_all_pou(); > isb(); > > /* Ignore ARM64_CB bit from feature mask */ > diff --git a/arch/arm64/kernel/efi-entry.S b/arch/arm64/kernel/efi-entry.S > index 72e6a580290a..6668bad21f86 100644 > --- a/arch/arm64/kernel/efi-entry.S > +++ b/arch/arm64/kernel/efi-entry.S > @@ -29,7 +29,7 @@ SYM_CODE_START(efi_enter_kernel) > */ > ldr w1, =kernel_size > add x1, x0, x1 > - bl __clean_dcache_area_poc > + bl dcache_clean_poc > ic ialluis > > /* > @@ -38,7 +38,7 @@ SYM_CODE_START(efi_enter_kernel) > */ > adr x0, 0f > adr x1, 3f > - bl __clean_dcache_area_poc > + bl dcache_clean_poc > 0: > /* Turn off Dcache and MMU */ > mrs x0, CurrentEL > diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S > index 8df0ac8d9123..6928cb67d3a0 100644 > --- a/arch/arm64/kernel/head.S > +++ b/arch/arm64/kernel/head.S > @@ -118,7 +118,7 @@ SYM_CODE_START_LOCAL(preserve_boot_args) > // MMU off > > add x1, x0, #0x20 // 4 x 8 bytes > - b __inval_dcache_area // tail call > + b dcache_inval_poc // tail call > SYM_CODE_END(preserve_boot_args) > > /* > @@ -268,7 +268,7 @@ SYM_FUNC_START_LOCAL(__create_page_tables) > */ > adrp x0, init_pg_dir > adrp x1, init_pg_end > - bl __inval_dcache_area > + bl dcache_inval_poc > > /* > * Clear the init page tables. > @@ -381,11 +381,11 @@ SYM_FUNC_START_LOCAL(__create_page_tables) > > adrp x0, idmap_pg_dir > adrp x1, idmap_pg_end > - bl __inval_dcache_area > + bl dcache_inval_poc > > adrp x0, init_pg_dir > adrp x1, init_pg_end > - bl __inval_dcache_area > + bl dcache_inval_poc > > ret x28 > SYM_FUNC_END(__create_page_tables) > diff --git a/arch/arm64/kernel/hibernate-asm.S b/arch/arm64/kernel/hibernate-asm.S > index ef2ab7caf815..81c0186a5e32 100644 > --- a/arch/arm64/kernel/hibernate-asm.S > +++ b/arch/arm64/kernel/hibernate-asm.S > @@ -45,7 +45,7 @@ > * Because this code has to be copied to a 'safe' page, it can't call out to > * other functions by PC-relative address. Also remember that it may be > * mid-way through over-writing other functions. For this reason it contains > - * code from __flush_icache_range() and uses the copy_page() macro. > + * code from caches_clean_inval_pou() and uses the copy_page() macro. > * > * This 'safe' page is mapped via ttbr0, and executed from there. This function > * switches to a copy of the linear map in ttbr1, performs the restore, then > @@ -87,7 +87,7 @@ SYM_CODE_START(swsusp_arch_suspend_exit) > copy_page x0, x1, x2, x3, x4, x5, x6, x7, x8, x9 > > add x1, x10, #PAGE_SIZE > - /* Clean the copied page to PoU - based on __flush_icache_range() */ > + /* Clean the copied page to PoU - based on caches_clean_inval_pou() */ > raw_dcache_line_size x2, x3 > sub x3, x2, #1 > bic x4, x10, x3 > diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c > index b40ddce71507..46a0b4d6e251 100644 > --- a/arch/arm64/kernel/hibernate.c > +++ b/arch/arm64/kernel/hibernate.c > @@ -210,7 +210,7 @@ static int create_safe_exec_page(void *src_start, size_t length, > return -ENOMEM; > > memcpy(page, src_start, length); > - __flush_icache_range((unsigned long)page, (unsigned long)page + length); > + caches_clean_inval_pou((unsigned long)page, (unsigned long)page + length); > rc = trans_pgd_idmap_page(&trans_info, &trans_ttbr0, &t0sz, page); > if (rc) > return rc; > @@ -381,17 +381,17 @@ int swsusp_arch_suspend(void) > ret = swsusp_save(); > } else { > /* Clean kernel core startup/idle code to PoC*/ > - __flush_dcache_area((unsigned long)__mmuoff_data_start, > + dcache_clean_inval_poc((unsigned long)__mmuoff_data_start, > (unsigned long)__mmuoff_data_end); > - __flush_dcache_area((unsigned long)__idmap_text_start, > + dcache_clean_inval_poc((unsigned long)__idmap_text_start, > (unsigned long)__idmap_text_end); > > /* Clean kvm setup code to PoC? */ > if (el2_reset_needed()) { > - __flush_dcache_area( > + dcache_clean_inval_poc( > (unsigned long)__hyp_idmap_text_start, > (unsigned long)__hyp_idmap_text_end); > - __flush_dcache_area((unsigned long)__hyp_text_start, > + dcache_clean_inval_poc((unsigned long)__hyp_text_start, > (unsigned long)__hyp_text_end); > } > > @@ -477,7 +477,7 @@ int swsusp_arch_resume(void) > * The hibernate exit text contains a set of el2 vectors, that will > * be executed at el2 with the mmu off in order to reload hyp-stub. > */ > - __flush_dcache_area((unsigned long)hibernate_exit, > + dcache_clean_inval_poc((unsigned long)hibernate_exit, > (unsigned long)hibernate_exit + exit_size); > > /* > diff --git a/arch/arm64/kernel/idreg-override.c b/arch/arm64/kernel/idreg-override.c > index 3dd515baf526..53a381a7f65d 100644 > --- a/arch/arm64/kernel/idreg-override.c > +++ b/arch/arm64/kernel/idreg-override.c > @@ -237,7 +237,7 @@ asmlinkage void __init init_feature_override(void) > > for (i = 0; i < ARRAY_SIZE(regs); i++) { > if (regs[i]->override) > - __flush_dcache_area((unsigned long)regs[i]->override, > + dcache_clean_inval_poc((unsigned long)regs[i]->override, > (unsigned long)regs[i]->override + > sizeof(*regs[i]->override)); > } > diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h > index bcf3c2755370..c96a9a0043bf 100644 > --- a/arch/arm64/kernel/image-vars.h > +++ b/arch/arm64/kernel/image-vars.h > @@ -35,7 +35,7 @@ __efistub_strnlen = __pi_strnlen; > __efistub_strcmp = __pi_strcmp; > __efistub_strncmp = __pi_strncmp; > __efistub_strrchr = __pi_strrchr; > -__efistub___clean_dcache_area_poc = __pi___clean_dcache_area_poc; > +__efistub_dcache_clean_poc = __pi_dcache_clean_poc; > > #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) > __efistub___memcpy = __pi_memcpy; > diff --git a/arch/arm64/kernel/insn.c b/arch/arm64/kernel/insn.c > index 6c0de2f60ea9..51cb8dc98d00 100644 > --- a/arch/arm64/kernel/insn.c > +++ b/arch/arm64/kernel/insn.c > @@ -198,7 +198,7 @@ int __kprobes aarch64_insn_patch_text_nosync(void *addr, u32 insn) > > ret = aarch64_insn_write(tp, insn); > if (ret == 0) > - __flush_icache_range((uintptr_t)tp, > + caches_clean_inval_pou((uintptr_t)tp, > (uintptr_t)tp + AARCH64_INSN_SIZE); > > return ret; > diff --git a/arch/arm64/kernel/kaslr.c b/arch/arm64/kernel/kaslr.c > index 49cccd03cb37..cfa2cfde3019 100644 > --- a/arch/arm64/kernel/kaslr.c > +++ b/arch/arm64/kernel/kaslr.c > @@ -72,7 +72,7 @@ u64 __init kaslr_early_init(void) > * we end up running with module randomization disabled. > */ > module_alloc_base = (u64)_etext - MODULES_VSIZE; > - __flush_dcache_area((unsigned long)&module_alloc_base, > + dcache_clean_inval_poc((unsigned long)&module_alloc_base, > (unsigned long)&module_alloc_base + > sizeof(module_alloc_base)); > > @@ -172,10 +172,10 @@ u64 __init kaslr_early_init(void) > module_alloc_base += (module_range * (seed & ((1 << 21) - 1))) >> 21; > module_alloc_base &= PAGE_MASK; > > - __flush_dcache_area((unsigned long)&module_alloc_base, > + dcache_clean_inval_poc((unsigned long)&module_alloc_base, > (unsigned long)&module_alloc_base + > sizeof(module_alloc_base)); > - __flush_dcache_area((unsigned long)&memstart_offset_seed, > + dcache_clean_inval_poc((unsigned long)&memstart_offset_seed, > (unsigned long)&memstart_offset_seed + > sizeof(memstart_offset_seed)); > > diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c > index 3e79110c8f3a..03ceabe4d912 100644 > --- a/arch/arm64/kernel/machine_kexec.c > +++ b/arch/arm64/kernel/machine_kexec.c > @@ -72,10 +72,10 @@ int machine_kexec_post_load(struct kimage *kimage) > * For execution with the MMU off, reloc_code needs to be cleaned to the > * PoC and invalidated from the I-cache. > */ > - __flush_dcache_area((unsigned long)reloc_code, > + dcache_clean_inval_poc((unsigned long)reloc_code, > (unsigned long)reloc_code + > arm64_relocate_new_kernel_size); > - invalidate_icache_range((uintptr_t)reloc_code, > + icache_inval_pou((uintptr_t)reloc_code, > (uintptr_t)reloc_code + > arm64_relocate_new_kernel_size); > > @@ -111,7 +111,7 @@ static void kexec_list_flush(struct kimage *kimage) > unsigned long addr; > > /* flush the list entries. */ > - __flush_dcache_area((unsigned long)entry, > + dcache_clean_inval_poc((unsigned long)entry, > (unsigned long)entry + > sizeof(kimage_entry_t)); > > @@ -128,7 +128,7 @@ static void kexec_list_flush(struct kimage *kimage) > break; > case IND_SOURCE: > /* flush the source pages. */ > - __flush_dcache_area(addr, addr + PAGE_SIZE); > + dcache_clean_inval_poc(addr, addr + PAGE_SIZE); > break; > case IND_DESTINATION: > break; > @@ -155,7 +155,7 @@ static void kexec_segment_flush(const struct kimage *kimage) > kimage->segment[i].memsz, > kimage->segment[i].memsz / PAGE_SIZE); > > - __flush_dcache_area( > + dcache_clean_inval_poc( > (unsigned long)phys_to_virt(kimage->segment[i].mem), > (unsigned long)phys_to_virt(kimage->segment[i].mem) + > kimage->segment[i].memsz); > diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c > index 5fcdee331087..9b4c1118194d 100644 > --- a/arch/arm64/kernel/smp.c > +++ b/arch/arm64/kernel/smp.c > @@ -122,7 +122,7 @@ int __cpu_up(unsigned int cpu, struct task_struct *idle) > secondary_data.task = idle; > secondary_data.stack = task_stack_page(idle) + THREAD_SIZE; > update_cpu_boot_status(CPU_MMU_OFF); > - __flush_dcache_area((unsigned long)&secondary_data, > + dcache_clean_inval_poc((unsigned long)&secondary_data, > (unsigned long)&secondary_data + > sizeof(secondary_data)); > > @@ -145,7 +145,7 @@ int __cpu_up(unsigned int cpu, struct task_struct *idle) > pr_crit("CPU%u: failed to come online\n", cpu); > secondary_data.task = NULL; > secondary_data.stack = NULL; > - __flush_dcache_area((unsigned long)&secondary_data, > + dcache_clean_inval_poc((unsigned long)&secondary_data, > (unsigned long)&secondary_data + > sizeof(secondary_data)); > status = READ_ONCE(secondary_data.status); > diff --git a/arch/arm64/kernel/smp_spin_table.c b/arch/arm64/kernel/smp_spin_table.c > index 58d804582a35..7e1624ecab3c 100644 > --- a/arch/arm64/kernel/smp_spin_table.c > +++ b/arch/arm64/kernel/smp_spin_table.c > @@ -36,7 +36,7 @@ static void write_pen_release(u64 val) > unsigned long size = sizeof(secondary_holding_pen_release); > > secondary_holding_pen_release = val; > - __flush_dcache_area((unsigned long)start, (unsigned long)start + size); > + dcache_clean_inval_poc((unsigned long)start, (unsigned long)start + size); > } > > > @@ -90,7 +90,7 @@ static int smp_spin_table_cpu_prepare(unsigned int cpu) > * the boot protocol. > */ > writeq_relaxed(pa_holding_pen, release_addr); > - __flush_dcache_area((__force unsigned long)release_addr, > + dcache_clean_inval_poc((__force unsigned long)release_addr, > (__force unsigned long)release_addr + > sizeof(*release_addr)); > > diff --git a/arch/arm64/kernel/sys_compat.c b/arch/arm64/kernel/sys_compat.c > index 265fe3eb1069..db5159a3055f 100644 > --- a/arch/arm64/kernel/sys_compat.c > +++ b/arch/arm64/kernel/sys_compat.c > @@ -41,7 +41,7 @@ __do_compat_cache_op(unsigned long start, unsigned long end) > dsb(ish); > } > > - ret = __flush_cache_user_range(start, start + chunk); > + ret = caches_clean_inval_user_pou(start, start + chunk); > if (ret) > return ret; > > diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c > index 1cb39c0803a4..c1953f65ca0e 100644 > --- a/arch/arm64/kvm/arm.c > +++ b/arch/arm64/kvm/arm.c > @@ -1064,7 +1064,7 @@ static int kvm_arch_vcpu_ioctl_vcpu_init(struct kvm_vcpu *vcpu, > if (!cpus_have_final_cap(ARM64_HAS_STAGE2_FWB)) > stage2_unmap_vm(vcpu->kvm); > else > - __flush_icache_all(); > + icache_inval_all_pou(); > } > > vcpu_reset_hcr(vcpu); > diff --git a/arch/arm64/kvm/hyp/nvhe/cache.S b/arch/arm64/kvm/hyp/nvhe/cache.S > index 36cef6915428..958734f4d6b0 100644 > --- a/arch/arm64/kvm/hyp/nvhe/cache.S > +++ b/arch/arm64/kvm/hyp/nvhe/cache.S > @@ -7,7 +7,7 @@ > #include > #include > > -SYM_FUNC_START_PI(__flush_dcache_area) > +SYM_FUNC_START_PI(dcache_clean_inval_poc) > dcache_by_line_op civac, sy, x0, x1, x2, x3 > ret > -SYM_FUNC_END_PI(__flush_dcache_area) > +SYM_FUNC_END_PI(dcache_clean_inval_poc) > diff --git a/arch/arm64/kvm/hyp/nvhe/setup.c b/arch/arm64/kvm/hyp/nvhe/setup.c > index 5dffe928f256..8143ebd4fb72 100644 > --- a/arch/arm64/kvm/hyp/nvhe/setup.c > +++ b/arch/arm64/kvm/hyp/nvhe/setup.c > @@ -134,7 +134,7 @@ static void update_nvhe_init_params(void) > for (i = 0; i < hyp_nr_cpus; i++) { > params = per_cpu_ptr(&kvm_init_params, i); > params->pgd_pa = __hyp_pa(pkvm_pgtable.pgd); > - __flush_dcache_area((unsigned long)params, > + dcache_clean_inval_poc((unsigned long)params, > (unsigned long)params + sizeof(*params)); > } > } > diff --git a/arch/arm64/kvm/hyp/nvhe/tlb.c b/arch/arm64/kvm/hyp/nvhe/tlb.c > index 83dc3b271bc5..38ed0f6f2703 100644 > --- a/arch/arm64/kvm/hyp/nvhe/tlb.c > +++ b/arch/arm64/kvm/hyp/nvhe/tlb.c > @@ -104,7 +104,7 @@ void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, > * you should be running with VHE enabled. > */ > if (icache_is_vpipt()) > - __flush_icache_all(); > + icache_inval_all_pou(); > > __tlb_switch_to_host(&cxt); > } > diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c > index 10d2f04013d4..e9ad7fb28ee3 100644 > --- a/arch/arm64/kvm/hyp/pgtable.c > +++ b/arch/arm64/kvm/hyp/pgtable.c > @@ -841,7 +841,7 @@ static int stage2_unmap_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, > if (need_flush) { > kvm_pte_t *pte_follow = kvm_pte_follow(pte, mm_ops); > > - __flush_dcache_area((unsigned long)pte_follow, > + dcache_clean_inval_poc((unsigned long)pte_follow, > (unsigned long)pte_follow + > kvm_granule_size(level)); > } > @@ -997,7 +997,7 @@ static int stage2_flush_walker(u64 addr, u64 end, u32 level, kvm_pte_t *ptep, > return 0; > > pte_follow = kvm_pte_follow(pte, mm_ops); > - __flush_dcache_area((unsigned long)pte_follow, > + dcache_clean_inval_poc((unsigned long)pte_follow, > (unsigned long)pte_follow + > kvm_granule_size(level)); > return 0; > diff --git a/arch/arm64/lib/uaccess_flushcache.c b/arch/arm64/lib/uaccess_flushcache.c > index 62ea989effe8..baee22961bdb 100644 > --- a/arch/arm64/lib/uaccess_flushcache.c > +++ b/arch/arm64/lib/uaccess_flushcache.c > @@ -15,7 +15,7 @@ void memcpy_flushcache(void *dst, const void *src, size_t cnt) > * barrier to order the cache maintenance against the memcpy. > */ > memcpy(dst, src, cnt); > - __clean_dcache_area_pop((unsigned long)dst, (unsigned long)dst + cnt); > + dcache_clean_pop((unsigned long)dst, (unsigned long)dst + cnt); > } > EXPORT_SYMBOL_GPL(memcpy_flushcache); > > @@ -33,6 +33,6 @@ unsigned long __copy_user_flushcache(void *to, const void __user *from, > rc = raw_copy_from_user(to, from, n); > > /* See above */ > - __clean_dcache_area_pop((unsigned long)to, (unsigned long)to + n - rc); > + dcache_clean_pop((unsigned long)to, (unsigned long)to + n - rc); > return rc; > } > diff --git a/arch/arm64/mm/cache.S b/arch/arm64/mm/cache.S > index b70a6699c02b..e799a4999299 100644 > --- a/arch/arm64/mm/cache.S > +++ b/arch/arm64/mm/cache.S > @@ -15,7 +15,7 @@ > #include > > /* > - * __flush_cache_range(start,end) [fixup] > + * caches_clean_inval_pou_macro(start,end) [fixup] > * > * Ensure that the I and D caches are coherent within specified region. > * This is typically used when code has been written to a memory region, > @@ -25,7 +25,7 @@ > * - end - virtual end address of region > * - fixup - optional label to branch to on user fault > */ > -.macro __flush_cache_range, fixup > +.macro caches_clean_inval_pou_macro, fixup > alternative_if ARM64_HAS_CACHE_IDC > dsb ishst > b .Ldc_skip_\@ > @@ -50,7 +50,7 @@ alternative_else_nop_endif > .endm > > /* > - * __flush_icache_range(start,end) > + * caches_clean_inval_pou(start,end) > * > * Ensure that the I and D caches are coherent within specified region. > * This is typically used when code has been written to a memory region, > @@ -59,13 +59,13 @@ alternative_else_nop_endif > * - start - virtual start address of region > * - end - virtual end address of region > */ > -SYM_FUNC_START(__flush_icache_range) > - __flush_cache_range > +SYM_FUNC_START(caches_clean_inval_pou) > + caches_clean_inval_pou_macro > ret > -SYM_FUNC_END(__flush_icache_range) > +SYM_FUNC_END(caches_clean_inval_pou) > > /* > - * __flush_cache_user_range(start,end) > + * caches_clean_inval_user_pou(start,end) > * > * Ensure that the I and D caches are coherent within specified region. > * This is typically used when code has been written to a memory region, > @@ -74,10 +74,10 @@ SYM_FUNC_END(__flush_icache_range) > * - start - virtual start address of region > * - end - virtual end address of region > */ > -SYM_FUNC_START(__flush_cache_user_range) > +SYM_FUNC_START(caches_clean_inval_user_pou) > uaccess_ttbr0_enable x2, x3, x4 > > - __flush_cache_range 2f > + caches_clean_inval_pou_macro 2f > mov x0, xzr > 1: > uaccess_ttbr0_disable x1, x2 > @@ -85,17 +85,17 @@ SYM_FUNC_START(__flush_cache_user_range) > 2: > mov x0, #-EFAULT > b 1b > -SYM_FUNC_END(__flush_cache_user_range) > +SYM_FUNC_END(caches_clean_inval_user_pou) > > /* > - * invalidate_icache_range(start,end) > + * icache_inval_pou(start,end) > * > * Ensure that the I cache is invalid within specified region. > * > * - start - virtual start address of region > * - end - virtual end address of region > */ > -SYM_FUNC_START(invalidate_icache_range) > +SYM_FUNC_START(icache_inval_pou) > alternative_if ARM64_HAS_CACHE_DIC > isb > ret > @@ -103,10 +103,10 @@ alternative_else_nop_endif > > invalidate_icache_by_line x0, x1, x2, x3 > ret > -SYM_FUNC_END(invalidate_icache_range) > +SYM_FUNC_END(icache_inval_pou) > > /* > - * __flush_dcache_area(start, end) > + * dcache_clean_inval_poc(start, end) > * > * Ensure that any D-cache lines for the interval [start, end) > * are cleaned and invalidated to the PoC. > @@ -114,13 +114,13 @@ SYM_FUNC_END(invalidate_icache_range) > * - start - virtual start address of region > * - end - virtual end address of region > */ > -SYM_FUNC_START_PI(__flush_dcache_area) > +SYM_FUNC_START_PI(dcache_clean_inval_poc) > dcache_by_line_op civac, sy, x0, x1, x2, x3 > ret > -SYM_FUNC_END_PI(__flush_dcache_area) > +SYM_FUNC_END_PI(dcache_clean_inval_poc) > > /* > - * __clean_dcache_area_pou(start, end) > + * dcache_clean_pou(start, end) > * > * Ensure that any D-cache lines for the interval [start, end) > * are cleaned to the PoU. > @@ -128,17 +128,17 @@ SYM_FUNC_END_PI(__flush_dcache_area) > * - start - virtual start address of region > * - end - virtual end address of region > */ > -SYM_FUNC_START(__clean_dcache_area_pou) > +SYM_FUNC_START(dcache_clean_pou) > alternative_if ARM64_HAS_CACHE_IDC > dsb ishst > ret > alternative_else_nop_endif > dcache_by_line_op cvau, ish, x0, x1, x2, x3 > ret > -SYM_FUNC_END(__clean_dcache_area_pou) > +SYM_FUNC_END(dcache_clean_pou) > > /* > - * __inval_dcache_area(start, end) > + * dcache_inval_poc(start, end) > * > * Ensure that any D-cache lines for the interval [start, end) > * are invalidated. Any partial lines at the ends of the interval are > @@ -148,7 +148,7 @@ SYM_FUNC_END(__clean_dcache_area_pou) > * - end - kernel end address of region > */ > SYM_FUNC_START_LOCAL(__dma_inv_area) > -SYM_FUNC_START_PI(__inval_dcache_area) > +SYM_FUNC_START_PI(dcache_inval_poc) > /* FALLTHROUGH */ > > /* > @@ -173,11 +173,11 @@ SYM_FUNC_START_PI(__inval_dcache_area) > b.lo 2b > dsb sy > ret > -SYM_FUNC_END_PI(__inval_dcache_area) > +SYM_FUNC_END_PI(dcache_inval_poc) > SYM_FUNC_END(__dma_inv_area) > > /* > - * __clean_dcache_area_poc(start, end) > + * dcache_clean_poc(start, end) > * > * Ensure that any D-cache lines for the interval [start, end) > * are cleaned to the PoC. > @@ -186,7 +186,7 @@ SYM_FUNC_END(__dma_inv_area) > * - end - virtual end address of region > */ > SYM_FUNC_START_LOCAL(__dma_clean_area) > -SYM_FUNC_START_PI(__clean_dcache_area_poc) > +SYM_FUNC_START_PI(dcache_clean_poc) > /* FALLTHROUGH */ > > /* > @@ -196,11 +196,11 @@ SYM_FUNC_START_PI(__clean_dcache_area_poc) > */ > dcache_by_line_op cvac, sy, x0, x1, x2, x3 > ret > -SYM_FUNC_END_PI(__clean_dcache_area_poc) > +SYM_FUNC_END_PI(dcache_clean_poc) > SYM_FUNC_END(__dma_clean_area) > > /* > - * __clean_dcache_area_pop(start, end) > + * dcache_clean_pop(start, end) > * > * Ensure that any D-cache lines for the interval [start, end) > * are cleaned to the PoP. > @@ -208,13 +208,13 @@ SYM_FUNC_END(__dma_clean_area) > * - start - virtual start address of region > * - end - virtual end address of region > */ > -SYM_FUNC_START_PI(__clean_dcache_area_pop) > +SYM_FUNC_START_PI(dcache_clean_pop) > alternative_if_not ARM64_HAS_DCPOP > - b __clean_dcache_area_poc > + b dcache_clean_poc > alternative_else_nop_endif > dcache_by_line_op cvap, sy, x0, x1, x2, x3 > ret > -SYM_FUNC_END_PI(__clean_dcache_area_pop) > +SYM_FUNC_END_PI(dcache_clean_pop) > > /* > * __dma_flush_area(start, size) > diff --git a/arch/arm64/mm/flush.c b/arch/arm64/mm/flush.c > index 143f625e7727..5fea9a3f6663 100644 > --- a/arch/arm64/mm/flush.c > +++ b/arch/arm64/mm/flush.c > @@ -17,14 +17,14 @@ > void sync_icache_aliases(unsigned long start, unsigned long end) > { > if (icache_is_aliasing()) { > - __clean_dcache_area_pou(start, end); > - __flush_icache_all(); > + dcache_clean_pou(start, end); > + icache_inval_all_pou(); > } else { > /* > * Don't issue kick_all_cpus_sync() after I-cache invalidation > * for user mappings. > */ > - __flush_icache_range(start, end); > + caches_clean_inval_pou(start, end); > } > } > > @@ -76,20 +76,20 @@ EXPORT_SYMBOL(flush_dcache_page); > /* > * Additional functions defined in assembly. > */ > -EXPORT_SYMBOL(__flush_icache_range); > +EXPORT_SYMBOL(caches_clean_inval_pou); > > #ifdef CONFIG_ARCH_HAS_PMEM_API > void arch_wb_cache_pmem(void *addr, size_t size) > { > /* Ensure order against any prior non-cacheable writes */ > dmb(osh); > - __clean_dcache_area_pop((unsigned long)addr, (unsigned long)addr + size); > + dcache_clean_pop((unsigned long)addr, (unsigned long)addr + size); > } > EXPORT_SYMBOL_GPL(arch_wb_cache_pmem); > > void arch_invalidate_pmem(void *addr, size_t size) > { > - __inval_dcache_area((unsigned long)addr, (unsigned long)addr + size); > + dcache_inval_poc((unsigned long)addr, (unsigned long)addr + size); > } > EXPORT_SYMBOL_GPL(arch_invalidate_pmem); > #endif > -- > 2.31.1.751.gd2f1c929bd-goog > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel