From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 92006C433F5 for ; Fri, 3 Dec 2021 10:14:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=uAVIAolfD6cnptDKKOXXGITVc4Jx/KKcTNq2VBLhiRs=; b=mCNGIaa9ERcfZj rXlF5wrNlrQLgfAGqV7L7pz4e5SHDNB2hoRtdlmBDotoQipD5NMLA5UEBaK4oKy3H1XM3qyvTeISK TQkWTkH80O0Y7Miw2mLML6TaUnP4077A+99nVXusRN8uNXhGvqFppXlW3fzKXrGCHevMceceHd+Wl uYrWgnmGi1G/Mg0sKTgL3woh1OZBuzPvAGNAP9EsiPiobLL1TOEJEL5gU5bjHAmpcU2YX6u32/wQ9 8a6uwnfEPDnCrVtWkgBP2jeG9Uoe3J852OeZ0dVJqlv9suuhzV3EqaMQrgrnNNXu5+fhpT4/lXqHo 03CObxNDyHUUcJrgCDaQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mt5Xz-00F5rD-AS; Fri, 03 Dec 2021 10:12:07 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mt5Vb-00F4ju-HC for linux-arm-kernel@lists.infradead.org; Fri, 03 Dec 2021 10:09:41 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 1E16362997; Fri, 3 Dec 2021 10:09:39 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E0132C53FC7; Fri, 3 Dec 2021 10:09:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638526178; bh=PhC6cgPQoRMJeeY3zWraiO5mByR1ipXljHxzSVWaL3A=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=W35JWZRsDO8wdv25YndXyR2Zs1FZVMOeSvC8sSVbbxaXeq+AiK5bUYEjcn1r4BrNm FRThR47fhwFHsbA42ufnqUtOwb+Tg3lET7XfOo13qKS9TnL+wNvLmJX37vtBIkWsMw Z8AigzmtPqltZcx2W4Jhkp/uHoEVm2ROSlYfhW27yfyUG6oFxv/myiXoo9FiJBUWBG NJwuH7JGapTgnO44jvucQIdL+i08IgqXwWO9udKFUQZ5KEh/vBqCP4L/uRt/qL06uh dr9onLPOhjbWXiBc7X2fOTfgajTI+fYEEv3aQb//AtEcS4g0aObVuut6c5PPMPSsmB ksCPzbe33A+Fw== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org, linux@armlinux.org.uk Cc: Ard Biesheuvel , Nicolas Pitre , Arnd Bergmann , Kees Cook , Keith Packard , Linus Walleij , Nick Desaulniers , Tony Lindgren , Marc Zyngier , Vladimir Murzin , Jesse Taube Subject: [PATCH v3 09/14] ARM: assembler: add optimized ldr/str macros to load variables from memory Date: Fri, 3 Dec 2021 11:08:58 +0100 Message-Id: <20211203100903.334206-10-ardb@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211203100903.334206-1-ardb@kernel.org> References: <20211203100903.334206-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=3862; h=from:subject; bh=PhC6cgPQoRMJeeY3zWraiO5mByR1ipXljHxzSVWaL3A=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBhqeyyFXyXaRSkaqrUa/r+5mQG/z4RCP1voiqXiEoH g/pSCmGJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYanssgAKCRDDTyI5ktmPJKUkC/ 9ZKP+l19E03zXoDhTZUwaOg8uVxy0VczCXO1+RQ745oHm+apzLq4j7WkI48PQdBIC3lJd3BtpHAyOx yViBMBChMStCf0rOOmba4Vvt2dG/cHAHdbvIJ5eoNZOB53+Heg36l8T3YEkpfPfeUKVpxyitNqaUMn 0cWK5Wf9fV+tgOZKDxMwt/UtYV2+wdRdGxCbVIhuFLy5IeumGXb9EaW7lRpBxU5P/fumKOcVCPFrZ5 WaBVOn96THDvRVtCm2PIxxW3KrqMoGjbi+AhjITrxguVFUk4gvtlRoCMpL2qqPjRY1qvtmz+6SqkGk EDy/qb5IYsmwuiCcDE/cG63MOEtYpIQ0f4MJpMvO61KkSjac+h2Aa2Y7YeMXWbL5Kba4KeuEv2vrKp MuTOUGBtTdNGwLkr3EssUxGAsrwmx0VDosJoOg/gqclyK3QWW51L2FbAQHpXbAVY8xcoWJcfNG1HJk hE8I0oXkQoyHhRQM1L4ZHh9xokTdiA5zNk5tted/8pPPY= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211203_020939_688766_0C477FDC X-CRM114-Status: GOOD ( 18.86 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org We will be adding variable loads to various hot paths, so it makes sense to add a helper macro that can load variables from asm code without the use of literal pool entries. On v7 or later, we can simply use MOVW/MOVT pairs, but on earlier cores, this requires a bit of hackery to emit a instruction sequence that implements this using a sequence of ADD/LDR instructions. Acked-by: Linus Walleij Acked-by: Nicolas Pitre Signed-off-by: Ard Biesheuvel --- arch/arm/include/asm/assembler.h | 45 ++++++++++++++++++-- arch/arm/kernel/entry-armv.S | 2 +- arch/arm/kernel/entry-header.S | 2 +- 3 files changed, 43 insertions(+), 6 deletions(-) diff --git a/arch/arm/include/asm/assembler.h b/arch/arm/include/asm/assembler.h index 1b9d4df331aa..2095638b7140 100644 --- a/arch/arm/include/asm/assembler.h +++ b/arch/arm/include/asm/assembler.h @@ -568,12 +568,12 @@ THUMB( orr \reg , \reg , #PSR_T_BIT ) /* * mov_l - move a constant value or [relocated] address into a register */ - .macro mov_l, dst:req, imm:req + .macro mov_l, dst:req, imm:req, cond .if __LINUX_ARM_ARCH__ < 7 - ldr \dst, =\imm + ldr\cond \dst, =\imm .else - movw \dst, #:lower16:\imm - movt \dst, #:upper16:\imm + movw\cond \dst, #:lower16:\imm + movt\cond \dst, #:upper16:\imm .endif .endm @@ -611,6 +611,43 @@ THUMB( orr \reg , \reg , #PSR_T_BIT ) __adldst_l str, \src, \sym, \tmp, \cond .endm + .macro __ldst_va, op, reg, tmp, sym, cond +#if __LINUX_ARM_ARCH__ >= 7 || \ + (defined(MODULE) && defined(CONFIG_ARM_MODULE_PLTS)) || \ + (defined(CONFIG_LD_IS_LLD) && CONFIG_LLD_VERSION < 140000) + mov_l \tmp, \sym, \cond + \op\cond \reg, [\tmp] +#else + /* + * Avoid a literal load, by emitting a sequence of ADD/LDR instructions + * with the appropriate relocations. The combined sequence has a range + * of -/+ 256 MiB, which should be sufficient for the core kernel and + * for modules loaded into the module region. + */ + .globl \sym + .reloc .L0_\@, R_ARM_ALU_PC_G0_NC, \sym + .reloc .L1_\@, R_ARM_ALU_PC_G1_NC, \sym + .reloc .L2_\@, R_ARM_LDR_PC_G2, \sym +.L0_\@: sub\cond \tmp, pc, #8 +.L1_\@: sub\cond \tmp, \tmp, #4 +.L2_\@: \op\cond \reg, [\tmp, #0] +#endif + .endm + + /* + * ldr_va - load a 32-bit word from the virtual address of \sym + */ + .macro ldr_va, rd:req, sym:req, cond + __ldst_va ldr, \rd, \rd, \sym, \cond + .endm + + /* + * str_va - store a 32-bit word to the virtual address of \sym + */ + .macro str_va, rn:req, sym:req, tmp:req, cond + __ldst_va str, \rn, \tmp, \sym, \cond + .endm + /* * rev_l - byte-swap a 32-bit value * diff --git a/arch/arm/kernel/entry-armv.S b/arch/arm/kernel/entry-armv.S index 1a6cf711a3b4..7f7ac963445c 100644 --- a/arch/arm/kernel/entry-armv.S +++ b/arch/arm/kernel/entry-armv.S @@ -53,7 +53,7 @@ UNWIND( .setfp fpreg, sp ) subs r2, sp, r0 @ SP above bottom of IRQ stack? rsbscs r2, r2, #THREAD_SIZE @ ... and below the top? #ifdef CONFIG_VMAP_STACK - ldr_l r2, high_memory, cc @ End of the linear region + ldr_va r2, high_memory, cc @ End of the linear region cmpcc r2, r0 @ Stack pointer was below it? #endif movcs sp, r0 @ If so, revert to incoming SP diff --git a/arch/arm/kernel/entry-header.S b/arch/arm/kernel/entry-header.S index 81df2a3561ca..268f7f4c5c05 100644 --- a/arch/arm/kernel/entry-header.S +++ b/arch/arm/kernel/entry-header.S @@ -445,7 +445,7 @@ THUMB( it ne ) @ in such cases so just carry on. @ str ip, [r0, #12] @ Stash IP on the mode stack - ldr_l ip, high_memory @ Start of VMALLOC space + ldr_va ip, high_memory @ Start of VMALLOC space ARM( cmp sp, ip ) @ SP in vmalloc space? THUMB( cmp r1, ip ) THUMB( itt lo ) -- 2.30.2 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel