From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DF790C433F5 for ; Tue, 30 Nov 2021 13:03:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=uAVIAolfD6cnptDKKOXXGITVc4Jx/KKcTNq2VBLhiRs=; b=qwUiV6Mgor6meA WF/MkIiw7qSj7Ieezs0DtzNJMxSaRPWipnBA9LXMcmIS38pw4rd7Fc00z20085UIvQSfDrKWP4g+5 V64pW5a6HbFqezPunb+OxgbkILD7sRpWkCCUAy5Tm0VD9RV4Bmoa36eE21Rcs7LZ5Xr0yko3roT5B OVtHMQpEZ0EI2dsa/LEecELTGlRe6PXZEclLEVZnmVusso4Ku3f88ZRHCjsH+0ukGYsonh5Dvxzew YFT2lVFRJwEyUOzw9KQWX3vvjy0NDpzwQCMmR/n7YA1POqHGf72HbiF8JcLFxiI7uN55QRuBP1Qzd SendZTrTI7X1qyxtlbSw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1ms2lk-005DTf-2y; Tue, 30 Nov 2021 13:02:01 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1ms2jN-005CQ7-6T for linux-arm-kernel@lists.infradead.org; Tue, 30 Nov 2021 12:59:34 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id E06B0B817AF; Tue, 30 Nov 2021 12:59:31 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 80503C53FC7; Tue, 30 Nov 2021 12:59:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638277170; bh=PhC6cgPQoRMJeeY3zWraiO5mByR1ipXljHxzSVWaL3A=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=UtGRqvhx5whBMK1YLNUy42GBzMlCaoECmWLHmAwzSFT5zfePDRZKiDP+jh/k+/2cl y0cXqgAHVP0LTa2GmkEe3GivKXd+a0wfmuu6J+wQtTvagMeqK5WGSGcN927XTvGEAk ky7WQztdaAZGivDRk1144NFsE2PHezwZQ471mI5QALG79phAG4Q2mOZRDBRcRjtadK 1+91Ct5p6WFsBhAvg+vF1+v0EWbz/X0Bdhib2dKidO4Jc1Gl94JeXzby4As8w0EL3o ZJF5ZyCUmQlWUKL/g9ghbGVJS410/DLveQ/MN0NLpWObhKSGgmtEFE0mippaMypKz0 cTd/xmB55cMEw== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org, linux@armlinux.org.uk Cc: Ard Biesheuvel , Nicolas Pitre , Arnd Bergmann , Kees Cook , Keith Packard , Linus Walleij , Nick Desaulniers , Tony Lindgren Subject: [PATCH v2 08/12] ARM: assembler: add optimized ldr/str macros to load variables from memory Date: Tue, 30 Nov 2021 13:58:57 +0100 Message-Id: <20211130125901.3054-9-ardb@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211130125901.3054-1-ardb@kernel.org> References: <20211130125901.3054-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=3862; h=from:subject; bh=PhC6cgPQoRMJeeY3zWraiO5mByR1ipXljHxzSVWaL3A=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBhpiAQFXyXaRSkaqrUa/r+5mQG/z4RCP1voiqXiEoH g/pSCmGJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYaYgEAAKCRDDTyI5ktmPJNP1C/ 4gmqZ4g1UsgAGg7yiIJIfTvrmQO4+jcGrR8MP/gH0DDVoN69k86/E+4msx4+3zEIRbkpxVYeotk7VN jKqIko35jJlkeBWlm7eF428ZDAhNpMluUIFeGDtVmPzXONlhfqft8EcApAk534NQNPSC8cFVVSTe7w DPeryobxAL1RibFuVBYkHq05dGXxnhT5SljnZSjn3cmmLGXQK36eS3YjeHFyOV5myZvNPfkY03tGLP WEax/P5KI0oCupy8nmIfujwTayS6qhA5xO+nZPWYUsM5hNBdRCg+QerVEl79Q/NU8ojPKe6qy1AZvM CJqAyuvZordMnfWuMu7V938kCnpiwDI3mxelUJlpC2+24BcvUk+2f+cSk+B4weuFQC6tTfVPAKIVZI 9jpY5BRoIhs6GyOvIkiBRlXLWqjDLPwLmmb/BsWL7F2XMfA/6M5H5f9O4VDYY7USFOydbj0AoEU9Cx 8Oztxb5vG8EAfPQGxGoB2GfKGgDIXXyGVS0j9qT1vsyHs= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211130_045933_547290_76C83D2D X-CRM114-Status: GOOD ( 19.66 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org We will be adding variable loads to various hot paths, so it makes sense to add a helper macro that can load variables from asm code without the use of literal pool entries. On v7 or later, we can simply use MOVW/MOVT pairs, but on earlier cores, this requires a bit of hackery to emit a instruction sequence that implements this using a sequence of ADD/LDR instructions. Acked-by: Linus Walleij Acked-by: Nicolas Pitre Signed-off-by: Ard Biesheuvel --- arch/arm/include/asm/assembler.h | 45 ++++++++++++++++++-- arch/arm/kernel/entry-armv.S | 2 +- arch/arm/kernel/entry-header.S | 2 +- 3 files changed, 43 insertions(+), 6 deletions(-) diff --git a/arch/arm/include/asm/assembler.h b/arch/arm/include/asm/assembler.h index 1b9d4df331aa..2095638b7140 100644 --- a/arch/arm/include/asm/assembler.h +++ b/arch/arm/include/asm/assembler.h @@ -568,12 +568,12 @@ THUMB( orr \reg , \reg , #PSR_T_BIT ) /* * mov_l - move a constant value or [relocated] address into a register */ - .macro mov_l, dst:req, imm:req + .macro mov_l, dst:req, imm:req, cond .if __LINUX_ARM_ARCH__ < 7 - ldr \dst, =\imm + ldr\cond \dst, =\imm .else - movw \dst, #:lower16:\imm - movt \dst, #:upper16:\imm + movw\cond \dst, #:lower16:\imm + movt\cond \dst, #:upper16:\imm .endif .endm @@ -611,6 +611,43 @@ THUMB( orr \reg , \reg , #PSR_T_BIT ) __adldst_l str, \src, \sym, \tmp, \cond .endm + .macro __ldst_va, op, reg, tmp, sym, cond +#if __LINUX_ARM_ARCH__ >= 7 || \ + (defined(MODULE) && defined(CONFIG_ARM_MODULE_PLTS)) || \ + (defined(CONFIG_LD_IS_LLD) && CONFIG_LLD_VERSION < 140000) + mov_l \tmp, \sym, \cond + \op\cond \reg, [\tmp] +#else + /* + * Avoid a literal load, by emitting a sequence of ADD/LDR instructions + * with the appropriate relocations. The combined sequence has a range + * of -/+ 256 MiB, which should be sufficient for the core kernel and + * for modules loaded into the module region. + */ + .globl \sym + .reloc .L0_\@, R_ARM_ALU_PC_G0_NC, \sym + .reloc .L1_\@, R_ARM_ALU_PC_G1_NC, \sym + .reloc .L2_\@, R_ARM_LDR_PC_G2, \sym +.L0_\@: sub\cond \tmp, pc, #8 +.L1_\@: sub\cond \tmp, \tmp, #4 +.L2_\@: \op\cond \reg, [\tmp, #0] +#endif + .endm + + /* + * ldr_va - load a 32-bit word from the virtual address of \sym + */ + .macro ldr_va, rd:req, sym:req, cond + __ldst_va ldr, \rd, \rd, \sym, \cond + .endm + + /* + * str_va - store a 32-bit word to the virtual address of \sym + */ + .macro str_va, rn:req, sym:req, tmp:req, cond + __ldst_va str, \rn, \tmp, \sym, \cond + .endm + /* * rev_l - byte-swap a 32-bit value * diff --git a/arch/arm/kernel/entry-armv.S b/arch/arm/kernel/entry-armv.S index 1a6cf711a3b4..7f7ac963445c 100644 --- a/arch/arm/kernel/entry-armv.S +++ b/arch/arm/kernel/entry-armv.S @@ -53,7 +53,7 @@ UNWIND( .setfp fpreg, sp ) subs r2, sp, r0 @ SP above bottom of IRQ stack? rsbscs r2, r2, #THREAD_SIZE @ ... and below the top? #ifdef CONFIG_VMAP_STACK - ldr_l r2, high_memory, cc @ End of the linear region + ldr_va r2, high_memory, cc @ End of the linear region cmpcc r2, r0 @ Stack pointer was below it? #endif movcs sp, r0 @ If so, revert to incoming SP diff --git a/arch/arm/kernel/entry-header.S b/arch/arm/kernel/entry-header.S index 81df2a3561ca..268f7f4c5c05 100644 --- a/arch/arm/kernel/entry-header.S +++ b/arch/arm/kernel/entry-header.S @@ -445,7 +445,7 @@ THUMB( it ne ) @ in such cases so just carry on. @ str ip, [r0, #12] @ Stash IP on the mode stack - ldr_l ip, high_memory @ Start of VMALLOC space + ldr_va ip, high_memory @ Start of VMALLOC space ARM( cmp sp, ip ) @ SP in vmalloc space? THUMB( cmp r1, ip ) THUMB( itt lo ) -- 2.30.2 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel