From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 00772C433EF for ; Mon, 6 Dec 2021 16:56:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=jo5iZ5LsGHpBtO0ptm+fXUvIlD54KvVPS0OWxpAn+rc=; b=wcMeHYY+54qxii Ha6f8cl+ACw49SrkPdsRgzBj1W4S6Pwjcl/OalbAW7CJB2tpG9upw+0e4oWj87aXTdqO1yARdqVEV l3UV5IGM1AtcXnVI2YH/Tu9ZaThhvkEtIYTWhgLEIY+p5z2G/6elf2jnbyUtDKDga0ScpTlteBHeB VdmHRQ2icJqMb+qUn648FJSTDN7B129DXPAlcjyWurA2FAfssZyojCUQkF7KDCLsm/TZeEdKXUOpL oWi7u8KUtYEm2ipJgCWh3Ko/thrTq9WtsXKf6aF5pHahLFdebP+Cda8ff/vrHE6rKpKKQUx2hcrcT 7z0brMwQolog4u8DhZkg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1muHFq-004tdx-CA; Mon, 06 Dec 2021 16:54:19 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1muH9c-004qlc-2A for linux-arm-kernel@lists.infradead.org; Mon, 06 Dec 2021 16:47:53 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id C2EA1B81184; Mon, 6 Dec 2021 16:47:50 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D289DC341C1; Mon, 6 Dec 2021 16:47:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1638809269; bh=cKWcKF+i/3lZtdU7a2NB3vYsXa0hBxUzRaicPyUll/Q=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=X5JjDfjqN4mAPerK23Nunz4v5UL1vWQ62MiN/tMWyGk6o+jZJAJDovkAnhsEE9gdK TvAHeJacqC2RL80dNo0A86fGmdZ9Iw44Q3Wy/lWx115XZbMVfW8j3J1t/A2aD1RntX CpQkG8T+HnQV2FMDoF4iAs1fz+Ttaqiv/HuhhHM8YkgU3SACd36F0kPiSq3xUOZRFg CYqs+80VTJoYdAuXK05ystzwHoniVGhunlpMsnjyvVqSZkNH5xeaboyRl4s+lek+S2 evxsJkrrFu1YnnjGijuOOpLjaxW/7XeQp/J1iyPElsaSK3i1KoCQwk7XLoW3RrPzGJ SLh15QFWuS7lg== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org, linux@armlinux.org.uk Cc: Ard Biesheuvel , Nicolas Pitre , Arnd Bergmann , Kees Cook , Keith Packard , Linus Walleij , Nick Desaulniers , Tony Lindgren , Marc Zyngier , Vladimir Murzin , Jesse Taube Subject: [PATCH v4 10/15] ARM: assembler: add optimized ldr/str macros to load variables from memory Date: Mon, 6 Dec 2021 17:46:54 +0100 Message-Id: <20211206164659.1495084-11-ardb@kernel.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211206164659.1495084-1-ardb@kernel.org> References: <20211206164659.1495084-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=3967; h=from:subject; bh=cKWcKF+i/3lZtdU7a2NB3vYsXa0hBxUzRaicPyUll/Q=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBhrj54tWHL40x8PlbJICEr8A+DkrLFtVWvP/pmhcXt xB1JNoGJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCYa4+eAAKCRDDTyI5ktmPJKkHC/ 0UHvg6digWr+W3yOwrvKkrN4/tEqMz5gvdicWJKvLe4PbuPKMI8IlmFy6rncYdTRFZTAFZp82RB/M/ O8gEwMJsr9t1kTaXor0MOVHVrQuCFeHGc6j2w17vYkT8ayffAnlyJtGdporq3Kc8y9nXA7bkg80/4s JgZDqHIGrePkwUytybOO8f33b2ojGZZgG1MAbhGiXuwnb2RWrOK6gtQ+OOZABIimewUTASucM5XZ1+ xGPra26jHTtyuJALjdyTJkuSCMPCe/J7zZNJXq5C4bpa/o+9Dw5g3LKVWV8E3xk8nyM4DZYsNlSR3K oWnQnRKRuMTqGyIX5wi1mbA4OT+PJgoArTPr9XQIBNkjTw4HnxpkyJuJwHbDyiWCG1uVnYm7QkhP1O RVtaNHXh7JGT6CGxLKSUZaEjsVg5lmEgl4A+kI3PmSfYkk3lVpGttV954vVZ8ZMZbqkM998gkkOMvv qAyszyG5Bj77x/+m5ZU9VUQa7spX0Ys3GRsBT8wkuauyY= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211206_084752_440164_36F89D7B X-CRM114-Status: GOOD ( 17.93 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org We will be adding variable loads to various hot paths, so it makes sense to add a helper macro that can load variables from asm code without the use of literal pool entries. On v7 or later, we can simply use MOVW/MOVT pairs, but on earlier cores, this requires a bit of hackery to emit a instruction sequence that implements this using a sequence of ADD/LDR instructions. Acked-by: Linus Walleij Acked-by: Nicolas Pitre Signed-off-by: Ard Biesheuvel Tested-by: Marc Zyngier Tested-by: Vladimir Murzin # ARMv7M --- arch/arm/include/asm/assembler.h | 45 ++++++++++++++++++-- arch/arm/kernel/entry-armv.S | 2 +- arch/arm/kernel/entry-header.S | 2 +- 3 files changed, 43 insertions(+), 6 deletions(-) diff --git a/arch/arm/include/asm/assembler.h b/arch/arm/include/asm/assembler.h index 1b9d4df331aa..2095638b7140 100644 --- a/arch/arm/include/asm/assembler.h +++ b/arch/arm/include/asm/assembler.h @@ -568,12 +568,12 @@ THUMB( orr \reg , \reg , #PSR_T_BIT ) /* * mov_l - move a constant value or [relocated] address into a register */ - .macro mov_l, dst:req, imm:req + .macro mov_l, dst:req, imm:req, cond .if __LINUX_ARM_ARCH__ < 7 - ldr \dst, =\imm + ldr\cond \dst, =\imm .else - movw \dst, #:lower16:\imm - movt \dst, #:upper16:\imm + movw\cond \dst, #:lower16:\imm + movt\cond \dst, #:upper16:\imm .endif .endm @@ -611,6 +611,43 @@ THUMB( orr \reg , \reg , #PSR_T_BIT ) __adldst_l str, \src, \sym, \tmp, \cond .endm + .macro __ldst_va, op, reg, tmp, sym, cond +#if __LINUX_ARM_ARCH__ >= 7 || \ + (defined(MODULE) && defined(CONFIG_ARM_MODULE_PLTS)) || \ + (defined(CONFIG_LD_IS_LLD) && CONFIG_LLD_VERSION < 140000) + mov_l \tmp, \sym, \cond + \op\cond \reg, [\tmp] +#else + /* + * Avoid a literal load, by emitting a sequence of ADD/LDR instructions + * with the appropriate relocations. The combined sequence has a range + * of -/+ 256 MiB, which should be sufficient for the core kernel and + * for modules loaded into the module region. + */ + .globl \sym + .reloc .L0_\@, R_ARM_ALU_PC_G0_NC, \sym + .reloc .L1_\@, R_ARM_ALU_PC_G1_NC, \sym + .reloc .L2_\@, R_ARM_LDR_PC_G2, \sym +.L0_\@: sub\cond \tmp, pc, #8 +.L1_\@: sub\cond \tmp, \tmp, #4 +.L2_\@: \op\cond \reg, [\tmp, #0] +#endif + .endm + + /* + * ldr_va - load a 32-bit word from the virtual address of \sym + */ + .macro ldr_va, rd:req, sym:req, cond + __ldst_va ldr, \rd, \rd, \sym, \cond + .endm + + /* + * str_va - store a 32-bit word to the virtual address of \sym + */ + .macro str_va, rn:req, sym:req, tmp:req, cond + __ldst_va str, \rn, \tmp, \sym, \cond + .endm + /* * rev_l - byte-swap a 32-bit value * diff --git a/arch/arm/kernel/entry-armv.S b/arch/arm/kernel/entry-armv.S index 1a6cf711a3b4..7f7ac963445c 100644 --- a/arch/arm/kernel/entry-armv.S +++ b/arch/arm/kernel/entry-armv.S @@ -53,7 +53,7 @@ UNWIND( .setfp fpreg, sp ) subs r2, sp, r0 @ SP above bottom of IRQ stack? rsbscs r2, r2, #THREAD_SIZE @ ... and below the top? #ifdef CONFIG_VMAP_STACK - ldr_l r2, high_memory, cc @ End of the linear region + ldr_va r2, high_memory, cc @ End of the linear region cmpcc r2, r0 @ Stack pointer was below it? #endif movcs sp, r0 @ If so, revert to incoming SP diff --git a/arch/arm/kernel/entry-header.S b/arch/arm/kernel/entry-header.S index 81df2a3561ca..268f7f4c5c05 100644 --- a/arch/arm/kernel/entry-header.S +++ b/arch/arm/kernel/entry-header.S @@ -445,7 +445,7 @@ THUMB( it ne ) @ in such cases so just carry on. @ str ip, [r0, #12] @ Stash IP on the mode stack - ldr_l ip, high_memory @ Start of VMALLOC space + ldr_va ip, high_memory @ Start of VMALLOC space ARM( cmp sp, ip ) @ SP in vmalloc space? THUMB( cmp r1, ip ) THUMB( itt lo ) -- 2.30.2 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel