From: Ard Biesheuvel <ardb@kernel.org>
To: linux-arm-kernel@lists.infradead.org, linux@armlinux.org.uk
Cc: Ard Biesheuvel <ardb@kernel.org>,
Nicolas Pitre <nico@fluxnic.net>, Arnd Bergmann <arnd@arndb.de>,
Kees Cook <keescook@chromium.org>,
Keith Packard <keithpac@amazon.com>,
Linus Walleij <linus.walleij@linaro.org>,
Nick Desaulniers <ndesaulniers@google.com>,
Tony Lindgren <tony@atomide.com>, Marc Zyngier <maz@kernel.org>,
Vladimir Murzin <vladimir.murzin@arm.com>,
Jesse Taube <mr.bossman075@gmail.com>
Subject: [PATCH v4 10/15] ARM: assembler: add optimized ldr/str macros to load variables from memory
Date: Mon, 6 Dec 2021 17:46:54 +0100 [thread overview]
Message-ID: <20211206164659.1495084-11-ardb@kernel.org> (raw)
In-Reply-To: <20211206164659.1495084-1-ardb@kernel.org>
We will be adding variable loads to various hot paths, so it makes sense
to add a helper macro that can load variables from asm code without the
use of literal pool entries. On v7 or later, we can simply use MOVW/MOVT
pairs, but on earlier cores, this requires a bit of hackery to emit a
instruction sequence that implements this using a sequence of ADD/LDR
instructions.
Acked-by: Linus Walleij <linus.walleij@linaro.org>
Acked-by: Nicolas Pitre <nico@fluxnic.net>
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Tested-by: Marc Zyngier <maz@kernel.org>
Tested-by: Vladimir Murzin <vladimir.murzin@arm.com> # ARMv7M
---
arch/arm/include/asm/assembler.h | 45 ++++++++++++++++++--
arch/arm/kernel/entry-armv.S | 2 +-
| 2 +-
3 files changed, 43 insertions(+), 6 deletions(-)
diff --git a/arch/arm/include/asm/assembler.h b/arch/arm/include/asm/assembler.h
index 1b9d4df331aa..2095638b7140 100644
--- a/arch/arm/include/asm/assembler.h
+++ b/arch/arm/include/asm/assembler.h
@@ -568,12 +568,12 @@ THUMB( orr \reg , \reg , #PSR_T_BIT )
/*
* mov_l - move a constant value or [relocated] address into a register
*/
- .macro mov_l, dst:req, imm:req
+ .macro mov_l, dst:req, imm:req, cond
.if __LINUX_ARM_ARCH__ < 7
- ldr \dst, =\imm
+ ldr\cond \dst, =\imm
.else
- movw \dst, #:lower16:\imm
- movt \dst, #:upper16:\imm
+ movw\cond \dst, #:lower16:\imm
+ movt\cond \dst, #:upper16:\imm
.endif
.endm
@@ -611,6 +611,43 @@ THUMB( orr \reg , \reg , #PSR_T_BIT )
__adldst_l str, \src, \sym, \tmp, \cond
.endm
+ .macro __ldst_va, op, reg, tmp, sym, cond
+#if __LINUX_ARM_ARCH__ >= 7 || \
+ (defined(MODULE) && defined(CONFIG_ARM_MODULE_PLTS)) || \
+ (defined(CONFIG_LD_IS_LLD) && CONFIG_LLD_VERSION < 140000)
+ mov_l \tmp, \sym, \cond
+ \op\cond \reg, [\tmp]
+#else
+ /*
+ * Avoid a literal load, by emitting a sequence of ADD/LDR instructions
+ * with the appropriate relocations. The combined sequence has a range
+ * of -/+ 256 MiB, which should be sufficient for the core kernel and
+ * for modules loaded into the module region.
+ */
+ .globl \sym
+ .reloc .L0_\@, R_ARM_ALU_PC_G0_NC, \sym
+ .reloc .L1_\@, R_ARM_ALU_PC_G1_NC, \sym
+ .reloc .L2_\@, R_ARM_LDR_PC_G2, \sym
+.L0_\@: sub\cond \tmp, pc, #8
+.L1_\@: sub\cond \tmp, \tmp, #4
+.L2_\@: \op\cond \reg, [\tmp, #0]
+#endif
+ .endm
+
+ /*
+ * ldr_va - load a 32-bit word from the virtual address of \sym
+ */
+ .macro ldr_va, rd:req, sym:req, cond
+ __ldst_va ldr, \rd, \rd, \sym, \cond
+ .endm
+
+ /*
+ * str_va - store a 32-bit word to the virtual address of \sym
+ */
+ .macro str_va, rn:req, sym:req, tmp:req, cond
+ __ldst_va str, \rn, \tmp, \sym, \cond
+ .endm
+
/*
* rev_l - byte-swap a 32-bit value
*
diff --git a/arch/arm/kernel/entry-armv.S b/arch/arm/kernel/entry-armv.S
index 1a6cf711a3b4..7f7ac963445c 100644
--- a/arch/arm/kernel/entry-armv.S
+++ b/arch/arm/kernel/entry-armv.S
@@ -53,7 +53,7 @@ UNWIND( .setfp fpreg, sp )
subs r2, sp, r0 @ SP above bottom of IRQ stack?
rsbscs r2, r2, #THREAD_SIZE @ ... and below the top?
#ifdef CONFIG_VMAP_STACK
- ldr_l r2, high_memory, cc @ End of the linear region
+ ldr_va r2, high_memory, cc @ End of the linear region
cmpcc r2, r0 @ Stack pointer was below it?
#endif
movcs sp, r0 @ If so, revert to incoming SP
--git a/arch/arm/kernel/entry-header.S b/arch/arm/kernel/entry-header.S
index 81df2a3561ca..268f7f4c5c05 100644
--- a/arch/arm/kernel/entry-header.S
+++ b/arch/arm/kernel/entry-header.S
@@ -445,7 +445,7 @@ THUMB( it ne )
@ in such cases so just carry on.
@
str ip, [r0, #12] @ Stash IP on the mode stack
- ldr_l ip, high_memory @ Start of VMALLOC space
+ ldr_va ip, high_memory @ Start of VMALLOC space
ARM( cmp sp, ip ) @ SP in vmalloc space?
THUMB( cmp r1, ip )
THUMB( itt lo )
--
2.30.2
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
next prev parent reply other threads:[~2021-12-06 16:56 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-12-06 16:46 [PATCH v4 00/15] ARM: enable IRQ stacks and vmap'ed stacks for UP Ard Biesheuvel
2021-12-06 16:46 ` [PATCH v4 01/15] ARM: riscpc: drop support for IOMD_IRQREQC/IOMD_IRQREQD IRQ groups Ard Biesheuvel
2021-12-06 16:46 ` [PATCH v4 02/15] ARM: riscpc: use GENERIC_IRQ_MULTI_HANDLER Ard Biesheuvel
2021-12-06 16:46 ` [PATCH v4 03/15] ARM: footbridge: " Ard Biesheuvel
2021-12-06 16:46 ` [PATCH v4 04/15] ARM: iop32x: offset IRQ numbers by 1 Ard Biesheuvel
2021-12-06 16:46 ` [PATCH v4 05/15] ARM: iop32x: use GENERIC_IRQ_MULTI_HANDLER Ard Biesheuvel
2021-12-06 16:46 ` [PATCH v4 06/15] ARM: remove old-style irq entry Ard Biesheuvel
2021-12-06 16:46 ` [PATCH v4 07/15] irqchip: nvic: Use GENERIC_IRQ_MULTI_HANDLER Ard Biesheuvel
2021-12-06 16:46 ` [PATCH v4 08/15] ARM: entry: preserve thread_info pointer in switch_to Ard Biesheuvel
2021-12-06 16:46 ` [PATCH v4 09/15] ARM: module: implement support for PC-relative group relocations Ard Biesheuvel
2021-12-06 16:46 ` Ard Biesheuvel [this message]
2021-12-06 16:46 ` [PATCH v4 11/15] ARM: percpu: add SMP_ON_UP support Ard Biesheuvel
2021-12-06 16:46 ` [PATCH v4 12/15] ARM: use TLS register for 'current' on !SMP as well Ard Biesheuvel
2021-12-06 16:46 ` [PATCH v4 13/15] ARM: smp: defer TPIDRURO update for SMP v6 configurations too Ard Biesheuvel
2021-12-06 16:46 ` [PATCH v4 14/15] ARM: implement THREAD_INFO_IN_TASK for uniprocessor systems Ard Biesheuvel
2021-12-06 16:46 ` [PATCH v4 15/15] ARM: v7m: enable support for IRQ stacks Ard Biesheuvel
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20211206164659.1495084-11-ardb@kernel.org \
--to=ardb@kernel.org \
--cc=arnd@arndb.de \
--cc=keescook@chromium.org \
--cc=keithpac@amazon.com \
--cc=linus.walleij@linaro.org \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux@armlinux.org.uk \
--cc=maz@kernel.org \
--cc=mr.bossman075@gmail.com \
--cc=ndesaulniers@google.com \
--cc=nico@fluxnic.net \
--cc=tony@atomide.com \
--cc=vladimir.murzin@arm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).