From: Frederic Weisbecker <frederic@kernel.org>
To: Peter Zijlstra <peterz@infradead.org>, Ard Biesheuvel <ardb@kernel.org>
Cc: LKML <linux-kernel@vger.kernel.org>,
James Morse <james.morse@arm.com>,
David Laight <David.Laight@ACULAB.COM>,
Frederic Weisbecker <frederic@kernel.org>,
Quentin Perret <qperret@google.com>,
Catalin Marinas <catalin.marinas@arm.com>,
Will Deacon <will@kernel.org>,
Mark Rutland <mark.rutland@arm.com>
Subject: [PATCH 2/4] arm64: implement support for static call trampolines
Date: Mon, 25 Oct 2021 14:21:00 +0200 [thread overview]
Message-ID: <20211025122102.46089-3-frederic@kernel.org> (raw)
In-Reply-To: <20211025122102.46089-1-frederic@kernel.org>
From: Ard Biesheuvel <ardb@kernel.org>
Implement arm64 support for the 'unoptimized' static call variety, which
routes all calls through a single trampoline that is patched to perform a
tail call to the selected function.
It is expected that the direct branch instruction will be able to cover
the common case. However, given that static call targets may be located
in modules loaded out of direct branching range, we need a fallback path
that loads the address into R16 and uses a branch-to-register (BR)
instruction to perform an indirect call.
Unlike on x86, there is no pressing need on arm64 to avoid indirect
calls at all cost, but hiding it from the compiler as is done here does
have some benefits:
- the literal is located in .text, which gives us the same robustness
advantage that code patching does;
- no performance hit on CFI enabled Clang builds that decorate compiler
emitted indirect calls with branch target validity checks.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Quentin Perret <qperret@google.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: James Morse <james.morse@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: David Laight <David.Laight@ACULAB.COM>
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
---
arch/arm64/Kconfig | 1 +
arch/arm64/include/asm/static_call.h | 25 ++++++++++++++
arch/arm64/kernel/patching.c | 51 ++++++++++++++++++++++++++--
arch/arm64/kernel/vmlinux.lds.S | 1 +
4 files changed, 75 insertions(+), 3 deletions(-)
create mode 100644 arch/arm64/include/asm/static_call.h
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index d13677f4731d..34b175b1e247 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -192,6 +192,7 @@ config ARM64
select HAVE_PERF_REGS
select HAVE_PERF_USER_STACK_DUMP
select HAVE_REGS_AND_STACK_ACCESS_API
+ select HAVE_STATIC_CALL
select HAVE_FUNCTION_ARG_ACCESS_API
select HAVE_FUTEX_CMPXCHG if FUTEX
select MMU_GATHER_RCU_TABLE_FREE
diff --git a/arch/arm64/include/asm/static_call.h b/arch/arm64/include/asm/static_call.h
new file mode 100644
index 000000000000..4871374d584b
--- /dev/null
+++ b/arch/arm64/include/asm/static_call.h
@@ -0,0 +1,25 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_STATIC_CALL_H
+#define _ASM_STATIC_CALL_H
+
+#define __ARCH_DEFINE_STATIC_CALL_TRAMP(name, insn) \
+ asm(" .pushsection .static_call.text, \"ax\" \n" \
+ " .align 4 \n" \
+ " .globl " STATIC_CALL_TRAMP_STR(name) " \n" \
+ "0: .quad 0x0 \n" \
+ STATIC_CALL_TRAMP_STR(name) ": \n" \
+ " hint 34 /* BTI C */ \n" \
+ insn " \n" \
+ " ldr x16, 0b \n" \
+ " cbz x16, 1f \n" \
+ " br x16 \n" \
+ "1: ret \n" \
+ " .popsection \n")
+
+#define ARCH_DEFINE_STATIC_CALL_TRAMP(name, func) \
+ __ARCH_DEFINE_STATIC_CALL_TRAMP(name, "b " #func)
+
+#define ARCH_DEFINE_STATIC_CALL_NULL_TRAMP(name) \
+ __ARCH_DEFINE_STATIC_CALL_TRAMP(name, "ret")
+
+#endif /* _ASM_STATIC_CALL_H */
diff --git a/arch/arm64/kernel/patching.c b/arch/arm64/kernel/patching.c
index 771f543464e0..f98127d92e1f 100644
--- a/arch/arm64/kernel/patching.c
+++ b/arch/arm64/kernel/patching.c
@@ -66,7 +66,7 @@ int __kprobes aarch64_insn_read(void *addr, u32 *insnp)
return ret;
}
-static int __kprobes __aarch64_insn_write(void *addr, __le32 insn)
+static int __kprobes __aarch64_insn_write(void *addr, void *insn, int size)
{
void *waddr = addr;
unsigned long flags = 0;
@@ -75,7 +75,7 @@ static int __kprobes __aarch64_insn_write(void *addr, __le32 insn)
raw_spin_lock_irqsave(&patch_lock, flags);
waddr = patch_map(addr, FIX_TEXT_POKE0);
- ret = copy_to_kernel_nofault(waddr, &insn, AARCH64_INSN_SIZE);
+ ret = copy_to_kernel_nofault(waddr, insn, size);
patch_unmap(FIX_TEXT_POKE0);
raw_spin_unlock_irqrestore(&patch_lock, flags);
@@ -85,7 +85,52 @@ static int __kprobes __aarch64_insn_write(void *addr, __le32 insn)
int __kprobes aarch64_insn_write(void *addr, u32 insn)
{
- return __aarch64_insn_write(addr, cpu_to_le32(insn));
+ __le32 i = cpu_to_le32(insn);
+
+ return __aarch64_insn_write(addr, &i, AARCH64_INSN_SIZE);
+}
+
+void arch_static_call_transform(void *site, void *tramp, void *func, bool tail)
+{
+ /*
+ * -0x8 <literal>
+ * 0x0 bti c <--- trampoline entry point
+ * 0x4 <branch or nop>
+ * 0x8 ldr x16, <literal>
+ * 0xc cbz x16, 20
+ * 0x10 br x16
+ * 0x14 ret
+ */
+ struct {
+ u64 literal;
+ __le32 insn[2];
+ } insns;
+ u32 insn;
+ int ret;
+
+ insn = aarch64_insn_gen_hint(AARCH64_INSN_HINT_BTIC);
+ insns.literal = (u64)func;
+ insns.insn[0] = cpu_to_le32(insn);
+
+ if (!func) {
+ insn = aarch64_insn_gen_branch_reg(AARCH64_INSN_REG_LR,
+ AARCH64_INSN_BRANCH_RETURN);
+ } else {
+ insn = aarch64_insn_gen_branch_imm((u64)tramp + 4, (u64)func,
+ AARCH64_INSN_BRANCH_NOLINK);
+
+ /*
+ * Use a NOP if the branch target is out of range, and rely on
+ * the indirect call instead.
+ */
+ if (insn == AARCH64_BREAK_FAULT)
+ insn = aarch64_insn_gen_hint(AARCH64_INSN_HINT_NOP);
+ }
+ insns.insn[1] = cpu_to_le32(insn);
+
+ ret = __aarch64_insn_write(tramp - 8, &insns, sizeof(insns));
+ if (!WARN_ON(ret))
+ caches_clean_inval_pou((u64)tramp - 8, sizeof(insns));
}
int __kprobes aarch64_insn_patch_text_nosync(void *addr, u32 insn)
diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S
index f6b1a88245db..ceb35c35192c 100644
--- a/arch/arm64/kernel/vmlinux.lds.S
+++ b/arch/arm64/kernel/vmlinux.lds.S
@@ -161,6 +161,7 @@ SECTIONS
IDMAP_TEXT
HIBERNATE_TEXT
TRAMP_TEXT
+ STATIC_CALL_TEXT
*(.fixup)
*(.gnu.warning)
. = ALIGN(16);
--
2.25.1
next prev parent reply other threads:[~2021-10-25 12:21 UTC|newest]
Thread overview: 30+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-10-25 12:20 [PATCH 0/4] arm64: Support dynamic preemption v2 Frederic Weisbecker
2021-10-25 12:20 ` [PATCH 1/4] sched/preempt: Prepare for supporting !CONFIG_GENERIC_ENTRY dynamic preemption Frederic Weisbecker
2021-10-25 12:21 ` Frederic Weisbecker [this message]
2021-10-25 13:56 ` [PATCH 2/4] arm64: implement support for static call trampolines Peter Zijlstra
2021-10-25 14:08 ` Ard Biesheuvel
2021-10-25 14:19 ` Peter Zijlstra
2021-10-25 14:44 ` Peter Zijlstra
2021-10-25 14:55 ` Ard Biesheuvel
2021-10-25 15:03 ` Peter Zijlstra
2021-10-25 15:10 ` Ard Biesheuvel
2021-10-26 10:36 ` Mark Rutland
2021-10-26 10:45 ` Peter Zijlstra
2021-10-26 11:06 ` David Laight
2021-10-27 12:47 ` Mark Rutland
2021-10-25 15:03 ` David Laight
2021-10-25 14:25 ` David Laight
2021-10-25 14:31 ` Ard Biesheuvel
2021-10-25 14:38 ` David Laight
2021-10-25 12:21 ` [PATCH 3/4] arm64: Implement IRQ exit preemption static call for dynamic preemption Frederic Weisbecker
2021-10-25 12:21 ` [PATCH 4/4] arm64: Implement HAVE_PREEMPT_DYNAMIC Frederic Weisbecker
-- strict thread matches above, loose matches on Subject: below --
2021-09-20 23:32 [PATCH 0/4] arm64: Support dynamic preemption Frederic Weisbecker
2021-09-20 23:32 ` [PATCH 2/4] arm64: implement support for static call trampolines Frederic Weisbecker
2021-09-21 7:09 ` Peter Zijlstra
2021-09-21 14:44 ` Ard Biesheuvel
2021-09-21 15:08 ` Peter Zijlstra
2021-09-21 15:33 ` Mark Rutland
2021-09-21 15:55 ` Ard Biesheuvel
2021-09-21 16:28 ` Mark Rutland
2021-09-25 17:46 ` David Laight
2021-09-27 8:58 ` Mark Rutland
2021-09-21 16:10 ` Ard Biesheuvel
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20211025122102.46089-3-frederic@kernel.org \
--to=frederic@kernel.org \
--cc=David.Laight@ACULAB.COM \
--cc=ardb@kernel.org \
--cc=catalin.marinas@arm.com \
--cc=james.morse@arm.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mark.rutland@arm.com \
--cc=peterz@infradead.org \
--cc=qperret@google.com \
--cc=will@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.