From: Peter Zijlstra <peterz@infradead.org>
To: Frederic Weisbecker <frederic@kernel.org>
Cc: Ard Biesheuvel <ardb@kernel.org>,
LKML <linux-kernel@vger.kernel.org>,
James Morse <james.morse@arm.com>,
David Laight <David.Laight@aculab.com>,
Quentin Perret <qperret@google.com>,
Catalin Marinas <catalin.marinas@arm.com>,
Will Deacon <will@kernel.org>,
Mark Rutland <mark.rutland@arm.com>
Subject: Re: [PATCH 2/4] arm64: implement support for static call trampolines
Date: Mon, 25 Oct 2021 15:56:59 +0200 [thread overview]
Message-ID: <YXa3q2AOH0T+smFy@hirez.programming.kicks-ass.net> (raw)
In-Reply-To: <20211025122102.46089-3-frederic@kernel.org>
On Mon, Oct 25, 2021 at 02:21:00PM +0200, Frederic Weisbecker wrote:
> +#define __ARCH_DEFINE_STATIC_CALL_TRAMP(name, insn) \
> + asm(" .pushsection .static_call.text, \"ax\" \n" \
> + " .align 4 \n" \
> + " .globl " STATIC_CALL_TRAMP_STR(name) " \n" \
> + "0: .quad 0x0 \n" \
> + STATIC_CALL_TRAMP_STR(name) ": \n" \
> + " hint 34 /* BTI C */ \n" \
> + insn " \n" \
> + " ldr x16, 0b \n" \
> + " cbz x16, 1f \n" \
> + " br x16 \n" \
> + "1: ret \n" \
> + " .popsection \n")
> +void arch_static_call_transform(void *site, void *tramp, void *func, bool tail)
> +{
> + /*
> + * -0x8 <literal>
> + * 0x0 bti c <--- trampoline entry point
> + * 0x4 <branch or nop>
> + * 0x8 ldr x16, <literal>
> + * 0xc cbz x16, 20
> + * 0x10 br x16
> + * 0x14 ret
> + */
> + struct {
> + u64 literal;
> + __le32 insn[2];
> + } insns;
> + u32 insn;
> + int ret;
> +
> + insn = aarch64_insn_gen_hint(AARCH64_INSN_HINT_BTIC);
> + insns.literal = (u64)func;
> + insns.insn[0] = cpu_to_le32(insn);
> +
> + if (!func) {
> + insn = aarch64_insn_gen_branch_reg(AARCH64_INSN_REG_LR,
> + AARCH64_INSN_BRANCH_RETURN);
> + } else {
> + insn = aarch64_insn_gen_branch_imm((u64)tramp + 4, (u64)func,
> + AARCH64_INSN_BRANCH_NOLINK);
> +
> + /*
> + * Use a NOP if the branch target is out of range, and rely on
> + * the indirect call instead.
> + */
> + if (insn == AARCH64_BREAK_FAULT)
> + insn = aarch64_insn_gen_hint(AARCH64_INSN_HINT_NOP);
> + }
> + insns.insn[1] = cpu_to_le32(insn);
> +
> + ret = __aarch64_insn_write(tramp - 8, &insns, sizeof(insns));
OK, that's pretty magical...
So you're writing the literal and the two instructions with 2 u64
stores. Relying on alignment to guarantee both are in a single page and
that copy_to_kernel_nofault() selects u64 writes.
By unconditionally writing the literal, you avoid there ever being an
stale value, which in turn avoids there being a race where you switch
from 'J @func' relative addressing to 'NOP; do-literal-thing' and cross
CPU execution gets the ordering inverted.
Ooohh, but what if you go from !func to NOP.
assuming:
.literal = 0
BTI C
RET
Then
CPU0 CPU1
[S] literal = func [I] NOP
[S] insn[1] = NOP [L] x16 = literal (NULL)
b x16
*BANG*
Is that possible? (total lack of memory ordering etc..)
On IRC you just alluded to the fact that this relies on it all being in
a single cacheline (i-fetch windows don't need to be cacheline sized,
but provided they're at least 16 bytes, this should still work given the
alignment).
But is I$ and D$ coherent? One load is through I-fetch, the other is a
regular D-fetch.
However, Will has previously expressed reluctance to rely on such
things.
> + if (!WARN_ON(ret))
> + caches_clean_inval_pou((u64)tramp - 8, sizeof(insns));
> }
next prev parent reply other threads:[~2021-10-25 13:57 UTC|newest]
Thread overview: 30+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-10-25 12:20 [PATCH 0/4] arm64: Support dynamic preemption v2 Frederic Weisbecker
2021-10-25 12:20 ` [PATCH 1/4] sched/preempt: Prepare for supporting !CONFIG_GENERIC_ENTRY dynamic preemption Frederic Weisbecker
2021-10-25 12:21 ` [PATCH 2/4] arm64: implement support for static call trampolines Frederic Weisbecker
2021-10-25 13:56 ` Peter Zijlstra [this message]
2021-10-25 14:08 ` Ard Biesheuvel
2021-10-25 14:19 ` Peter Zijlstra
2021-10-25 14:44 ` Peter Zijlstra
2021-10-25 14:55 ` Ard Biesheuvel
2021-10-25 15:03 ` Peter Zijlstra
2021-10-25 15:10 ` Ard Biesheuvel
2021-10-26 10:36 ` Mark Rutland
2021-10-26 10:45 ` Peter Zijlstra
2021-10-26 11:06 ` David Laight
2021-10-27 12:47 ` Mark Rutland
2021-10-25 15:03 ` David Laight
2021-10-25 14:25 ` David Laight
2021-10-25 14:31 ` Ard Biesheuvel
2021-10-25 14:38 ` David Laight
2021-10-25 12:21 ` [PATCH 3/4] arm64: Implement IRQ exit preemption static call for dynamic preemption Frederic Weisbecker
2021-10-25 12:21 ` [PATCH 4/4] arm64: Implement HAVE_PREEMPT_DYNAMIC Frederic Weisbecker
-- strict thread matches above, loose matches on Subject: below --
2021-09-20 23:32 [PATCH 0/4] arm64: Support dynamic preemption Frederic Weisbecker
2021-09-20 23:32 ` [PATCH 2/4] arm64: implement support for static call trampolines Frederic Weisbecker
2021-09-21 7:09 ` Peter Zijlstra
2021-09-21 14:44 ` Ard Biesheuvel
2021-09-21 15:08 ` Peter Zijlstra
2021-09-21 15:33 ` Mark Rutland
2021-09-21 15:55 ` Ard Biesheuvel
2021-09-21 16:28 ` Mark Rutland
2021-09-25 17:46 ` David Laight
2021-09-27 8:58 ` Mark Rutland
2021-09-21 16:10 ` Ard Biesheuvel
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=YXa3q2AOH0T+smFy@hirez.programming.kicks-ass.net \
--to=peterz@infradead.org \
--cc=David.Laight@aculab.com \
--cc=ardb@kernel.org \
--cc=catalin.marinas@arm.com \
--cc=frederic@kernel.org \
--cc=james.morse@arm.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mark.rutland@arm.com \
--cc=qperret@google.com \
--cc=will@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.